For the purposes of this post, let's take The Time Traveller's Wife, Never Let Me Go and The Handmaid's Tale as examples of literary science fiction (sorry, Margaret). I'm not reviewing any of these here - I'll make a case for why everyone should read them another day.
Literary sci-fi novels that get a very wide readership and critical acclaim outside the sci-fi genre, still having sci-fi themes at their heart. They're usually acclaimed as being of quality authorship. I wouldn't question this but many people in sci-fi, fantasy and related genres also have a very high quality of writing. The style of writing can be deceptive - Stephen King may write brief MTV soundbite style chapters but there's a lot of depth to many of his books. I also find it hard to think of a single sci-fi book I've read that doesn't use sci-fi as a way to explore deeper issues and questions or at least to attempt this. I'll accept that this is often not true of films.
I've just finished reading Lifeboat. My final word - most of the book is perfectly paced, however in the last chapter there's suddenly a series of convoluted and unlikely plot twists - I think the enjoyment to be had from this book was definitely the journey rather than the destination. I was reflecting on this question as I started my next book - Jack Vance's classic The Dying Earth, a novel I've been meaning to read for ages - as I think the answer may be in the first chapter or even paragraph.
Basically, writers have audiences in mind when they write - and they try to direct the book towards them from the first word onwards. The authors of the three above novels all introduce the main themes of their novels - love and the agony of being apart, caring for each other and making sacrifices, and the role of women - in the first chapters, but they do so with only gradual hints at the sci-fi concepts to come. Dying Earth, on the other hand, begins outright as a fantasy novel with the protagonist's failing attempts to create magical life. This is an opening gambit to intrigue and draw in a sci-fi or fantasy reader - it worked for me! - but a more general reader might be turned away by this point. Perhaps they need some reassurance that the book is going to tackle Big Issues or reach Great Depths before they'll accept their six impossible things before breakfast.