After a rich year for fiction, the novel most likely to be placed under the Christmas tree will surely be Julian Barnes's Booker winner, The Sense of an Ending (Jonathan Cape, £12.99). A meditation on memory and regret slyly conveyed through the unreliable voice of a complacent man whose past gives him a nasty surprise, it's slim enough to gobble at a sitting and slips down with deceptive ease, but leaves plenty to ponder in its wake. The hardback is also a thing of beauty in its own right.

Also small but perfectly wrought, At Last by Edward St Aubyn (Picador, £16.99) is the fifth and final volume in his series about abuse, addiction and other bad behaviours among the English upper classes. It's savagely funny stuff, and a fitting conclusion to a saga that's been one of the literary highlights of our time. Alan Hollinghurst teased out the literary establishment's path through the 20th century in The Stranger's Child (Picador, £20), elegantly unpicking myths and customs of Englishness as he traces the secret life and afterglow of a country house and a Georgian poem.

For more rambunctious fare, turn to Carol Birch's Jamrach's Menagerie (Canongate, £7.99), the most irrepressible read of the year. A young boy is plucked from the streets of 19th-century Wapping and launched on the high seas, joining a quest to capture and bring back a komodo dragon. His story is full of wonder, peril and discovery. Animals also cavort through the picaresque Orange winner, The Tiger's Wife (Phoenix, £7.99) by Téa Obreht. Through a mixture of folklore and autobiography, she paints a vivid portrait of Yugoslavia's history and the Balkan wars.

This year saw several fine novels that opened on to parallel dimensions, including the much-anticipated 1Q84 (Harvill Secker, £20 & £14.99) by Haruki Murakami (which in two volumes means double the wrapping). There are cults, conspiracies and lost lovers aplenty in this vast labyrinth of a novel, in which the parallel universe is lit by a second moon. Beautifully strange, too, is John Burnside's A Summer of Drowning (Jonathan Cape, £16.99), now on the Costa shortlist. An impressionistic study of a withdrawn young woman living on a small island in the Arctic Circle, it brings an eerie glow to the colours and sounds, flora and foodstuffs of the far north. With Mr Fox (Picador, £12.99), meanwhile, Helen Oyeyemi tumbled together fairy stories, feminism and screwball comedy in a phantasmagoria that's both witty and weird.

In A Visit from the Goon Squad (Corsair, £7.99), American author Jennifer Egan takes on time and compromise, old music and new technology, in a mosaic of styles and voices. It follows a gang of friends and enemies linked by the music business from the late 1970s to the near future, and is required reading for anyone worried about a visit from that goon, the relentless passing of time. There's more wry humour of a dark, dry kind in Anne Enright's The Forgotten Waltz (Jonathan Cape, £16.99), the tale of an affair and its repercussions related over one snowbound winter day at the end of the Irish boom years, while Ali Smith is in playful mood with There but for the (Hamish Hamilton, £16.99), about a dinner guest who rebels.

Sherlock Holmes fans have been well served by Anthony Horowitz's new adventure, The House of Silk (Orion, £18.99), in which an elderly Watson sets down an early case too shocking to be published in his lifetime. More modern thrills were provided by Robert Harris's The Fear Index (Hutchinson, £18.99), a financial what-if melding artificial intelligence with hedge funds, and Neal Stephenson's Reamde (Atlantic, £18.99), a globetrotting jeu d'esprit rejoicing in espionage, virtual reality and plenty of guns. Terror on a domestic scale, meanwhile, was ingeniously evoked by SJ Watson in Before I Go to Sleep (Doubleday, £12.99), about an amnesiac struggling to work out  and then to remember  what has happened to turn her world upside down.

Two masters of the fantasy behemoth were on fighting form this year: George RR Martin with A Dance With Dragons (Harper Voyager, £25), the latest in his Ice and Fire series, and Stephen King with 11.23.63 (Hodder & Stoughton, £19.99), a time-travel epic about 60s America and an attempt to avert the death of JFK. China Miéville turned to science fiction in Embassytown (Macmillan, £17.99), a remarkable investigation into the difficulties of communication and the possibilities of language set on a far-off planet where humans and aliens coexist; while The Night Circus (Harvill Secker, £12.99) by debut author Erin Morgenstern conjured a delicate fin-de-siècle world of enchantment in a tale of magicians and circuses that will appeal to fans of Jonathan Strange.

Three short story collections stand out this year: Roddy Doyle's tales of deceptively "ordinary" middle-aged men in Bullfighting (Jonathan Cape, £12.99); Don DeLillo's The Angel Esmeralda (Picador, £16.99), nine pieces spanning 30 years; and Sarah Hall's fierce and sensuous The Beautiful Indifference (Faber, £12.99). Each will surprise, move and provoke. But if you want something more left-field, snap up Diego Marani's New Finnish Grammar (Dedalus, £9.99). The story of a mysteriously wounded man sent to Helsinki amid the chaos of the second world war to try to rebuild his memory and identity, it's a strange, tender portrait of Finnish legends and language-learning, loneliness and human connection.

 Which novels would you give this year?

The Sense of an Ending is a short novel, but one that packs in a lot. Full of insight and intelligence, it is in some ways a more intellectual version of Ian McEwan's On Chesil Beach, touching on the same themes of youthful sex, inhibition, class, regret and false recollection. It is the story of a retired sixtysomething man, Tony Webster, a relatively dull and "peaceable" character, once in arts administration, who seems, while broadly accepting his own decline, to be trying to impose a pattern on his past. Barnes has taken his title from Frank Kermode, who in his 1965 book, The Sense of an Ending, explored the way in which writers use "peripeteia"  the unexpected twist in the plot  to force readers to adjust their expectations. Barnes has visited the subject of death two or three times recently, most directly in his 2008 nonfiction work, Nothing to Be Frightened Of, and he is fascinated by how people deal with death, and the changed circumstances it can bring to the surviving partner. So we rightly come to suspect that this novel is setting the reader up for peripeteia.

As the book opens, Tony is spending a good deal of his time thinking about his relationship with his schoolfriend Adrian Finn, who committed suicide as a young man. Tony is Pooterishly content with his life and mediocrity (exemplified by his 2:1 from Bristol), which he contrasts with Finn's burning and forensic intelligence. Even as a schoolboy, Finn demonstrated a precocious understanding of philosophy and history. He said that "he hates the way the English have of not being serious about being serious"  words quoted twice in the novel. This is a Barnesian theme too, lying behind much of his work, fiction and nonfiction, even when it is at its most playful.

Tony attended school in the 1960s, and many of his memories centre on him and his friends grappling with the new sexual freedoms. Not that these always arrived in their neck of the woods quite in time: one of their schoolmates killed himself after getting a girl pregnant. At the end of their school careers, Adrian Finn goes to Cambridge and Tony to Bristol, where he meets Veronica Ford. Veronica invites him to stay for a few days with her family in Chislehurst, and he feels himself to have been humiliated by her disdainful father and supercilious brother, both of whom are intimidatingly posh. But Veronica's mother, Sarah, takes to him, and even appears to offer him a mysterious warning about her daughter.

Back in Bristol, the relationship between Veronica and Tony is full of awkward skirmishing around "full sex" and the meaning of love. After a year or two and a single, unappealing sexual experience, they split up; Veronica is very bitter. Finn later writes to Tony to say that he is going out with Veronica, and hopes that his friend doesn't mind. He does. Finn and Veronica marry, and eventually, Tony learns, Finn slits his wrists in a bath. The circumstances are unclear but Tony is sure they must have some sort of logic to them.

As he pores over his relationship with Veronica it becomes clear that this is a novel largely concerned with how, in the course of a life, we edit and erase our memories. Tony, we begin to suspect, is very prone to self-redaction. Now, in the present, he suddenly receives a letter from a solicitor dealing with the estate of Sarah Ford, Veronica's mother, saying that she has left him £500 and a diary kept by Adrian Finn. He is puzzled, and tries to get in touch with Veronica. She claims to have burnt the diary after her mother's death. Very quickly, vertiginously, the whole story becomes much darker and ultimately shocking, as Tony finally understands  as do we  what Veronica meant when she said repeatedly: "You just don't get it, do you?"

Deservedly longlisted for the Man Booker prize, this is a very fine book, skilfully plotted, boldly conceived, full of bleak insight into the questions of ageing and memory, and producing a very real kick  or peripeteia  at its end. As Kermode wrote: "At some very low level we all share certain fictions about time, and they testify to the continuity of what is called human nature" Barnes has achieved, in this shortish account of a not very attractive man, something of universal importance.

In Nothing to Be Frightened Of, his family memoir cum meditation on mortality, Julian Barnes admits that he and his brother disagree about many details of their childhood. His brother, a philosopher, maintains that memories are so often false that they cannot be trusted without independent verification. "I am more trusting, or self-deluding," writes Barnes, "so shall continue as if all my memories are true."

The narrator of his Booker longlisted new novella has always made that same reasonable assumption, but the act of revisiting his past in later life challenges his core beliefs about causation, responsibility and the very chain of events that make up his sense of self. This concise yet open-ended book accepts the novelistic challenge of an aside in Nothing to Be Frightened Of: "We talk about our memories, but should perhaps talk more about our forgettings, even if that is a more difficult  or logically impossible  feat."

Like so many of Barnes's narrators, Tony Webster is resigned to his ordinariness; even satisfied with it, in a bloody-minded way. In one light, his life has been a success: a career followed by comfortable retirement, an amiable marriage followed by amicable divorce, a child seen safely into her own domestic security. On harsher inspection, "I had wanted life not to bother me too much, and succeeded  and how pitiful that was." Barnes is brutally incisive on the diminishments of age: now that the sense of his own ending is coming into focus, Tony apprehends that "the purpose of life is to reconcile us to its eventual loss", that he has already experienced the first death: that of the possibility of change.

But like all of us, he has carried his youth inside him into adulthood, fixed in vivid memory. Looming largest in his personal mythology is his brilliant, tragic, Camus-reading schoolfriend Adrian (another echo of Nothing to Be Frightened Of here: in that book Barnes remembers a similar friend by the fitting but unlikely name of Alex Brilliant). It is a solicitor's letter informing him that, 40 years on, he has been left Adrian's diary in a will, that sets Tony to examining what he thinks his life has been.

The novella divides into two parts, the first being Tony's memoir of "book-hungry, sex-hungry" sixth form days, and the painful failure of his first relationship at university, with the spiky, enigmatic Veronica. It's a lightly sketched portrait of awkwardness and repression at a time when yes, it was the 60s, "but only for some people, only in certain parts of the country". In one of the book's many slow-rumbling ironies, the second section undermines the veracity of these expertly drawn memories, as Tony reopens his relationship with Veronica, a woman he had previously edited out of his life story.

It was a "slightly odd thing", he cautiously admits, to pretend to his ex-wife when they first met that Veronica had never existed (and then later give such a one-sided account of her that she's known within their marriage as "The Fruitcake"). Barnes builds a powerful atmosphere of shame and silence around the past as Tony tries to track down the elusive diary, which promises, as missing diaries tend to do, some revelation or closure. In a book obsessed with evidence and documentation  verification for unreliable, subjective memory  the most powerful depth charge turns out to be something forgotten yet irrefutable that Tony has kept from himself for 40 years. With it Barnes puts the rest of the narrative, and his unreliable yet sincere narrator, tantalisingly into doubt.

There's the atmosphere of a Roald Dahl short story to Tony's quest; the sense that, with enigmatic emails and mysterious meetings in the Oxford Street John Lewis brasserie, he is somehow being played or manipulated by others. "You don't get it. You never did," Veronica tells him repeatedly. A secret permeates the text, heavily withheld. But this schematic element pales beside the emotional force of Tony's re-evaluation of the past, his rush of new memories in response to fresh perspectives, and the unsettling sense of the limits of self-knowledge. As ever, Barnes excels at colouring everyday reality with his narrator's unique subjectivity, without sacrificing any of its vivid precision: only he could invest a discussion about hand-cut chips in a gastropub with so much wry poignancy.

With its patterns and repetitions, scrutinising its own workings from every possible angle, the novella becomes a highly wrought meditation on ageing, memory and regret. But it gives as much resonance to what is unknown and unspoken  lost to memory  as it does to the engine of its own plot. Fiction, Barnes writes in Nothing to Be Frightened Of, "wants to tell all stories, in all their contrariness, contradiction and irresolvability". The Sense of an Ending honours that impossible desire in a way that is novel, fertile and memorable.