The game is a familiar one. Six Degrees of Kevin Bacon, they call it in the film world. Or, in the words of the old song: "I've danced with a man, who's danced with a girl, who's danced with the Prince of Wales."

One on One daisy-chains 101 stories of chance encounters between well-known figures, beginning with the story of a man who (alas, not going fast enough) ran over Adolf Hitler in 1931, and ending with Hitler's 1937 meeting with the Duchess of Windsor. In between, through a series of handshakes and bows and snogs and awkward silences, it ranges from the late 19th century to the early 21st, from Moscow to California, and from Rudyard Kipling to George Lazenby.

In a pleasingly Oulipian conceit, as Craig Brown explains in an afterword, "I have described each of the 101 meetings in exactly 1001 words, which makes One on One 101,101 words long. The acknowledgements, prefacing quotes, author's blurb, book's blurb and list of my other books each consist of 101 words, as does this note." Lovely to imagine that Brown's publication history prior to this book could have been deliberately engineered to ensure his back-catalogue was 101 words long.

For those who know Brown best as a parodist, this book will come as a surprise. Though often very funny, it's a work of straight non-fiction whose great virtue is not excess, but restraint. The humour is biscuit-dry. "Any meeting between the living and the dead is inevitably one-sided," Brown opens one story, "Do they know something we don't know?" GI Gurdjieff, he writes, "is the self-proclaimed pioneer of a revolutionary new school of 'objective' music, the first ever to produce exactly the same reaction in all its listeners." Throwaway details are allowed to speak for themselves: "One night, the beat poet Gregory Corso drops by, and falls asleep while reading Patti [Smith]'s poems."

Incongruity is one of its pleasures. The quintessential Craig Brown encounter would probably be Ted Heath and Wee Jimmy Krankie, or George Steiner and Benny from Crossroads. Here we get Marianne Faithfull chatting to WH Auden about how to smuggle drugs ("Do you pack them up your arse?" "Oh no, Wystan. I stash them in my pussy"), and Bertrand Russell making cucumber sandwiches for Sarah Miles: "Two years ago, his ninetieth birthday was celebrated with a festschrift with contributions from, among others, Dr Martin Luther King, Leonard Bernstein, Jawaharlal Nehru, Kenneth Kaunda, U Thant, Albert Schweitzer and David Ben-Gurion. But Sarah mixes in different circles: Robert Morley, Eric Sykes, Terry-Thomas, Flora Robson and Benny Hill are all in her next film."

Brown relishes folly and absurdity: his tone is at once amused and wanly forgiving. Frauds and cranks bring out the best in him. Blood-soaked dictators are juxtaposed with half-forgotten figures of light entertainment, great writers and composers with gadabouts and chancers. It feels almost like a medieval story-cycle designed to show the whole human comedy, the fickleness of Fortune, the vanity of human wishes. Celebrity  which is what all Brown's characters have in common  is a stuff that won't endure.

The detail is all. Brown has the certainty of touch, when talking about Kenneth Tynan and Truman Capote, to inform us in parentheses that their middle names are, respectively, "Peacock" and "Streckfus". He quotes sharply, too. The Duke of Windsor "always had something of  riveting stupidity to say on any subject"; or the British ambassador, of Theodore Roosevelt: "You must always remember that the President is about six."

Footnotes are used to land quick, precise jabs. Of the downfall of the chat-show host Simon Dee, a footnote tells us: "A brand-new late-night chat show is hosted by a relative unknown, whose name is Michael Parkinson. Dee's slot on the BBC is given to the actor Derek Nimmo, in If It's Saturday, it Must be Nimmo. Among Nimmo's first guests is Basil Brush, a leading glove puppet." That's comic  but, to me at least, it is also loaded with pathos. The great deeds that live in history are never so poignant as the little human sillinesses that are rescued from it, and what's most unexpected about this book is how moving these stories are, both in their particulars and in aggregate.

When Helen Keller visits her friend Mark Twain's house after his death, she rescues a geranium and plants it in her own garden: "It always seems to say the same thing to me, 'Please don't grieve.' But I grieve, nevertheless." There's Madonna, as a young dancer, encountering the formidable Martha Graham in a corridor and being too afraid to speak  then, 10 years later, writing the cheque that saves the Martha Graham Dance School from bankruptcy: "When Martha Graham, now aged ninety-four, is presented with the cheque, she bursts into tears." There's Cecil Beaton's farewell to the gaga Harold Nicolson. And there's the disgraced Michael Barrymore, "spotted working part-time at a vehicle bodyworks business in Epping".

My only beef is that Brown's determination to tell all his stories in the present tense snarls him up. Because most of the encounters contain some back-story for each participant, some subsequent material, and some discussion of how we know what we know, tenses get muddled. One story opens by telling us that "James Dean is sitting at a table in his favourite little restaurant"; then we get a snatch of Alec Guinness's back-story in the pluperfect, a further snatch in the past simple, then Guinness "arrives in Hollywood", then he and companion "are having difficulty finding a table", then James Dean runs out and invites him in.

But that's a minor technical cavil about a rich and hugely enjoyable book that looks with affection and melancholy on the whirling roundabouts of history and celebrity, and reminds us that the paths of glory lead, handshake by handshake, pratfall by pratfall, to the grave.

Sam Leith's You Talkin' to Me? is published by Profile Books.