While all novelists are called upon to fictionalise reality, Jake Arnott stands out as a dark prince of confabulation. In books such as The Long Firm and He Kills Coppers he yoked together the actual and the invented with such verve and authority that it was often difficult to know where history stopped and his stories began.

Both those novels were set in or against the London underworld of the 1960s and 1970s, a cliche-rich milieu from which Arnott unearthed fresh nuggets of subversion and transgression. Yet with his potent blend of forensic excavation and renegade imagination, it seemed odd that Arnott should restrict himself to London crime.

His last book, The Devil's Paintbrush, duly ventured further afield, into the occult world of Aleister Crowley and Belle Epoque Paris. Though a pleasing digression, it lacked the unsettling verisimilitude of its predecessors. But instead of retreating to familiar territory, Arnott has returned with his most ambitious work to date.

The House of Rumour is a novel that seeks to fold time and space into a series of linked situations: Rudolph Hess's 1941 flight to Scotland; early rocket science; the British wartime intelligence service, and, once again, Crowley's paranormal myth-making and sexual libertarianism. There are walk-on parts for Hess, Crowley, Cyril Connolly, Ian Fleming, Jim Jones (of drink-the-Kool Aid cult-death infamy) and L Ron Hubbard; but the plot focuses on a transsexual with a detachable penis, a closeted British pop singer and an American pulp science-fiction writer named Larry Zagorski.

I say "plot", but it's a multiple-narrator jumble of ideas, outlandish theories and literary spoofs loosely tied to a stolen classified file on Hess's mysterious wartime "peace" mission to Britain. As though appealing to one of his conspiracy-fed characters, Arnott floats the possibility of narrative convergence, the idea that all these apparently disparate people and random events are secretly connected. The notion of a shadowy organising force is further suggested by the structure  22 chapters that follow the 22 major arcana cards in the tarot deck. Like much else in this teasingly entertaining book, it's a clever conceit, a game of smoke and mirrors.

The novel opens with Zagorski recalling how he came by the confidential document that was intelligence officer Sir Marius Trevelyan's personal account of the Hess affair. It was stolen from Trevelyan, Zagorski explains, by a transvestite prostitute named Vita Lampada (itself a literary reference to Sir Henry Newbolt's famous poem). This being Arnott, we learn, soon enough, that Lampada is based on Vikki de Lambray, a real transvestite prostitute who died in suspicious circumstances in August 1986.

The book could almost work as a compendious excuse to exhume neglected figures from the past, because Arnott assembles a fascinating cast of misfits who have been consigned to the margins of history. There is Jack Parsons, the rocket scientist and Crowley devotee who set up a hedonistic commune in California in the 1940s, before accidentally blowing himself up. And Katharine Burdekin, the British feminist who in 1937 wrote Swastika Night, a work of visionary dystopia that has been compared to Nineteen Eighty-Four. But this novel is more than a collection of obscure biographies; it's also about timing and dislocation, and how life and history rest on what sci-fi readers may know as a "Jonbar Hinge", a point at which the future could have taken a different path.

Perhaps most of all, it's a book about the uncertainty of writing. In mid-career, Zagorski publishes "a cycle of 22 interconnecting stories structured around the trump cards in the tarot deck". The Village Voice hails it a "meta-fictional masterpiece", while the New York Times calls it "a confused and self-indulgent mess".

If this is the dark prince Arnott's Jonbar Hinge, the future looks impressively bright.

Marshalling a cast list crammed with famous authors actual and invented, The House of Rumour also features some cheeky self-references to the writer whose sixth novel we are reading. An American scifi novelist abandons an idea because she doesn't want to get caught "in that world of fiction where reality and fantasy start to coincide", a genre that, Arnott readers smilingly appreciate, would prominently include his novels The Long Firm (1999), in which imagined gangsters mix with historical figures from the 1960s, and The Devil's Paintbrush (2009), which dramatises an incident in the life of the writer and occultist Aleister Crowley.

At another point in The House of Rumour, a made-up novelist  a Californian pulp fiction writer called Larry Zagorski  reads a real book, Nightmare Alley, published in 1947 by William Lindsay Gresham. Zagorski comments that "using the Major Arcana as a structure looks like a gimmick at first but in the end the Tarot bestows an ominous gravity on the narrative". In a typical example of the games this novel plays, this analysis comes in a rare chapter in the book that doesn't take its title from the Major Arcana of the Tarot cards. Gresham's wife, Joy Davidman, left him to marry CS Lewis, an affair fictionalised in William Nicholson's film and play Shadowlands, so he is a particularly appropriate figure to turn up in a story that delights in unexpected connections between literary figures and historical events.

As the book opens, in 2011, the ageing Zagorski, a nonagenarian relic of the mid-20th-century golden age of scifi, is reading a newspaper obituary. It is of Sir Marius Trevelyan, a former British intelligence officer, who was involved in the bizarre incident in 1941 when the Nazi Rudolph Hess crash-landed in Scotland, apparently seeking to negotiate peace terms.

Loosely linked by the literary and romantic career of Zagorski over 70 years, the novel constantly switches points of view, a fragmentary technique Arnott has favoured since The Long Firm. Some sections are told from the perspective of Hess, both on his Scottish flight and subsequently as a prisoner in Berlin, in the joint custody of four powers.

The most significant participants in the story, though, are all famous writers. Continuing the fascination he showed in The Devil's Paintbrush, Arnott again employs Aleister Crowley as a catalyst. These scenes are heavily informed by Richard Spence's recent biography, Secret Agent 666, which suggested that the Satanist was a wartime triple agent. It's probably sensible that the novelist only alludes in one line to claims that Crowley held a black mass in Sussex, attended by Churchill, in an attempt to influence the war effort.

What most engages Arnott are the suggested connections between Crowley and Ian Fleming, another in the book's relay of historical antagonists, who, as an intelligence officer, may have worked with the necromantic in a false propaganda unit during the second world war. Crowley also influenced another of the characters, the American scifi writer L Ron Hubbard, who went on to form the Church of Scientology.

However, the most compelling of the real-life personae whom Arnott animates is Katharine Burdekin (1896-1963), a peripheral member of the Bloomsbury Group who, under the pseudonym Murray Constantine, in 1937 published Swastika Night, a future fantasy in which the Nazis have achieved global domination and Hitler is worshipped as a god. Although written before the war, the book can be read as prophesying various events, including even the flight of Hess; one of The House of Rumour's recurrent concerns is the possibilities of predicting or affecting the future, through scifi or darker arts.

In a novel filled with open or oblique cultural references and allusions, readers may also detect some undeclared ones. The structure, in which apparently unrelated narratives accumulate into an alternative history of the 20th century, hints at the sympathetic effect of Don DeLillo's epic of American paranoia, Underworld. Like DeLillo, Arnott makes clever, subtle connections between the discrete narratives: words such as "mistletoe" and "labyrinth", for example, echo resonantly between the sections.

I also suspect the influence of the work of the documentary-maker Adam Curtis. As in Curtis's series such as The Power of Nightmares, there is a gleeful, teasing joining of improbable dots. The creator of James Bond interrogates, in wartime, the author of an apparently prophetic scifi novel; an American scifi writer spots an opportunity in American tax laws to create a powerful religion with a grip on Hollywood stars; and Rudolph Hess, in jail in Germany, develops an obsession with the Apollo moon shots, on which some of his former German colleagues worked. For Arnott, as for Curtis, history does not fall neatly into periods and presidencies, as it is taught at school, but accretes secretly and strangely in a sweep across decades or even centuries.

Unlike Curtis, though, whose softly ironic narrating voice drives his films, Arnott vanishes behind an impressive facility with voices and pastiche. Most chapters introduce a new style as well as viewpoint: the doings of Commander Ian Fleming are recounted in a James Bond parody; the story of Katharine Burdekin is framed as Quentin Bell-like diary entries; and the career of Zagorski is told through a perfectly captured fan magazine profile.

The fact that Arnott's early books used the form of gangster yarns encouraged a tendency to categorise him as a sort of bookish Guy Ritchie. Recent novels, though, have broken free from this narrow definition of his range and this novel confirms his escape. A conspiracy thriller filled with bewildering connections, dark conjecture and arcane information, The House of Rumour perhaps most resembles The Da Vinci Code, rewritten by an author with the gifts of characterisation, wit and literacy. It may be the ideal holiday read for those who like to take their brains with them on vacation.

 Mark Lawson's Enough is Enough, or the Emergency Government is published by Picador.