Who cares who wrote Shakespeare? What matters is that the plays exist, and the author  whether he was a glover's son from Stratford or some better-connected and more fancily educated courtier  is necessarily absent from them. Novelists can editorialise in the margins of a narrative; a dramatist is unable to comment from the wings, and must allow actors to entice from the text whatever meanings they fancy. Plays are playful and relativistic by nature, and ask questions to which the only answers are provisional. Hamlet doesn't know why he delays, and Iago has no idea why he hates Othello. Shakespeare, whoever he was, couldn't have helped them out.

"Others abide our question. Thou art free," said Matthew Arnold in the poem he addressed to Shakespeare  free to be Francis Bacon or the Earl of Oxford, but equally free to be the mercenary actor who retired to Stratford after making his packet and spent his last years dealing in malt. Literary theory delights in what Roland Barthes called "the death of the author", because the writer's annihilation licenses the critic's self-display. But Shakespeare the author was stillborn: he fudged his own identity or conceded its irrelevance. As James Shapiro points out, he had scant interest in publishing his plays, and left his name off the title pages of his bestselling narrative poems.

Shapiro doesn't doubt that Shakespeare wrote Shakespeare. What interests him  in this wily, absorbing study  is how and why his authorship became a problem, and what ancillary purposes, political and religious as much as literary, the controversy has served. Justices of the US supreme court have offered verdicts on the legal conundrum, and amateurs like Malcolm X feel entitled to weigh in with ill-informed opinions. Many of the doubters Shapiro unearths are cranks, with one learned dunce persuading an ectoplasmic Shakespeare to dictate a confessional autobiography during a seance.

The debate involves conspiracy theories and cryptic cover-ups elaborate enough for a novel by Dan Brown. Those pressing the Earl of Oxford's claim allege that he was both the lover and the son of Elizabeth I, who had been incestuously impregnated at the age of 14 by her own stepfather. (This kinky lineage presumably qualified the earl to sympathise with family life at Elsinore.) At their oddest, the Oxfordians  a reprehensibly reactionary lot  surmise that recognition of their man's authorial rights might have averted England's collapse into civil war and revolution: if the Earl of Southampton, Oxford's son and brother, had ascended the throne after the death of Elizabeth I, social hierarchy might never have foundered and the country could have remained a forelock-tugging utopia.

During the 19th century, Shakespeare's supposed unworthiness laid bare other misgivings  adjustments necessary in a period when literature was promoted to the status of scripture and writers were expected to be sages, evangelists, national heroes. Dante or Goethe or Tolstoy possessed that lofty mental and moral superiority, but Shakespeare  with his real-estate deals and his puns, his rancorous marriage and his fondness for lewd farces  fell short. It was easier to believe in Bacon, a statesman and a philosopher, though his devout followers seldom mentioned that he fell from grace after sleazily pocketing bribes: the difficulty encountered again and again by Shapiro's controversialists is the "unbridgeable rift" between art and life, between creative genius and human imperfection.

As Mark Twain understood, the argument about Shakespeare was "curiously theological", and the most intellectually thrilling episodes in Shapiro's book concern efforts to comprehend a teasingly absent god. Freud's advocacy of Bacon matched his demolition of Moses in an essay that exposed the Jewish prophet as an Egyptian priest and thus deprived his co-religionists of "the greatest of their sons"; he advanced to another reckless act of deicide in his attempts to persuade the English to stop worshipping Shakespeare. After studying the Chandos portrait, Freud even declared that Shakespeare's face was "completely un-English", and proposed that he was actually French, his name a corruption of "Jacques Pierre". The notion would be absurd if the garbling of the transliterated words weren't so ingeniously Shakespearean.

The spurious quest for the true author of the plays had its origins in snobbery. Bardolaters found it hard to explain how a low-born fellow who never travelled and had small Latin could write about modern Venice or ancient Rome. Mark Twain disbelieved in Shakespeare because he thought that fiction had to be the finessing of fact, grounded in personal history. Summarising Shakespeare's life, Twain concluded  his stupefaction registered in capital letters  that "NOTHING HAPPENED TO HIM"; he therefore had nothing to write about. Yet Twain knew well enough that actors, like the busking mountebanks who perform bits of Shakespeare's history plays in Huckleberry Finn, can invent experience and conjure up alternative existences. That's why Coleridge's tribute to "myriad-minded" Shakespeare may be the truest thing ever said about him.

After all, it does matter who wrote Shakespeare, because the case Shapiro makes for him doubles as a defence of art. As he complains, the argument about the poverty of Shakespeare's experience refuses to acknowledge what he learned from books: all his plays are retold tales or commentaries on recorded history. More damningly, the doubters fail to credit Shakespeare with possessing imagination which, as Theseus says in A Midsummer Night's Dream, means the capacity to mould "airy nothing" into matter and create alternative worlds out of words. Contested Will ends with a challenge, issued as much to theory-addled academics as to deluded cultists: scepticism about Shakespeare signals an agnostic disrespect for what Shapiro bravely, bracingly calls "the mystery of literary creation". Some puzzles  like that of how this nondescript provincial came to be the greatest and most elusively polymorphous of writers  are best left unsolved.

If you seek his monument, wear a hard hat. For some years Stratford-on-Avon has been a building site while a new theatre grows by the riverside; traffic snarls on the bridge, and puzzled tourists mill glumly outside McDonald's, wondering where Shakespeare is to be found and why they're looking for him. There are no letters, James Shapiro says, no diaries, no authenticated portraits except the posthumous. The mystery man is almost 400 years dead, and yet still so powerful that his words can collapse an audience in gales of laughter or make them walk out of the theatre in nauseated shock.

History missed its chances with Shakespeare. His daughter Judith was still alive in 1662, at a time when scholars were beginning to take an interest in his life, but no one collected her testimony. Survivors remembered him: his fellow-actors, his rivals, his sometime collaborators. Ben Jonson laughed at his shaky geography  shipwrecks in Bohemia? He testified to the frantic pace of Will's invention, and said he loved him "on this side idolatry". But only a few dubious anecdotes are left. John Aubrey was told that Shakespeare preferred a quiet life; he was no "company keeper", and if his friends wanted to go on the town he would slide off home, saying he was "in pain". His grave keeps its secrets, and his monument, Shapiro admits, makes him look more like an accountant than an artist. The absence of frank autobiography is a source of pain to romantics. In his brilliantly readable 1599, a study of a decisive year in the playwright's life, Shapiro put it like this: "Shakespeare held the keys that opened the hearts and minds of others, even as he kept a lock on what he revealed about himself."

In that book Shapiro showed that, though we may have no access to the poet's inner workings, we do know quite a lot about the public career of the man who made a living in London as actor and playwright. We know enough to persuade a reasonable sceptic that there is only one, economical explanation for the plays: Shakespeare wrote them, mostly by himself, sometimes in collaboration. But why do so many people insist that the man from Stratford is an imposter, a fraud, a cover for some more illustrious name? Where did the controversy arise? What are its roots, and how did it grow and sustain itself?

It's a tale of snobbery and ignorance, of unhistorical assumptions, of myths about the writing life sometimes fuelled by bestselling authors who ought to know better. The trail is strewn, Shapiro says, with "fabricated documents, embellished lives, concealed identity, calls for trial, pseudonymous authorship, contested evidence, bald-faced deception, and a failure to grasp what could not be imagined". It is failure of imagination that has led successive generations of sceptics to imagine Shakespeare as their contemporary and assimilate his world to theirs, their judgments on his life and times guided by values that are anachronistic. Shakespeare's supporters, exasperated by the lack of traces he has left behind, have been tempted to forge some; but luckily for later generations, anachronism traps them too. One 18th-century poem, allegedly written by Shakespeare to Queen Elizabeth, described titled ladies drinking tea.

The argument from snobbery is basic to the debate and runs roughly as follows: Shakespeare was a glover's son from a provincial town, and therefore not very intelligent. He didn't go to university and had never travelled anywhere, or at least, not that we know. (Gaps in the record are by their nature suspicious, in this worldview.) Since the plays are sophisticated products of a finely tuned and knowledgeable mind, they could only have been written by a courtier with a lofty spirit and superb education, as well as superior experience of life. Step forward Francis Bacon, step forward Edward de Vere, Earl of Oxford. Shapiro concentrates on these claimants, once fashionable; nowadays Marlowe is gaining on them. He extends unfailing courtesy to the Shakespeare sceptics, both living and dead: which is more than the sceptics extend to the man from Stratford. Delia Bacon (no relation) was a 19th-century Baconian who called Shakespeare a "stupid, illiterate, third-rate play actor". Delia, who died in an asylum, had clinching evidence concerning a Baconian cipher, but refused to share it. Her views  which, as Shapiro says, embrace some provocative and original readings of the texts  were internationally disseminated, and influenced Mark Twain, who thought not only that Shakespeare didn't write Shakespeare, but that Milton, not Bunyan, wrote The Pilgrim's Progress. He also thought Queen Elizabeth was a man.

Twain had an admiring and eloquent relationship with the facts of his own life. He didn't believe that authors could produce work out of what they "only know about by hearsay", so Shakespeare's limited life-experience disqualified him. This belief did not stop Twain employing a sort of stunt-writer to prospect for diamonds in South Africa and gather material that Twain could use. The venture was thwarted by the surrogate's death from blood poisoning, after he stabbed himself in the mouth with a fork. Shapiro keeps an admirably straight face. But it does seem that, once you stop believing in Shakespeare, you'll believe in anything.

All the world is encoded; nothing is what it appears to be; the authorities are trying to deceive you; there is a gigantic conspiracy stretching from the playwright's contemporaries to the present-day heirs of the Shakespeare industry, the academics, the actors and the custodians of heritage tea shoppes. In the late 1890s a Shakespeare sceptic called Orville Ward Owen, a Detroit physician, built a decoding machine, a cumbersome apparatus involving rotating drums and a 1,000ft-long canvas sheet, a sort of intellectual mangle designed to wring out key words from texts not only of Shakespeare but of Marlowe, Spenser, Robert Greene and others. "There was," Shapiro says mildly, "a great deal of interpretive latitude."

As he conducts us through the pretensions of the Baconians, the Marlovians, the Oxfordians, and on through the latest internet conspiracy theories, larded with pompous quasi-legal language about "reasonable doubt" and "prima facie case", Shapiro sprinkles his text with glinting, steely facts, about the actors of Shakespeare's company, about Elizabethan printers and their methods, about what Shakespeare's manuscripts reveal about how his plays and stagecraft worked. These details, in the chapter which he devotes to Shakespeare himself, are the most riveting part of his book. The contrarian theories, faithfully and respectfully reported, become less interesting as they slide beyond parody. Francis Bacon was the love-child of Elizabeth and Leicester? The Earl of Southampton was the son of Elizabeth and the Earl of Oxford? The Virgin Queen, it seems, was never out of the labour ward. She had a child at 14 by Thomas Seymour; this child was Oxford, who was also her incestuous lover.

Shapiro does not waste words on the preposterous, but he does uncover the mechanism of fantasy and projection that go to make up much of the case against Shakespeare. His book lays bare, too, assumptions about the writing life that come to us from the 18th-century romantics. Those who made Shakespeare a demigod have much to answer for. They played into the hands of those who believed a writer could not also be, as Twain put it, a "grossly commercial wool-stapler". Shakespeare's retirement to Stratford causes problems to refined souls. His afterlife, Henry James sniffed, was "supremely vulgar". But if many of the surviving documents about him concern money, that does not mean that money was all he made.

Shapiro is at his most combative when he engages with the autobiographical approach to Shakespeare studies. Here, William must be saved from his friends as well as his foes. Are the plays encoded episodes from his life? Do the sonnets reveal his soul? Self-revelation, Shapiro persuades us, was not an early modern mode. What Shakespeare demonstrates is the authority of the human imagination. He commands the transpersonal; that is why he is a genius. If the scant facts of his life disappoint, that's our problem. A genius is also a man who needs to eat. As Thomas Heywood put it: "Mellifluous Shake-speare, whose enchanting Quill / Commanded Mirth or Passion, was but Will."

Hilary Mantel's Wolf Hall is published by Fourth Estate.