Science is one of humanity's greatest cultural achievements. By testing ideas against evidence, our species has been able to trace the course of evolution, to devise medicines that prolong lives, and even to glimpse the first microseconds of the universe. For all its success, though, there remains no shortage of people who doubt science's insights and achievements, sometimes preferring bizarre beliefs that count as unproven or disproved under its lens. Even the most compelling evidence cannot always dent the popularity of attractive but groundless stories.

This human reluctance to spurn a tall tale that we would like to be true is worth understanding, and the project on which Will Storr embarks in The Heretics: Adventures with the Enemies of Science is therefore an admirable one. In seeking out creationists and homeopaths, Holocaust-deniers and past-life regression therapists, he seeks not to mock strange convictions, but to get inside the minds of those who hold them.

The result is an entertaining journey dotted with some fascinating reportage. Storr tells good stories, which sometimes shed interesting light on the psychology and delusion of belief. His account of a tourist trip around concentration camps, with David Irving as a grumpy and self-absorbed guide, is outstanding, exposing both the historian's staggering ability to deceive himself, and a surprising diversity of behaviour among his neo-Nazi acolytes.

Other strong chapters detail morgellons  an itching syndrome that sufferers claim is caused by mysterious fibres, but which medicine does not recognise  and a damaged woman who disowned her family after "recovering" memories of Satanic abuse. In each case, Storr remains sufficiently dubious of improbable claims, but sufficiently open to the genuine distress of those who make them, to explore alternative explanations for their plight that fall the right side of Occam's razor.

It is the very quality of these chapters, however, that makes Storr's book disappointing and infuriating. For having shown he can achieve the kind of sceptical distance that makes Jon Ronson and Louis Theroux such effective chroniclers of weirdness, he casts it aside in an indulgence of wishful thinking. It is not enough for Storr to consider why people believe weird things; he also wants to challenge whether these things really are weird. He seems to accept, deep down, that they are, but he doesn't want to admit this. He is like the child who still wants to believe in Father Christmas, but who is just old enough to know better. Life would be more magical, more fun, if the story were true.

So it is that homeopaths are given a more sympathetic hearing than sceptics, with no discussion of the harm that unscientific medicine can do: Storr's storytelling does not extend to children like Neon Roberts, denied timely radiotherapy for his brain tumour as his mother sought an "alternative". Rupert Sheldrake, who posits that dogs can sense when their owners are coming home, shakes Storr's certainty that telepathy does not exist. "A new grey space has been nudged between the black and the white," he says. "And it is invigorating to have some mystery back."

For a book about the "enemies of science", Storr does remarkably little to engage with what science is. He explores the psychology and neuroscience of belief, explaining how all of us are prone to confirmation bias and cognitive dissonance: scientists are people, who fall into these traps from time to time. But he never considers that the great strength of the scientific approach is its recognition of these weaknesses of human thinking, and its attempts, albeit partial and imperfect, to erect defences against them. This is why randomised controlled trials, statistical significance and fair experimental design matter. They help us to avoid seeing only that which we find invigorating.

Also absent from The Heretics are the ideas of Thomas Bayes, the great statistician celebrated in Nate Silver's The Signal and the Noise. Bayes's insight was that a claim should be judged not only against new information about it, but also according to the prior probability of its truth. An observation that appears to contradict a well-attested phenomenon  such as faster-than-light neutrinos  is more likely to be noise than signal. To win support for hypotheses that are scientifically implausible  such as homeopathy, telepathy and morgellons  exceptional evidence is therefore needed. The onus is on the believer to find it, not on the sceptic to show it does not exist.

Without Bayes, Storr wastes time and space on trivialities that don't really matter. While his skill as an interviewer successfully paints James Randi, the sceptic champion, as an unpleasant bully who can be creative with the truth, he fails to recognise that this does not refute his arguments, nor confirm those of the heretics Randi debunks. "Bring on the psychics, bring on the alien abductees," says Storr. Yet even if Randi exaggerates from time to time, it remains far more probable that he, not the psychics, is right.

It is all very well to approach weird beliefs with an open mind, but as the aphorism goes, you must be careful not to leave it so open that your brains fall out.

Mark Henderson is author of The Geek Manifesto: Why Science Matters

Bertrand Russell's essay "On the Value of Scepticism" was published 85 years ago this year. Its opening lines are as relevant today as they were then: "I wish to propose a doctrine which may, I fear, appear wildly paradoxical and subversive. The doctrine in question is this: that it is undesirable to believe a proposition when there is no ground whatever for supposing it true." His proposal was "wildly paradoxical" because Russell understood that most of us do not use reason to form our beliefs. Most of what we believe is formed instead by an eclectic brew of formative influences, emotional hunches and selective concentration on evidence that supports our own prejudices. We are all victims of this "confirmation bias", but most of us like to think we'd have the good grace to change our minds if faced with incontrovertible evidence.

Enter Will Storr, whose first book Will Storr vs the Supernatural (2006) was a journalistic investigation of "the truth about ghosts". With The Heretics he has returned to similar ground and, as if taking on Russell's mantle, sets out to question individuals who persist in beliefs about the world that are, in his view, demonstrably false. "I like to write about these people," Storr confesses, "it is like being a tourist in another universe." At first glance the chapters of this book seem like a straightforward series of magazine pieces, with Storr in the role of objective journalist driven to uncover the truth, then persuade others of his arguments. We expect him to poke fun at some of the wackier believers  the UFO-spotters, yoga fundamentalists and past life regression fanatics have some rollicking arguments with the belligerents among them, and emerge crowned in the conclusion, a champion for science. But as you read on, something darker, more complex, and potentially more interesting begins to unfold. The other universes he tours are brought rather closer to home: the heretics "are magic makers", he tells us, and he feels "kinship" with them. This is not what we expect of the sceptical journalist. He confounds other expectations by being naturally shy, and by refusing to champion anything. He confides that he likes being a journalist because it frees him from the demands of social conversation, and that he is at ease in only two situations: with a voice recorder and a list of questions, or lying on his therapist's couch.

Storr can open chapters like a stage conjurer, and his prose has an easy, laconic style embracing Jon Ronson's taste for the fabulously weird and Louis Theroux's ability to put his subjects at ease. He is a funny and companionable guide (though occasionally he breaks his flow with excessively transcribed dialogue). Current neuropsychology is a theme, and the two chapters he spends exploring theories of mind, emotion and intuition are among the best.

As the book goes on, we understand it to be as much about the author's own therapy as it is about the subjects of his investigations. We hear how he dropped out of college, fought with his parents, stalked a former girlfriend, fell into drug abuse and petty crime. We see him anguished, contemplating suicide, and masturbating on a meditation retreat. For the most part this level of candour isn't distracting, but endearing. But by the end of the book I began to worry a little for him, too.

The book opens with a quote from the psychologist Jerome Bruner: "A self is probably the most impressive work of art we ever produce." Storr wants us to understand that it is not just "heretics" but all of us who are riddled with untested beliefs. In his search to understand how we construct our inner worlds he talks to chronic schizophrenics who have fought against the prevailing medical model of their disease, and attempted to view the particularities of the way they see the world more creatively. He spends time with a creationist, trying to understand how he can explain away the fossil record. He has a chilling encounter with a therapist whom he suspects of implanting false memories of demonic sexual abuse in the minds of vulnerable patients. At a "sceptics" conference he questions some self-righteous individuals with a passionate hatred of homoeopathy, only to discover that none of them has read the research that they refer to. One of the most pitiable groups he encounters are the sufferers of delusional parasitosis, who scratch their skins until they bleed because of a belief that they are infested by insects ("morgellons") despite the lack of any evidence to support them. But it is towards the end of the book, when he meets the right-wing ideologue Lord Monckton and the Holocaust denier David Irving, that he begins to excavate darker and more dangerous territory  the places where humanity's wilful self-delusion can lead. Going undercover with some neo-Nazis he finds himself standing in a gas chamber, listening to Irving point out how the chamber is a fake, a "typical Polish botch job". Shuddering with horror, Storr turns away from the group and begins to cross himself.

And so the fearless defender of reason finds himself warding off evil with a gesture from a religion that he professes to have disowned. This is a revelatory moment: he has demonstrated that we are all capable of prejudice, and each of us is an "enemy of science" in our own way. But what matters most to Storr is that we don't find ourselves enemies of compassion and understanding as well. We can believe what we like, as long as we don't hurt others with our beliefs. The book could have ended there, but there are two more chapters (on ESP and the arch-sceptic James Randi) and then an epilogue, none of which manages to build on Storr's moment of clarity in the gas chamber  the real climax of this uneven but thought-provoking book.

I began reading The Heretics thinking that Bertrand Russell, that old pipe-smoking pacifist, would have approved of it, but by the end I was doubtful. About the widespread adoption of sceptical rationality Russell wrote: "these propositions may seem mild, yet, if accepted, they would absolutely revolutionise human life." He wanted to see reason adopted more thoroughly, despite knowing that it would "diminish the incomes of clairvoyants, bookmakers, bishops, and others who live on  irrational hopes". But Storr doesn't want to do away with his heretics  their beliefs "glitter and pulse and enchant" while the cold rationalists look on glumly, touting "glitter-extinguishers". His enthusiasm for the straggling cavalcade of human beliefs is at odds with some of his other conclusions, but maybe it's churlish to expect consistency. "Bring on the psychics," he declares towards the end. "Bring on the alien abductees  bring on a hundred of them  Wrongness is a human right."

 Gavin Francis's Empire Antarctica is published by Chatto.