Obama aside, the indubitable hero of the 2012 US presidential election was the statistician and political forecaster Nate Silver. His blog, FiveThirtyEight.com, syndicated by the New York Times since 2010, correctly predicted the results of the election in 50 out of 50 states. When the worldwide media was universally proclaiming the race too close to call and the pundits were deriding mathematical models, FiveThirtyEight.com steadily argued that the odds made clear that Obama would win. On election day, Silver's final forecast was that Obama had a 90.9% chance of winning.
Reflecting on the electoral impact of Hurricane Sandy, Silver was the voice of sanity in the last few days of the race. On 5 November he suggested that "historical memory" might consider Sandy pivotal, but in fact Obama had been rebounding slowly but surely in the polls since his lows in early October. Listing eight alternative explanations for Obama's gains after the storm hit including recent encouraging economic news Silver concluded that the gains were "over-determined": a lot of variables might have contributed to the one result.
As the votes were counted and the states declared themselves, vindicating the FiveThirtyEight.com predictions in every single case, Silver's newly published book became an overnight bestseller.
The first thing to note about The Signal and the Noise is that it is modest not lacking in confidence or pointlessly self-effacing, but calm and honest about the limits to what the author or anyone else can know about what is going to happen next. Across a wide range of subjects about which people make professional predictions the housing market, the stock market, elections, baseball, the weather, earthquakes, terrorist attacks Silver argues for a sharper recognition of "the difference between what we know and what we think we know" and recommends a strategy for closing the gap.
Recognition of the gap is not new: there are plenty of political theorists and scientists droning on about it already, in the manner of the automated voice on the tube when train and platform don't quite meet. Strategies for closing, or at least narrowing, the gap between what we know and what we think we know in specific contexts, are rarer, specialised, and probably pretty hard for anyone outside a small circle of experts to understand.
What Silver has to offer is a lucid explanation of how to think probabilistically. In a promising start, he claims that his model based on a theorem inspired by Thomas Bayes, the 18th-century English mathematician has more in common with how soldiers and doctors think than with the cognitive habits of TV pundits. "Much of the most thoughtful work I have found on the use and abuse of statistical models, and on the proper role of prediction, comes from people in the medical profession," Silver reports. You can quite easily get away with a stupid model if you are a political scientist, but in medicine as in war, "stupid models kill people. It has a sobering effect".
Silver is not a medical doctor, even if a version of the Hippocratic oath Primum non nocere (First, do no harm) is the guiding principle of his probabilistic thinking: "If you can't make a good prediction, it is very often harmful to pretend that you can." After graduating from Chicago with a degree in economics in 2000, he worked as a transfer-pricing consultant for the accounting firm KPMG: "The pay was honest and I felt secure," but he soon became bored. In his spare time, on long flights and in airports, he started compiling spreadsheets of baseball statistics that later became the basis for a predictive system called Pecota.
Silver delivers a candid account of the hits and misses of Pecota, the lessons learned and the system's limitations: "It's hard to have an idea that nobody else has thought of. It's even harder to have a good idea and when you do, it will soon be duplicated."
After his interest in baseball peaked, he moved on to predicting electoral politics. The idea for FiveThirtyEight (named after the 538 votes in the electoral college) arrived while Silver was waiting for a delayed flight at New Orleans airport in 2008. Initially, he made predictions about the electoral winners simply by taking an average of the polls after weighting them according to past accuracy. The model gradually became more intricate: his method centres on crunching the data from as many previous examples as possible; imagine a really enormous spreadsheet. He accurately forecast the outcome of 49 out of 50 states in the 2008 presidential election and the winner of all 35 senate races.
Challenged by the economist Justin Wolfers and his star student David Rothschild as to why he continues to make forecasts through FiveThirtyEight despite fierce competition from larger prediction websites such as Intrade (which covers "everything from who will win the Academy Award for Best Picture to the chance of an Israeli air strike on Iran") Silver replies: "I find making the forecasts intellectually interesting and they help to produce traffic for my blog." His unabashed honesty seems the open secret of his success.
Bayes, who lends his name to Silver's theorem, was "probably born in 1701 although it might have been 1702". Silver is a statistician, not a historian, so he reports the fact of the uncertainty without elaboration. As a Nonconformist, Bayes could not go to Oxford or Cambridge, but was eventually elected a fellow of the Royal Society. His most famous work, "An Essay toward Solving a Problem in the Doctrine of Chances", was published posthumously in 1763. Silver summarises it as: "a statement expressed both mathematically and philosophically about how we learn about the universe: that we learn about it through approximation, getting closer and closer to the truth as we gather more evidence."
The attraction of Bayes's theorem, as Silver presents it, is that it concerns conditional probability: the probability that a theory or hypothesis is true if some event has happened. He applies the theorem to 9/11. Prior to the first plane striking the twin towers, the initial estimate of how likely it was that terrorists would crash planes into Manhattan skyscrapers is given as 0.005%. After the first plane hit, the revised probability of a terror attack comes out at 38%. Following the second plane hitting the revised estimate that it was a deliberate act jumps to 99.99%. "One accident on a bright sunny day in New York was unlikely enough, but a second one was almost a literal impossibility, as we all horribly deduced."
Fastidiously aware of the gap between what we know and what we think we know, Silver proceeds wryly to delineate the limits of what he has achieved with this application of Bayes theorem to 9/11: "It's not that much of an accomplishment, however, to describe history in statistical terms."
Silver ends by advocating a balance between curiosity and scepticism when it comes to making predictions: "The more eagerly we commit to scrutinising and testing our theories, the more readily we accept that our knowledge of the world is uncertain, the more willingly we acknowledge that perfect prediction is impossible, the less we will live in fear of our failures, and the more freedom we will have to let our minds flow freely. By knowing more about what we don't know, we may get a few more predictions right."
More modesty and effort, in other words, would improve the predictive performance of everyone from the TV pundits to the political scientists, and members of the public trying to understand what is likely to happen next. Just do not expect, Silver warns, to fit a decent prediction on a bumper sticker. "Prediction is difficult for us for the same reason that it is so important: it is where objective and subjective reality intersect." You would probably need to be a stat geek to drive around with that on the back of your car, but it might just fit if the lettering were small.
Ruth Scurr's Fatal Purity: Robespierre and the French Revolution is published by Vintage.
