The key ideas

The Signal and the Noise: Why So Many Predictions Fail, but Some Don't

Nate Silver, 2012

Why were the rating agencies off by a factor of two hundred in 2008, while weather forecasters get more accurate every year? Nate Silver looks at finance, weather, earthquakes, baseball, poker, climate and terrorism. And everywhere he shows the same thing: a forecast fails when noise is mistaken for signal, and it works when people think in probabilities.

Who it is for

For anyone who is tired of confident experts and wants to know which forecasts to trust. And for anyone who has to make decisions without knowing how things will turn out.

The signal is the truth. The noise is what distracts us from the truth.

More data, more mistakes

In the middle of the fifteenth century, Gutenberg set his printing press running, and suddenly there were books everywhere. You'd think people would have grown wiser. But first, print spread religious quarrels across Europe, and those quarrels turned into wars. It took a long time before all that information started doing any good. This is where Nate Silver begins his book: when knowledge grows, telling the truth from the noise gets harder at first, not easier.

Silver is a statistician who built a forecasting system for baseball, made his living at poker for a few years, and then became famous for predicting American elections. He published this book in 2012. In it, he goes through the fields where people try to guess the future one by one: finance, politics, weather, earthquakes, sport, climate, terrorism. And in each one he asks the same question: why do forecasts fail here and work there?

His answer: it isn't about the amount of data. The signal is what really tells you something about the future. The noise is everything else that looks like it. The people who get it wrong mistake noise for signal and are too sure of their model. The people who win think in probabilities and honestly update their views when new facts come in.

Ahead are twelve stories, from the 2008 crisis to the poker table. Each one shows a single trap or a single good move. By the end, you'll see more clearly which forecast deserves your trust and which one just sounds confident.

Top grades for securities that collapsed

Before the 2008 crisis, banks sold investors complex securities built from thousands of mortgages. Each one was called a CDO, a collateralized debt obligation. Rating agencies such as S&P gave many of them the top grade, AAA. That meant: almost perfectly safe. And the agencies were paid for those grades by the very banks that issued the securities. By the agency's own reckoning, the chance that such a security would fail to pay out within five years was about 0.12 percent.

In reality, according to S&P's own internal data, around 28 percent of these securities failed to pay. The mistake was more than two hundredfold. Silver calls it close to the most complete failure a forecast can possibly have: trillions of dollars that were thought to be safe turned out to have almost no protection at all.

To explain how it happened, Silver borrows a distinction drawn by the economist Frank Knight. Risk is something you can calculate, like the odds at a roulette wheel. Uncertainty is something you can't calculate, because you don't even know the rules of the game. The new mortgage securities were pure uncertainty: nobody had ever seen them go through a crisis.

But the agencies acted as if it were ordinary risk, and handed out a precise number. A precise number is reassuring. But if there's no understanding behind it, it's more dangerous than an honest "we don't know".

Read in Livria

About the author

Nate Silver is an American statistician, the creator of the PECOTA baseball system and of FiveThirtyEight, the site known for its election forecasts.

All books by this author: Nate Silver

Terms from the book

  • Pearl Harbor
  • Bayes
  • calibration
  • Hansen
  • Uncertainty
  • CDO
  • rating agencies
  • Lorenz
  • Fukushima
  • overfitting
  • greenhouse effect
  • Armstrong
  • Gutenberg: Johannes Gutenberg (about 1400 to 1468), the German inventor of printing with movable type. Not to be confused with the seismologist Beno Gutenberg.
  • S&P: Standard & Poor's, one of the largest rating agencies in the world. It judges how safe the debts of companies, countries and securities are.
  • Frank Knight: Frank Knight (1885 to 1972), an American economist. In Risk, Uncertainty and Profit (1921) he separated measurable risk from uncertainty that cannot be measured.
  • out of sample: An event that was not in the data a model was built on. The model knows nothing about it and predicts it badly.
  • McLaughlin Group: An American political TV show (1982 to 2016) where commentators argued about the news and made predictions. It was hosted by John McLaughlin.
  • Katrina: The hurricane that struck New Orleans and the Gulf Coast in August 2005. One of the most destructive natural disasters in American history.
  • chaos theory: The science of systems where tiny changes at the start greatly change the end result. This is why weather is hard to predict far ahead.
  • National Weather Service: The United States government agency that issues weather forecasts and warnings about dangerous weather.
  • The Weather Channel: An American commercial TV channel about the weather, on the air since 1982.
  • Richter: Charles Richter (1900 to 1985), an American seismologist who created the scale for earthquake strength. Together with Beno Gutenberg he found the law of how often earthquakes occur.
  • PECOTA: A system for forecasting how baseball players will perform, built by Silver in the early 2000s. It compares a player with similar players from the past.
  • Pedroia: Dustin Pedroia, an American baseball player for the Boston Red Sox. American League Rookie of the Year in 2007 and its Most Valuable Player in 2008.
  • scouts: In sport, people who travel to games and assess young players so a club can decide whom to sign.
  • water level: An image from the book, not a scientific term: the level of skill above which a player starts to win. It depends on how strong the opponents are.
  • prior probability: Your estimate of how likely something is before a new fact appears. In Bayes' theorem, the calculation starts from it.
  • IPCC: The Intergovernmental Panel on Climate Change, a United Nations body. It has reviewed the science of climate since 1988.
  • Schelling: Thomas Schelling (1921 to 2016), an American economist who studied strategy and conflict, winner of the 2005 Nobel Prize in Economics.
  • Clauset: Aaron Clauset, an American scientist who studies complex systems. He analysed statistics on terrorist attacks around the world since the late 1960s.
  • power law: A pattern in which an event becomes steadily rarer as it gets bigger: small cases are very common, large ones are rare, but they never disappear.

All terms

Similar books