The key ideas

Superforecasting: The Art and Science of Prediction

Philip Tetlock, Dan Gardner, 2015

Why do famous experts predict the future no better than chance, while retirees and engineers beat intelligence analysts? Philip Tetlock spent twenty years checking forecasts, then ran a tournament where thousands of volunteers predicted world events. You will learn how the best of them think, and which of their habits you can pick up right now.

Who it is for

For anyone who has to guess about the future: about work, money, deadlines or the news. And for anyone tired of loud predictions that nobody ever checks afterwards.

The average expert was roughly as accurate as a dart-throwing chimpanzee.

A retiree from Nebraska and a columnist from New York

Bill Flack lives in Kearney, a small city in Nebraska. He used to work for the US Department of Agriculture, and now he is retired. He answers questions about world politics: will the head of some foreign government resign, will a currency fall, will a new round of talks begin. Ministers do not quote Flack, and nobody invites him to Davos to share a stage with the columnist Tom Friedman. But every one of his forecasts is written down with a date and later checked against what actually happened. And Flack's record is excellent.

You cannot say the same about Friedman. Presidents read his columns, and his views on the future are worth a lot of money, but nobody has ever properly checked how often he turns out to be right. That awkward comparison is where the psychologist Philip Tetlock and the science journalist Dan Gardner begin the book.

Tetlock calls himself an optimistic skeptic. The skeptic in him knows that much of the future is unpredictable and that most loud forecasts are never checked. The optimist sees something else: over a short horizon, months rather than decades, prediction can be done better, the skill can be measured, and, judging by his experiments, it can be taught. Flack is no seer. What he does can be broken down into steps.

In this book you will travel from the famous joke about a chimpanzee to a tournament run by American intelligence, and you will see how the people who beat the professionals think. At the end you will get their rules, which work in ordinary life too: at work, with money, in any decision where you have to guess what comes next.

A dart-throwing chimpanzee and twenty years of checking

In 1984 Tetlock started an experiment that most scientists would not have had the patience for. He gathered 284 experts: professors, journalists, analysts, people who make a living explaining politics and economics. Over the next twenty years or so they made about 28,000 predictions. Will a country break apart? Will the economy grow? Will a leader stay in power? Every answer was recorded as a probability and later checked against reality.

Tetlock published the results in 2005, and one line from them spread through the newspapers. The average expert did about as well as a chimpanzee throwing darts at a board of possible answers. The public took it simply: experts are useless.

Tetlock himself disagrees with that conclusion, and this matters. The word “average” hides the main point. When he looked at the results more closely, the experts split into two groups. One really did guess no better than chance. The other clearly beat the chimpanzee, though it was far from all-seeing.

So the difference is not whether you are an expert, and not how much you know. It is something else, and that something could be found. Tetlock described what set the two groups apart with the help of two animals.

Read in Livria

About the author

Philip Tetlock is an American psychologist and professor at the University of Pennsylvania who has studied the accuracy of forecasts for many years, and a co-founder of the Good Judgment Project. Dan Gardner is a Canadian journalist and the author of books on risk and prediction.

All books by this author: Philip Tetlock, Dan Gardner

Terms from the book

  • Kahneman
  • Friedman
  • calibration
  • Fermi
  • Outside view
  • Anchor
  • Flack: Bill Flack, one of the superforecasters in Tetlock's project and a former employee of the US Department of Agriculture. His story opens the book.
  • Kearney: A small city in the state of Nebraska, in the middle of the United States.
  • Davos: A Swiss resort that hosts the World Economic Forum every year, gathering politicians, business leaders and well-known commentators.
  • Chimpanzee: A great ape. In the book, the dart-throwing chimpanzee is a joking image of random guessing, not a real experiment.
  • Archilochus: An ancient Greek poet of the seventh century BC. The line about the fox and the hedgehog is attributed to him.
  • Kudlow: Larry Kudlow, an American economic commentator and TV host who champions tax cuts. Tetlock uses him as a typical hedgehog.
  • Great Recession: A deep economic downturn in which output and incomes fell for many months in a row. In the United States it began in December 2007 and lasted until mid-2009.
  • State Department: The foreign ministry of the United States.
  • Kent: Sherman Kent (1903–1986), a historian and one of the founders of the analytic service of American intelligence. He fought to have probability estimates stated clearly.
  • Weather forecaster: A person who makes the weather forecast.
  • Brier: Glenn Brier, an American meteorologist. In 1950 he proposed a way to score the accuracy of probability forecasts, which now bears his name.
  • Resolution: The ability to give high probabilities to things that happen and low ones to things that do not, instead of cautious numbers in the middle.
  • IARPA: The Intelligence Advanced Research Projects Activity of the United States. It funds high-risk research for the intelligence community.
  • Mellers: Barbara Mellers, an American psychologist, professor at the University of Pennsylvania and one of the leaders of the Good Judgment Project.
  • Good Judgment Project: The team led by Tetlock and Mellers in the IARPA forecasting tournament, made up of volunteers.
  • Superforecaster: A word from the book, not a scientific term: the authors' name for participants who made it into the top two percent for forecasting accuracy.
  • Lorch: Doug Lorch, a retired IBM programmer from Santa Barbara and one of the first superforecasters.
  • Polonium: A rare radioactive chemical element, highly poisonous even in tiny doses. It decays quickly over time.
  • Arafat: Yasser Arafat (1929–2004), leader of the Palestine Liberation Organization. After his death there was a long debate over whether he had been poisoned.
  • Renzetti: A made-up family from an example in the book, used by the authors to explain the outside view.
  • Base rate: How often an event happens among similar cases in general, before you take the details of a particular case into account.
  • Minto: Tim Minto, one of the project's superforecasters; the authors use him as an example of updating a forecast in small steps.
  • Bayes' theorem: A mathematical rule for how to recalculate the probability of a hypothesis when new evidence arrives.
  • Perpetual beta: A beta is a version of a program that already works but is still being improved. “Perpetual beta” is an image from the book, not a scientific term.
  • Moltke: Helmuth von Moltke the Elder (1800–1891), a Prussian field marshal and military theorist.

All terms

Similar books