The key ideas

Weapons of Math Destruction

Cathy O'Neil, 2016

Why does a good teacher get fired on a program's say-so, while a driver with a spotless record pays more for insurance than a driver with a drunk driving conviction? Cathy O'Neil, a mathematician who worked at a hedge fund, shows how models nobody can check decide who gets hired, who gets a loan and who gets a longer sentence. You'll learn the three signs of a dangerous algorithm and how to spot someone's opinion hiding behind an impartial number.

Who it is for

For anyone who has ever been scored by a program: when applying for a job, at the bank, by an insurer or at school. And for anyone who builds rankings, reports and models and doesn't want them to become weapons.

The privileged, we'll see time and again, are processed more by people, the masses by machines.

The best teacher, fired by a program

In 2011, Washington, D.C. fired a fifth-grade teacher named Sarah Wysocki. Her principal praised her lessons, parents were happy, and one review called her one of the best teachers its writer had ever come across. But they weren't the ones who decided. The decision came from a score, issued by a computer system that evaluated teachers. On the strength of that same score, more than two hundred other people were fired along with her. Nobody could clearly explain why the program judged her so differently from everyone who had actually watched her teach. We'll come back to that question.

Cathy O'Neil is a mathematician with a PhD from Harvard. She taught, then left for the hedge fund D.E. Shaw to build models for the financial markets. After the 2008 crisis she became one of the best known critics of how mathematics gets applied to people. This book came out in 2016.

O'Neil's central discovery is a particular kind of model that decides people's fates. She gave these models a name: weapons of math destruction, a play on weapons of mass destruction. They have three signs. They are opaque: you don't know exactly how you were scored. They work at scale: the same calculation is applied to millions. And they do damage, most often to the people who already have the fewest options.

Ahead are stories from schools, courts, banks, hiring departments and even elections. In each one, a number everyone trusts quietly decides someone's fate. You'll learn to ask any score the awkward questions, and to see where someone's opinion is hiding behind an objective number.

Dinner for the kids is a mathematical model too

O'Neil starts with the most everyday scene at home. Every evening she decides what to feed her family. In her head there's a model: who likes what, what's in the fridge, what things cost, what day it is. When the kids make a face or ask for seconds, the model updates itself.

The most interesting part is what counts as success. Her kids would happily eat Pop-Tarts every day. If success meant their delight, the model would feed them sweets. But for O'Neil, success means kids who are full and who eat their vegetables too. Defining success is a choice, not a calculation, and the person who builds the model makes it.

This leads to her famous line: models are opinions embedded in mathematics. Every model simplifies the world and decides what matters and what can be left out. There's nothing wrong with simplifying as such: without it a model is useless. The danger is when someone's opinion is passed off as an impartial number.

The author finds the opposite case in baseball. There the models are transparent: any fan can see the statistics. The data is plentiful, it describes the game itself, and every match shows whether the model got it wrong. A good model keeps learning where it was mistaken. A bad one never finds out.

Read in Livria

About the author

Catherine (Cathy) O'Neil is an American mathematician with a PhD from Harvard. She taught at Barnard College and worked as a quant at a hedge fund, and after the 2008 crisis she became one of the best known critics of opaque algorithms.

All books by this author: Cathy O'Neil

Terms from the book

  • Obama
  • Pop-Tarts
  • Bipolar disorder
  • Hedge fund
  • feedback loop
  • Taylor
  • Clifford
  • Starbucks
  • rating agencies
  • FICO
  • Wysocki: Sarah Wysocki, a Washington, D.C. teacher fired in 2011 because of her score in the IMPACT teacher evaluation system. Her story opens the book.
  • D.E. Shaw: An American hedge fund known for trading with mathematical models. O'Neil worked there as a quant from 2007.
  • Weapons of math destruction: An image from the book, not a scientific term: O'Neil's name for models that are opaque, used at scale and cause harm.
  • IMPACT: The teacher evaluation system in Washington, D.C. schools, introduced in 2009. Half of a teacher's score came from a value-added model.
  • Value-added model: A way of rating a teacher by how much their students' results grew over the year beyond what was predicted.
  • Mortgage-backed securities: Financial securities built from bundles of mortgage loans. Their income comes from borrowers' payments. They were one of the causes of the 2008 crisis.
  • RiskMetrics: An American company that calculated risk for banks and investment funds. O'Neil worked there after D.E. Shaw.
  • U.S. News & World Report: An American magazine that since 1983 has published the most influential ranking of US universities.
  • Proxy: An indirect measure: something easy to count, used in place of the thing you actually want to measure.
  • Texas Christian University: A private university in Fort Worth, Texas. It climbed the rankings by investing in sports and in its appeal to applicants.
  • For-profit colleges: Private colleges in the US run to make a profit. A large share of their money comes through federal student loans.
  • Leads: A marketing word: the contact details of people who might become customers. They are collected and sold to companies.
  • Reading: A city in Pennsylvania, USA. Its police were among the first to use the PredPol program.
  • Heim: William Heim, the Reading police chief who brought in the PredPol crime prediction program.
  • PredPol: An American predictive policing program: based on past data, it suggests where a crime is most likely in the next few hours.
  • Stop-and-frisk: A New York police practice of stopping and searching passersby on suspicion. It was used on a massive scale in the 2000s and early 2010s.
  • Recidivism: When a person who has already been convicted commits another crime.
  • LSI-R: A questionnaire for assessing the risk of reoffending, used by courts and prisons in the US. It stands for Level of Service Inventory, Revised.
  • Behm: Kyle Behm, a student screened out by personality tests when applying for jobs, and his father Roland Behm, a lawyer who took a stand against such tests.
  • St. George's: A London hospital with a medical school whose applicant screening program discriminated against women and people with foreign names in the 1980s.
  • Navarro: Jannette Navarro, a Starbucks barista whose work schedule The New York Times wrote about in 2014.
  • Clopening: A shift in which a worker closes up late at night and opens up again early the next morning.
  • E-scores: Opaque scores of a person built from indirect data: zip code, purchases, online behavior.
  • Microtargeting: Showing each voter or shopper a message of their own, tailored to the data about them.
  • Ghani: Rayid Ghani, a data scientist who led the data work on Obama's 2012 campaign.

All terms

Similar books