In this article
I've been citing her for years in arguments about AI, and almost everyone gets her wrong: shrunk down to a clever slogan about bad algorithms. Cathy O'Neil was inside Wall Street when the 2008 crash blew up, watched a set of opaque mathematical models sign off mortgages that would ruin entire families, and out of that came a category worth understanding whole, not in three words. Her weapons of math destruction predate ChatGPT. And they are exactly the problem that generative AI inherits and multiplies.
Who Cathy O'Neil is
Where she came from matters, because it defines what she wrote later. O'Neil earned her PhD in mathematics at Harvard in 1999 and taught at Barnard College before crossing over to finance. In 2007 she joined the quant fund D. E. Shaw, one of Wall Street's giants, and from there moved to the risk-software firm RiskMetrics. She was inside the sector when the credit-risk models that had been used for a decade collapsed in 2008. Lehman fell. The crisis began.
She wasn't a victim of all that. She was a witness from the side that caused it, and that vantage point is what gives the book its authority. She watched how sophisticated mathematics —Value at Risk, subprime mortgage scoring, default probability— served as technical cover for reckless decisions, and how a number with the look of objectivity let an executive say "the model said so" and wash his hands.
She left. She worked afterward as a data scientist in the New York online-advertising ecosystem —a stint at the startup Intent Media— and got involved with the alternative-banking group of Occupy Wall Street. Later she founded ORCAA, an algorithmic-auditing consultancy. In 2016 she published Weapons of Math Destruction, which made the longlist for that year's National Book Award and later won the Euler Book Prize from the American mathematical association. Nearly ten years on it remains the cleanest explanation in English of why certain algorithms do so much harm and are so hard to stop.
The precise definition
O'Neil calls a Weapon of Math Destruction (WMD) an algorithm that gathers three traits at once. The conjunction is what matters: drop one, and you have at most a mediocre algorithm, not a WMD.
The first is opacity. No outsider can inspect how it decides. Sometimes because the company that built it declares it a trade secret, which is the usual case; sometimes because the system itself —a neural network with hundreds of millions of parameters— is so tangled that not even its creators can explain a specific decision.
The second is scale. We're not talking about a judge or a recruiter deciding face to face, with their human margin of error and their chance to reconsider. We're talking about a decision applied automatically to entire populations, and at that scale any tiny bias turns into mass harm.
The third is asymmetric harm, and it's the one almost no one gets right. The algorithm doesn't spread the damage evenly: it punishes hardest those who already started at a disadvantage. It learns the bias dragged in by the data it was trained on, and since that historical data carries prior social inequalities inside it, the algorithm reproduces them and hardens them.
Three conditions stacking up. The definition is good, above all, for telling the bank's credit scoring —a WMD candidate— apart from the algorithm that recommends you songs, where there is scale and there is opacity but the asymmetric harm is negligible.
Case one. COMPAS
COMPAS is a recidivism-risk assessment system used by judges in the United States to rule on parole, bail and sentencing. Northpointe —today Equivant— developed it, and it has been in use since the early 2000s in states like Florida, New York, Wisconsin and Arizona.
It works like this. The defendant answers a questionnaire of more than 130 questions about their record, their family, their neighbourhood, their schooling and their relationships; the algorithm cross-references those answers with administrative data and returns a risk score from 1 to 10 that the judge has in front of them when deciding.
In 2016, the ProPublica journalists —Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner— analysed around 7,000 cases in Broward County and published what they found. The false-positive rate for Black people was double that of white people: the system predicted nearly twice as often that a Black person would reoffend when they then did not. And the reverse, labelling as safe white people who did go on to reoffend. This is not an abstract bias to be debated in a seminar. It's years in prison for some and the street for others, decided in part by an opaque number with a documented bias.
COMPAS meets all three traits. The opacity is textbook, because Northpointe shelters behind trade secrecy to avoid showing the formula. The scale reaches hundreds of thousands of people assessed. And the harm falls on the Black population, already overrepresented in the penal system.
Northpointe pushed back against ProPublica, and the technical argument over which fairness metric to use —predictive calibration versus error parity— is legitimate and still alive among statisticians. But what that fight does not touch is the core: a tool that decides over the freedom of specific people works inside in a way that no outsider can review.
Case two. The Washington teachers
In 2009, the Washington D. C. public schools rolled out IMPACT, a teacher-evaluation system that leaned in part on a statistical "value-added" model. The idea was to estimate how much each teacher contributed to their students' progress on standardised tests: the model predicted the grade each student should get given their circumstances —neighbourhood, family income, prior performance— and at the end of the year compared the predicted with the real. The difference was attributed to the teacher.
The problem is statistical before it's moral. With small samples, a teacher of twenty or thirty students a year, the margin of error is enormous, to the point where many teachers came out indistinguishable from one another inside the noise. It didn't matter: the score was used in earnest, for bonuses, promotions and firings.
O'Neil tells the case of Sarah Wysocki, a Washington teacher whom colleagues and families considered excellent. Her IMPACT score came out low and she was fired. It later emerged that the previous year's grades, the ones the prediction rested on, had been inflated by administrative cheating elsewhere in the system. The input was false, so the prediction was false, and the model had no way of knowing it. Wysocki lost her job over noise she hadn't generated.
In 2015, in New York, the Board of Regents approved a four-year moratorium on using test scores in teacher evaluation: a reversal, forced by public pressure, of the policy Governor Andrew Cuomo had pushed. For Wysocki it came too late.
Case three. ZIP code and credit
The book's third central example is credit scoring, but not the classic FICO, which is relatively transparent, rather the alternative scoring of the fintechs, the online lenders and the big data platforms.
Those models feed on variables that look innocent: the ZIP code, browsing habits, the device you log in from, how long you take to read each screen, the hour you ask for the loan. Seemingly neutral. But in the United States the ZIP code has for decades been correlated with race thanks to residential segregation, so your neighbourhood gives away what the law forbids using outright.
The result is that the highest interest rates land systematically on historically marginalised communities, not because of each person's financial characteristics but because of the aggregate statistics of their neighbourhood. It's what has been called algorithmic redlining: the present-day version of mid-twentieth-century bank redlining, that practice of denying financial services based on a neighbourhood's racial makeup. They made it illegal, and the algorithm rebuilds it without naming race, inferring it from correlated variables.
Is it a WMD? The opacity is there, because that scoring is a trade secret. The scale is there, with millions of applications a year. And the asymmetric harm is exactly what O'Neil's book documents as a structural pattern of this kind of model.
The European case. The toeslagenaffaire
So it doesn't look like a purely American vice, it's worth looking at the Netherlands.
Between 2005 and 2019, the tax authority (the Belastingdienst) used a risk algorithm to decide which childcare-benefit claims to investigate for suspected fraud. The model treated foreign nationality, ethnic origin and low income as indicators of suspicion. Some 26,000 families were falsely accused. They were ordered to repay the benefits collected, in many cases tens of thousands of euros at once. There were ruined families, divorces, children removed by social services, suicides.
Behind it was a mechanism O'Neil describes very well. The algorithm amplified the bias of origin: at the signal of a few fraudsters of one nationality, it put almost every resident of that nationality under the microscope, and by looking only at that group it generated more cases in that group, which confirmed the initial suspicion. A feedback loop that feeds on its own prejudice.
In January 2021, the entire government of Mark Rutte resigned over the scandal. Five years on, the victims are still fighting for compensation that measures up. Opacity: those affected didn't even know why they were being investigated. Scale: tens of thousands of families. Asymmetric harm: the blow fell on migrant and poor households. And it happened in one of the countries with the best democratic reputation and the biggest administrative budget in Europe, which is the part that should cost you sleep.
The test O'Neil proposes
Faced with an algorithm being applied to you, or that you apply yourself, O'Neil leaves three questions. Is it auditable, can someone outside look at what it does and why? If the answer is no, whether through trade secrecy or technical complexity, you already have an opacity problem. How many people does it affect? An individual human decision keeps a margin for review; an automatic decision over millions does not. Does it punish hardest those who were already at the bottom, and is there any analysis of performance disaggregated by protected groups? When that analysis doesn't exist, the prudent move is to presume the bias is there, because historical data almost always carries it inside.
Three yeses in a row and what you've got in front of you is no longer a tool but a weapon, and the harm it's going to do won't be an accident: it was in the design.
The connection to generative AI
WMDs predate ChatGPT, and generative AI doesn't change the category, it widens it. There's opacity over how your CV is filtered before a human sees it, now that applicant-tracking systems extract and score applications with generative models. There's opacity over how your credit application is enriched with automatically gathered data, over how the background reports employers consult are written, over what a legal assistant suggests to a judge ahead of a hearing.
Each of those systems puts the logic O'Neil described in 2016 inside a new scale. She laid down the frame nearly ten years ago; the cases are laid down, without pause, by the sector itself. And the three traits remain the cleanest filter for locating where the problems that really matter sit, right when the noise about AI invites you to look anywhere else.
Definitions
Weapon of Math Destruction (WMD). An algorithm that combines opacity, scale and asymmetric harm. The category is Cathy O'Neil's, in her 2016 book.
COMPAS. Short for Correctional Offender Management Profiling for Alternative Sanctions, a recidivism-risk assessment system used in United States courts.
Value-added model. A statistical model that tries to estimate how much a teacher (or any agent) contributes to students' measured performance, controlling for prior characteristics.
Redlining. Historical practice of denying financial services to the residents of certain neighbourhoods based on their racial composition; today it reappears in its algorithmic version.
Toeslagenaffaire. The Dutch childcare-benefit scandal: some 26,000 families falsely accused of fraud between 2005 and 2019 by a biased algorithm.
ORCAA. O'Neil Risk Consulting & Algorithmic Auditing, the algorithmic-auditing consultancy founded by Cathy O'Neil.
Biased feedback loop. A situation in which an algorithm, by acting only on a selected subgroup, generates data that confirm and reinforce its own selection bias.
References
Weapons of Math Destruction. How Big Data Increases Inequality and Threatens Democracy, by Cathy O'Neil (Crown, 2016), is the source of the conceptual framework and of the IMPACT and credit-scoring cases.
Machine Bias, by Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner (ProPublica, 23 May 2016), is the investigation into COMPAS in Broward County.
Cathy O'Neil's biographical entry on Wikipedia (en.wikipedia.org/wiki/Cathy_O'Neil) backs her path through D. E. Shaw, RiskMetrics, the New York advertising sector and Intent Media, and the founding of ORCAA.
The Wikipedia entry on Weapons of Math Destruction (en.wikipedia.org/wiki/Weapons_of_Math_Destruction) records the book's inclusion on the 2016 National Book Award longlist and the Euler Book Prize.
North Country Public Radio's report on the teacher-evaluation moratorium in New York (northcountrypublicradio.org) documents that the 2015 moratorium was approved by the Board of Regents, reversing the policy pushed by Governor Cuomo.
Xenophobic Machines. Discrimination through unregulated use of algorithms in the Dutch childcare benefits scandal, by Amnesty International (2021), documents the toeslagenaffaire.
Automating Inequality, by Virginia Eubanks (St. Martin's Press, 2018), broadens the pattern of algorithmic harm against vulnerable populations.
You might also like
- Invisible AI. The one that decides about you without you knowing the decision ever existed
- Invisible biases. The declared bias gets filtered, the baseline one stays intact
- Timnit Gebru and her exit from Google
- The dataset as ideology. The quietest political choice in the sector
Comments0
No comments yet.
Leave a comment