Misleading metrics. When the exam was already in the study material

A model that scores ninety-five on an exam may be brilliant or may have seen the exam before. When the student is a language model and the exam is a public benchmark (standardised evaluation test), the second hypothesis goes from probable to nearly certain. The industry publishes the number as an achievement. The paper showing it was in the training corpus gets read at EMNLP and doesn't leave campus. The technical discussion is boring and that's why nobody wants it. That's why it's the important one.

The figure worth pinning up before going on

There are a couple of figures worth nailing to the wall at the start, because afterwards it's easy for the conversation to turn abstract. In the study Inference-Time Decontamination. Reusing Leaked Benchmarks for Large Language Model Evaluation (Findings of EMNLP 2024, arXiv 2406.13990), the authors report that their technique reduces the inflated accuracy by 22.9 per cent on GSM8K and 19.0 per cent on MMLU. That is, a substantial part of the figure the industry publishes as performance can be explained by examples the model had already seen during training.

Read it again.

A good part of the performance we celebrate isn't the model's capacity. It's memory of what it had already seen. It isn't a rough estimate, it isn't an acid critic's suspicion. It's the difference the paper measures between what a model answers when the question forms part of its training corpus and what it answers when that shortcut is taken away.

What data contamination is

The technical name for the problem is data contamination (also called data leakage when framed as a leak from the test set into the training set). The most complete survey published to date, Benchmark Data Contamination of Large Language Models. A Survey by Xu and others (arXiv 2406.04244, 2024), distinguishes several forms.

Direct, partial, by paraphrase

The most obvious is the direct kind. The evaluation set, or parts of it, appears verbatim in the training corpus. It's what a human would call cheating, though it's rarely deliberate.

The partial kind. Individual examples, fragments, solutions discussed on forums, bits of the prompt mixed with other users' code, show up.

And the indirect or paraphrase kind. The corpus contains rewordings, translations, blog comments, repositories with explanations of the problem, academic articles citing the item. This last one is the hardest to detect and the most widespread, because any sufficiently important benchmark generates a textual ecosystem around it within months. Whoever builds the model isn't obliged to cheat for the cheating to happen. It's enough not to filter the corpus. And web-scale corpora are impossible to filter exhaustively.

Contamination is the default state. Decontamination is the expensive exception almost no one pays for.

When the measure becomes the target

There's a mechanism here older than AI. Marilyn Strathern formulated it in 1997, in Improving ratings. Audit in the British University system, generalising an earlier observation by Charles Goodhart, who in 1975 had written that any observed statistical regularity tends to collapse the moment pressure is put on it for control purposes. The line that caught on, simpler, is Strathern's. When a measure becomes a target, it ceases to be a good measure.

In AI the dynamic plays out with an uncomfortable clarity.

MMLU (Massive Multitask Language Understanding, a battery covering fifty-seven different areas) is published in 2020 as a multitask test, varied enough and hard enough to be a useful yardstick. Five years later, MMLU is a declared target of training runs. Models are evaluated, fine-tuned and re-evaluated against it until the number comes out. SWE-bench, another important benchmark built on real GitHub tickets, follows the same path. It's born as a diagnostic of capacity. It becomes the quarter's KPI (key indicator). And, in the process, it stops being a diagnostic and turns into product.

What it measures is no longer what the model can do in real programming. It's what the model can do over the fixed sample the entire industry has read, commented on, cited and retrained on.

A blog isn't the one providing the proof

Whoever wants to check it doesn't need to trust the voice of the one writing this. Gary Marcus, one of the field's most insistent critics, returns to the problem in his newsletter Marcus on AI, outside the academic paper, in a format readable by anyone who doesn't want to wrestle with table 4 of the appendix. Leak, Cheat, Repeat. Data Contamination and Evaluation Malpractices in Closed-Source LLMs documents the problem in closed models, where external audit is impossible by definition because the corpus isn't published. LessLeak-Bench. A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks (arXiv 2502.06215) does the same exercise over eighty-three software-engineering benchmarks. Its finding is finer than an alarmist headline would like. On average the leakage is low, around 4.8 per cent in Python, 2.8 in Java and 0.7 in C/C++. But there are extreme cases: QuixBugs leaked at a hundred per cent, BigCloneBench over half. And where the leakage appears, it inflates brutally: in the APPS benchmark, StarCoder-7B obtains a score almost five times higher on the leaked examples than on the clean ones. Cheng, Chang and Wu (A Survey on Data Contamination for Large Language Models, arXiv 2502.14425) survey the landscape a year after Xu, and the picture doesn't improve. The detection techniques advance. The models advance faster. The corpora grow. The balance is still tipped on contamination's side.

The technical argument, in the cold, is textbook. What's interesting is what happens when you look at the whole incentive apparatus.

The asymmetry that holds the business up

The lab that trains a model knows its corpus. Not entirely, because at web scale no one knows their corpus entirely, but it knows it far better than any external evaluator.

Why nobody audits

If it wanted to audit itself, it could. It could run membership inference tests (membership inference, the technique that estimates whether a specific example formed part of the training). It could decontaminate against the known benchmarks. It could retrain excluding specific domains and report the drop in performance. Some do it partially. Most don't.

Why would they.

The audit isn't requested by the buyer, because the buyer doesn't know the audit exists. Not by the regulator, because the regulator works with the published figure. Not by the investor, because the investor wants to see the quarter's number rise. Not by the press, because the press headlines the big percentage and not the real decontaminated one. The one who should request it is the academic community, which indeed requests it, and that's why it publishes the papers no one outside the field reads.

The system works because the informational asymmetry is functional to the business. Calling it an accident is a courtesy the business hasn't earned.

The leaderboard industry

There's an aggravating factor. The leaderboard (the public ranking table that orders models by their score on a benchmark). The quarter's rankings, the weekly comparisons on Twitter, the Hacker News threads, the charts showing how model X beats Y by six tenths on MMLU.

The leaderboard industry rewards over-training to the test with brutal efficiency, because the difference between first and second place in the weekly ranking translates into investment, into contracts, into the TechCrunch front page. The team that decides not to optimise against the benchmark, for reasons of methodological purity, loses the investment. The one that optimises, wins.

The field's natural selection is aligned against metric honesty. Not because the researchers are dishonest. Because the incentive structure rewards one specific kind of practice and punishes the other. What comes out at the end is what the system selects.

The reasonable objection

Someone will say that contamination is a known technical problem, that work is being done on it, that there are dynamic benchmarks, private validation sets, live evaluations, tests with data generated on purpose after the model's cutoff (the cut-off date of the training corpus). It's true and it's relevant.

It's also true that those advanced techniques are applied to a minority of the evaluations that appear in the press kits, and that the general public consumes the MMLU, GSM8K and company figures with no asterisk. The existence of good practices at the margins doesn't rescue the bulk of the numbers underpinning the public discourse on AI.

It's a situation like nineties nutrition. There were nutritionists who understood the difference between saturated and unsaturated fat, and there were packages that put "light" in big letters. What reached the consumer was the package.

Public decisions on false numbers

If the metrics are inflated, the industry knows it and the regulators don't understand it, then adoption and regulation decisions are made on false numbers. Not on approximate numbers, nor on optimistic numbers. On numbers with a known bias in a known direction and a magnitude estimated at around twenty per cent on some tests, according to the papers already cited.

When a public administration decides to adopt a model for processing case files on the basis of its performance on a benchmark, it's deciding on a figure of which an appreciable part, in the rough, is memorisation noise. When a regulatory board discusses whether to impose particular obligations on "frontier" models, and the frontier is defined by a capacity threshold measured on these same benchmarks, it's drawing the frontier with an instrument the industry has learned to fool.

The result isn't a bad measurement. It's a bad policy derived from a bad measurement.

The nuance that mustn't be lost

It's worth insisting on a point that's easy to misread. It isn't being said that the models are stupid nor that their capabilities are null. They do things that five years ago looked like science fiction, and this blog isn't going to join the comfortable revisionism that denies reality to avoid the noise.

Something else is being said. That the measure the industry publishes to quantify those capabilities is systematically inflated. That the inflation isn't random noise but bias in one direction. That the direction is always the one that favours the figure that makes the press.

The model is good. The measure is broken. Both sentences can be true at once. Usually they are.

Boredom as the business frontier

There remains a less academic, dirtier question. Why is the technical discussion of benchmark contamination so boring that practically no one outside the field wants it.

Because it's built to be.

The business frontier

Because really understanding what MMLU measures demands patience, statistical knowledge, parallel reading of several papers and a willingness not to settle for the simple version. Because the simple version, the big-percentage one, is optimised to go in through the ear. Because whoever wants to sell the reader something needs the reader not to ask questions, and boredom is the most efficient method for stopping them.

The business frontier isn't in the performance. It's in the programmed boredom of whoever should audit the performance. That's why the asymmetry is stable. That's why the money lives there.

A model that scores ninety-five on a leaked exam and ninety-five on a clean exam would be good news. A model that scores ninety-five on the leaked one and seventy-five on the clean one is something very different, and the industry publishes the first number without clarifying whether it has even verified the difference. The exam was already in the study material. The number is in the press release. The difference between the two sentences is the only relevant thing in this text, and the only thing that won't appear in any headline about next quarter's model.

Definitions

Benchmark. A standardised test that measures some capacity of a model over a fixed set of examples, usually public. MMLU, GSM8K, APPS and SWE-bench are benchmarks in common use for evaluating language models.

Data contamination. A situation in which examples from the evaluation set, or information derived from them, were present in the corpus the model was trained on. Inflates the model's score relative to what it would yield on truly new examples.

Data leakage. A particular case of contamination in which information that should have been reserved for the test set leaks into the training one, whether verbatim or by paraphrase.

Decontamination. A set of techniques that detect and remove from the training corpus, or from the evaluation, the contaminated examples. When applied at evaluation time and not before training it's called inference-time decontamination.

Membership inference. A technique that estimates, from a model's behaviour towards an example, whether that example formed part of its training set. It's one of the main tools for auditing contamination in closed models.

Goodhart's law. The principle that a measure ceases to be reliable as soon as it becomes the target orienting the system being measured. The modern formulation is attributed to Marilyn Strathern in 1997.

Leaderboard. A public ranking table ordering models by their score on one or more benchmarks. It works as a mechanism of visibility and, in practice, of capital allocation.

Cutoff. The cut-off date of a model's training corpus. Examples generated after that date are, in principle, outside the corpus, which makes them candidates for cleaner evaluations.

References

Xu, C. et al. (2024). Benchmark Data Contamination of Large Language Models. A Survey. arXiv 2406.04244. A general survey of the problem, contamination typologies and detection techniques. The main source for the body of the article.

Inference-Time Decontamination. Reusing Leaked Benchmarks for Large Language Model Evaluation. Findings of EMNLP 2024 (arXiv 2406.13990). The study's abstract reports that its technique reduces the inflated accuracy by 22.9 % on GSM8K and 19.0 % on MMLU; those two figures in the body come from there.

Balloccu, S., Schmidtová, P., Lango, M. and Dušek, O. (2024). Leak, Cheat, Repeat. Data Contamination and Evaluation Malpractices in Closed-Source LLMs. EACL 2024, ACL Anthology 2024.eacl-long.5 (arXiv 2402.03927). Documentation of the problem in closed commercial models, where no training corpus is published.

LessLeak-Bench. A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks. arXiv 2502.06215. A systematic investigation of leakage in eighty-three software-engineering benchmarks. Its abstract reports average leakage rates of 4.8 % in Python, 2.8 % in Java and 0.7 % in C/C++, and extreme cases like QuixBugs (100 %) and BigCloneBench (55.7 %). The article's text also notes that StarCoder-7B obtains in APPS a Pass@1 4.9 times higher on the leaked examples than on the clean ones.

Cheng, Y., Chang, Y. and Wu, Y. (2025). A Survey on Data Contamination for Large Language Models. arXiv 2502.14425. A review later than Xu, cited to show that the landscape hasn't improved despite advances in detection.

Strathern, M. (1997). Improving ratings. Audit in the British University system. European Review 5, 305–321. On page 308 it carries the formulation of Goodhart's law that caught on, "when a measure becomes a target, it ceases to be a good measure", cited as the framework for understanding why benchmarks degrade once they become targets.

Goodhart, C. (1975). Problems of Monetary Management. The U.K. Experience. Papers in Monetary Economics, Reserve Bank of Australia. The monetary-policy work from which the original observation derives into its eponymous law, later generalised by Strathern. Cited as the antecedent of the principle applied to benchmarks.

Marcus, G. Marcus on AI. The author's newsletter on Substack, cited as an example of accessible critical writing outside the academic paper.

You might also like

• Measuring artificial intelligence
• MMLU, GPQA, HELM, a manual for understanding AI benchmarks
• What is the Chatbot Arena?
• Statistics as a substitute for knowledge

Elsewhere

• Goodhart's law (Wikipedia): when a measure becomes a target
• AI: A Guide for Thinking Humans (Melanie Mitchell)