In this article
Today we're talking about the acronyms the Spanish press copies without understanding. Every time a lab presents a new model it brings a table in its press release. "92.3% on MMLU. 85.1% on GPQA. 78.4% on HumanEval." As if they were university-entrance grades. What the hell does each one measure and why does it matter? I admit that for months I repeated those figures myself too, in talks and in articles, until a colleague at MIT pointed out to me that half were saturated and the other half miscounted. It's embarrassing, and that's why I'm writing it. Form your own opinion calmly, since this time mine comes with the bias of a recent convert.
I've spent three months with a notebook where I jot down, every time I see a benchmark figure in a headline, which benchmark exactly and when it was published. Half the figures come from tests saturated two years ago. A quarter come from tests the lab itself designed. The rest are legitimate data. Keeping the distinction, once you have it, is very easy. Not having it is the norm in media coverage.
MMLU, the general exam
Massive Multitask Language Understanding is a set of roughly 15,900 multiple-choice questions spread across 57 disciplines, from elementary maths to jurisprudence and bioethics. Dan Hendrycks and collaborators published it in 2020 (arXiv:2009.03300, presented at ICLR 2021), and for several years it was the central reference for comparing general models.
The idea was good. The questions come from real exams —SAT, GRE, AP, medical exams, civil-service exams— and cover enough disciplines that passing the whole requires something like broad culture. When GPT-3 came out in 2020 it scored around 43%. When GPT-4 arrived in March 2023 it jumped to 86%. And then came saturation.
Saturation, in evaluation jargon, means the benchmark's ceiling approaches the limit of what makes sense to measure. The frontier models of 2025 and 2026 routinely score between 88% and 92%. The difference between the first and fifth model —on MMLU— sits inside the margin of noise caused by errors in the set's own questions and answers. Several independent analyses have documented that MMLU contains between 6% and 10% of questions with a debatable official answer, imprecise translation between validation sets, or genuine ambiguity in the wording. No one is going to reach that ceiling.
MMLU is still useful as a minimum filter. If a model scores under 70%, it's very likely no good for general cases. If it scores over 85%, it's inside the frontier family, and little more can be said without moving to other tests.
GPQA, the PhD trap
Graduate-Level Google-Proof Q&A Benchmark, published by David Rein and collaborators in November 2023 (arXiv:2311.12022), tightens where MMLU loosens. It's 448 questions in physics, chemistry and biology written by PhD candidates in each field, validated by other experts, and explicitly designed so that not even a layperson with Google access can solve them by searching. Hence the Google-proof.
The most-used reduced set is called GPQA Diamond, 198 questions filtered by difficulty and consensus among validators. On GPQA Diamond, experts in the field itself —physics PhDs answering physics questions— score around 65% under time-limited conditions. That's the reasonable human ceiling. It isn't 95%, it isn't 100%, it's 65%, because the questions are deliberately at the limit of what a human in the field can solve without recourse to specialised literature.
A model scoring 70% on GPQA Diamond is serious. Scoring 85% indicates technical-reasoning capacity that was science fiction in 2022. And even so it's worth seeing how the figures are obtained. Models with extended reasoning —the ones that allow themselves thousands of tokens of chain of thought before answering— can have ten to thirty times more compute per question than fast models. Comparing the GPQA of a reasoning model with that of a conversational model without qualifying the inference regime is comparing apples to bananas.
Unlike MMLU, GPQA isn't saturated yet. The difference between the best and the fifth model in the ranking remains significant. That's why it has become the reference benchmark for serious presentations in 2025 and 2026.
HumanEval and MBPP, the code benchmarks that aged
HumanEval was published by Mark Chen and the OpenAI team in 2021 (arXiv:2107.03374). It's 164 small problems where the model receives a Python function signature and a comment explaining what it has to do, and has to return the function body. It's evaluated with automatic unit tests. MBPP —Mostly Basic Python Problems, Austin et al., arXiv:2108.07732, 2021— is similar but shorter: natural-language descriptions and a function that passes the tests.
Both have been saturated since 2024. Frontier models score above 90%, and the difference between first and fifth is a few points. Citing them today in a press release to show off is a clear symptom of lazy marketing. It doesn't lie, but it doesn't inform.
What does measure something is the honest successor: SWE-bench, presented by Carlos Jimenez and collaborators in October 2023 (arXiv:2310.06770, published at ICLR 2024). Instead of isolated functions, SWE-bench takes real issues from open-source GitHub projects —Django, scikit-learn, sympy, requests— and asks the model to produce the patch that closes the issue. Then it's evaluated whether the patch passes the project's tests.
The benchmark's real difficulty shows in the foundational figure. In the original 2024 paper version, GPT-4 solved 1.7% of the issues and Claude 2 4.8% using an oracle retriever that already handed it the right files to edit. That is: at the moment MMLU was hovering around 90%, this benchmark returned the best models of that year to the reality of single-digit figures. Three years later, OpenAI published a curated variant called SWE-bench Verified in August 2024, and in the months that followed Anthropic and OpenAI's frontier models have climbed to the 70%–77% range. It remains the most informative programming benchmark of those routinely cited.
HELM, what doesn't appear in the press release
Holistic Evaluation of Language Models isn't a single benchmark. It's a meta-framework published by Percy Liang and the Stanford CRFM team in November 2022 (arXiv:2211.09110), later expanded in successive updates up to the current HELM Lite and HELM Classic. The underlying idea is that evaluating a model with a single metric is deceptive. HELM evaluates against dozens of scenarios and returns for each model not a number but a vector: accuracy, robustness to perturbations, fairness, bias, efficiency, calibration, toxicity.
The result is awkward to communicate. A press release can say "92.3% on MMLU." It can't say "6th in general accuracy, 4th in robustness, 11th in fairness, 2nd in efficiency." The complexity of HELM evaluation is the reason almost no one cites it in marketing. And it's the reason it's worth seeking out when you want to know where a model breaks under adverse conditions.
The operational use of HELM isn't reading the general ranking. It's seeking the model already in production and seeing in which sub-scenarios it falls. If the application is translation into Spanish, look at the relevant sub-scenario. If the application is summarising long text, look at the relevant sub-scenario. The granularity is the usefulness.
The benchmarks that still measure something hard
There are two more tests worth keeping on the radar because they're still unsaturated and resist the general inflation.
ARC-AGI, proposed by François Chollet in 2019 as part of On the Measure of Intelligence (arXiv:1911.01547), is a set of abstract visual puzzles where the model must infer a rule from few examples and apply it to new cases. The puzzles can be solved by an eight-year-old, and even so for years frontier models stayed at ridiculous figures. In December 2024, OpenAI announced that o3 had reached 75.7% on the semi-private evaluation with the competition's standard compute limit, rising to 87.5% with a very-high-compute configuration (172x). That leap was legitimate news. Before o3, the record hovered around 30%.
What ARC-AGI measures —abstract reasoning with few cues— is what for decades was considered the mark of general intelligence. That a model solves it consuming massive compute doesn't settle the debate over whether it understands or brute-forces. But it shifts it to far more interesting ground than MMLU.
The other serious candidate is FrontierMath, proposed by Elliot Glazer and collaborators in November 2024 (arXiv:2411.04872). It's hundreds of original, unpublished maths problems, written by mathematicians in the field, and designed to resist memorisation. At its launch, frontier models solved under 2%. Like GPQA in its day, FrontierMath still has a long way to run.
How to read a benchmark table without swallowing the ad
When a company publishes a new model and shows a table, there are three things worth checking before trusting the headline.
The first is which benchmark it cites and which it omits. If the announcement only cites the benchmarks where the model wins, there's selection bias. If it cites a representative set —at least one reasoning benchmark (GPQA), one realistic programming one (SWE-bench Verified), one multidimensional one (HELM or equivalent), one hard unsaturated one (ARC-AGI or FrontierMath)—, there's method behind it. The difference between the two ways of presenting says more about the launcher than about the model.
The second is under what inference regime it was measured. A given model evaluated with extended reasoning and thirty minutes per question can score twenty points more than the same model evaluated in fast conversational mode. Comparing results between labs without matching the regime is a rigged comparison.
The third is whether the benchmark was in the training data. Benchmark contamination is a documented problem since 2022. Public test sets circulate online, end up in training corpora, and the model learns to answer them without learning to generalise. That's the reason GPQA Diamond was designed Google-proof and the reason FrontierMath keeps most of its problems unpublished. A benchmark's reliability is inversely proportional to how long it has been public.
The political question
This is personal opinion but I back it with data. The proliferation of benchmarks favours the big lab and harms the small one. Designing a serious benchmark costs hundreds of thousands of dollars in fees to specialists. Evaluating a model against all the relevant benchmarks consumes amounts of compute only the big labs can afford. The result is an ecosystem where whoever evaluates wins, and whoever can't pay for the evaluation is left out of the public conversation.
To this is added the perverse incentive of funding. Investment rounds and commercial deals are closed with benchmark tables. If raising three points on MMLU unlocks fifty million in capital, the incentive to optimise precisely those three points —training on the set, or near it— is enormous. The sector has known it for years and very few actors say it out loud because the current status quo suits almost everyone.
What I'd ask the Spanish press —and I say it knowing the English-language press barely does it better— is that when it reproduces a benchmark figure it add three pieces of data. Which benchmark exactly and from what year. Under what inference regime. What the runner-up's figure was. Those three pieces of data wouldn't make the article less sellable. They'd simply make it useful.
The latest documented saturation state, in Stanford HAI's AI Index 2026 published on 8 April 2025, shows that on MMLU the global top-5 range fits within less than 2.7 percentage points, while on SWE-bench Verified that range exceeds 12 points and on FrontierMath exceeds 18. That difference between the saturated and the discriminative is, today, the difference between following the noise and reading the signal.
Definitions
Saturated benchmark: a test on which frontier models obtain scores so close to the maximum that the difference between them no longer distinguishes capabilities. MMLU and HumanEval have been saturated since 2024. When a benchmark saturates, it's worth swapping it for a harder one, not insisting.
Expert human ceiling: the percentage of correct answers people with specific training obtain on the benchmark under reasonable conditions. On GPQA Diamond it hovers around 65%. It's the useful reference; when a model clears it by a margin, it's doing something non-trivial.
Benchmark contamination: the presence, in a model's training data, of questions or answers from the test set itself. If it happens, the result is artificially inflated. It's a growing risk as public sets circulate online.
Inference regime: the conditions under which a model is evaluated —reasoning tokens allowed, number of attempts, available external tools. Two figures from the same model under different regimes are not comparable.
References
Dan Hendrycks et al., Measuring Massive Multitask Language Understanding (arXiv:2009.03300, 2020, presented at ICLR 2021). The original MMLU paper.
David Rein et al., GPQA: A Graduate-Level Google-Proof Q&A Benchmark (arXiv:2311.12022, November 2023). The original GPQA and GPQA Diamond paper.
Mark Chen et al., Evaluating Large Language Models Trained on Code (arXiv:2107.03374, 2021). The original HumanEval paper.
Jacob Austin et al., Program Synthesis with Large Language Models (arXiv:2108.07732, 2021). The original MBPP paper.
Carlos Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv:2310.06770, ICLR 2024). The original SWE-bench paper and its later variants.
Percy Liang et al., Holistic Evaluation of Language Models (arXiv:2211.09110, November 2022, Stanford CRFM). The original HELM paper.
François Chollet, On the Measure of Intelligence (arXiv:1911.01547, 2019). The conceptual foundation and original proposal of ARC-AGI.
Elliot Glazer et al., FrontierMath: A Benchmark for Advanced Mathematical Reasoning in AI (arXiv:2411.04872, November 2024). The original FrontierMath paper.
Stanford HAI, Artificial Intelligence Index Report 2026 (April 2025). Source of the cited saturation state.
To go deeper
Stuart Russell & Peter Norvig, Artificial Intelligence: A Modern Approach (Pearson, 4th ed., 2020). The chapter on evaluating learning systems and classic validation problems.
Kevin P. Murphy, Probabilistic Machine Learning (MIT Press, 2022). A statistical framework for understanding why a single metric deceives.
OpenAI, Introducing SWE-bench Verified (openai.com, August 2024). Documentation of the benchmark's curated variant.
ARC Prize, ARC Prize 2024: Technical Report (arxiv.org/abs/2412.04604, December 2024). The technical document of the 2024 edition of the prize and o3's performance.
You might also like
- What is the Chatbot Arena?
- Measuring artificial intelligence
- Deceptive metrics. When the exam was already in the study material
- Thinking without understanding. If it works the same, what did we think thinking was
Comments0
No comments yet.
Leave a comment