How to read an AI benchmark: ECI, Arena and the numbers in launch posts

A benchmark score is the share of a fixed set of tasks a model got right, under conditions someone chose. To read one, ask what the test asks, whether it is still hard, who ran it and with what settings.

What is a static benchmark?

A static benchmark is an exam with an answer key: fixed questions, marked automatically. Most launch numbers come from them.

MMLU (2020) is the classic. It has 15,908 multiple-choice questions across 57 subjects, from elementary maths to law. In the original paper, smaller models scored at random chance, 25% with four options, and the largest GPT-3 model reached 43.9%.

GPQA (November 2023) went for depth: 448 multiple-choice questions in biology, physics and chemistry, written by domain experts. People with or pursuing PhDs in those fields scored 65%. Skilled non-experts scored 34%, despite spending over 30 minutes on average with unrestricted web access. The strongest GPT-4 baseline reached 39%.

Humanity's Last Exam (January 2025) exists because models were scoring over 90% on MMLU. It has 2,500 multiple-choice and short-answer questions across dozens of subjects, each with a verifiable answer that cannot be found quickly online.

How do benchmarks go wrong?

In two main ways: they saturate, or their questions leak into training data.

Saturation. When every top model scores near the ceiling, the test stops telling them apart.

Contamination. Benchmark questions are public, and models are trained on large parts of the internet. If test questions leak into training data, a model can recall answers instead of working them out. A 2024 study checked this by writing GSM1k, a fresh set of grade-school maths problems matched to the widely used GSM8k. Some models scored up to 8 percentage points lower on the new set, and models more likely to reproduce GSM8k examples showed bigger gaps. Coding tests have their own version of the story, covered in our guide to coding benchmarks.

How do arena leaderboards work?

Arena leaderboards rank models by votes, not answer keys. On Chatbot Arena, a user types any prompt, gets answers from two anonymous models and votes for the better one. The names appear only after the vote. The 2024 paper describing it had collected over 240,000 votes. Rankings come from a Bradley-Terry model, a statistical method that turns many head-to-head results into one rating per model, and each rating comes with a confidence interval.

Fresh prompts make memorised test questions matter less. The authors list the weak points: voters were mostly AI hobbyists and researchers, prompts come from a chat website rather than real work, and safety is not measured.

What is the Epoch Capabilities Index?

The Epoch Capabilities Index (ECI), from the research group Epoch AI, combines scores from over 50 benchmarks, including GPQA Diamond, Humanity's Last Exam and MMLU. It infers how hard each test is from models that took several, and rewards doing well on harder ones.

The scale is arbitrary. Epoch fixes Claude 3.5 Sonnet at 130 and GPT-5 at 150, and there is no maximum, so only differences mean anything. It mixes developer-reported scores with its own runs and independent leaderboards, which it says limits cherry-picking.

In Epoch's scores file updated on 16 October 2026, the top three were Claude Opus 5.5 at 166.8, GPT-6 Astra at 166.4 and GPT-6.1 Sol at 166.3. Their confidence intervals run from about 163 to 171 and almost entirely overlap. The data cannot say which of them is ahead.

Why does the same model get different scores?

Because the score depends on how the model is run. The footnotes on Anthropic's Claude Opus 5.5 launch page, published on 22 September 2026, show what to look for:

• Unless noted, Opus 5.5 results use adaptive thinking at maximum effort, not the default medium setting. Some tests also get a separate default-effort figure.
• Its Humanity's Last Exam score, 67.7%, is with tools only. No score without tools is shown.
• One comparison uses a rival's figure as reported by that rival, not a run by Anthropic.
• One test lists a standard error of plus or minus 2.6 points.

How big is the margin of error?

On a test the size of GPQA, close to plus or minus 5 points. By the standard formula, a model scoring around 50% on GPQA's 448 questions has a standard error of about 2.4 points, so a 95% interval is close to plus or minus 5 points. Few launch posts show this.

A 2024 paper by Evan Miller of Anthropic argues that evaluations are experiments and should report standard errors, cluster them when questions come in related groups, and compare two models question by question.

What are the main types of benchmark?

There are five main types, and each has its own weak spot.

Our take

No single number ranks models. Static tests describe a skill until they saturate, arenas describe what one crowd prefers, and the ECI gives the steadiest overall picture, which at the top is now a near tie. A vote can also reward answers that read well over answers that are right, so check an arena rank against a test with an answer key. A headline score is the best case under conditions the seller chose.

Five questions for a launch post

1. Who ran it? The company's own run, or Epoch or another independent leaderboard?
2. At what setting? Maximum effort, tools on, several attempts, or the default you will actually use?
3. Is the test still hard? If every top model scores above 90%, a 1-point lead means little.
4. Is the gap bigger than the error bar? If none is shown, assume a few points either way.
5. Does it match your work? A science exam score says little about drafting contracts or answering customers.