For eleven weeks this fall, a small team from METR, an independent AI evaluation group, worked inside Anthropic to answer one question: how much is AI speeding up the work of building AI? Their answer, published today, is that it already is, by a lot. METR’s staff estimate that as of October 2026, Anthropic is improving its models on well-specified tasks at about twice the rate it managed between March 2024 and October 2025.

The headline number comes from AECI, Anthropic’s version of the Epoch Capabilities Index shown below, which scores models across a set of benchmarks. Before October 2025, Anthropic’s models gained about 15 AECI points a year. Since then, the trend has steepened to about 29 points a year. That is a 1.9x acceleration, with a 95% confidence interval of 1.5x to 2.8x, which means Anthropic made about two years of historical progress in the last twelve months.

The estimate moves with the method. Different assumptions about how benchmark difficity is distributed produce anything from 1.1x to 2.9x. Adding models from OpenAI, Google and xAI cuts the figure to 1.4x (chart below), partly because Anthropic’s earlier models trailed the industry frontier, which makes its own history look slower than it was. METR says the biggest reason its number beats Anthropic’s published 1.7x is the inclusion of recent internal-only models.

METR says it is confident the speedup did not come from faster growth in training compute, and METR’s best guess is that around 80% of the current acceleration is due to AI, with the caveat that non-AI factors probably played a role in the jump in late 2025. The supporting evidence is soft but consistent. In interviews, researchers described speedups centered near 2x, and of the eleven interviewees whose answers could be coded, five said or leaned yes to 2x or more, four were unclear, and two said no. In informal Slack polls, staff rated the newest internal model at about 12x uplift, up from roughly 1.2x for Claude 3.7 Sonnet. METR treats that as loose evidence, since the responses are low-effort and may overstate real gains. It also stresses a key distinction: a person being 12x faster does not mean the whole research effort is accelerating 12x. Sustained acceleration in output requires sustained growth in inputs like labor and labor productivity, so what matters most is whether individual uplift keeps growing.

That leads to the risk METR cares about most, which is compounding: If better models speed up the research that produces the next models, the gains could feed on themselves. METR looked for the things that would normally break that loop, such as limits on algorithmic progress, training compute, experiment compute or data, and found no convincing bottleneck over the next 12 months. METR it concludes there is a significant risk that further progress leads to even faster acceleration over the coming year.

The report spends real space on how it could be wrong, in both directions:

The number could be too low if growth is accelerating continuously, if automation is closer than researchers believe, or if benchmarks understate real-world impact. METR points to math, where benchmark scores have not jumped but long-standing problems are now falling.

It could also be too high if non-AI factors explain most of the gain, if Anthropic’s models are overfit to benchmark-style tasks, or if the historical rate was unusually slow.

Measuring is getting increasingly harder meanwhile. Anthropic’s newest internal model, Model 4, scores above 50% on all 15 benchmarks it was tested on, which makes it harder to measure the next jump. METR also notes that Anthropic spends fewer tokens per benchmark task than would fully elicit a model, though it guesses this affects old and new models similarly.

Then there is the matter of what the public can actually see. Anthropic gave METR more access than the group expected, including internal benchmark data, headcount, code and agent spending figures, and about 15 staff interviews. But Anthropic held unilateral redaction rights and used them heavily. Intermediate internal measures, algorithmic progress estimates, bottleneck analysis, the breakdown of progress between compute, data and algorithms, and forecasts for future models were all cut from the public report. Anthropic says the redactions were meant to remove information that could speed up other developers. METR says much of what it saw had several plausible readings, and that its own staff and Anthropic’s researchers disagree on how to put the evidence together. That leaves readers leaning on METR’s judgment more than anyone would like. A planned check of how Anthropic splits compute between capabilities and safety was dropped midway through. Anthropic said it wanted to stay focused on risk and would revisit it later.

METR's full report is available here.

© Raydn · raydn.us