The most misunderstood graph in AI just got a major update

A new report from the AI evaluation nonprofit METR finds that some AI capabilities are continuing to improve along an exponential trajectory—but it also indicates that models aren’t quite as good at tasks that require judgment as they are at coding. 

The report is an update to the organization’s time horizon plot, which was widely shared—and widely misinterpreted, due to the complexity of the measures it used. METR evaluated models by having them attempt a large suite of tasks, each of which had also been completed by humans. Each task’s difficulty was scored by how long it took the human testers to finish it—and then models were rated according to the length of the tasks that they could complete. That plot showed that AI systems were rapidly improving—every four months, the length of the tasks that frontier models could successfully complete doubled.

The updated plot adds more tasks—some that are estimated to take humans far longer than the tasks in the original suite, and others that are more open-ended. One of the biggest questions in AI right now is the extent to which progress on “verifiable” tasks—where it’s possible to check automatically whether an AI system has succeeded or failed—carry over into tasks on which success or failure is harder to measure. We know how to teach neural networks how to do verifiable tasks: Researchers have the systems attempt the task many times, strengthen the neural pathways that contribute to success, and weaken the pathways that contribute to failures. That’s why AI systems have become so good at coding and math, two domains where it’s possible to determine automatically whether a solution does or doesn’t work.

It’s much harder to apply this technique, known as reinforcement learning, to tasks where success is less well defined or takes longer to ascertain—such as choosing a new research direction. For the same reason, it’s tough to evaluate AI performance on these tasks, so METR’s original time horizon plot didn’t include any of them. But it’s precisely this type of task that will prove essential to AI systems automating research and development.

And AI research and development is an especially hot topic right now. Parts of the AI community are still reeling from a different, heavily redacted METR report from earlier this month weeks ago, which claimed that AI might be speeding up the pace of AI capabilities progress at Anthropic by a factor of two. That report sparked fears that the company (which just had a blockbuster IPO) could soon achieve full recursive self-improvement, a regime in which AI systems autonomously design their own successors with no human involvement.

METR researchers have worked to incorporate some of these fuzzier, judgment-laden tasks that could be most relevant to AI research and development into their new suite. In one task, they ask models to design new AI evaluations that METR could use in its research and then have human METR employees score the results. Even though the most advanced models are now better than almost every human at software engineering, here they fall short: Not one of the AI-designed evaluations was good enough for METR employees to consider it useful for their research.

In another task, models have to score philosophical arguments. This task is intended to capture the extent to which models can exercise good judgment—though METR writes in the report that employees disagree about the extent to which the task is relevant for AI R&D automation. On this task, too, models fell short of human performance.

Though it’s difficult to score these new tasks—and even more so to determine what these scores mean—experts I spoke with appreciate that METR is incorporating new types of evaluations into their work. “We’re at the point where AI evaluations need to get more creative,” says Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley. “Trying to push the envelope of the space of evaluations is a welcome change.”

Kapoor says he wasn’t surprised to see that models fall short of performance on more open-ended tasks, such as designing an AI evaluation—in a preprint from August, he and his collaborators reported that AI models are not yet able to reproduce human computer science research on their own. And in a Wednesday blog post, OpenAI reported that its models don’t do as well on the real-world tasks its employees ask them to do as they do on the easier, more verifiable tasks in METR’s original task suite.

Still, people disagree about how to interpret current performance differences on easily verifiable and hard-to-verify tasks, Kapoor says. Though he thinks that research taste—the ability to choose promising research questions, hypotheses, and experiments—will prove a major bottleneck to AI R&D automation, others contend that models trained on easy-to-verify tasks will eventually be able to successfully complete open-ended tasks as well. And progress on verifiable tasks doesn’t seem to be slowing down. 

Even if the models do lag behind human performance on open-ended tasks right now, the new report suggests that they might be getting better. Some of the newest models, such as Opus 5.5, tend to achieve what METR calls “borderline above the bar” performance on the evaluation design tasks—not yet usable for METR’s research, but promising. Just a year ago, every model was scoring in the “clearly below the bar” category.

But because these open-ended and judgment-heavy tasks are so tough to design and score, METR includes few of them in its report, and that makes it difficult to predict whether and how models might continue to progress in these areas. “The improvement we observe, I think that does tell us something for the current moment,” says Najoung Kim, a linguist and computer scientist at Boston University. “I’m not sure just yet how to think about how it would extrapolate.”

Matt Putz, a member of the technical staff at METR, says that predicting progress on these kinds of tasks is likely to remain difficult. “By their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify,” he said in an email. “Ultimately, I don’t think we’ll be able to establish trends as clean and robust as for the easy-to-specify [tasks].”

In general, METR is clearly butting up against the practical difficulties of testing AI performance on difficult and underspecified tasks—precisely the tasks most relevant to AI R&D automation. METR’s time horizon plots score task difficulty by evaluating how long it would take a human to complete the task, and for the original plot they measured this quantity by actually having people try each task. 

In the new report, however, the completion times are mostly estimates, because paying people to complete tasks that might take hundreds of hours, or that only a few experts are qualified to complete, is prohibitively expensive and time-consuming. Other tasks have more specific shortcomings—for example, for the philosophical argument task, just one person rated most of the arguments, and agents are judged by how well they agree with that particular individual. It’s possible that the scores would change if more experts were surveyed—but that, too, is time consuming.

It’s still early days for evaluating models on tasks that are hard to validate or require judgment. And while the experts I spoke with see the inclusion of these tasks in METR’s newest time horizon plot as a clear step in the right direction, they agree that we are still far from achieving any definitive answers about whether, and how quickly, automated AI R&D might be coming. “This move towards open-ended tasks is promising,” Kapoor says. But, he adds, “there is just such a lot more to be done to actually get a fuller understanding.”

Scroll to Top