ZICQ
中 Log in / Sign up
Newsroom LLMs & Foundation Models #METR #AI Evaluations #Time Horizons #Verifiable Tasks #Automated AI R&D

METR Updates AI Time Horizons: Verifiable Tasks Double Every Four Months, Judgment Still Lags

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:METR’s updated AI time-horizon report finds that frontier models can complete verifiable tasks whose human reference duration is roughly doubling every four months. The same update broadens the suite with long-running and open-ended work: models failed to produce an evaluation that METR researchers considered useful, and underperformed humans on philosophical-argument scoring. The results point to a two-speed capability profile rather than one universal curve. Some recent models, including Opus


Core Finding: The Growth Curve and the Capability Boundary Must Be Read Separately

METR has updated its AI time-horizon plot. The human-reference duration of tasks completed by frontier models still follows an exponential pattern, roughly doubling every four months on automatically verifiable work. The new suite, however, also includes long-running tasks estimated to require hundreds of human hours, as well as open-ended work such as designing evaluations and scoring philosophical arguments. The result is not a single curve showing that every dimension of AI improves at the same rate. It is a two-speed capability profile:

  • Verifiable tasks: Success can be checked quickly by a compiler, test harness, or deterministic scorer. Model capability shows a comparatively stable doubling trend.
  • Judgment-heavy tasks: Goals are underspecified, feedback is delayed, and quality is subjective. Models remain below human performance, while the available progress signals are too sparse for a robust trend.

METR therefore demonstrates both the leverage of automatic verification and the gap between solving a well-defined problem and choosing the right problem, designing a valid evaluation, or judging what matters.

What the Time-Horizon Metric Actually Measures

METR assigns each task a human-completion-time baseline, lets models attempt the tasks, and uses a passing threshold to determine how long a human-equivalent task a model can reliably complete. The task duration is not how long the model thinks; it is how long the work normally takes for a human expert.

If H(t) denotes the time horizon, the reported four-month doubling can be approximated as:

H(t) ≈ H₀ × 2^(t/4)

That corresponds to a logarithmic growth rate of about ln(2)/4 = 0.173 per month, or an eightfold expansion over a year if the trend remained unchanged. This is a descriptive fit to the current sample and task distribution—not a universal constant, a causal law, or a guaranteed arrival date for autonomous AI R&D.

The metric is intuitive because it translates model capability into human-readable work duration, but it has three structural limitations:

  1. Human duration is not a pure cognitive measure. It can depend on domain expertise, tool proficiency, information retrieval, waiting, and collaboration.
  2. The task distribution changes. The update adds longer and more open-ended work, so the newer curve does not represent exactly the same scope as the original one.
  3. Results depend on the system configuration. Tool access, inference budget, retry policy, prompt scaffold, and external orchestration can all change whether a model crosses the threshold.

Why Reinforcement Learning Favors Verifiable Work

Coding and mathematics are unusually well suited to scaled training because success can be checked automatically. The model makes an attempt, a test program returns pass or fail, and reinforcement learning updates the policy: successful trajectories receive more reward while failure modes are suppressed. As validators cover more edge cases, reward signals become denser and can scale with additional compute.

Open-ended tasks face a different engineering constraint. The objective of evaluation design is not merely to generate plausible test questions; it is to produce measurements that distinguish model capabilities accurately, prove useful to researchers, and resist gaming. Scoring philosophical arguments similarly has no unique answer, so useful feedback arrives only after an expert reads and interprets the response.

The limiting factor is therefore not reasoning alone, but verification economics:

  • Is validation cheaper than the task itself?
  • Is feedback fast enough for frequent policy updates?
  • Does the validator measure outcomes, processes, or the underlying value the task is meant to capture?
  • Can the model exploit the reward function, as in reward hacking?

Even a task that is verifiable in principle can still have a weak training signal if validation is expensive, incomplete, or easy to game.

Benchmark Validity: Why the Open-Ended Trend Cannot Yet Be Extrapolated

The most important contribution of the update is not a claim that open-ended performance has improved. It is METR's recognition that the current evidence is insufficient.

According to the report, none of the AI-designed evaluations was considered useful by METR's researchers, and models underperformed humans on philosophical-argument scoring. Yet most completion times for the new long tasks were estimated rather than measured through hundreds of hours of expert execution. The philosophical task relied primarily on one rater, so a model's score largely represented agreement with that individual's preferences rather than consensus among independent experts.

This weakens both the statistical basis and the external validity of an open-ended trend:

  • A small number of tasks and models cannot separate real progress from prompt sensitivity and scoring noise.
  • One expert's preference can be mistaken for a general capability ceiling.
  • A “borderline above the bar” classification is directional evidence, not enough for exponential extrapolation.
  • Public or contaminated tasks may reward memorization rather than general capability.
  • Comparisons become impure if models use different tools, inference budgets, or agent scaffolds.

The updated plot is consequently most useful as a diagnostic for the gap between verification-driven and judgment-driven capabilities. It is not yet a reliable forecasting instrument for automated AI R&D.

Reported Performance

| Evaluation dimension | Reported result | Appropriate interpretation | |---|---|---| | Automatically verifiable tasks | The human-reference duration of tasks completed by frontier models roughly doubles every four months | A strong trend, but only for the current task distribution and evaluation configuration | | AI evaluation design | No model-generated evaluation was considered useful by METR researchers | Software-engineering strength does not imply the ability to design valid research evaluations | | Philosophical-argument scoring | Models underperform humans | Highly dependent on the rater and rubric, with little group-consensus evidence | | Newer models | Opus 5.5 and some peers reach “borderline above the bar” on evaluation design | A year earlier all tested models were “clearly below the bar,” but the sample is too small for a trend |

The most revealing result is the transfer gap. Advanced models can already outperform nearly every human participant in software-engineering tasks, yet they still cannot produce an evaluation that METR is willing to use. Completing clearly specified work does not automatically imply defining which work should be completed.

Practical Guidance for Developers and Agent Evaluation

1. Avoid a Single Composite Score

Model selection should separate at least three layers: executable validation, rubric-based scoring, and blinded expert review. Code agents can be checked with hidden and property-based tests, but whether those tests cover the important risks still requires independent review. Collapsing these layers into one number hides the most actionable failure mode.

2. Calibrate Scoring for Open-Ended Tasks

A single LLM judge can amplify order, wording, and self-preference biases. A stronger workflow combines explicit rubrics, pairwise comparison, randomized or blinded presentation, multiple independent raters, and adjudication. Report inter-rater reliability so that agreement among models is not confused with agreement among judges.

3. Evaluate Agents End to End

Agent evaluation should record final task success, tool-call accuracy, retries, budget consumption, latency, regression results, and human acceptance—not merely whether the final text sounds polished. For open-ended tasks, also test whether the artifact changes a downstream decision or research workflow.

4. Hold Comparison Conditions Fixed

Every evaluation run should record the model snapshot, prompt template, tool set, inference budget, retry limit, and task-suite version. A change in any of these can move the measured time horizon. Trends should be refitted on a stable task distribution instead of simply extending an older curve.

5. Keep Human Gates in AI R&D Pipelines

A practical architecture today is: models propose candidate research directions or experiments; deterministic tools validate what can be verified; experts assess value, assumptions, and missing risks; and the system runs only low-cost experiments. Treating executable code as equivalent to a worthwhile research direction would skip the weakest layer in the current pipeline.

Bottom Line

METR's update does not prove that AI has acquired general judgment or that recursive self-improvement is imminent. It shows that verifiable work continues to expand along a clear exponential trajectory, while open-ended performance remains a sparse and difficult-to-extrapolate signal.

From an engineering perspective, the bottleneck is no longer only whether a model can produce an answer. It is whether the system can define success cheaply, detect reward hacking, obtain high-quality feedback, and convert that feedback into reliable training signal. Verification remains the main engine of automated AI R&D; judgment and evaluation determine whether that engine can move beyond closed tasks into open-ended research.


Source: MIT Tech Review AI (2026-11-20)

— END —

Tags: #METR #AI Evaluations #Time Horizons #Verifiable Tasks #Automated AI R&D

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…