AI Agent Time Horizon Measurement Is Not a Uniform Capability Scale

The time horizon is useful, but its ruler is uneven

AI agent time horizon measurement does not provide a uniform scale of capability. A new statistical audit finds that equal movements in human task duration can correspond to unequal movements in AI task difficulty.

The result matters because the METR time horizon is unusually legible. It expresses agent performance in minutes or hours of equivalent human work rather than an abstract benchmark score. Yet that clarity can invite an interpretation the data does not support: that moving from three to 30 minutes is comparable with moving from 30 minutes to five hours because both are tenfold increases.

Drew T. Nguyen and William Fithian tested that assumption using results from 26 AI systems on 228 software tasks. Their fitted relationship between human completion time and latent AI difficulty was nearly flat from roughly two to 30 minutes, then closer to linear elsewhere. Their conclusion is precise: the two tenfold jumps above do not represent equal capability changes (Nguyen and Fithian, 2026).

This is a critique of interpretation and estimation, not evidence that agent progress stopped. The audit's alternative model still produces a strong rising trend. It also finds that a rescaled latent-capability series closely matches METR's original estimates, with an R² of 0.996. The familiar plot may therefore track a real capability signal while labelling parts of that signal with a misleadingly uniform time scale.

What the 50% METR time horizon actually measures

METR defines the 50% time horizon as the human completion time at which a fitted model predicts that an agent will succeed half the time. It is not the agent's wall-clock runtime. It is also not a claim that the agent can perform every task below that duration (METR, 2026).

The original study combined HCAST, RE-Bench and 66 short Software Atomic Actions. Its July 2026 revision reports 170 tasks spanning seconds to hours. Skilled professionals attempted most tasks, producing more than 800 human baselines and 2,529 recorded hours. The researchers then ran agents repeatedly and fitted success probability against the logarithm of human completion time (Kwa et al., 2025).

For each agent, the original form was a logistic curve. The estimated horizon is where that curve crosses 50% success. This construction compresses many observations into one readable number:

1. Define self-contained tasks with automated scoring.
2. Estimate how long qualified humans need to complete each task.
3. Run each model-and-scaffold combination several times per task.
4. Fit success probability as a function of human task time.
5. Invert the fitted curve at 50% or another reliability threshold.

That metric has several boundaries. METR's current page says estimates above 16 hours are unreliable with its task suite. Its tasks concentrate on software engineering, machine learning and cybersecurity. Human baseliners also have less project context than an established employee, so the resulting hours resemble low-context contractor work more than ordinary work inside a familiar organisation (METR, 2026).

The 50% threshold is consequential too. In the original analysis, 80% horizons were four to six times shorter than 50% horizons, although their historical doubling rates were similar. A buyer who needs dependable execution should therefore not translate a headline 50% horizon directly into unattended operating time (Kwa et al., 2025).

The audit replaces a straight line with a learned conversion

The original method assumes that AI difficulty changes linearly with the logarithm of human time. In plain terms, each doubling of human duration is assumed to add the same amount of difficulty for the agent. That is the hidden regularity behind reading equal horizontal steps on a log-time chart as equal capability gains.

Nguyen and Fithian relaxed that assumption in two ways. Their first model replaced the straight line with a shared monotone cubic spline. Their second used an explanatory item-response model with task-family effects, task-level variation and overdispersion between repeated runs. Both models used the same underlying public observations: outcomes for each AI, task and run, plus the task's human-time estimate.

The overdispersion adjustment is material. Under an independence assumption, the fitted baseline would predict that about 60% of repeated-run vectors were unanimous. The observed share was 83%. Runs on the same AI-task pair therefore carried more redundant information than an independent-trial model implied (Nguyen and Fithian, 2026).

The authors compared models with five-fold cross-validation over 79 task families. They used 12 combinations of proper scoring rules and weighting schemes. Both proposed models beat the shared-slope baseline on every reported point estimate. The improvement was concentrated among systems whose original horizons fell between two and 15 minutes. Individual-system intervals were often wide, while the pooled comparison supplied the stronger evidence.

This distinction prevents overclaiming. The audit does not prove one universal nonlinear law for all agent tasks. It shows that a more flexible relationship predicted held-out outcomes better on this version of METR's software-task data.

Equal gains in minutes can hide unequal gains in difficulty

The fitted spline is the article's central result. Between approximately two and 30 minutes of human time, estimated AI difficulty changed little. A model could move its 50% crossing point a long way through that interval after a comparatively small change in underlying ability. Outside that interval, the conversion was closer to linear.

This is a comparability problem. Human time remains predictive overall, but the conversion rate changes along the scale. A minute is still a minute for a person; it is not a constant unit of difficulty for an AI.

The same issue becomes sharper across domains. METR's cross-domain analysis found that human task length correlated strongly with model difficulty in some software and programming benchmarks. It correlated weakly in video understanding, where longer videos did not reliably require more fundamental steps. METR explicitly warns that a time horizon may be inappropriate when the anchor does not predict agent success (Kwa and Cheng, 2025).

Benchmark setup can also move results. Epoch AI notes that prompt and evaluation-setting changes can produce meaningful score differences, and that contamination may inflate some benchmark results (Epoch AI, 2026). A time horizon belongs to the evaluated model, scaffold, tools, budget and task distribution together. It should not be attributed to the base model alone.

The broad trend survives, but the middle of the chart changes meaning

The original paper estimated a horizon doubling about every 207 days, with a bootstrapped 95% interval of 166 to 240 days for its 2019–2025 data. It also stressed that task-selection errors were correlated across models, making the trend more stable than any one model's absolute horizon (Kwa et al., 2025).

METR later expanded the suite from 170 to 228 tasks in Time Horizon 1.1. It raised the number of tasks estimated at eight hours or longer from 14 to 31 and moved evaluation infrastructure from Vivaria to Inspect. The stitched 2019–2025 trend still doubled in 196 days. Since 2023, however, the revised suite produced a 131-day doubling estimate versus 165 days under the older setup (METR, 2026).

Those changes already showed that task composition and infrastructure affect the fitted rate. The new audit adds another layer: the statistical conversion from human time to AI difficulty affects how parts of the curve should be read. Under its preferred item-response model, the fitted frontier time-horizon trend still doubled in about six months, with R² of 0.931.

The responsible conclusion is therefore narrow. Evidence still supports rapid improvement on this software-task distribution. Evidence does not support treating every time-horizon doubling as an equal increment of capability, especially around the two-to-30-minute region. Nor does it justify a direct forecast of job automation.

METR's own limitations note makes the operational point plainly. Time horizon is not how long an AI can work independently, absolute estimates have historically carried error bars of roughly a factor of two in each direction, and domain-specific horizons can differ substantially (Kwa, 2026).

What to measure before acting on a horizon claim

A time horizon can help select systems for deeper testing. It cannot replace that testing. For an organisation considering AI employees, a custom AI workflow or agent features inside custom software, the relevant unit is the production workflow.

Build the decision around a small measurement stack:

• Task-distribution fit: Sample actual tickets, documents, tools and exception paths. Report where these differ from the benchmark.
• Success curves: Keep results by task bucket rather than publishing only one 50% crossing point. A curve reveals whether performance falls gradually or abruptly.
• Repeated-run reliability: Run each case enough times to estimate variance. Our related guide explains why AI agent consistency testing needs repeated runs.
• Higher reliability thresholds: Measure 80%, 90% or the threshold demanded by the workflow. Do not extrapolate it from 50%.
• Failure severity: Separate harmless retries from irreversible actions, privacy breaches, incorrect payments and destructive changes.
• Human context: Compare the agent with the employee or contractor who would really perform the work, including repository familiarity and organisational knowledge.
• Cost and latency: Record model, token budget, tool calls, retries, reviewer time and wall-clock duration. Capability without acceptable unit economics is not deployment readiness.
• Change control: Re-run the suite when the model, prompt, scaffold, tools, permissions or environment changes.

A useful evaluation report should show the headline horizon, the full conditional success curve, uncertainty intervals and a time-to-difficulty diagnostic. If the human-time relationship flattens, split the tasks by meaningful operational features or report a direct success rate instead.

The immediate next step is simple: take 30 to 100 representative production tasks, define pass criteria before testing, and measure repeated outcomes at the reliability level the workflow requires. If you need an implementation partner for that process, talk to Cloudex about a workflow-specific agent evaluation.