AI agent consistency testing needs repeated runs

AI agent consistency testing exposes what average pass rates hide

AI agent consistency testing must run the same task more than once. A mean pass rate measures average success across attempts; it does not show whether a task that passed today will pass again tomorrow.

That distinction is already visible in interactive-agent research. In a September 2026 study, a ReAct agent using GPT-4.1 completed 77.4% of AppWorld attempts across five runs. Yet it passed all five runs on only 53.0% of tasks. The 24.4 percentage-point difference is what the authors call the "consistency gap" (Duesterwald et al., 2026). About 32% of the 168 tasks mixed passes and failures across those runs.

The benchmark matters here. AppWorld contains 750 tasks across nine simulated apps and 457 APIs. Its state-based tests accept different valid solutions and can detect collateral changes. That makes the repeated-run result harder to dismiss as a quirk of exact-string grading.

The finding is still narrow. It covers one ReAct setup, two models and one AppWorld split. The paper is also a preprint. It does not establish a universal consistency rate for agents. It does establish that a respectable mean can coexist with much weaker repeatability.

A broader 2026 reliability study reaches the same measurement conclusion from another direction. It argues that one success score compresses away consistency, robustness, predictability and error severity. Its evaluation covered 15 models on two benchmarks (Rabanser et al., 2026). The useful lesson is not that every deployment needs 12 dashboard tiles. It is that "accuracy" is too overloaded to carry an operational reliability claim.

Pass@k, Mean@k and Pass^k answer different questions

Repeated run agent evaluation needs notation that separates capability from dependability. Three metrics use the same trials but answer different product questions.

For independent trials with per-run success probability p, the expected values are 1-(1-p)^k, p and p^k. Anthropic gives a plain example: a 75% per-trial rate implies only about 42% probability that all three trials succeed (Anthropic, 2026).

Do not calculate a benchmark's Pass^k by raising its overall mean to the power k. Tasks differ in difficulty, and failures may be correlated. Measure Pass^k per task from actual reruns, then average those task results.

The choice of k changes the question. Pass^5 is a stricter test than Pass^3, so figures with different rerun budgets are not directly comparable. Report k, the number of tasks, the number of total trials and uncertainty intervals beside every result.

Pass^k also has a hard limitation: it sees verdicts, not behaviour. Five consistent failures score zero, just like a task that fails in five materially different ways. That difference matters when an agent can send messages, modify records, issue refunds or place orders.

The same verdict can conceal different actions

The sharper failure mode appears below the final score. An agent can receive the same pass or fail verdict while choosing different tools, arguments, destinations or payloads.

A September 2026 clinical-agent preprint tested this directly on MedAgentBench, a virtual electronic health record benchmark with 300 tasks in ten categories. The researchers selected 50 write-capable tasks and ran each five times under four configurations: Qwen3-4B and Llama-3.1-8B, each at temperatures 0.05 and 0.7. Both models were four-bit quantised. The grid produced 1,000 runs (Bellibatlu et al., 2026).

Under Llama-3.1-8B at temperature 0.7, 43 task groups emitted at least one order. All 43 produced more than one canonical order set across five identical runs. Twenty-six sometimes emitted an order and sometimes emitted none. Twelve changed the destination endpoint, while 28 changed a clinical field such as the code, recorded value, medication or dose.

The benchmark verdict did not expose all of this movement. In 22 of those 43 groups, the order changed while all five runs retained the same failing verdict. Qwen3-4B at temperature 0.7 showed the same pattern in all ten groups whose orders diverged.

One environment check made the operational consequence concrete. The same service request returned 201 Created at the typed FHIR endpoint and 400 Bad Request at the collection root. MedAgentBench's original scaffold did not execute writes. It checked whether the payload parsed as JSON and then told the agent the action succeeded. A verdict could therefore miss both the changed endpoint and the real server's rejection.

These results do not show that clinical agents as a category have those failure rates. The authors state that the rates do not generalise. The panel used two small, locally served models, an original scaffold with hand-built HTTP requests and one task prefix. A frontier model or revised scaffold could behave differently. The study supports the existence of hidden action divergence and supplies a protocol for measuring it.

A useful rerun protocol controls more than the prompt

Running a benchmark five times is not enough if each run inherits different state. The test must hold the task input fixed and control the surrounding system.

1. Freeze the model identifier, system prompt, agent scaffold, tool schemas and grader version. Record provider-side version changes when the provider exposes them.
2. Start every trial from a clean snapshot. Reset databases, files, queues, clocks and credentials that can affect the route through the task.
3. Preserve the full trace: model messages, tool names, arguments, tool results, retries, state changes, token use, latency and final outcome.
4. Execute enough independent trials to match the risk. Five is a useful diagnostic start, not a universal standard.
5. Compare results at three layers: final outcome, action sequence and resulting state.
6. Report confidence intervals and retain task-level results. An aggregate without denominators cannot distinguish a broad issue from one unstable task family.

Clean isolation is easy to overlook. Anthropic reports that shared files, cached data and resource pressure can create correlated failures or give later trials an unfair advantage. Its guidance defines a trial as one attempt at a task and recommends multiple trials because model outputs vary between runs (Anthropic, 2026).

Temperature zero does not remove the need to test. A 2024 study ran five models on eight tasks over ten repetitions under nominally deterministic settings. It observed run-level accuracy variation of up to 15 percentage points and a best-to-worst task-performance gap as high as 70 percentage points (Atil et al., 2024). The authors identify continuous batching, chunk prefilling and prefix caching as possible causes, but say they can only speculate because several tested models were accessed through closed APIs. Local serial inference may be more repeatable, but that property belongs to the whole serving stack, not to the temperature field alone.

Action-level metrics show where consistency breaks

Outcome metrics should remain the top line, because users care whether the task finished. The diagnostic layer beneath them should compare actions in a domain-aware form.

Canonicalisation determines whether these metrics are useful. Sort query parameters. Remove timestamps only when time is irrelevant to meaning. Treat aliases as equal only when the destination system does. Compare sets where order is immaterial and sequences where ordering changes the outcome. Publish these rules because each choice changes the reported divergence rate.

Pairwise disagreement is useful across different trial budgets. It measures the share of run pairs whose canonical actions differ. An "ever diverged across five runs" rate can only rise as more trials are added; pairwise disagreement is less tied to that budget. Both still need their denominators.

Trace divergence is evidence for debugging, not automatically a defect. Two paths can produce the same valid state. Conversely, identical final text can hide different side effects. This is why AI agent observability and consistency testing should share one trace format rather than operate as separate projects.

The IBM consistency-gap study adds another possible diagnostic. It resampled each inference point 30 times at temperature 0.5 and compared action names and parameters with pairwise Jaccard similarity. It then generated prompt guidelines for steps below a 0.85 consistency threshold. On ReAct with GPT-4.1, those guidelines raised Pass^5 from 53% to 69% on the same tasks and to 66% on similar tasks (Duesterwald et al., 2026). The method is promising but expensive: the authors estimate about 30 times the token cost per analysed trajectory, and a stable decision can still be consistently wrong.

What to do before an agent reaches production

A team deciding whether to deploy an agent should set the reliability target from the workflow's consequence, not from the benchmark's average. A research assistant that produces several candidates can tolerate retries. An agent that changes a customer record, sends a payment or schedules care needs a much higher all-runs-success threshold and explicit action checks.

Start with 20 to 50 tasks drawn from real workflows and known failures. Run each task five times in isolated state. Report Mean@5, Pass^5, the mixed-outcome task rate and pairwise action disagreement. For state-changing work, add omission, endpoint, argument and postcondition checks. Stratify results by task family; a pooled mean can hide one dangerous operation.

Budgeting is practical. Five runs multiply model and environment cost by five before trace grading. A staged design controls that cost:

• Run repeated trials on every high-consequence task.
• Sample lower-risk tasks and expand any task that shows mixed outcomes or action divergence.
• Re-run the consistency suite after model, prompt, tool, retrieval or scaffold changes.
• Move stable, high-value tasks into continuous regression testing.

Teams building AI employees should treat this suite as a release gate. Teams implementing custom AI solutions should include the rerun system, state reset and trace schema in the design rather than add them after launch.

The decision rule is simple: do not approve a consequential agent because its average pass rate looks acceptable. Approve the tested workflow only when repeated outcomes, actions and resulting state remain inside a defined tolerance. The next useful step is to select one real task, run it five times from the same clean snapshot and inspect the first point where the traces disagree.