Beyond Monitor Accuracy: A Decision-Value Framework for AI Agent Restarts

Summary:This work evaluates AI-agent restart monitors by their decision value rather than classification accuracy alone. An exact alarm-to-action identity for an independent, zero-overhead clean restart shows that a monitor with 90% accuracy can either reduce final success by five percentage points or increase it by 35 points, depending on the replacement attempt's residual-budget success probability. The authors introduce a decision-sufficiency criterion for comparing progress representations and a fi

---

Core Finding: Classification Is Not the Action Objective

This work does not introduce another failure classifier. It addresses a more fundamental evaluation mismatch: a monitor can identify failures with 90% accuracy and still trigger restarts that reduce overall task success. Whether restarting is worthwhile depends on whether the replacement attempt has a higher success probability than continuing the current trajectory—not on accuracy, recall, or false-positive rate alone.

Under an independent clean restart with zero monitoring overhead, the authors derive an exact alarm-to-action identity. Let (q_c) denote the probability that the current trajectory ultimately succeeds if continued, and let (q_r) denote the probability that a replacement attempt succeeds under its residual budget. The change in overall success probability can be expressed as:

[ \Delta S = P(\text{false alarm with successful retry}) - P(\text{missed failure with successful continuation}) ]

The identity shows that the same monitor with 90% accuracy can produce either a five-percentage-point loss or a 35-percentage-point gain. The decisive variables are the replacement attempt's residual-budget success probability and how monitor errors align with recoverable failures.

High recall therefore does not imply high decision value. If most failures are unrecoverable, or restarting discards useful work and context, frequent restarts may merely waste tokens and wall-clock time.

Decision Sufficiency: What a Progress Representation Must Preserve

The first contribution is a decision-sufficiency criterion, a binary-action specialization of Blackwell's comparison-of-experiments framework. It asks whether a compressed progress representation preserves the information required to choose between continue and restart, rather than whether it reconstructs the original trace faithfully.

For two representations (S_A) and (S_B), decision sufficiency requires that the expected value of the optimal action under (S_B) is no worse than under (S_A):

[ \mathbb{E}[V(a^\mid S_B)] \geq \mathbb{E}[V(a^\mid S_A)] ]

Here, (a^*) is the best action supported by the available information, and (V) is the action value. This separates trace-reconstruction fidelity from decision sufficiency. A compressed state need not preserve every historical detail; it only needs to avoid discarding information that would reverse the restart decision.

For engineering teams, the criterion can be used to reassess progress summaries, failure signals, compressed memory, and trajectory states. The relevant test is not only how closely a summary resembles the full context, but also whether both representations induce the same optimal action.

Finite-Sample, Model-Free Decision Audits

The second contribution is a model-free decision audit over existing agent traces. It uses binary outcomes from past executions and simultaneous binomial intervals to compare the conditional success of continue and restart, without additional calls to the evaluated model or new model training.

Under explicit sampling assumptions, the audit can certify a preference for one action when its confidence interval remains above the other. If evidence is weak or the intervals overlap, it returns abstain rather than forcing a restart decision.

This design reduces the risk of selective reporting and multiple-comparison errors. A team cannot simply select one successful restart and declare the policy effective; failed trajectories and the full denominator must remain part of the audit. In production, the decision layer can use the result to preserve the current trajectory when evidence is insufficient.

Finite-sample inference still depends on exchangeability and the sampling mechanism. Test tasks enriched with recoverable errors, or live traffic dominated by persistent infrastructure faults, may not generalize to future distributions.

SWE-bench Verified Results: Apparent Gains Fail to Replicate

The study covers 194 externally evaluated model executions on 19 selected SymPy issues from SWE-bench Verified, with a retrospective audit of 130 traces. The initial apparent restart gain was only 2.29 percentage points. It was traced primarily to a single surviving failed attempt rather than to a reproducible recovery mechanism.

On a second model, restarting instead produced a 42.29-percentage-point loss. No positive replication appeared in prospective task cohorts, and none of 103 live comparisons met the specified certification threshold. The available evidence therefore does not support a robust improvement in task success.

This result highlights the rare-trajectory problem in agent evaluation. Agent runs are often correlated: unusually strong or weak outcomes may reflect a particular model, tool path, or failure-recovery mechanism rather than the monitor itself. Averages can make a small number of extreme trajectories look like a stable gain.

Engineering Trade-offs and Deployment Guidance

Restart policies trade model calls and execution latency for recovery opportunities. Their value must include:

• Tokens, tool calls, and wall-clock time spent on retries;
• Repeated localization costs caused by lost context;
• Repeated triggering on unrecoverable failures;
• The monitoring model's own inference overhead;
• Effects of longer runtimes on scheduling and task completion.

A practical architecture separates three layers:

1. Monitoring layer: estimates failure or recovery opportunity without directly controlling restarts.
2. Decision layer: compares the conditional value of continue and restart, and abstains when evidence is insufficient.
3. Execution layer: accounts for residual budget, recoverable workspace state, and tool state before deciding whether to preserve or discard the current run.

Offline evaluation should pre-register the action set, sampling unit, and interval procedure to prevent metric selection after seeing results. At minimum, it should compare never restart, threshold-based restart, and decision-value restart, then validate the preferred policy on held-out tasks.

Conclusion

The work reframes agent monitoring as a decision problem rather than a classification problem. A monitor should not merely describe what went wrong; it should preserve enough evidence to support an action. More importantly, the study does not turn a small finite-sample gain into a definitive improvement. Its abstain mechanism and zero certified live comparisons expose the current lack of statistical support for restart policies.

---

Source: Hugging Face Daily Papers (2026-10-27)

— END —

Tags: #AI Agents Follow #Restart Monitors Follow #Decision Value Follow #Finite-Sample Auditing Follow #SWE-bench Follow