Evidence-Guided Recovery for AI Agents Facing Ambiguous Tool Outcomes

Summary:This research addresses a difficult failure mode in tool-using agents: after a timeout, the system may not know whether an action failed, committed, or is still in flight. The proposed framework constrains recovery to approved obligations, declared provider capabilities, and states compatible with public evidence. A shared admissibility check handles credential expiry, authorization timing, dependencies, and residual side effects, while terminal certificates distinguish actual goal satisfaction

---

The Core Problem: A Timeout Is Not Equivalent to Failure

When an agent calls an external tool—such as a payment API, ticketing system, or resource-management endpoint—and the request times out, the server may already have committed the operation even though the client received no success response. A blind retry can therefore duplicate charges, create duplicate records, or trigger conflicting state changes. Treating the timeout as a failure is equally unsafe: it may report an already completed task as incomplete.

This research formulates recovery over three explicit sources of constraint: approved obligations, declared provider capabilities, and states compatible with publicly available evidence. It does not assume that an arbitrary third-party tool provides exactly-once execution, nor does it infer success from a timeout alone.

Architecture: Separating Planning from Effect-History Adjudication

In the known-model implementation, two relatively separate components cooperate:

1. An observation-contingent planner proposes actions based on currently visible evidence.
2. An effect-history adjudicator checks whether previous calls produced verifiable side effects and whether the current state permits further execution.

This separation prevents one language model from both proposing an action and serving as the sole judge of its safety. For deployment, the adjudicator can be implemented as a deterministic policy layer around an Agent Runtime, adding structured state checks without requiring changes to the underlying foundation model.

Shared Admissibility Check

Every candidate recovery action passes through a common admissibility check that evaluates:

• whether API credentials have expired or are about to expire;
• whether authorization is active and the action remains inside its permitted time window;
• whether prerequisite dependencies have been satisfied;
• whether retrying could conflict with an already committed side effect;
• whether residual effects remain after the proposed step completes.

At the engineering level, this component is an execution-policy validator. It need not understand the complete business objective, but it must represent credentials, dependency graphs, idempotency keys, authorization scopes, and compensating actions as machine-checkable constraints.

Terminal Certificates

The system does not use “the agent believes the task is complete” as its termination condition. Instead, it introduces terminal certificates that distinguish:

• Actual goal satisfaction: sufficient evidence confirms that the objective was achieved.
• Publicly supported safe completion: the true business state cannot be confirmed, but the declared capabilities and available evidence establish that continuing would not create a worse outcome.

That distinction is essential. The first can support a final business receipt; the second is better used to end a timed-out interaction safely. If a query API explicitly guarantees that an operation remains in flight after a timeout, the agent can wait or hand off to a human rather than risk a duplicate retry.

Experimental Results and Statistical Boundaries

The study contains two experiments.

The first compares the planner with three active controls across 37 task assignments. All four certify 28 completions. The planning method saves four synthetic fee units but incurs additional computation, so it is not a simple win under a fixed-cost constraint.

The second experiment is a preregistered, assignment-held-out comparison between planning advice and evidence prompting. It covers 19 previously developed task families, 58 paired units, and two local model bundles:

• Planning advice certifies 30 completions.
• Evidence prompting certifies 9.
• The difference is 36.2 percentage points.
• An exact family sign-flip test reports p=0.0078.
• The methods produce 272 and 399 proposals, respectively.
• All 116 trajectories pass independent replay.

Using a sign-flip test at the task-family level is more appropriate than treating every trajectory as an independent sample, because it partially controls for correlation within task families. Nevertheless, this remains a limited-scale study. A positive difference across two local model bundles does not establish generalization to every model, toolchain, or production workload.

Why This Is More Than a Retry Policy

A conventional retry policy mainly asks whether a call can be repeated. The proposed framework addresses a broader recovery decision:

• Did the original call actually fail?
• What evidence is sufficient to support that conclusion?
• Which recovery actions remain valid under permission and dependency constraints?
• When should the agent query the provider, hand off to a human, or terminate safely?
• How can compensating actions avoid amplifying the original side effect?

Its guarantees are also deliberately bounded. Conclusions rely on capabilities declared by the provider. If a tool exposes no idempotency semantics, query interface, or state commitment, the system can offer only conditional conclusions—not a guarantee that the business objective was completed.

Engineering Trade-offs

The approach provides a clear safety boundary, but it introduces meaningful costs:

1. Additional inference and adjudication overhead: planning, evidence collection, and state checks increase latency and model usage.
2. Reliability depends on provider contracts: without stable queries, idempotency keys, or meaningful state guarantees, recovery must remain conservative.
3. Certificates can be misinterpreted: publicly supported safe completion is not actual business completion, and the product interface must preserve that distinction.
4. Rule maintenance is ongoing: credentials, authorization, dependencies, and compensation logic must evolve with provider APIs.
5. The evaluation remains bounded: 19 task families and two local model bundles do not yet cover high-risk production scenarios such as payments or irreversible infrastructure changes.

Deployment Recommendations

Teams adopting this mechanism in a production Agent Runtime can proceed in the following order:

1. Define provider capability contracts: specify idempotency, query interfaces, post-timeout state, cancellation, and compensation semantics.
2. Record complete effect histories: retain request IDs, idempotency keys, returned states, side-effect types, and retry history.
3. Implement a shared admissibility layer: validate credentials, authorization, dependencies, and residual effects before every tool execution.
4. Separate planning from adjudication: give the adjudicator structured state rather than relying on free-form model judgment.
5. Expose three result classes: actual completion, publicly supported safe completion, and unknown or human handoff.
6. Evaluate by risk tier: prioritize fault injection and deterministic replay for high-value writes, irreversible operations, and workflows with multi-tool dependencies.

The deployable idea is not to make an agent guess more accurately whether a tool succeeded. It is to turn uncertain execution into an auditable, constrained, and transferable state machine.

---

Source: Hugging Face Daily Papers (2026-10-30)

— END —

Tags: #AI Agents Follow #Tool Use Follow #Task Recovery Follow #Evidence-Based Reasoning Follow #Runtime Safety Follow