Agent Harness Awareness Gives Tool-Using Agents Verified Runtime State Before Dispatch
Summary:The paper introduces Agent Harness Awareness, a trust-separated architecture in which the execution harness gives an agent a small, fresh, verified runtime-state snapshot before consequential tool calls while retaining monitoring, enforcement, evaluation, and dispatch authority. Using the ToolRoute benchmark, the authors report a 96.9% reduction in mean observable policy regret across 216 decisions and an increase in single-dispatch completion from 69.4% to 95.8%. A separate live TCP/HTTP replic
Core Idea: Turn Runtime State into a Decision Signal
A tool description usually explains what an action can do, but not whether the action is currently healthy or appropriate. The execution harness may know this through monitors, traces, and recent failures, while the model must select from static descriptions and stale conversational context. The paper calls this gap operational blindness.
Agent Harness Awareness introduces a trust-separated design. Before a consequential action, the host supplies the agent with a small, fresh, verified state snapshot. The model can use that information to choose a candidate tool, but measurement, enforcement, evaluation, and final dispatch authority remain with the host.
The architecture is not merely another prompt injection mechanism. It establishes a constrained state-delivery path:
- The host collects tool health, recent failures, and network state.
- It validates and compresses those signals into a bounded view.
- It injects the view before the call.
- The model selects a candidate action.
- The host enforces policy and dispatches the final request.
Benchmark Design and Ablation Logic
The authors introduce ToolRoute, an isolated benchmark that holds the task and candidate tool semantics constant while varying the operational information visible to the model. This design attempts to attribute performance differences to state awareness rather than tool descriptions or search-space changes.
The study covers 216 decisions across GPT-5.6 Sol, Claude Sonnet 5, and Gemini 3.7 Flash. With verified tool-health state:
- Mean observable policy regret fell from 4,252 ms to 131 ms, a 96.9% reduction.
- Single-dispatch completion increased from 69.4% to 95.8%.
- Models stopped relying only on tool names, descriptions, or static suitability.
Policy regret is expressed in milliseconds and represents the observable delay or loss of the selected route relative to a better observable alternative. It is useful for evaluating runtime routing, but it is not equivalent to task correctness, financial cost, or user value.
Live Network-Fault Replication
A second experiment covers 27 end-to-end decisions using live local TCP/HTTP execution, OpenTelemetry client spans, and Toxiproxy-injected faults. The verified-state condition reduced mean regret from 8,956 ms to 0 ms and completed all nine model-by-fault combinations, compared with five out of nine for the neutral control.
This test is more engineering-relevant than a purely simulated router because it exercises the connection between state snapshots and an actual execution path. Nevertheless, the live replication is small, and the zero mean regret may depend on the selected faults and test distribution. It should not be extrapolated directly to production networks with long-tail failures, concurrency jitter, or partial degradation.
Engineering Trade-offs
Runtime awareness is not free. The host must continuously collect telemetry, verify snapshot freshness, and constrain its size. If state collection blocks the critical path, end-to-end latency can increase rather than decrease. An oversized snapshot can also consume context budget and make the model more likely to misinterpret irrelevant fields.
A practical deployment can use a tiered policy:
- Keep a fast path for low-cost and reversible calls.
- Require a preflight health check for expensive, irreversible, or dependency-critical calls.
- Protect snapshots with TTLs, sequence numbers, and source authentication.
- Retain full traces on the host and expose only the minimum task-relevant view.
- Define explicit fallback behavior for missing, stale, or conflicting state.
Trust separation is crucial here. Runtime state may guide a recommendation, but it should never bypass host-side authorization, rate limits, or execution policy. Otherwise, a noisy or manipulated status feed could be misread by the model as permission to act.
Evaluation Limitations
The results support the narrower claim that selected operational state can improve tool-and-network routing. They do not yet show that the mechanism benefits every agentic workload. The paper leaves runtime compute pressure, economics, retry-aware recovery, and heterogeneous tool selection as open extensions.
The benchmark also needs a production-oriented follow-up: Which fields are genuinely useful? Can stale but richer state be misleading? How should conflicting telemetry sources be resolved? How much additional token and provider cost does state injection create? These questions determine whether the architecture scales beyond controlled routing tasks.
Deployment Guidance
When integrating the approach into an existing agent harness, start with pre-dispatch state gating rather than rebuilding the entire framework. Maintain a signed health snapshot for each tool, route high-risk calls through preflight checks, and log whether state information changed the selected action. Replay production traffic to compare single-dispatch completion, P95 latency, telemetry overhead, and misrouting rates.
Before deployment, validate freshness thresholds, the marginal value of each field, fallback behavior under untrusted telemetry, and selection consistency across multiple candidate tools. The paper's central engineering lesson is straightforward: model parameters determine whether an agent can choose; trustworthy runtime state determines whether it chooses correctly.
— END —Source: ArXiv AI (cs.AI) (2026-11-05)
Tags: #Agent Harness #Runtime Awareness #Tool Use #Fault Injection #OpenTelemetry
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments