SilentGuard Enforces API Invariants Inside the Agent Harness, Cutting Targeted Error Rates to 1.7%
Summary:SilentGuard moves semantic validation of tool results out of the LLM and into the harness that executes API calls. The system evaluates each response against API-specific invariants and withholds any result that violates them, preventing silent failures that return HTTP 200 with plausible but incorrect data. Across 15 public research APIs, the study found silent failures in 11, with some counts inflated by as much as 691 times. On targeted faults, wrong-answer rates fell from 74.0% to 1.7% acros
Core finding
SilentGuard targets one of the most difficult failure modes in agent toolchains: an API returns HTTP 200 but does not fully execute the request. A misspelled or unsupported filter may be ignored, after which the service returns a structurally valid and plausible count that is nevertheless wrong. Rather than adding another reviewer model, the paper moves semantic validation into the harness that executes tool calls and prevents noncompliant results from entering the answer-generation context.
This changes the system-level question from whether an LLM will heed a warning to whether the runtime permits bad data to pass. On targeted faults, wrong-answer rates fell from 74.0% to 1.7% across seven open models and from 70.2% to 0.5% across three frontier models, with no observed degradation on fault-free tasks. The trade-off is equally important: SilentGuard avoids errors by abstaining, not by repairing the failing API call.
Mechanism: executable invariants at the harness boundary
The core mechanism confirmed by the abstract is to define invariants that every correct response from a given API must satisfy. Let q denote a request, r its response, and I_j(q, r) the result of invariant j. The release condition can be modeled as:
allow(q, r) = all(I_j(q, r) for j in mandatory_invariants)
If any mandatory invariant fails, the harness withholds the result, records the violation, and aborts the answer path. Architecturally, this consists of four layers:
- Request normalization: preserve the actual parameters, filters, authorization scope, and API version used for the call.
- Invariant registry: maintain machine-checkable constraints per API version instead of relying on natural-language reminders injected into the model context.
- Response validation: verify schema, types, ranges, aggregate relationships, and request semantics before releasing the payload.
- Enforcement gate: block the result on failure and emit an abstention or error state rather than attaching a warning and continuing.
Potential invariants include schema conformance, consistency between counts and paginated rows, evidence that a requested filter affected the result, and monotonic relationships across fields or calls. These are deployment patterns rather than a claim about the paper's undisclosed invariant taxonomy.
A crucial asymmetry is that violating an invariant can be strong evidence that a result is wrong, while passing all invariants does not prove that it is correct. The gap between a 1.7% error rate on targeted faults and a 24.5% rate on unseen faults is consistent with this limitation.
Why enforcement beats another warning
The abstract reports that checklists, reviewer models, and LLM judges still left more than 70% of answers wrong in the tested conditions. For counts flagged by a judge, agents reported only 77.4%. When the same diagnostic was delivered merely as a warning, 70.1% of answers remained wrong.
An LLM is optimizing multiple objectives at once: completing the user request, making a tool call, and interpreting an anomalous response. It may treat a warning as a low-confidence suggestion or lose track of it during generation. A harness gate removes that discretion. The invalid value never reaches the context, so it cannot be quoted, summarized, or rationalized in the final answer.
Reported results
| Evaluation setting | Error rate without protection | Error rate with SilentGuard | Interpretation | |---|---:|---:|---| | Seven open models, targeted faults | 74.0% | 1.7% | Large reduction with no observed loss on fault-free tasks | | Three frontier models, targeted faults | 70.2% | 0.5% | Enforcement also works with closed frontier models | | Same check delivered only as a warning | 70.1% remained wrong | — | The decisive factor is enforcement rather than another prompt | | Unseen fault types | 61.1% | 24.5% | Generalization helps, but substantial residual risk remains |
The API audit found silent failures in 11 of 15 public research APIs, with some counts inflated by as much as 691 times. This demonstrates that successful transport and correct execution are separate properties; HTTP status and plausible response structure are insufficient evidence for an agent.
The reported 1.7% and 0.5% are wrong-answer rates, not end-to-end task-success rates. Converting invalid results into abstentions prevents error propagation but reduces answer availability. Production evaluation must therefore report error rate, abstention rate, answer coverage, and a risk-coverage curve together.
Engineering trade-offs and limitations
- False-rejection cost: A strict invariant can block a valid result because of an overbroad rule, schema drift, or a validator defect. Abstention is safer than an incorrect answer but can damage completion rates.
- Coverage cost: Manually enumerated invariants cannot capture every semantic failure. Near-zero errors on known faults do not generalize automatically; the 24.5% residual rate on new faults makes this explicit.
- Maintenance burden: Invariants must be versioned with API behavior, schemas, and permission semantics. Field aliases, default changes, and pagination updates can silently invalidate a previously correct check.
- Composition risk: Passing local checks does not guarantee consistency across multiple tool calls. Production systems need cross-call checks for referential integrity, count conservation, ordering, and state transitions.
- Latency and cost: Deterministic code checks usually add little CPU overhead. Semantic verifiers, additional model calls, or supplementary queries can materially increase cost and tail latency. Teams should measure p50 and p99 latency and the total tool cost per successful answer.
- Security boundary: A faulty or adversarial API may produce a result that satisfies superficial rules while remaining semantically wrong. If the checker itself uses a model, prompt injection and verifier manipulation remain possible. Deterministic enforcement improves control but does not replace authorization, provenance, and audit controls.
- No recovery mechanism: The paper explicitly uses abstention rather than repair. High-value workflows may also need quarantine, safe retry, or fallback data sources, but a result already known to violate an invariant must never be re-released through another path.
How to interpret the evaluation
Fifteen public research APIs are a limited convenience sample and do not necessarily represent production failures in enterprise databases, search services, or payment systems. The available material does not disclose the fault-injection protocol, reference truth, task counts, model identities, random seeds, or confidence intervals. The 691-fold extreme should therefore be treated as evidence of potentially severe system risk rather than a universally expected frequency.
The targeted-versus-unseen-fault gap is especially important. The reduction from 74.0% to 1.7% demonstrates that known invariants can work, while the reduction from 61.1% to 24.5% shows that SilentGuard is not yet a universal semantic validator for arbitrary APIs. The matched warning comparison is one of the paper's strongest pieces of evidence because it isolates enforcement from the act of showing the model another diagnostic.
Finally, no degradation on fault-free tasks does not by itself prove a low false-rejection rate. Deployment decisions also require abstention rates, confidence intervals, and breakdowns by API and invariant.
Deployment blueprint
- Build an API failure inventory first: Capture real incidents involving ignored filters, inconsistent pagination, missing fields, incorrect time windows, and similar behavior instead of beginning with generic schemas.
- Separate hard and soft invariants: High-risk workflows should only release results that pass hard invariants. Run soft checks in shadow mode initially. Not every diagnostic deserves to become a blocking gate.
- Start in shadow mode: Log valid results that would have been blocked and invalid results that would have passed. Estimate invariant coverage, false-rejection rate, and false-pass rate before enforcing production decisions.
- Bind checks to requests and versions: Store a normalized request hash, API version, authorization scope, invariant version, and violation ID. This reduces errors caused by stale caches or version drift.
- Use tiered failure policies: Choose among fail-closed behavior, quarantine and retry, fallback tools, or explicit abstention according to risk. Every path should preserve auditable evidence.
- Evaluate selective risk, not one accuracy number: Track semantic error rate, dangerous-result pass rate, abstention precision, answer coverage, p99 latency, and cost per successful answer.
- Continue fault injection: Extend beyond the paper's target faults to misspelled parameters, permission filters, pagination drift, field aliases, concurrent updates, and cross-API inconsistencies.
SilentGuard's most important contribution is not another detector but the location of enforcement: it turns advice that an LLM may ignore into a runtime constraint. The approach sharply blocks known API silent failures from contaminating agent answers, but its coverage of unknown faults remains limited. It should be deployed as an observable, versionable API-contract layer rather than marketed as a universal firewall for tool results.
— END —Source: Hugging Face Daily Papers (2026-11-16)
Tags: #LLM Agents #Tool Use #API Reliability #Invariant Checking #Runtime Guardrails
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments