ArXiv Introduces Proxy Confidence: A Novel Auditing Method for Black-Box LLM Agents
Summary:ArXiv has released a new research paper introducing Proxy Confidence, a novel method for auditing black-box Large Language Model (LLM) agents. This approach employs a low-cost surrogate model running in parallel to score the agent's outputs based on log-probabilities through complementary readouts, such as teacher forcing, request-PMI discriminative verdicts, and tool-choice competition. The method requires no access to the agent's internals and can effectively identify errors, providing real-ti
Background and Motivation
When deploying Large Language Model (LLM) agents, these agents may emit erroneous tool calls, queries, or code, with such issues often surfacing only after execution. Existing frontier chat APIs hide the model's token probabilities, and the agent's stated confidence barely outperforms random guessing on critical errors. Additionally, resampling does not help because frontier models are highly repetitive, reproducing the same call across samples.
Methodology and Innovation
Proxy Confidence addresses these challenges by running a low-cost open-weight surrogate model in parallel. This surrogate reads the same context, schema, and proposed action as the target agent and scores the agent's outputs through complementary readout mechanisms, including:
- Teacher Forcing: Evaluating the likelihood of each argument value.
- Request-PMI: Measuring the association between the request and the argument values.
- Discriminative Verdict: Making a holistic judgment on the call.
- Tool-Choice Competition: Comparing the function with its sibling functions.
These mechanisms effectively identify erroneous argument values and holistically wrong calls. When the error type is unknown, an ensemble method serves as the low-regret default.
Experimental Results
On difficult coding tasks, Proxy Confidence achieves an AUROC of 0.825, while the actor's stated confidence is near chance (0.598). The generative readouts outperform the actor's confidence by +0.07 to +0.28 across three different actors. On near-deterministic actors, Proxy Confidence gains +0.14 to +0.19 in self-consistency tests, at 1/K the cost.
Deployment Modes
Proxy Confidence supports two deployment modes:
- Real-time Gate: Escalating the least-trustworthy calls for review, with accepted-action accuracy increased by +0.05 to +0.30 at 50% coverage.
- Confidence Feedback: Returning the tool result with the score so the agent can adapt its next step, thereby lifting task success on live-execution benchmarks (+0.119 and +0.137, p <= 1e-4) and outperforming a random-value control where step errors are silent (+0.078, p = 0.003).
Industry Impact and Future Outlook
Proxy Confidence offers an efficient and practical solution for auditing and error detection in LLM agents. Its ability to function without accessing the agent's internals makes it easy to integrate into existing AI systems. This method not only enhances the reliability and safety of agents but also opens new possibilities for AI applications in complex tasks.
— END —Source: ArXiv AI (cs.AI) (2026-10-06)
Tags: #LLMs & Foundation Models #AI Auditing #Agentic #Proxy Confidence #ArXiv
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments