ZICQ
中 Log in / Sign up
Newsroom Research & Papers #SearchAuditor #Smart audit #Long-range Search #Disaster recovery #Benchmark testing

SearchAuditor: Multi-Perspective Auditing Framework Localizes and Repairs Failures in Long-Horizon Search Agents, Boosti

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 7 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:This arXiv paper introduces SearchAuditBench, a benchmark for evaluating LLM auditors' ability to localize, attribute, and repair failures in long-horizon search agents, and SearchAuditor, a multi-perspective auditing framework. SearchAuditBench comprises 1,243 failed trajectories (avg 73.1 messages, 65.1K tokens) from eight open-weight models on five deep-search benchmarks, with expert annotations. SearchAuditor outperforms strong baselines, achieving a 32.3% end-to-end pass rate vs. 26.6% for


Overview

Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers. Diagnosing such failures is difficult, requiring manual inspection of extremely long execution traces, which could be beyond human capacity. This arXiv paper introduces SearchAuditBench, a benchmark for evaluating LLM auditors' ability to localize, attribute, and repair these failures, and SearchAuditor, a multi-perspective auditing framework.

SearchAuditBench: A Benchmark for Auditing Search-Agent Failures

SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks. Each trajectory is expert-annotated with the critical error step, a search-specific root cause, and a reference repair with grading rubrics. This benchmark provides a standardized testbed for assessing LLM auditors' capabilities in failure localization, attribution, and repair.

SearchAuditor: Multi-Perspective Auditing Framework

SearchAuditor is a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures through evidence-grounded adjudication. It examines failed trajectories from multiple angles, cross-validates evidence, and generates reliable repair suggestions.

Experimental Results: Significant Improvement in Auditing and Repair

Experimental results show that even the strongest baseline (e.g., GPT-5.5) attains only a 26.6% end-to-end pass rate. In contrast, SearchAuditor consistently outperforms all baselines across different frontier models, achieving an end-to-end pass rate of 32.3%. Furthermore, resuming failed runs with its repairs enables agents to better recover from errors, further improving overall performance.

Industry Impact and Developer Recommendations

This research provides new tools for improving the reliability of long-horizon search agents. Developers can use SearchAuditBench as a standard test set to evaluate their agents' failure diagnosis capabilities, and adopt SearchAuditor as a practical auditing solution to reduce manual inspection costs and enhance self-healing capabilities. Future work may integrate this approach with observability tools and automated repair pipelines, advancing the maturity of agent engineering.

Source

This article is based on the arXiv paper "SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents" (arXiv:2608.05212), available at https://arxiv.org/abs/2608.05212.

— END —

Tags: #SearchAuditor #Smart audit #Long-range Search #Disaster recovery #Benchmark testing

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…