Hugging Face Releases SWEeper-Bench: Revolutionizing Interactive Software Bug Detection for Agents

Summary:Hugging Face introduces SWEeper-Bench, a benchmark suite comprising 200 tasks designed to evaluate agents' ability to detect and fix bugs in interactive software. Built on real bug fixes from 200 widely-used open-source web applications, the benchmark challenges agents to identify and repair issues through interactive testing based on open-ended requests. The results show that the best-performing agent passes only 59.0% of the tasks, highlighting the current limitations in agents' ability to han

---

Background and Challenge

As the mode of interaction between agents and users evolves, the generation of interactive software is increasingly replacing traditional text-based responses. However, many software bugs manifest only after multi-step interactions, requiring users to actively identify and wait for fixes, making the development cycle slow and costly. Existing benchmarks typically assume that bugs have already been identified, leaving the assessment of agents' ability to detect interaction-dependent failures largely unexplored.

Innovation of SWEeper-Bench

Hugging Face's SWEeper-Bench comprises 200 tasks based on real bug fixes from 200 widely-used open-source web applications. Each task provides only the codebase and an open-ended request, such as 'Find and fix issues in Focalboard's board sharing and membership.' Due to the difficulty of detecting these bugs from code alone, agents must design interactive tests to uncover them.

Technical Mechanism and Evaluation

SWEeper-Bench employs an 'agentic verifier' mechanism that evaluates each repair by simulating the user's observation in the browser. In tests involving 15 frontier agents, the best-performing agent passed only 59.0% of the tasks.

Through stage-by-stage analysis of agent trajectories, the study identifies the biggest bottleneck as the identification of buggy behavior: agents often reach the relevant feature but fail to trigger the failure or recognize it.

Engineering Trade-offs and Practical Applications

The design of SWEeper-Bench emphasizes the importance of interactive testing, which poses higher demands on the adaptability of agents in dynamic environments. The main challenges include:

1. Interaction Complexity: Agents need to handle multi-step interaction logic, which challenges their planning and reasoning capabilities.
2. Diversity of Faults: Different types of software bugs require different detection strategies, testing the generalization capabilities of agents.
3. Efficiency and Accuracy Trade-off: While ensuring detection accuracy, agents must also complete tasks within a reasonable time frame, which requires resource allocation and optimization.

Developer Recommendations

For developers, SWEeper-Bench provides a platform to evaluate and improve agents' capabilities in interactive software bug detection. It is recommended that developers:

1. Focus on Interactive Test Design: Design more effective test cases to cover more potential defect scenarios.
2. Strengthen Fault Identification: Use reinforcement learning and data augmentation techniques to enhance agents' ability to identify faults in complex interaction scenarios.
3. Optimize Resource Allocation: Optimize the resource usage efficiency of agents to achieve more efficient defect detection while ensuring detection accuracy.

Conclusion

SWEeper-Bench provides a critical benchmark for advancing agents' capabilities in dynamic software development and interactive testing, driving the application and development of agents in the field of software engineering.

---

Source: Hugging Face Daily Papers (2026-10-08)

— END —

Tags: #Hugging Face Follow #Intelligent Agents Follow #Software Bug Detection Follow #Interactive Testing Follow #AI Benchmarking Follow