Hugging Face Releases SWE-Duel: A Self-Scaling Arena for Coding Agents

Summary:Hugging Face introduces SWE-Duel, a self-scaling benchmark platform for coding agents. Unlike traditional static coding benchmarks, SWE-Duel dynamically generates tasks and evaluation criteria to prevent task leakage and saturation. The arena employs a Red-Blue turn-based mechanism where agents alternately program and review, simulating real-world collaborative and competitive development environments. Experiments show that SWE-Duel effectively distinguishes agent capabilities and provides a mor

---

1. Background and Motivation

Traditional coding benchmarks are fixed at release, leading to task leakage into training data over time and eventual saturation by stronger agents. To address this, SWE-Duel introduces a self-scaling arena mechanism that dynamically generates tasks and evaluation criteria, preventing task leakage and saturation.

2. SWE-Duel Mechanism

The core mechanisms of SWE-Duel include:

• Red-Blue Turn-Based Programming and Review: In each turn, the Red agent works on a real repository, proposes and implements a new feature with a subtle bug, while the Blue agent reviews the pull request (PR). If the Blue agent's revision passes all test suites, it wins, and the roles are swapped.

• Task Generation and Evaluation: Tasks are generated at evaluation time to ensure no pre-existing content leaks. Task difficulty is determined by the participating agents' capabilities, not fixed at release.

• Unsolvable Task Exclusion: By requiring agents to repair bugs blind, SWE-Duel excludes unsolvable tasks, ensuring fair competition.

3. Experiments and Results

SWE-Duel conducted a tournament with 11 model-driven agents across 15 repositories for 1,500 turns. Key findings include:

• GPT-5.5 with Codex CLI excelled in defect repair, fixing 98.3% of the defects it reviewed.

• Claude Opus 4.8 with mini-swe-agent was the strongest task author, achieving a 27.3% Red win rate.

• Model Version Differences: Newer versions of the same model family outperformed older versions in task authoring (42.9% vs. 4.0%), providing preliminary evidence that task difficulty scales with authoring capability.

4. Technical Trade-offs and Engineering Challenges

Designing SWE-Duel involves several challenges:

• Complexity of Task Generation and Evaluation: Dynamically generating tasks requires complex logic and evaluation criteria, increasing system complexity.

• Fairness of Agent Interaction: Ensuring fairness in the Red-Blue turn-based mechanism is crucial for maintaining competitive integrity.

• Exclusion of Unsolvable Tasks: Precise mechanisms are needed to identify and exclude unsolvable tasks to avoid unreasonable pressure on agents.

5. Developer Adoption and Deployment Recommendations

SWE-Duel offers new avenues for training and evaluating coding agents. Developers can:

• Leverage SWE-Duel for Agent Capability Assessment: Use dynamic tasks and evaluation criteria to more accurately assess coding agent capabilities.

• Build Self-Scaling Coding Benchmarks: Adapt SWE-Duel's mechanisms to create tailored coding benchmark platforms.

• Focus on Agent Interaction Mechanism Design: SWE-Duel's turn-based mechanism provides insights into agent collaboration and competition.

6. Future Outlook

SWE-Duel demonstrates the potential of self-scaling coding benchmarks. Future work could extend it to more programming languages and complex tasks. Additionally, integrating reinforcement learning and multi-agent collaboration could further enhance SWE-Duel as a key tool for training and evaluating coding agents.

---

Source: Hugging Face Daily Papers (2026-10-08)

— END —

Tags: #Hugging Face Follow #SWE-Duel Follow #Coding Agents Follow #Self-Scaling Benchmark Follow #Multi-Agent Arena Follow