Hugging Face Releases SWE-Bench ProMax: A Multilingual Code Refactoring Benchmark
Summary:Hugging Face has released SWE-Bench ProMax, a multilingual code refactoring benchmark designed to address the shortcomings of existing benchmarks, such as flawed test cases and over-reliance on training data. The benchmark includes 170 curated instances across seven programming languages, with an average of 11.4 modified files and 261.6 lines of code per task, significantly raising the bar for task complexity and scale. Experiments show that the best-performing AI model achieves only a 41.2% res
Hugging Face Releases SWE-Bench ProMax: A Multilingual Code Refactoring Benchmark
As AI coding agents advance in handling complex, long-horizon software engineering tasks, existing benchmarks are increasingly revealing limitations in the quality of test cases and the over-reliance of models on training data. A recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed test cases, either being too narrow and rejecting correct solutions or too broad and checking unstated requirements.
To address these challenges, Hugging Face has introduced SWE-Bench ProMax, an expert-curated multilingual code refactoring benchmark comprising 170 instances across seven programming languages: Python, Java, TypeScript, Go, C, C++, and Rust. Each instance undergoes rigorous, multi-stage curation to resolve the quality issues identified in prior benchmarks: problem descriptions are rewritten to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests.
Key Technical Highlights
- Multilingual Support: Covers seven major programming languages, providing a unified benchmark for cross-language code refactoring.
- High-Quality Test Cases: All test cases are reviewed by experts to ensure their accuracy and effectiveness.
- Large-Scale Tasks: Tasks average 11.4 modified files and 261.6 lines of code per instance, significantly raising the bar for task complexity and scale.
- Challenging Task Design: Tasks with insufficient complexity or limited cross-file scope are filtered out, ensuring the benchmark's challenge.
Experimental Results and Industry Impact
Experiments with frontier models under two agent scaffolds show that the best model achieves only a 41.2% resolution rate on SWE-Bench ProMax, indicating that current AI coding agents still face significant challenges in handling complex code refactoring tasks. The release of SWE-Bench ProMax not only provides a more reliable benchmark for evaluating AI coding agents but also pushes the boundaries of AI's application in software engineering.
Recommendations for Developers
- Use SWE-Bench ProMax for Model Evaluation: Developers can use this platform to conduct a more comprehensive evaluation of AI coding agents, uncovering the shortcomings of models in handling complex code refactoring tasks.
- Focus on Multilingual Code Refactoring Technologies: As the importance of multilingual support grows, developers should pay attention to the progress of related technologies and try to apply these technologies in their projects.
Conclusion
The release of SWE-Bench ProMax marks an important milestone in the benchmarking of AI coding agents. It not only addresses many of the issues in existing benchmarks but also provides new challenges and opportunities for the application of AI in software engineering.
Original Source
Hugging Face Releases SWE-Bench ProMax
— END —Tags: #Hugging Face #AI Coding Agents #Code Refactoring #Multilingual Support #Benchmarking
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments