arXiv Research: Limitations of Multi-Agent Code Judges and a Label-Free Evaluation Method
Summary:A new arXiv study examines the limitations of multi-agent code judging systems, highlighting that current methods confidently return verdicts even when lacking evidence, leading to poor accuracy. The researchers found that an unmodified published framework achieved only 4.4% accuracy on code judging benchmarks, compared to 43.7% for a direct model. They propose a label-free evaluation method based on pipeline logs, which identifies cases where the judge cannot make a reliable decision. This appr
Background and Problem
Multi-Agent Systems (MAS) have been increasingly applied in tasks such as code judging. However, current methods often return overconfident verdicts even when lacking evidence, leading to poor accuracy. This issue is particularly pronounced in code judging, as determining code correctness requires solid evidence.
Methodology and Findings
The research team analyzed the performance of the MARCH framework on two code judging benchmarks and found that it declared both solutions equally good in 78% to 95% of comparisons, achieving only 4.4% accuracy. In contrast, a direct model achieved 43.7% accuracy on the same task. The study identifies that the problem lies in the lack of strict consideration for the independence and difference of evidence in the judging process.
To address this, the researchers propose a label-free evaluation method based on pipeline logs. This method analyzes key indicators in the judging process, such as the independence and difference of evidence, to identify cases where the system cannot make a reliable judgment. Experimental results show that this approach improves the system's accuracy from 20.7% to 36.9% while still addressing half of all comparisons.
Technical Highlights
- Label-Free Evaluation Method: Assess the quality of code judging systems without human labels.
- Evidence Independence and Difference Analysis: Improve judging accuracy by analyzing the independence and difference of evidence.
- Log-Based Analysis: Utilize the system's own log data for analysis, avoiding reliance on external data.
Industry Impact and Developer Recommendations
This study provides new insights into the reliability of AI code judging systems and offers a method for assessing judge quality without human labels. For developers, the following recommendations can be considered:
- Introduce Evidence Independence Checking Mechanisms: Incorporate checks for evidence independence and difference into code judging systems to improve accuracy.
- Use Log Data for Self-Assessment: Analyze the system's own log data to identify and address issues in the judging process.
- Explore Advanced Judging Methods: Combine MAS with advanced technologies such as reinforcement learning to develop more reliable code judging systems.
Conclusion
This research reveals the limitations of current multi-agent code judging systems and proposes an effective label-free evaluation method, providing a new direction for the development of AI code judging.
— END —Source: ArXiv AI (cs.AI) (2026-09-28)
Tags: #Multi-Agent Systems #Code Judging #AI Reliability #Label-Free Evaluation #arXiv
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments