Benchmark Contamination Unsolvable via Decontamination Reports: Evaluator-Controlled Testing as a New Paradigm
By Mr.Xu Community Post
Published: · 12 views
Summary:In February, OpenAI halted reporting on SWE-bench Verified, recommending other labs do the same due to the ease with which frontier models could reproduce human-written reference fixes or verbatim problem statements. This stagnation in progress, coupled with the ineffectiveness of decontamination reports—due to self-checking by labs, copyright issues, and the prevalence of paraphrased or synthetic data—has led to a proposal for evaluator-controlled testing. This new paradigm involves running eva
The Dilemma of AI Benchmarking
Benchmarking is a cornerstone of AI model evaluation. However, in February, OpenAI halted reporting on SWE-bench Verified and recommended other labs do the same. The reason? Frontier models could easily reproduce human-written reference fixes or verbatim problem statements, leading to stalled progress.
The Limitations of Decontamination Reports
Decontamination reports, which attempt to identify and remove contaminated data by searching training datasets, face several challenges:
- Self-checking by labs: Only the lab has access to the data, so no external party can verify the search.
- Copyright issues: Disclosing the list of copyrighted works poses legal risks.
- Matching failures: Models can learn answers from paraphrased text, forum discussions, GitHub solutions, or synthetic data generated from the benchmark, without sharing n-grams.
Evaluator-Controlled Testing: A New Paradigm
The proposed solution is evaluator-controlled testing, which includes:
- Offline execution: Submissions run without network access to prevent external data interference.
- Code built from commits: Evaluators build the code from a specific commit to ensure reproducibility.
- Test data generated after submissions freeze: Generating test data after submissions are frozen further reduces the risk of contamination.
This approach aims to ensure the reproducibility and reliability of results by placing control in the hands of evaluators.
Practice and Challenges
The author has built a small-scale version of this approach (for tabular models and private test sets) and acknowledges several unresolved issues, such as the validity of the benchmark itself, the risk of hidden test sets being cracked through repeated submissions, the possibility of label leakage by funders, and the difficulty of third-party reproduction. The author prioritizes addressing the risk of hidden test sets being cracked.
Conclusion
The evolution of AI evaluation methods is imperative. Evaluator-controlled testing offers a new direction, but further research and practice are needed to validate its effectiveness and feasibility.
— END —Source: Reddit r/MachineLearning (2026-09-20)
Tags: #AI Evaluation #Benchmarking #Data Contamination #Model Evaluation #AI Safety
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments