ZICQ
中 Log in / Sign up
ZICQ Info Research & Papers #AI Evaluation #Benchmarking #Data Contamination #Model Evaluation #AI Safety

Benchmark Contamination Unsolvable via Decontamination Reports: Evaluator-Controlled Testing as a New Paradigm

Avatar of Mr.Xu

By Mr.Xu Community Post

Published: · 12 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:In February, OpenAI halted reporting on SWE-bench Verified, recommending other labs do the same due to the ease with which frontier models could reproduce human-written reference fixes or verbatim problem statements. This stagnation in progress, coupled with the ineffectiveness of decontamination reports—due to self-checking by labs, copyright issues, and the prevalence of paraphrased or synthetic data—has led to a proposal for evaluator-controlled testing. This new paradigm involves running eva


The Dilemma of AI Benchmarking

Benchmarking is a cornerstone of AI model evaluation. However, in February, OpenAI halted reporting on SWE-bench Verified and recommended other labs do the same. The reason? Frontier models could easily reproduce human-written reference fixes or verbatim problem statements, leading to stalled progress.

The Limitations of Decontamination Reports

Decontamination reports, which attempt to identify and remove contaminated data by searching training datasets, face several challenges:

  1. Self-checking by labs: Only the lab has access to the data, so no external party can verify the search.
  2. Copyright issues: Disclosing the list of copyrighted works poses legal risks.
  3. Matching failures: Models can learn answers from paraphrased text, forum discussions, GitHub solutions, or synthetic data generated from the benchmark, without sharing n-grams.

Evaluator-Controlled Testing: A New Paradigm

The proposed solution is evaluator-controlled testing, which includes:

  • Offline execution: Submissions run without network access to prevent external data interference.
  • Code built from commits: Evaluators build the code from a specific commit to ensure reproducibility.
  • Test data generated after submissions freeze: Generating test data after submissions are frozen further reduces the risk of contamination.

This approach aims to ensure the reproducibility and reliability of results by placing control in the hands of evaluators.

Practice and Challenges

The author has built a small-scale version of this approach (for tabular models and private test sets) and acknowledges several unresolved issues, such as the validity of the benchmark itself, the risk of hidden test sets being cracked through repeated submissions, the possibility of label leakage by funders, and the difficulty of third-party reproduction. The author prioritizes addressing the risk of hidden test sets being cracked.

Conclusion

The evolution of AI evaluation methods is imperative. Evaluator-controlled testing offers a new direction, but further research and practice are needed to validate its effectiveness and feasibility.


Source: Reddit r/MachineLearning (2026-09-20)

— END —

Tags: #AI Evaluation #Benchmarking #Data Contamination #Model Evaluation #AI Safety

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…