Open-source AI Leaderboard whatmodel.sh Reveals Labs' Benchmark Scores Are 6.4% Higher on Average

Summary:Reddit user Jake Manger has released an open-source AI leaderboard, whatmodel.sh, which analyzed 1,629 benchmark results and found that AI labs report scores an average of 6.4% higher than independent testers for the same models on the same tests. The study is based on 611 comparisons of 180 models across 103 benchmarks, with all datasets and code made publicly available, offering valuable insights into AI model evaluation transparency.

---

Key Findings

Reddit user Jake Manger has developed an open-source AI leaderboard called whatmodel.sh, aimed at systematically analyzing discrepancies between AI labs and independent testers in benchmark scores. The main findings of the study are as follows:

1. Data Scale and Coverage: The research analyzed 1,629 benchmark results, comparing 180 models across 103 different benchmarks.
2. Significant Bias: AI labs reported scores an average of 6.4% higher than independent testers. This bias suggests a potential systemic optimism in labs' self-assessment of their models.
3. Data and Code Open-Sourcing: All relevant datasets are released under the CC BY-SA 4.0 license, and the code under the MIT license, ensuring reproducibility and transparency of the study.

Technical Mechanism Analysis

whatmodel.sh achieves its in-depth analysis of AI model evaluation through the following methods:

• Data Collection and Matching: Gathering benchmark results from public sources and using automated tools to match models and tests, ensuring the accuracy of comparisons.
• Statistical Analysis: Calculating the score differences between labs and independent testers using statistical methods and conducting significance tests to rule out random errors.
• Visualization Tool: Providing an interactive chart (https://whatmodel.sh/claims) where users can intuitively view the performance differences of different models across various benchmarks.

Engineering Trade-offs and Performance

• Transparency Improvement: The study reveals potential biases in AI model evaluation, offering the AI community a more reliable evaluation benchmark.
• Resource Requirements: Although data collection and analysis require substantial computational resources, the open-sourced code and datasets enable other researchers to easily replicate and extend the study.
• Limitations: The research relies primarily on publicly available data, which may introduce biases or omissions. Additionally, differences in testing environments and methodologies between labs and independent testers can affect the interpretation of results.

Developer Implementation and Deployment Recommendations

• Model Evaluation Improvement: Developers should refer to independent testers' results and combine them with lab reports for a more accurate understanding of model performance.
• Data Sharing and Collaboration: The AI community should strengthen data sharing and collaboration, establishing a more comprehensive benchmark testing system to improve evaluation transparency and reliability.
• Continuous Monitoring and Updates: As AI technology evolves, benchmark testing methods and datasets also need to be continuously updated to adapt to new models and tasks.

Conclusion

The findings of whatmodel.sh indicate a systemic optimism in AI labs' benchmark testing. This discovery serves as a reminder for us to be more cautious in evaluating AI models and calls on the AI community to enhance transparency and data sharing to promote the healthy development of AI technology.

---

Source: Reddit r/LocalLLaMA (2026-10-19)

— END —

Tags: #AI Evaluation Follow #Benchmarking Follow #Open-Source AI Follow #Data Transparency Follow #Model Bias Follow