arXiv Research Reveals Luce's Choice Axiom Failure in Chatbot Arena
Summary:A new arXiv study examines the limitations of the Bradley-Terry model in Chatbot Arena, which assumes Luce's choice axiom to rank language models based on a single score. The research reveals that when models are similar in version, size, or originating lab, user votes are more decisive than the model scores imply, indicating a failure of Luce's axiom. By analyzing 1.8 million votes, the study proposes a correction model with four to six parameters that better predicts ties but does not signific
Background and Problem
The Bradley-Terry model is the primary method used in Chatbot Arena to rank language models, based on Luce's choice axiom, which assumes that user choices can be explained by a unified score. However, this approach overlooks the impact of model similarities on user decisions.
Methodology and Findings
The research team analyzed 1.8 million user votes and found that when models are similar in version, size, or originating lab, user votes are more decisive than the model scores imply. Specifically:
- Between model versions: User votes are 1.37 times more decisive than Bradley-Terry predicts.
- Between model sizes: User votes are 1.26 times more decisive than Bradley-Terry predicts.
- Between models of the same lab: User votes are 1.12 times more decisive than Bradley-Terry predicts.
Additionally, the similarity in wording and formatting of model answers was found to correlate with this effect.
Correction Model and Results
The study proposes a correction model with four to six parameters that better predicts ties but does not significantly alter the leaderboard order. While the model performs better in handling ties, it does not establish better head-to-head odds.
Engineering Trade-offs and Practical Applications
- Strengths: The correction model provides more accurate predictions when dealing with similar models, especially in tie situations.
- Weaknesses: The model does not significantly change the overall leaderboard order and has higher computational complexity, which may increase evaluation costs.
Developer Recommendations
For developers, when using the Bradley-Terry model for model evaluation, it is important to consider the impact of model similarities on user decisions. If models are highly similar, it is recommended to use the correction model for more accurate evaluations. Additionally, developers should pay attention to the wording and formatting of model answers to reduce unnecessary biases.
Conclusion
This research reveals the limitations of the Bradley-Terry model in handling similar models and proposes a correction method. Although the correction model performs better in certain aspects, its overall impact is limited. Future research can further explore more complex model evaluation methods to better reflect user preferences.
— END —Source: ArXiv NLP/LLM (cs.CL) (2026-10-15)
Tags: #Bradley-Terry model #Chatbot Arena #Luce's choice axiom #model evaluation #user votes
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments