BoRa_EM: A New Algorithm for Jointly Learning Rewards and Worker Reliability Based on Boltzmann-Rational Model

Summary:This research introduces BoRa_EM, a novel algorithm designed to address the challenge of learning item rewards from pairwise comparisons while accounting for worker unreliability. Based on the Boltzmann-rational model, BoRa_EM incorporates worker competencies and employs Polya-Gamma latent variables to transform the logistic likelihood into a conditionally Gaussian form, enabling tractable optimization and theoretical convergence guarantees. Experiments demonstrate that BoRa_EM outperforms sever

---

Background and Challenges

Learning rewards from pairwise comparisons is a well-studied problem in domains such as recommendation systems, social choice, and fine-tuning large language models. However, in practical scenarios, these comparisons are often elicited from crowdworkers on platforms like Amazon Mechanical Turk and Scale AI, who may be unreliable due to limited domain knowledge or spamming behavior aimed at maximizing revenue. Therefore, the challenge of assessing worker reliability while learning rewards becomes crucial.

Key Contributions

1. BoRa_EM Algorithm: The study introduces BoRa_EM, a new algorithm based on the Boltzmann-rational model, which jointly learns item rewards and worker reliability.
2. Polya-Gamma Latent Variables: By introducing Polya-Gamma latent variables, the algorithm transforms the logistic likelihood into a conditionally Gaussian form, simplifying the optimization process.
3. Theoretical Guarantees: The study provides theoretical convergence guarantees for the algorithm, ensuring its effectiveness in practical applications.
4. Experimental Validation: Extensive experiments on real-world and synthetic datasets demonstrate that BoRa_EM outperforms several baselines and exhibits strong robustness against unreliable workers and adversarial behavior.

Technical Highlights

• Boltzmann-Rational Model: Extends the Bradley-Terry-Luce model by incorporating worker competencies.
• EM Algorithm Optimization: Utilizes the EM algorithm for efficient optimization, with the Q-function in the E-step simplified to a matrix sensing problem.
• Polya-Gamma Trick: The core innovation of the algorithm, enabling the transformation of the logistic likelihood.

Industry Impact and Recommendations for Developers

BoRa_EM offers a more reliable solution for crowdsourcing platforms and reward learning systems, particularly in handling unreliable data sources. Developers can consider the following recommendations:

• Expand Application Scenarios: Apply BoRa_EM to broader domains such as online education, user feedback analysis, etc.
• Algorithm Optimization: Further optimize the algorithm to handle larger datasets and more complex tasks.
• Leverage Open-Source Tools: Utilize the publicly available code and data to quickly integrate BoRa_EM into existing systems.

Conclusion

BoRa_EM significantly enhances the robustness and accuracy of reward learning systems by jointly learning rewards and worker reliability, providing strong support for real-world applications.

References

• Code and data available at: https://github.com/KaustubhShejole/BoRa_EM

---

Source: ArXiv Machine Learning (cs.LG) (2026-08-12)

— END —

Tags: #Crowdsourcing #Reward Learning #Boltzmann Model #EM Algorithm #Polya-Gamma