Hugging Face Releases JockeyBench: Evaluating LLM Agents on Historical Japanese Horse Racing Data
Summary:Hugging Face introduces JockeyBench, a benchmark and evaluation protocol for repeated event forecasting, which utilizes historical Japanese horse racing data to study the extraction of predictive signals from noisy evidence and their evaluation across successive decisions. The protocol integrates time-gated evidence, priced tools, paired frozen-program controls, and two workflows: executing an LLM-written predictor or acquiring evidence for a fresh per-race LLM forecast. Evaluation metrics inclu
1. Background and Motivation
In the field of artificial intelligence, repeated event forecasting is a complex and challenging problem. Agents need to extract valuable predictive signals from noisy data and evaluate their effectiveness across successive decisions. The introduction of JockeyBench aims to provide a new solution to this problem by analyzing historical Japanese horse racing data.
2. Technical Mechanism Analysis
JockeyBench integrates the following key technologies:
- Time-gated Evidence: Controls the acquisition of evidence through time windows to ensure the timeliness of predictions.
- Priced Tools: Prices different types of evidence to evaluate their contribution to predictions.
- Frozen Program Controls: Uses frozen programs as control groups to ensure the fairness of evaluations.
- Two Workflows: (a) Execute an LLM-written predictor; (b) Acquire evidence for a fresh per-race LLM forecast.
3. Engineering Trade-offs and Actual Performance
Experiments evaluated 28 model-scaffold configurations across 324 horse races. Evaluation metrics included forecast quality, benchmark-specific net utility, and retrospective betting returns and risk.
- Key Findings:
- Under capped-Kelly settlement, all configurations yielded negative mean profits.
- The reusable-program workflow demonstrated higher mean betting profit and lower research fees.
- Mean probability bias reversed on selected bets, and profitable individual runs could depend on concentrated returns.
4. Developer Implementation and Deployment Recommendations
- Predictive Capability Evaluation: It is recommended to consider predictive capability, adaptability, and resource use separately when evaluating agents.
- Resource Management: In resource-constrained environments, the reusable-program workflow can be prioritized to reduce research fees.
- Risk Management: When deploying agents, it is important to consider the risks of prediction bias and concentrated returns and to develop corresponding risk management strategies.
5. Future Outlook
JockeyBench provides a new approach to evaluating agents' performance in repeated event forecasting. Future research can further optimize evaluation metrics and explore datasets from more domains to verify its general applicability.
— END —Source: Hugging Face Daily Papers (2026-10-14)
Tags: #Hugging Face #JockeyBench #LLM Agents #Repeated Event Forecasting #Benchmark
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments