Hugging Face Research: Tracing the Origins of 'Wait' in Reasoning Models and Its Relationship to RL Training

Summary:Hugging Face's research team analyzed the provenance of the 'wait' marker in reasoning models, revealing the influence of reinforcement learning (RL) and training data on model behavior. The study found that RL does not directly coin the 'wait' marker but inherits it from training data, with varying behavioral patterns across different models. Additionally, the research highlights the limitations of using marker frequency as the sole indicator of RL effectiveness and advises against using LLMs a

---

Background and Motivation

Reinforcement Learning (RL) is often used to teach reasoning models to write the 'wait' marker before reconsidering a step. However, do the accuracy gains from RL truly depend on this marker? This study aims to explore the origins of the 'wait' marker and its relationship with RL training, uncovering the true drivers behind model behavior.

Methodology

The research team used the following methods to trace the 'wait' marker:

1. Frequency Analysis: Counting the occurrences of the 'wait' marker in different models.
2. Position and Form Analysis: Analyzing the position and form changes of the marker in the text.
3. Injected-Error Test: Testing the model's dependence on the 'wait' marker by injecting errors.
4. No-RL Baseline Comparison: Comparing models with and without RL training in terms of 'wait' marker usage.

Key Findings

1. RL is Not the Creator of the 'wait' Marker: On baselines lacking the 'wait' marker, RL training significantly improves MATH-500 task accuracy, but the 'wait' marker remains near absent.
2. Impact of Training Data: In the OLMo-3 model, the 'wait' marker appears mid-training, and fine-tuning on reasoning traces introduces it to models whose base lacks it.
3. Influence of RL-Zero Recipes: The RL-Zero recipes of OLMo-3 and MiMo amplify the use of the 'wait' marker, while KL-free GRPO from the same base does not.
4. Limitations of Marker Frequency: Relying solely on marker frequency does not accurately reflect RL's impact on reasoning ability, and automated judges may contradict human annotations.

Engineering Trade-offs and Practical Applications

1. Training Cost and Efficiency: While RL training can lead to accuracy improvements, it requires substantial computational resources. The study suggests that in some cases, markers inherited from training data may be more efficient than RL training.
2. Challenges in Model Evaluation: Using LLMs as automated judges poses risks, potentially leading to biased evaluation results. A combination of human annotation and automated tools is recommended for comprehensive evaluation.
3. Model Deployment Recommendations: When deploying reasoning models, consider the origins and impact of the 'wait' marker and optimize based on the specific application scenario.

Conclusion

This study reveals the true origins of the 'wait' marker in reasoning models, emphasizing the complex relationship between RL training and training data. The findings have important implications for model training and evaluation and provide a new perspective for further research in the AI field.

---

Source: Hugging Face Daily Papers (2026-10-22)

— END —

Tags: #Hugging Face Follow #Reinforcement Learning Follow #Reasoning Models Follow #Model Evaluation Follow #Training Data Follow