Hugging Face Unveils 'Thinking Leakage' Phenomenon in RL for Large Language Models

Summary:Hugging Face researchers have identified a 'thinking leakage' phenomenon in reinforcement learning (RL) training for large language models (LLMs). Their study demonstrates that improvements in reasoning performance under the NO-THINK mode are largely due to the partial reactivation of the original THINK mode's reasoning abilities, rather than the acquisition of enhanced reasoning behavior. Experiments with the Qwen3 series models show that as RL training progresses, responses become longer, more

---

Background and Motivation

Large Language Models (LLMs) often need to switch between different reasoning modes to handle complex reasoning tasks. The Qwen3 series models switch between THINK and NO-THINK modes using a few chat-template tokens. Reinforcement Learning (RL) is commonly used to train these models in NO-THINK mode to improve their reasoning capabilities. However, Hugging Face researchers have discovered that this improvement may not be due to genuine enhancement of reasoning abilities but rather due to a 'thinking leakage' phenomenon, where the model partially reactivates the reasoning abilities of the original THINK mode during RL training.

Methodology and Findings

The research team trained Qwen3-1.7B, 4B, and 8B models using GRPO (a type of RL algorithm) in NO-THINK mode and evaluated their performance. As RL training progressed, the responses became longer, more reflective, and increasingly judged by the original model to resemble its own THINK mode behavior. Further analysis revealed:

1. Internal Activation Shift: The internal activations of the RL-trained model gradually shifted toward the THINK mode.
2. Behavioral Similarity: The model's behavior increasingly resembled the THINK mode.
3. Replacement Experiment: In the Qwen3-8B model, replacing the low-dimensional THINK-related activation subspace of the RL-trained model with the original model's value eliminated most of the RL gains, and the reverse exchange transferred a considerable fraction of the gains to the original model.
4. Training Signal Mechanism: The GRPO training signal favored responses that were more THINK-like, pushing the model toward the THINK state.

Engineering Trade-offs and Implications

This study highlights the limitations of current RL training methods in genuinely enhancing LLM reasoning capabilities and underscores the importance of understanding training mechanisms. While RL training can improve model performance to some extent, this improvement is largely due to the reactivation of existing abilities rather than the acquisition of new reasoning behavior. This has important implications for future model training and optimization:

• Training Method Improvement: New training methods need to be developed to truly enhance the reasoning capabilities of models, rather than relying on the reactivation of existing abilities.
• Model Evaluation Standards: When evaluating model performance, more attention should be paid to the intrinsic improvement of reasoning abilities, rather than just relying on superficial metrics.
• Engineering Application Recommendations: When applying RL training methods, the actual effects should be carefully assessed to avoid over-reliance on superficial improvements.

Conclusion

Hugging Face's research reveals the 'thinking leakage' phenomenon in RL training of LLMs, providing new directions for future model training and optimization. The study shows that RL training gains mainly reflect the reactivation of existing reasoning abilities rather than the acquisition of new reasoning behavior. This research provides important theoretical insights into understanding LLM reasoning mechanisms and improving training methods.

---

Source: Hugging Face Daily Papers (2026-10-09)

— END —

Tags: #Hugging Face Follow #LLMs & Foundation Models Follow #Reinforcement Learning Follow #Reasoning Follow #Thinking Leakage Follow