arXiv Research Reveals 'Thinking Leakage' Phenomenon in LLM Reasoning via Reinforcement Learning

Summary:A new arXiv study delves into the 'thinking leakage' phenomenon observed in hybrid reasoning models like the Qwen3 series when trained with reinforcement learning (GRPO) in NO-THINK mode. The research demonstrates that performance gains largely stem from the reactivation of the original model's THINK mode reasoning abilities rather than the acquisition of enhanced reasoning behavior. Through a detailed analysis at the levels of weights, activations, and attention heads, the study reveals how the

---

Background and Motivation

In recent years, significant progress has been made in the reasoning capabilities of Large Language Models (LLMs), with hybrid reasoning models like the Qwen3 series switching between THINK and NO-THINK modes to enhance performance. However, the mechanisms behind the improvements observed when training these models with reinforcement learning (RL) in NO-THINK mode remain poorly understood, particularly regarding the source of the performance gains.

Key Findings

1. Thinking Leakage Phenomenon: The study reveals that performance improvements in NO-THINK mode training with GRPO largely result from the reactivation of the original model's THINK mode reasoning abilities rather than the acquisition of new reasoning behaviors. This phenomenon is termed 'thinking leakage.'
2. Multi-Level Analysis: Analysis at the levels of weights, activations, and attention heads shows that the RL signal biases the model toward the THINK state. For instance, in the Qwen3-8B model, replacing the low-dimensional THINK-related activation subspace of the RL-trained model with the original model's value causes most of the RL gains to disappear, and vice versa.
3. Training Signal Mechanism: The study further uncovers that the GRPO training signal favors responses that are more THINK-like, thereby pushing the model toward the THINK state.

Engineering Trade-offs and Limitations

• Source of Performance Gains: While RL can enhance model performance in some cases, the gains are not due to the acquisition of new reasoning abilities but rather the reactivation of existing ones. This suggests potential bottlenecks in using RL to improve LLM reasoning capabilities.
• Training Cost and Efficiency: RL training typically requires substantial computational resources, and the existence of 'thinking leakage' suggests that its cost-effectiveness may need to be re-evaluated.

Developer Recommendations

1. Careful Evaluation of RL Applications: When applying RL to LLM reasoning tasks, developers should carefully assess its actual effects, avoiding the misinterpretation of the reactivation of existing abilities as the acquisition of new ones.
2. Explore Other Training Methods: Developers may consider combining other training methods, such as supervised or self-supervised learning, to more effectively enhance model reasoning capabilities.
3. Focus on Model Interpretability: A deeper understanding of model behavior and internal mechanisms can help better design and optimize training strategies.

Future Research Directions

Future research could further explore how to more effectively use RL to improve LLM reasoning capabilities or develop new training methods to overcome the limitations posed by the 'thinking leakage' phenomenon.

---

Source: ArXiv NLP/LLM (cs.CL) (2026-10-14)

— END —

Tags: #Reinforcement Learning Follow #LLMs & Foundation Models Follow #Reasoning Follow #Thinking Leakage Follow #GRPO Follow