Hugging Face Introduces Proximal KL: Revolutionizing Importance Sampling in Off-Policy RLVR

Summary:Hugging Face has introduced Proximal KL, a novel method addressing importance sampling bias in off-policy reinforcement learning with verifiable rewards (RLVR). By decoupling the proximal reference from the sampling policy and replacing PPO clipping with bounded future KL credit, Proximal KL enhances policy gradient accuracy and training stability. Experiments on Qwen3 models demonstrate its superiority over the existing GRPO method across various benchmarks, particularly excelling in periodic r

---

Technical Mechanism Analysis

Proximal KL achieves its performance improvements through the following core mechanisms:

1. Decoupling Proximal Reference and Sampling Policy: Proximal KL separates the proximal reference from the sampling policy in the policy optimization process, avoiding the bias issues caused by the coupling of the two in traditional methods.
2. Introducing Bounded Future KL Credit: Replacing the clipping mechanism in PPO, Proximal KL uses bounded future KL credit to constrain policy updates, thereby enhancing gradient accuracy while maintaining training stability.
3. Theoretical Proof and Convergence Analysis: The research team has proven that under coverage and local expressiveness conditions, the Proximal KL update strategy is the unique stationary point, and further analyzes the convergence of continuous-time tabular population updates and the finite-sample reward guarantee for linear logistic models under additional assumptions.

Engineering Trade-offs and Experimental Performance

• Performance Improvement: In seven offline comparison experiments on Qwen3-1.7B and Qwen3-4B models, Proximal KL achieved up to 2.58% improvement in micro-averaged accuracy over eight math benchmarks compared to the existing GRPO method.
• Training Stability: In periodic refresh experiments, Proximal KL avoided the training collapses observed in PG and CISPO methods and surpassed GRPO's training accuracy.
• Ablation Study: The ablation study shows that token-wise correction outperforms sequence-level and unweighted controls, while future KL feedback is superior to PPO clipping in maintaining sensitivity to baseline perturbations.

Developer Deployment and Implementation Recommendations

The release of Proximal KL provides developers with a more efficient off-policy reinforcement learning solution, particularly suitable for the following scenarios:

• Resource-Constrained Environments: Due to its improvement in training stability, Proximal KL performs exceptionally well in resource-constrained hardware environments.
• Complex Task Processing: In scenarios requiring long-term continuous learning and complex task processing, Proximal KL can provide more reliable performance guarantees.

Developers can refer to the following recommendations for deployment:

1. Model Selection: It is recommended to apply Proximal KL on Qwen3 series models or other models with similar architectures to achieve the best performance.
2. Hyperparameter Tuning: Adjust the KL credit bounds and learning rate according to the specific task requirements to optimize training effectiveness.
3. Experimental Validation: Conduct sufficient experimental validation before deployment, especially in tasks with high-dimensional state spaces and complex reward functions, to ensure the applicability of the method.

---

Source: Hugging Face Daily Papers (2026-10-13)

— END —

Tags: #Hugging Face Follow #Reinforcement Learning Follow #Proximal KL Follow #RLVR Follow #Qwen3 Follow