Hugging Face Proposes DLC Method to Address Training-Inference Mismatch in Large Language Models
Summary:Hugging Face has introduced the Discrepancy-Lag Correction (DLC) method to address the training-inference mismatch in large language models (LLMs). DLC disentangles numerical discrepancies from policy lags and incorporates a dynamic resynchronization mechanism to enhance training stability. Evaluated on the Qwen3-1.7B model, DLC demonstrates its effectiveness in handling asynchronous sampling and numerical errors, providing a more robust solution for LLM training.
Background and Problem
In reinforcement learning for large language models (LLMs), training data is typically assumed to be sampled from the current policy. However, in practice, the inference engine and the trainer are separate, leading to off-policy training data and a mismatch between training and inference. This mismatch arises from two sources: numerical discrepancies in the inference engine and stale weights from asynchronous sampling. Existing correction methods often address only one of these issues, failing to handle both simultaneously.
Core Mechanism of DLC
The Discrepancy-Lag Correction (DLC) method proposed by Hugging Face addresses these challenges through the following steps:
-
Disentangling Numerical Discrepancies and Policy Lags: DLC first decomposes each sampled token's error into a numerical discrepancy of the inference engine and a policy lag of the trainer. This separation allows for independent analysis and correction of each error source.
-
Dynamic Resynchronization Mechanism: To address the policy lag caused by asynchronous sampling, DLC incorporates a dynamic resynchronization mechanism. When the effective sample size falls below a certain threshold, DLC resynchronizes the weights of the inference engine and the trainer, preventing training collapse.
-
Centering Numerical Discrepancies: DLC centers the numerical discrepancies at each prefix to eliminate the leading bias.
Experiments and Results
Experiments on the Qwen3-1.7B model show that DLC keeps the KL divergence within twice that of independent on-policy trainings and prevents training collapses with its resynchronization mechanism. In contrast, traditional per-token correction methods tend to fail when dealing with asynchronous sampling, while DLC provides a more stable training process.
Engineering Trade-offs and Deployment Recommendations
- Advantages: DLC excels in handling training-inference mismatches, particularly in asynchronous sampling and large-scale model training scenarios.
- Disadvantages: DLC introduces additional computational overhead, requiring a trade-off between training speed and stability.
- Deployment Recommendations: For tasks requiring high stability, the adoption of DLC is recommended. Additionally, the resynchronization threshold can be adjusted based on specific task requirements to optimize training efficiency.
Future Directions
The DLC method provides a new approach to addressing data mismatches in LLM training. Future research could further optimize the resynchronization mechanism and explore its application in other model types and tasks.
— END —Source: Hugging Face Daily Papers (2026-10-29)
Tags: #Hugging Face #LLM Training #Reinforcement Learning #DLC #Training-Inference Mismatch
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments