Hugging Face Releases DeltaTTG: Revolutionizing Multi-Turn Agent Training in Reinforcement Learning

Summary:Hugging Face has introduced DeltaTTG, a novel reinforcement learning approach designed to address the challenges posed by sparse terminal rewards in multi-turn agent training. DeltaTTG leverages a shared progress meter to learn a reusable progress signal and optimizes the training process using low-cost supervision data. On tasks like ALFWorld, PaperSearchQA, and BFCLv3, DeltaTTG outperforms the existing method GRPO by 5.7 and 7.2 points in average scores, respectively, while maintaining similar

---

Technical Mechanism Analysis

The core innovation of DeltaTTG lies in the introduction of Token-to-Go (TTG) as a progress signal, which is a reusable metric learned through a shared progress meter. TTG represents the number of tokens remaining to be generated until task completion and serves as a proxy for the remaining completion effort. During training, DeltaTTG redistributes outcome-derived advantages through the frozen progress meter's estimates without altering the reward itself, thus maintaining the advantage sign and token-average scale of each trajectory. This approach offers the following advantages in multi-turn agent training:

1. Reusability of Progress Signal: TTG, as a progress signal, can be reused across different tasks and policies, reducing the dependency on task-specific rules or external evaluators.
2. Low-Cost Supervision: DeltaTTG utilizes diverse policy trajectories for supervision, eliminating the need for expensive external evaluations or human annotations.
3. Advantage Redistribution Mechanism: By freezing the progress meter's estimates, DeltaTTG redistributes advantages without affecting the reward, optimizing the training process.

Engineering Trade-offs and Performance

Experiments on tasks like ALFWorld, PaperSearchQA, and BFCLv3 show that DeltaTTG improves the average score over GRPO by 5.7 and 7.2 points, respectively, while maintaining a similar number of GPU training hours as GRPO. This indicates that DeltaTTG enhances performance without significantly increasing training costs.

Furthermore, DeltaTTG demonstrates superior sample efficiency and out-of-domain performance. The results show that DeltaTTG can more effectively utilize training samples and maintain high performance across different domains, indicating strong generalization and robustness.

Developer Implementation and Deployment Recommendations

For developers, DeltaTTG offers an efficient multi-turn agent training method with the following advantages:

1. Easy Integration: DeltaTTG can be seamlessly integrated with existing reinforcement learning frameworks without the need for major modifications to the existing system.
2. Resource Efficiency: Due to the shared nature of TTG, DeltaTTG has lower computational resource requirements during training, making it suitable for resource-constrained environments.
3. High Flexibility: TTG, as a progress signal, can be customized according to the specific needs of the task, enhancing training effectiveness.

Developers can refer to the open-source code provided by Hugging Face (https://github.com/Xuyan923r/DeltaTTG) for rapid deployment and testing.

Conclusion

DeltaTTG, by introducing TTG as a progress signal, revolutionizes the training method for multi-turn agent reinforcement learning. Its excellent performance on multiple benchmark tasks demonstrates the great potential of this method in improving the efficiency and performance of agent training.

---

Source: Hugging Face Daily Papers (2026-10-13)

— END —

Tags: #Hugging Face Follow #Reinforcement Learning Follow #Agent Training Follow #DeltaTTG Follow