Hugging Face Releases TRACE Framework: Revolutionizing Quantization Alignment in FP4 Reinforcement Learning

Summary:Hugging Face introduces the TRACE (Train-Rollout Quantization Alignment via Compact GuidancE) framework to address the limitations of existing FP4 reinforcement learning (RL) methods in quantization accuracy. TRACE incorporates rollout-guided quantization-aware training to align training-side and rollout-side quantization outcomes, reducing discrepancies between the two execution paths. Additionally, TRACE employs an efficient quantization-information caching scheme that selectively retains mant

---

Background and Challenges

Reinforcement Learning (RL) for post-training large language models (LLMs) faces significant computational and memory overhead, prompting researchers to explore low-precision quantization techniques for more efficient RL training. However, existing FP4 RL methods suffer from a critical limitation: they primarily optimize quantization accuracy on the training and rollout paths independently rather than directly reducing the discrepancy between the two quantized execution paths. This limitation restricts the overall efficiency and performance of RL training.

Core Innovations of the TRACE Framework

Hugging Face's TRACE framework addresses these challenges through the following innovations:

1. Rollout-guided quantization-aware training: TRACE leverages rollout-side quantization outcomes to guide training-side FP4 rounding decisions, directly reducing discrepancies between the training and rollout paths.
2. Efficient quantization-information caching: TRACE employs a selective caching mechanism that retains mantissa and scale information from deeper layers, reducing storage and communication overhead.

Experimental Results and Performance

TRACE was evaluated on four large-scale MoE language models across reasoning, coding, and long-horizon RL tasks. The results demonstrate that:

• TRACE enables joint FP4 weight/activation and FP4 KV-cache rollout with RL performance comparable to BF16 rollout.
• TRACE achieves up to 5.4x rollout speedup compared to traditional methods.
• In terms of final performance, TRACE exhibits strong FP4 performance, significantly outperforming post-hoc FP4 quantization of BF16-trained policies.

Technical Highlights

• Quantization Path Alignment: TRACE aligns training and rollout paths through rollout-guided quantization-aware training.
• Efficient Caching Mechanism: TRACE's caching scheme reduces overhead while maintaining quantization accuracy.
• Multi-Task Applicability: TRACE demonstrates strong performance across reasoning, coding, and RL tasks, showcasing its broad applicability.

Industry Impact and Developer Recommendations

The introduction of TRACE provides a new technical solution for low-precision RL training, particularly in scenarios where computational and memory resources are constrained. Developers can leverage TRACE to enhance the training efficiency and scalability of RL models while maintaining high performance. Furthermore, TRACE's success underscores the ongoing importance of quantization innovation in AI, with potential breakthroughs in a wider range of applications in the future.

---

Source: Hugging Face Daily Papers (2026-10-06)

— END —

Tags: #Hugging Face Follow #FP4 Quantization Follow #Reinforcement Learning Follow #MoE Architecture Follow #TRACE Framework Follow