Hugging Face Releases PACE: Revolutionizing Linear-Attention Models with Efficient 4-bit State Quantization

Summary:Hugging Face has introduced PACE, a novel post-hoc four-bit state quantizer designed to enhance the inference efficiency of linear-attention language models. By reducing the frequency of state rounding, introducing column-wise shared quantization scales, and retaining key heads at eight bits, PACE significantly mitigates precision loss while conserving memory. Experiments demonstrate that PACE outperforms existing methods like Q-Mamba in terms of accuracy and inference speed, using only two-thir

---

Technical Mechanism Analysis

PACE is a state quantization scheme for linear-attention language models with the following core mechanisms:

1. Reduced Quantization Frequency: Unlike traditional methods that round the state after every generated token, PACE rounds the state only once every few tokens, thereby reducing error accumulation.
2. Column-wise Shared Quantization Scales: PACE identifies that the large entries of the gated-delta state gather in columns and thus shares quantization scales within each column, further reducing quantization error.
3. Mixed-Bit Head Retention: PACE retains the heads that accumulate rounding error longest at eight bits while using four bits for other heads, balancing precision and memory usage.

Engineering Trade-offs and Performance

• Memory Efficiency: Tested on four hybrid models ranging from 4B to 35B parameters, PACE uses only two-thirds of the memory compared to the eight-bit DAMP scheme.
• Accuracy and Speed: With four-bit heads, PACE outperforms Q-Mamba in accuracy while maintaining the decoding speedup of the buffer.
• Applicability: PACE is applicable to various linear-attention models and does not require model fine-tuning, reducing deployment costs.

Developer Deployment Recommendations

1. Hardware Adaptation: Due to its focus on linear-attention models, deploying PACE on hardware with efficient memory management features is recommended to maximize its benefits.
2. Model Selection: For applications requiring high precision and low latency, the mixed-bit head scheme is recommended; for memory-constrained environments, the full four-bit head scheme can be chosen.
3. Performance Monitoring: During deployment, it is advisable to continuously monitor quantization error and inference speed to ensure the model meets performance expectations.

PACE offers a new solution for improving the inference efficiency of linear-attention models, particularly in resource-constrained environments.

---

Source: Hugging Face Daily Papers (2026-10-21)

— END —

Tags: #Hugging Face Follow #PACE Follow #Linear Attention Follow #State Quantization Follow #Inference Efficiency Follow