THUNLP Introduces Latent Compaction: Revolutionizing KV Cache Compression for Long-Context Language Models
Summary:THUNLP introduces Latent Compaction, a novel method for compressing key-value (KV) caches in long-context language models through test-time training. By initializing cache entries with reconstruction importance, teacher attention, and key-space diversity, and then optimizing the entire cache via end-to-end distillation, this approach achieves significant compression without sacrificing retrieval accuracy. Experiments on RULER, LongBench, and Phonebook Lookup demonstrate its superiority, with a 3
Technical Mechanism Analysis
Latent Compaction achieves KV cache compression through the following steps:
- Cache Entry Initialization: It initializes cache entries using reconstruction importance, teacher attention, and key-space diversity to ensure efficient information retention.
- Training Process:
- First, it fits the attention outputs and log-normalizers to those of the full cache, while keeping the reconstruction-selected entries fixed.
- Then, it optimizes the entire cache via end-to-end distillation from the full-cache model.
- Model Weight Freezing: Throughout the training process, model weights remain frozen, and only generic prompts about the document are used, without relying on downstream questions or answers.
Engineering Trade-offs and Performance
- Advantages:
- Efficient Compression: On the RULER dataset, Latent Compaction achieves 96.7% accuracy at a 32x compression rate, compared to 40.0% for KVZip+.
- Cross-Model Transferability: The method performs well on different models like Qwen3 and MiniCPM5, demonstrating its broad applicability.
- No Additional Data Required: It uses only generic document prompts for training, reducing dependency on specific data.
- Challenges:
- Training Time: Test-time training may introduce additional computational overhead, requiring a trade-off between compression efficiency and training time.
- Applicable Scenarios: While it excels in long-context tasks, its advantages may be less pronounced in scenarios with very short sequences or low memory requirements.
Developer Implementation and Deployment Recommendations
- Applicable Scenarios: Suitable for tasks that require handling long contexts, such as document summarization, dialogue systems, and legal text analysis.
- Deployment Recommendations:
- In resource-constrained environments, prioritize using Latent Compaction for KV cache compression to reduce memory requirements.
- In multi-model deployments, leverage its cross-model transferability to simplify the deployment process.
- Combine with other compression techniques (e.g., quantization, pruning) to further optimize model performance.
Conclusion
Latent Compaction, with its innovative test-time training approach, significantly enhances the compression efficiency of KV caches in long-context large language models while maintaining high retrieval accuracy. This technology opens new possibilities for the practical application of long-context models, particularly in resource-constrained environments.
— END —Source: Hugging Face Daily Papers (2026-10-14)
Tags: #THUNLP #KV Cache Compression #Long-Context #Test-Time Training
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments