MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

Summary:MoEZip is a routing-aware KV cache compression technique designed for sparse Mixture-of-Experts (MoE) language models, addressing the memory bottleneck in long-context inference. Traditional KV cache compression methods fail to address the issue of expert selection changes caused by cache eviction in MoE architectures, which degrades generation quality. MoEZip introduces a novel sensitivity metric based on the Fisher information matrix of routing distributions and combines it with attention weig

---

Technical Mechanism Analysis

The core innovation of MoEZip lies in its routing-aware cache compression method tailored for sparse MoE architectures. Traditional KV cache compression methods are primarily designed for dense Transformer models and fail to adequately address the issue of expert selection changes caused by cache eviction in MoE models. MoEZip addresses this through the following mechanisms:

1. Routing Sensitivity Metric: Based on the Fisher information matrix of routing distributions, MoEZip quantifies the sensitivity of expert selection to perturbations. This metric provides finer guidance for cache compression, ensuring that cache data critical to generation quality is prioritized during compression.

2. Routing-Aware Scoring: MoEZip combines attention weights with the routing sensitivity metric to score contexts. This scoring mechanism ensures that cache data crucial to the current generation task is retained during compression.

3. Complementary Scoring System: MoEZip employs a hybrid scoring system that combines eviction methods for both dense and sparse-MoE architectures. This system allows for flexible adjustment of cache compression strategies in different scenarios, further enhancing compression efficiency.

Engineering Trade-offs and Performance

MoEZip demonstrates strong performance across multiple benchmarks, particularly in long-context inference tasks. At a 10% KV retention ratio, MoEZip improves by +24 and +20 points on LongBench and RULER benchmarks, respectively, significantly outperforming existing techniques. This performance gain comes at the cost of additional computational overhead: MoEZip requires extra routing sensitivity calculations and scoring during cache compression, which may introduce some computational latency. However, for long-context inference tasks, this overhead is justified, as the memory savings from cache compression can significantly boost inference speed.

Developer Implementation and Deployment Recommendations

For developers, MoEZip offers an effective solution to the memory bottleneck in sparse MoE models during long-context inference. Here are some deployment recommendations:

1. Hardware Requirements: Due to the additional computational resources required by MoEZip, it is recommended to deploy it on servers equipped with high-performance GPUs.

2. Cache Strategy Adjustment: Depending on the specific application scenario and hardware conditions, developers can adjust the KV retention ratio to find the optimal balance between memory savings and generation quality.

3. Integration and Optimization: It is recommended to deeply integrate MoEZip with the existing training and inference workflows of MoE models and perform performance optimization to fully leverage its advantages.

Conclusion

MoEZip is a significant breakthrough in sparse MoE models for long-context inference. Its innovative routing-aware cache compression method significantly improves cache compression efficiency while maintaining generation quality, opening new possibilities for the widespread application of sparse MoE models.

---

Source: ArXiv Machine Learning (cs.LG) (2026-10-23)

— END —

Tags: #MoE Architecture Follow #KV Cache Compression Follow #Long-Context Inference Follow