LRMoE: Memory-Aware Training System for Sparse Mixture-of-Experts Boosts Efficiency by 21.2-44.8%

Summary:LRMoE is a memory-aware training system for Sparse Mixture-of-Experts (MoE) models that enhances training efficiency by jointly recalibrating backward demand and balancing the retained workload. It plans retention quotas and expert placement during forward execution while preserving all routed computations. During backpropagation, a load recalibration metric selects token-expert pairs within these quotas, and a capacity-aware routing algorithm distributes them among eligible instances of their o

---

Technical Mechanism Analysis

The core mechanisms of LRMoE include the following key steps:

1. Memory-aware Forward Execution: During the forward pass, LRMoE plans retention quotas and expert placement under resource constraints while preserving all routed computations. This approach avoids the memory competition caused by expert replication in traditional MoE models.

2. Load Recalibration in Backpropagation: In the backpropagation stage, LRMoE introduces a load recalibration metric to evaluate each token-expert pair's contribution to the expert update and selects pairs with higher contributions for retention. This method effectively balances the retained workload and avoids unnecessary computation and communication overhead.

3. Capacity-aware Routing Algorithm: This algorithm distributes the retained token-expert pairs among eligible instances of their original experts, further optimizing communication traffic and computational efficiency.

4. Physical Compaction: By forming smaller communication payloads and variable-sized grouped GEMMs, LRMoE reduces backward communication traffic, computation, and execution-buffer demand.

Engineering Trade-offs and Performance

LRMoE achieves a good balance between memory usage and computational efficiency. Its main advantages are:

• Training Speed Improvement: LRMoE reduces training time by 21.2-44.8% compared to the SYMI method on Qwen3-30B-A3B and GLM4.5-106B-A12B models.

• Memory Optimization: By reducing expert replication and optimizing communication loads, LRMoE effectively reduces GPU memory usage pressure.

• Scalability: This method is suitable for large-scale MoE model training and can maintain efficiency in resource-constrained environments.

However, the disadvantages of LRMoE include its high complexity, which requires a certain degree of modification to the existing training process. Additionally, the implementation of the load recalibration and capacity-aware routing algorithms may introduce additional computational overhead.

Developer Implementation and Deployment Recommendations

For developers who wish to adopt LRMoE, the following recommendations are provided:

1. Evaluate Existing Infrastructure: Before deploying LRMoE, evaluate the memory and computational resources of the current training infrastructure to ensure it can support LRMoE's requirements.

2. Gradual Integration: It is recommended to gradually integrate LRMoE, first testing it on small models and then expanding to larger models.

3. Monitor Performance: During training, closely monitor LRMoE's performance metrics, such as training time, memory usage, and computational efficiency, to adjust parameters in a timely manner.

4. Optimize Communication: Further optimize communication loads, for example, by using compression algorithms or more efficient network topologies, to improve overall training efficiency.

---

Source: ArXiv AI (cs.AI) (2026-10-30)

— END —

Tags: #LRMoE Follow #Sparse MoE Follow #Memory Optimization Follow #Training Efficiency Follow