MGO-SC Optimizer: Enhancing Robustness of Deep Learning in Noisy Environments

Summary:MGO-SC (Moment-Guided Optimizer with Sign-Consistency) is a lightweight modification to Adam-style optimizers that introduces a sign exponential moving average (EMA) to assess the stability of gradient signs, enabling coordinate-wise trust allocation. This method assigns higher trust to coordinates with stable gradient signs while down-weighting those with fluctuating signs, all while maintaining the overall step size. Experiments on CIFAR-10 demonstrate that MGO-SC outperforms the AdamW baselin

---

Technical Mechanism Analysis

MGO-SC (Moment-Guided Optimizer with Sign-Consistency) improves upon traditional Adam methods through the following mechanisms:

1. Sign Exponential Moving Average (Sign EMA): Calculates the EMA of gradient signs to assess the stability of each coordinate's sign.
2. Trust Allocation Mechanism: Based on the stability of the sign EMA, it assigns higher trust to coordinates with stable gradient signs and down-weights those with fluctuating signs.
3. Mean-Preserving Variant: Reallocates trust across coordinates without changing the net per-tensor step magnitude, avoiding the loss of effective step size due to down-weighting.

Engineering Trade-offs and Experimental Performance

• Advantages:

  • Significantly improves model performance in noisy environments, particularly under high label noise conditions.
  • Does not require second-order information or per-sample filtering, resulting in low computational overhead.
  • Easy to integrate into existing Adam optimizer frameworks.

• Disadvantages:

  • Limited performance improvement in very low noise environments.
  • Sensitive to the choice of learning rate and other hyperparameters, requiring fine-tuning.

Experimental results on the CIFAR-10 dataset using the ResNet-18 model show that MGO-SC outperforms the AdamW baseline across clean, symmetric, and asymmetric label noise conditions. For example, at 40% symmetric noise, it achieves a +1.4 point improvement, with a Wilcoxon p-value less than 0.01, confirming its statistical significance.

Developer Implementation and Deployment Recommendations

1. Integration Recommendations: MGO-SC can be used as a replacement for AdamW, especially in scenarios where data noise is high or label quality is poor.
2. Hyperparameter Tuning: It is recommended to fine-tune the learning rate and EMA coefficient during the initial training phase to fully leverage the advantages of MGO-SC.
3. Resource-Constrained Environments: Due to its low computational overhead, MGO-SC is suitable for device-side or edge computing scenarios with limited resources.

Conclusion

MGO-SC is an effective optimizer improvement that enhances the robustness of deep learning models in noisy environments without significantly increasing computational costs. Its lightweight design and ease of integration make it a valuable addition to existing optimizer frameworks.

---

Source: ArXiv Machine Learning (cs.LG) (2026-10-30)

— END —

Tags: #Deep Learning Follow #Optimizer Follow #Noise Robustness Follow