ZICQ
中 Log in / Sign up
Newsroom Research & Papers #Transformer #Grokking #Distributed Utility #Spectral Recoding #MLP

arXiv Research Reveals Mechanism of Memorization to Generalization in Transformers: Distributed Utility and Spectral Rec

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 10 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:A new arXiv study investigates the transition from memorization to generalization in Transformer models, introducing 'Transition Games'—behavior-aligned activation games with paired non-generalizing controls. The research reveals that generalization is achieved through distributed utility gains and spectral recoding rather than module switching. Specific modes account for 67-92% of the addition contrast in replacement games, and the MLP layer mediates more of the effect than other downstream pat


Background and Motivation

Transformer models have achieved remarkable success in natural language processing and other fields, but the exact mechanism of their transition from memorization to generalization remains unclear. The traditional view posits that MLP layers are responsible for memorization, while the attention mechanism handles generalization. However, this study challenges this notion through the innovative 'Transition Games' approach.

Methodology

The research team designed 'Transition Games,' a behavior-aligned activation game with paired non-generalizing controls. This method allows researchers to precisely track the transition from memorization to generalization and analyze the contributions of different components.

Key Findings

  1. Distributed Utility Gains: The generalization process is not dominated by a single module but is achieved through a distributed approach.
  2. Spectral Recoding: The study reveals that generalization is accomplished through spectral recoding of an existing distributed circuit, rather than module switching.
  3. MLP Layer Mediation: The MLP layer exhibits stronger mediation in the path study compared to the attention mechanism, which contradicts the traditional view.
  4. Mode Contribution: Specific modes contribute 67-92% of the addition contrast in replacement games.

Technical Highlights

  • Transition Games Method: An innovative behavior-aligned activation game method that can precisely track the transition from memorization to generalization.
  • Distributed Utility Gains Analysis: By analyzing the utility gains of different components, the distributed nature of generalization is revealed.
  • Spectral Recoding Mechanism: A new generalization mechanism is proposed, challenging the traditional module switching paradigm.

Industry Impact and Developer Recommendations

This research provides new directions for the optimization of Transformer models, particularly in enhancing their generalization capabilities and reducing memorization bias. Developers should consider the following:

  • Optimizing MLP Layer Design: Given the important role of MLP layers in generalization, consider optimizing their structure.
  • Exploring Spectral Recoding Techniques: Investigate the application of spectral recoding in model training to improve generalization performance.
  • Distributed Utility Gains Analysis: Introduce distributed utility gains analysis in model evaluation to more comprehensively understand the model's generalization capabilities.

Conclusion

This study provides new insights into the generalization mechanism of Transformer models, challenging the traditional 'MLP memorizes, attention generalizes' paradigm and offering new ideas for future model optimization and design.


Source: ArXiv Machine Learning (cs.LG) (2026-09-17)

— END —

Tags: #Transformer #Grokking #Distributed Utility #Spectral Recoding #MLP

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…