ArXiv Research: Optimizing Transformer Attention Mechanisms to Match Human Reading Times

Summary:This research investigates the introduction of memory constraints into Transformer models' attention mechanisms to better align with human reading time behavioral data. The study systematically compares multiple attention-based memory mechanisms and evaluates their performance in predicting human reading times and grammatical competence. The results show that constraints sensitive to the content of intervening tokens consistently achieve the highest alignment with human reading times, outperform

---

Background and Motivation

Transformer models have achieved great success in the field of natural language processing, but their attention mechanisms allow for lossless access to the full context, which differs significantly from the limited memory systems of humans. This study aims to introduce memory constraints into the attention mechanisms of Transformer models to better align with human reading time behavioral data.

Methodology

The research team systematically compared multiple attention-based memory mechanisms, including static constraints and dynamic memory curricula. Static constraints maintain a fixed constraint strength throughout training, while dynamic memory curricula adjust the constraint strength according to the training progress. The study evaluated the performance of these mechanisms in predicting human reading times and grammatical competence.

Key Findings

1. Constraints Sensitive to Intervening Tokens Perform Best: These constraints outperform distance-based constraints in matching human reading times.
2. Dissociation Under Dynamic Memory Curricula: There is a dissociation between psychometric fit and grammatical competence under dynamic memory curricula, suggesting that Transformer models cannot serve as a one-size-fits-all cognitive model.
3. Effectiveness of Memory Constraints: Introducing memory constraints can significantly improve the model's ability to predict human reading times, but it may also affect the model's grammatical competence.

Industry Impact and Developer Recommendations

1. Limitations in Cognitive Modeling: Transformer models have limitations as cognitive models, and researchers should be cautious when using these models for cognitive simulation.
2. Application of Memory Constraints: In scenarios where a closer alignment with human cognitive processes is required, introducing memory constraints can improve model performance.
3. Future Research Directions: Further research on different types of memory constraint mechanisms and how to balance memory constraints with other performance metrics of the model is needed.

---

Source: ArXiv cs.CL (2026-08-24)

— END —

Tags: #Transformer #Memory Constraints #Human Reading Times #Cognitive Modeling #Attention Mechanism