ZICQ
中 Log in / Sign up
Newsroom Research & Papers #Language Model #Probe Design #Duplicate Effect #Attention mechanisms #arXiv

New Study Reveals Position-Dependent Repetition Effects in Language Models, Challenging Conventional Assumptions

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 16 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:A new arXiv paper reveals that language models' predictions of repeated target tokens are not position-invariant but significantly affected by the readout position. Using a two-probe design, the study finds that adjacent repetition follows priming intuition, while displaced repetition exhibits an inverted-U curve. The effect is robust across 13 open-source models and replicates in four languages. Causal ablations isolate the effect to exact lexical repetition, and internal attention analysis sho


Background

Cloze-style probes are commonly used in NLP to evaluate language models' predictive abilities for target tokens. Such probes typically vary the number of occurrences (N) of a target token in the context to study memory and utilization mechanisms. However, existing research implicitly assumes that the effect of repetition count on prediction is the same regardless of where the readout slot is located.

Core Findings

This study challenges this assumption using a two-probe design. The researchers fixed a prefix containing repeated targets and varied only the readout position:

  • Adjacent probe: The readout slot is placed immediately after the repeated block.
  • Displaced probe: The readout slot is placed inside a fresh sentence frame.

The results reveal strikingly different patterns:

  • Adjacent repetition: Follows priming intuition; target probability increases with N and plateaus.
  • Displaced repetition: Exhibits an inverted-U curve; target probability rises to an early peak and then declines as N increases.

Experimental Validation

  • Model coverage: The displaced inverted-U effect is significant in all 13 open-access encoder and decoder models tested (per-word drop with bootstrap CI excluding zero).
  • Cross-lingual replication: Replicates in 42 of 42 multilingual cells across Spanish, Chinese, German, and French.
  • Causal ablation: A six-condition ablation isolates the effect to exact lexical repetition, not length, generic redundancy, or semantic-neighbor exposure.
  • Frame control: A frame-pragmatics control rules out an artifact of the readout frame.

Internal Mechanism Analysis

Analysis of internal attention reveals:

  • In causal language models (LMs), per-target-token attention decreases with N, while the total budget assigned to the repeated block grows.
  • This pattern is not observed in the masked LM probed.

This suggests that causal LMs may allocate attention across the repeated block, but individual token attention is diluted, leading to degraded prediction at displaced positions.

Implications

  1. Probe design: Any probe varying repetition count cannot treat the readout position as orthogonal to what it measures. Researchers must control for readout position to avoid misleading conclusions.
  2. Model evaluation: The findings are crucial for evaluating memory mechanisms and context utilization in LMs, especially in long-context and retrieval-augmented scenarios.
  3. Theoretical modeling: The inverted-U curve hints at a possible "over-repetition suppression" mechanism, akin to semantic satiation in humans, warranting further theoretical exploration.

Conclusion

This study systematically reveals the complex dynamics of position-dependent repetition effects in language models, providing key empirical evidence for understanding internal mechanisms and designing more rigorous evaluation probes.


Original paper: arXiv:2608.04021v1, titled "When More Becomes Less: Position-Dependent Repetition Effects in Language Models"


Source: ArXiv NLP/LLM (cs.CL) (2026-08-06)

— END —

Tags: #Language Model #Probe Design #Duplicate Effect #Attention mechanisms #arXiv

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…