When More Becomes Less: Position-Dependent Repetition Effects in Language Models
Quick Answer
This study reveals that the effect of repeated tokens in language models is position-dependent, with adjacent repetitions showing a plateau in prediction probability, while displaced repetitions exhibit an inverted-U effect.
Quick Take
The findings are consistent across 13 models, including multilingual tests in Spanish, Chinese, German, and French, indicating that readout position significantly influences prediction outcomes.
Key Points
- Adjacent repetitions increase prediction probability, plateauing with more tokens.
- Displaced repetitions show an inverted-U effect, peaking before declining.
- Results replicated across 42 multilingual cells in various languages.
- Causal ablation isolates effects to lexical repetition, not length or redundancy.
- Attention per target token decreases with more repetitions in causal LMs.
DeepSignal Analysis
What happened
The study investigates how the position of repeated tokens affects prediction probabilities in language models. It finds that adjacent repetitions lead to a plateau in prediction probability, while displaced repetitions show an inverted-U effect. This pattern is consistent across 13 models and multiple languages.
Key evidence
- The research utilized a two-probe design, varying only the readout position while keeping a repeated-target prefix fixed.
- Adjacent repetitions resulted in a steady increase in prediction probability, which then plateaued, while displaced repetitions peaked and subsequently declined.
- The findings were replicated across 42 multilingual cells, confirming the influence of readout position on prediction outcomes in models tested.
Why it matters
Understanding the effects of token repetition and position can refine how language models are trained and evaluated. The findings challenge the assumption that repetition impacts predictions uniformly, suggesting that the design of language model evaluations should account for position effects. This could lead to more accurate assessments of model performance and improvements in model architecture.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Cloze-style probes that vary how often a target token appears implicitly assume that more copies of a target affect prediction the same way regardless of where the readout slot sits. We show this assumption fails. Our two-probe design holds a repeated-target prefix fixed and varies only the readout position: the adjacent probe places the slot immediately after the repeated block; the displaced probe places it inside a fresh sentence frame. Adjacent repetition behaves as priming intuition predicts: $P(\text{target})$ climbs with $N$ and plateaus. Displaced repetition produces an inverted-U: $P(\text{target})$ rises to an early peak and then declines as more copies are added. The displaced inverted-U shows a per-word drop with bootstrap CI excluding zero in all 13 open-access encoder and decoder models we test, and replicates across Spanish, Chinese, German, and French in 42 of 42 multilingual cells. A six-condition causal ablation isolates the effect to exact lexical repetition rather than length, generic redundancy, or semantic-neighbour exposure. A frame-pragmatics control rules out an artefact of the readout frame. Internally, per-target-token attention falls with $N$ while the total budget assigned to the repeated block grows in causal LMs but not in the masked LM we probe. Probes that vary repetition count cannot treat the readout position as orthogonal to what they measure.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2608.04021 [cs.CL] |
| (or arXiv:2608.04021v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.04021 arXiv-issued DOI via DataCite |
Submission history
From: Han-Yu Wang [view email]
[v1]
Wed, 24 Jun 2026 23:57:21 UTC (132 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.