RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
Quick Answer
The study reveals that Rotary Positional Embeddings (RoPE) in long-context Transformers lose key properties as context length increases, leading to unpredictable attention scores and a failure to distinguish between positions and tokens.
Quick Take
This necessitates the exploration of new mechanisms for encoding position and token order in future models.
Key Points
- RoPE loses locality bias and token relevance consistency as context length increases.
- Attention scores become unpredictable, approaching random guessing probabilities.
- Adjusting RoPE base can distinguish tokens but sacrifices position distinction.
- Multi-head, multi-layer architectures do not overcome RoPE limitations.
- New encoding mechanisms for position and token order are needed in Transformers.
Paper Resources
📖 Reader Mode
~2 min readAbstract:We identify intrinsic limitations of Rotary Positional Embeddings (RoPE) in Transformer-based long-context language models. Our theoretical analysis abstracts away from the specific content of the context and depends only on its length. We prove that as context length increases, RoPE-based attention becomes unpredictable and loses two properties that are central to its effectiveness. First, it loses its locality bias: RoPE is no more likely to favor nearer positions than substantially farther ones. Second, it loses consistency in token relevance: a key vector that receives a higher attention score than an alternative at one position may receive a lower score at another. In both cases, the probability of failure approaches 0.5, no better than random guessing. We further prove that the attention score can remain unchanged when a key token is moved to a different position, or even replaced by a different token, indicating a failure to distinguish positions or tokens. Adjusting the RoPE base trades off distinguishing positions against distinguishing tokens but cannot preserve both at the same time. Increasing the RoPE base hyperparameter, a common practice in today's long-context models, helps distinguish different tokens, but inevitably sacrifices the ability to distinguish positions. Our empirical analysis shows that multi-head, multi-layer architectures are insufficient to overcome these limitations. Our findings suggest that fundamentally new mechanisms for encoding position and token order may be needed in future Transformer long-context language models.
| Comments: | 35 pages, 11 figures, submitted to NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.15514 [cs.CL] |
| (or arXiv:2605.15514v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15514 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yufeng Du [view email]
[v1]
Fri, 15 May 2026 01:16:16 UTC (1,216 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.