Context Injection and Long-Horizon Reliability: Separating Duplicate Recognition, Local Processing, and Counter Tracking
Quick Answer
The study evaluates long-horizon reliability in Qwen3-8B by scoring duplicate recognition, local processing, and counter tracking over 400 steps.
Quick Take
Results show that duplicate-recognition errors increase significantly late in the task, while counter tracking fails under irrelevant content, highlighting the importance of context re-injection for model performance.
Key Points
- Duplicate-recognition errors rise from 0.0019 to 0.011 per step after step 200.
- Counter tracking remains error-free across 8,000 steps without irrelevant content.
- Inserting irrelevant content causes 100% counter failure across 20 seeds.
- External reminders can realign counter values within one to two steps.
- Aggregate error classification may obscure critical differences in model performance.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Long-horizon failures of language models are usually measured as end-task success or as a horizon length. We instead score, step by step, three components of a 400-step ledger task in which every step is easy in isolation and the model's own answers are re-injected into its context: recognising a record already seen 5 to 300 steps earlier, processing the current record locally, and writing a counter incremented at every step. On Qwen3-8B (non-thinking, 20 seeds, replicated on a second machine and backend), with every step scored component by component, two results hold. First, duplicate-recognition errors rise late in the episode -- from 0.0019 to 0.011 per step at risk across step 200 -- while the counter makes no error in 8,000 steps and local processing is descriptively stable; the rise combines long-distance recall errors and rising false alarms. Second, a block of irrelevant content inserted before each step makes the counter fail in 20 of 20 seeds on each machine, against none without it, while its effect on local processing is much smaller and not distinguishable from none. Three pre-registered follow-ups probe the counter. Its value depends on the value present in the response history: a corrupted value persists without correction in every seed, and an exact external reminder, repeated before each message, re-aligns it within one to two steps. Under the irrelevant block, no counter error was observed when an exact compact reminder of the state, supplied externally by an oracle, preceded each message (0 of 20 seeds with an error, against 15 without it), and this reminder outperformed a length-matched neutral text; whether a single reminder suffices was not conclusive, and the absence of damage to the other components was not established. An aggregate or exclusive error classification can hide the difference we report.
| Comments: | 17 pages, 5 figures. Code, data and reproduction script: this https URL (tag v1.0-arxiv) |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.20633 [cs.CL] |
| (or arXiv:2610.20633v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20633 arXiv-issued DOI via DataCite |
Submission history
From: Thomas Serval [view email]
[v1]
Wed, 30 Sep 2026 14:08:47 UTC (144 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →The "10th Juror": Open-Set Standpoint Screening for Bureaucratic Bias Detection
MARS-Gov introduces a framework for detecting bureaucratic bias in Dutch government documents, achieving a new state-of-the-art F1 score of 0.880. This model outperforms existing zero-shot detectors by 20.2 points and reduces unnecessary interventions to just 2.5%. The framework's dynamic '10th juror' adapts to emerging biases, enhancing legal language processing.