Self-Guided Test-Time Training for Long-Context LLMs
Quick Answer
The paper introduces Self-Guided Test-Time Training (S-TTT) for long-context LLMs, enhancing accuracy by up to 15% on benchmarks like LongBench-v2.
Quick Take
The paper introduces Self-Guided Test-Time Training (S-TTT) for long-context LLMs, enhancing accuracy by up to 15% on benchmarks like LongBench-v2. S-TTT selectively trains on identified evidence spans, addressing the inefficiencies of traditional test-time training methods that suffer from irrelevant context noise. This approach significantly improves performance for models like Qwen3-4B-Thinking-2507 and Llama-3.1-8B-Instruct.
Key Points
- S-TTT improves long-context utilization by training on relevant evidence spans.
- Achieves up to 15% relative accuracy improvement on LongBench-v2.
- Traditional test-time training struggles with irrelevant context leading to performance degradation.
- Models Qwen3-4B-Thinking-2507 and Llama-3.1-8B-Instruct benefit from S-TTT.
- The method addresses the high cost of adapting to entire long contexts.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Xinyu Zhu, Zhe Xu, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Kaushik Rangadurai, Hua Zhi, Frank Shyu, Sandeep Pandey, Luke Simon, Yu Meng, Xi Liu
Abstract:Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs. As input length grows, accuracy often degrades, indicating that models still struggle to identify and use the evidence most relevant to a question. A promising way to improve long-context utilization is test-time training (TTT), which treats the test context as a training example for instance-specific parameter adaptation. However, applying TTT to the entire long context is prohibitively expensive, while adapting on randomly sampled spans introduces severe noise. Because most spans in a long context are irrelevant to the specific question, training on them may even degrade the base model's performance. Our preliminary study shows that TTT is highly sensitive to training-span quality: on LongBench-v2, TTT on randomly sampled spans hurts performance, whereas TTT on oracle spans substantially improves it. Motivated by this, we propose a simple method, Self-Guided TTT (S-TTT): before adaptation, the model identifies the evidence spans it should learn from, and the standard language-modeling training objective is applied only to those selected spans. On two challenging long-context reasoning benchmarks, LongBench-v2 and LongBench-Pro, S-TTT improves accuracy for both Qwen3-4B-Thinking-2507 and Llama-3.1-8B-Instruct, achieving up to a 15% relative improvement.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.09415 [cs.CL] |
| (or arXiv:2607.09415v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.09415 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xinyu Zhu [view email]
[v1]
Fri, 10 Jul 2026 13:45:56 UTC (990 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.