WILDTRACE: Benchmarking Natural Evidence Trails in Long-Context Reasoning
Quick Answer
WILDTRACE introduces a benchmark of 481 tasks across 214 long-form sources, focusing on integrating source-internal evidence for complex reasoning in long documents.
Quick Take
WILDTRACE introduces a benchmark of 481 tasks across 214 long-form sources, focusing on integrating source-internal evidence for complex reasoning in long documents. This benchmark addresses the challenge of distinguishing genuine reasoning from artifacts, crucial for high-stakes analytical tasks in AI.
Key Points
- WILDTRACE benchmarks 481 tasks from 214 long-form sources like incident reports and narratives.
- It defines seven evidence geometries for analytical reading in long documents.
- The benchmark emphasizes source-internal evidence integration over traditional methods.
- Multi-stage validation ensures the quality of evidence trails and question relevance.
- This work highlights a critical gap in long-context reasoning for AI models.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Answering complex questions over long documents frequently requires integrating evidence that the source itself disperses naturally across distant passages. In an incident report, the operating condition, design flaw, and missed safety check that jointly explain a disaster may appear dozens of sections apart; in a novel, a character's true motive may surface only through scenes far removed from the moment it becomes relevant. This source-internal evidence integration is central to real-world long-document analysis, yet existing benchmarks largely sidestep it. Needle probes, planted facts, and reverse-engineered multi-hop chains embed evidence that may differ from the host text in distribution, placement, or register, making it unclear whether strong performance reflects genuine source reasoning or distributional artifacts. We introduce WILDTRACE, a benchmark of 481 tasks over 214 naturally occurring long-form sources such as technical incident reports and lesser-known literary narratives, where all evidence trails arise from the document's own causal, temporal, and narrative logic. Drawing on Pearl's causal hierarchy and prior multi-hop reasoning typologies, we define seven source-internal evidence geometries that characterize the distinct relational demands of analytical reading in long documents. A source-first construction pipeline mines candidate trails from document structure before writing questions; each item then undergoes multi-stage validation covering clue necessity, answer groundedness, rubric fidelity, contamination resistance and answerability. As models are increasingly entrusted with real-world high-stakes analytical tasks, this gap between accessing information and reasoning over naturally dispersed evidence emerges as a defining challenge for the next stage of long-context research.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.09328 [cs.CL] |
| (or arXiv:2607.09328v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.09328 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zixin Chen [view email]
[v1]
Fri, 10 Jul 2026 12:09:21 UTC (4,134 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.