WILDTRACE: Benchmarking Natural Evidence Trails in Long-Context Reasoning
Quick Answer
WILDTRACE introduces a benchmark of 481 tasks across 214 long-form sources, focusing on integrating source-internal evidence for complex reasoning in long documents.
Quick Take
This benchmark addresses the challenge of distinguishing genuine reasoning from artifacts, crucial for high-stakes analytical tasks in AI.
Key Points
- WILDTRACE benchmarks 481 tasks from 214 long-form sources like incident reports and narratives.
- It defines seven evidence geometries for analytical reading in long documents.
- The benchmark emphasizes source-internal evidence integration over traditional methods.
- Multi-stage validation ensures the quality of evidence trails and question relevance.
- This work highlights a critical gap in long-context reasoning for AI models.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Answering complex questions over long documents frequently requires integrating evidence that the source itself disperses naturally across distant passages. In an incident report, the operating condition, design flaw, and missed safety check that jointly explain a disaster may appear dozens of sections apart; in a novel, a character's true motive may surface only through scenes far removed from the moment it becomes relevant. This source-internal evidence integration is central to real-world long-document analysis, yet existing benchmarks largely sidestep it. Needle probes, planted facts, and reverse-engineered multi-hop chains embed evidence that may differ from the host text in distribution, placement, or register, making it unclear whether strong performance reflects genuine source reasoning or distributional artifacts. We introduce WILDTRACE, a benchmark of 481 tasks over 214 naturally occurring long-form sources such as technical incident reports and lesser-known literary narratives, where all evidence trails arise from the document's own causal, temporal, and narrative logic. Drawing on Pearl's causal hierarchy and prior multi-hop reasoning typologies, we define seven source-internal evidence geometries that characterize the distinct relational demands of analytical reading in long documents. A source-first construction pipeline mines candidate trails from document structure before writing questions; each item then undergoes multi-stage validation covering clue necessity, answer groundedness, rubric fidelity, contamination resistance and answerability. As models are increasingly entrusted with real-world high-stakes analytical tasks, this gap between accessing information and reasoning over naturally dispersed evidence emerges as a defining challenge for the next stage of long-context research.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.09328 [cs.CL] |
| (or arXiv:2607.09328v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.09328 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zixin Chen [view email]
[v1]
Fri, 10 Jul 2026 12:09:21 UTC (4,134 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.