A Four-Condition Diagnostic Protocol for Evidence Utilization in Long-Context and Retrieval-Augmented Language Models
Quick Answer
This paper introduces a four-condition diagnostic protocol for evaluating evidence utilization in long-context and retrieval-augmented language models, revealing that failures differ by task type.
Quick Take
The study assesses models from Qwen, Gemma, Llama, and Mistral across various benchmarks, highlighting that controlled settings expose full-context failures while realistic settings reveal retrieval-chain issues.
Key Points
- Proposes a four-condition protocol: no evidence, full context, retrieved evidence, oracle evidence.
- Evaluates five models from Qwen, Gemma, Llama, and Mistral across 18,000 predictions.
- Finds controlled settings expose full-context utilization failures.
- Realistic multi-hop settings reveal retrieval-chain coverage failures.
- Focuses on separating different types of evidence utilization rather than a single-score leaderboard.
Paper Resources
Source Excerpt
arXiv:2606. 06758v1 Announce Type: new Abstract: Final-answer accuracy, retrieval recall, and citation overlap do not by themselves identify whether a long-context or retrieval-augmented language model used the evidence it was given. A model can answer from parametric memory, fail despite receiving the right passages, or cite evidence without converting it into the requested answer.
This paper proposes a matched four-condition evidence-availability protocol--no evidence, full context, retrieved evidence, and oracle-evidence reference--for diagnosing evidence utilization under fixed examples, prompts, score fields, retrieval settings, and validity checks. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.