LayerRAG-Bench: A Cross-Layer Reliability Benchmark for Agentic Retrieval-Augmented Generation
Quick Answer
LayerRAG-Bench introduces a cross-layer reliability benchmark for agentic retrieval-augmented generation systems, comprising 240 tasks across 8 domains and 9 models from OpenAI, Anthropic, and Gemini.
Quick Take
The study reveals that schema normalization significantly improves schema-drift success but fails to address issues like stale evidence and wrong-session context, underscoring the need for layer-specific evaluation in reliability interventions.
Key Points
- LayerRAG-Bench includes 240 tasks and 9 fault scenarios across 8 enterprise domains.
- Schema normalization improved schema-drift success from 0.000 to 0.913.
- Issues like stale evidence and wrong-session context remain unaddressed by schema normalization.
- Groundedness-only evaluation leads to significant false positives under specific conditions.
- The study advocates for layer-specific evaluations in reliability interventions.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Musa Shams (Independent Researcher)
Abstract:Agentic retrieval-augmented generation systems can produce answers that appear grounded while failing at the evidence, tool-contract, authorization, or session-state layer. We introduce LayerRAG-Bench, a controlled cross-layer reliability benchmark with 8 enterprise domains, 240 tasks, 9 fault scenarios, 2 contract modes, and 38,880 live task-level records across nine models from OpenAI, Anthropic, and Gemini. Schema normalization raises schema-drift success from 0.000 to 0.913, but stale evidence, missing tool output, denied permissions, and wrong-session context are not recovered by schema normalization. Groundedness-only evaluation also produces substantial false positives under stale and wrong-session evidence. These results support a layer-specific evaluation principle: a reliability intervention should be credited for repairing its target layer without being mistaken for a universal fix.
| Comments: | 10 pages, 9 tables. Code and data: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.27353 [cs.CL] |
| (or arXiv:2607.27353v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.27353 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Musa Shams [view email]
[v1]
Wed, 29 Jul 2026 18:09:17 UTC (28 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.