Evidence Graph Consistency in Retrieval-Augmented Generation: A Model-Dependent Analysis of Hallucination Detection
Quick Answer
This paper shows that The Evidence Graph Consistency (EGC) framework identifies hallucination patterns in Retrieval-Augmented Generation (RAG) models, revealing a model-family split: while Llama-2 shows expected graph consistency with hallucinations, GPT-4, GPT-3.5, and Mistral-7B exhibit a reversal, indicating distinct hallucination characteristics across models.
Quick Take
This suggests that embedding-based consistency cannot serve as a universal detection method.
Key Points
- EGC computes five structural consistency measures as hallucination indicators.
- Evaluated on RAGTruth with 5,767 responses across six .
- Llama-2 models show expected graph consistency; others do not.
- Reversal in GPT-4 and GPT-3.5 indicates different hallucination patterns.
- Embedding-based graph consistency is not a model-independent detection signal.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Retrieval-Augmented Generation (RAG) reduces but does not eliminate hallucination in large language models. Existing detection methods rely on flat similarity between generated answers and retrieved passages, ignoring structural relationships among evidence pieces and answer claims. We propose Evidence Graph Consistency (EGC), a framework that constructs a local evidence graph per response and computes five structural consistency measures as hallucination indicators. Evaluated on the full question answering split of RAGTruth across six LLMs (5,767 responses), EGC reveals a consistent model-family split: graph consistency features show the expected diagnostic direction for hallucinations in Llama-2 models but exhibit systematic reversal in GPT-4, GPT-3.5, and Mistral-7B. This reversal suggests qualitatively different hallucination patterns across model families and indicates that embedding-based graph consistency cannot serve as a model-independent hallucination detection signal.
| Comments: | Accepted at the International Conference on Advanced Machine Learning and Data Science; to appear in the IEEE Xplore proceedings |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.06748 [cs.CL] |
| (or arXiv:2606.06748v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.06748 arXiv-issued DOI via DataCite |
Submission history
From: Jianru Shen [view email]
[v1]
Thu, 4 Jun 2026 22:19:39 UTC (361 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.