Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitations
Quick Answer
This study evaluates the relevance of various RAG metrics on a question-answering dataset, comparing them to human evaluators and standard metrics like recall.
Quick Take
This study evaluates the relevance of various metrics on a question-answering dataset, comparing them to human evaluators and standard metrics like recall. The findings reveal correlations and limitations in the methodology, suggesting directions for future research in RAG evaluation.
Key Points
- Evaluated RAG metrics using a dataset created by human annotators from business data.
- Compared metrics from Ragas, DeepEval, RAGChecker, and Opik against human scores.
- Conducted correlation analysis with standard metrics like recall.
- Highlighted limitations of the methodology compared to existing literature.
- Suggested future research avenues for improving RAG evaluation.
Paper Resources
📖 Reader Mode
~1 min readAbstract:This paper reports an empirical study evaluating the relevance of several RAG metrics. The experiment is based on a question-answering dataset created by human annotators from business data. The generated responses and retrieved spans of a RAG system are scored using evaluation metrics from four libraries (Ragas, DeepEval, RAGChecker, Opik). These metrics are compared to scores given by two evaluators, as well as to standard metrics such as recall. An analysis of correlations is conducted. Finally, we highlight certain limitations of our methodology, compare it to those used in the literature, and suggest some avenues for future research. This paper is an English translation of a paper originally published in the French-speaking workshop EvalLLM (Brabant, 2026).
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.07302 [cs.CL] |
| (or arXiv:2607.07302v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.07302 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Quentin Brabant [view email]
[v1]
Wed, 8 Jul 2026 11:44:28 UTC (16 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.