When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG
Quick Take
A large-scale study on biomedical question answering reveals that retrieval-augmented generation (RAG) models, ranging from 7B to 72B parameters, show only minor improvements (1-2 points) over no-retrieval baselines across various datasets. The backbone model choice significantly impacts performance more than retrieval methods or sources, indicating that the main challenge lies in the model's ability to effectively utilize retrieved evidence.
Key Points
- Study analyzes five models across ten biomedical QA datasets.
- Retrieval methods yield minimal improvements, typically 1-2 points.
- Model backbone choice has a greater impact than retrieval sources.
- Expert and layman retrieval sources perform similarly in most cases.
- Main bottleneck identified as model's ability to use retrieved evidence.
Article Excerpt
From source RSS / original summaryarXiv:2606. 04127v1 Announce Type: new Abstract: Medical question answering is a high-stakes setting where factual errors can have serious consequences. Retrieval-augmented generation (RAG) is widely viewed as a promising solution, and prior work has reported substantial gains for large medical QA models. We revisit this assumption across a broad range of open-weight instruction-tuned models spanning 7B to 72B parameters.
Across five models, ten biomedical QA datasets, four retrieval methods, and four retrieval corpora, we find that retrieval yields only small and inconsistent improvements over a no-retrieval baseline, typically within 1-2 points. In contrast, the choice of backbone model has a much larger effect than the choice of retriever or corpus, and expert and layman retrieval sources perform similarly in most settings.
These results suggest that the main bottleneck is not retrieval quality alone, but the model's limited ability to use retrieved evidence effectively.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The REFLECT benchmark reveals that current LLM judges are unreliable, achieving below 55% accuracy in evaluating reasoning and evidence use, highlighting the need for improved evaluation methods for deep research agents.