When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering
Quick Answer
The OGCaReBench benchmark evaluates LLMs like GPT-5.2 on clinical questions beyond guidelines, achieving a 56% accuracy, improved to 82% with retrieved articles.
Quick Take
This highlights the need for evidence-based reasoning in rare medical scenarios.
Key Points
- OGCaReBench focuses on free-form clinical question answering beyond standard guidelines.
- GPT-5.2 achieves 56% accuracy on the benchmark, with specialized models at 42%.
- Retrieving medical articles boosts performance to 82% for GPT-5.2.
- The benchmark is derived from validated medical case reports.
- This work aims to enhance reliability in challenging clinical contexts.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Doeun Lee, Muge Zhang, Yi Yu, Ashish Manne, Stephen Koesters, Frank Wen, Brady Buchanan, Lynda Villagomez, Oluwatoba Moninuola, James Lim, Kathryn Tobin, Andrew Srisuwananukorn, Ping Zhang, Sachin Kumar
Abstract:Across medical specialties, clinical practice is anchored in evidence-based guidelines that codify best studied diagnostic and treatment pathways. These pathways routinely fall short for the long tail of real-world care not covered by guidelines. Most medical large language models (LLMs), however, are trained to encode common, guideline-focused medical knowledge in their parameters. Current evaluations test models primarily on recalling and reasoning with this memorized content, often in multiple-choice settings. Given the fundamental importance of evidence-based reasoning in medicine, it is neither feasible nor reliable to depend on memorization in practice. To address this gap, we introduce OGCaReBench, a free-form retrieval-focused benchmark aimed at evaluating LLMs at answering clinical questions that require going beyond typical guidelines. Extracted from published medical case reports and validated by medical experts, OGCaReBench contains long-form clinical questions requiring free-text answers, providing a systematic framework for assessing open-ended medical reasoning in rare, case-based scenarios. Our experiments reveal that even the best-performing baseline (GPT-5.2) correctly answers only 56% of our benchmark with specialized models only reaching 42%. Augmenting models with retrieved medical articles improves this performance to up to 82% (using GPT-5.2) highlighting the importance of evidence-grounding for real-world medical reasoning tasks. This work thus establishes a foundation for benchmarking and advancing both general-purpose and medical LLMs to produce reliable answers in challenging clinical contexts.
| Comments: | 34 pages, 20 figures |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.21807 [cs.CL] |
| (or arXiv:2605.21807v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.21807 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Doeun Lee [view email]
[v1]
Wed, 20 May 2026 23:04:48 UTC (1,142 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.