Retrieval-Augmented Large Language Models for Schema-Constrained Clinical Information Extraction
Quick Answer
The study presents a retrieval-augmented generation (RAG) pipeline for extracting clinical observations from nurse-patient transcripts, achieving an 80.36% F1 score using GPT-5.2.
Quick Take
The method incorporates schema-constrained prompting and second-pass auditing, demonstrating that enhances performance while optimal schema constraints vary by model.
Key Points
- Utilizes a modular RAG pipeline with Llama-4-Scout-17B-16E-Instruct and GPT-5.2.
- Achieved an 80.36% F1 score with full schema and second-pass auditing.
- RAG consistently improves performance in clinical information extraction.
- Optimal schema constraints depend on the specific model used.
- Second-pass auditing provides modest gains by correcting schema errors.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Conversational nurse-patient transcripts contain actionable observations, but converting these transcripts into structured representations at scale remains challenging. Documentation burden is substantial, with prior studies showing clinicians spend large portions of their workday on documentation and related desk work rather than direct patient care. MEDIQA-SYNUR focuses on observation extraction from conversational nurse-patient transcripts, requiring systems to normalize these narratives into a predefined schema with value-type constraints. We propose a modular retrieval-augmented generation (RAG) pipeline that uses the training set as an exemplar corpus, combines schema-constrained prompting (full schema vs. pruned candidate schema), deterministic schema-based postprocessing, and a second-pass audit, with two LLM backbones: Llama-4-Scout-17B-16E-Instruct and GPT-5.2 with corresponding embedding models for RAG. Our best configuration uses GPT-5.2 with full schema, RAG, and a second-pass auditing, achieving 80.36% F1 score. Overall, our results show that RAG consistently improves performance, while the optimal degree of schema constraint depends on the model, and second-pass auditing yields modest additional gains by correcting residual schema-adherence errors.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.15467 [cs.CL] |
| (or arXiv:2605.15467v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15467 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: A H M Rezaul Karim [view email]
[v1]
Thu, 14 May 2026 23:13:05 UTC (555 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.