Toward Automated Detection of Documentation Inconsistencies in Electronic Health Records
Quick Answer
This study employs a two-stage LLM pipeline (Gemini 2.5 Pro and Gemini 2.5 Flash) to identify 3,460 documentation inconsistencies in 3,000 MIMIC-IV-Note discharge summaries, affecting 69.7% of admissions.
Quick Take
The findings highlight the context-dependent nature of inconsistency detection and propose a graded ontology for better classification.
Key Points
- 3,460 inconsistencies identified, impacting 69.7% of discharge summaries.
- Inconsistencies span demographics, allergies, procedures, and more.
- Expert review revealed limitations in temporal reasoning and outpatient knowledge.
- Proposed a graded ontology for categorizing inconsistencies.
- Study lays groundwork for future large-scale EHR analysis.
DeepSignal Analysis
What happened
A study utilized a two-stage LLM pipeline to identify documentation inconsistencies in discharge summaries from the MIMIC-IV-Note dataset. The analysis revealed 3,460 inconsistencies across 3,000 summaries, impacting 69.7% of admissions. The findings indicate that inconsistency detection is context-dependent and propose a new ontology for classification.
Key evidence
- The study identified 3,460 candidate inconsistencies in 3,000 MIMIC-IV-Note discharge summaries, affecting 69.7% of admissions.
- The two-stage pipeline involved candidate identification using Gemini 2.5 Pro and context-grounded verification with Gemini 2.5 Flash.
- Expert review highlighted recurring failure modes, particularly in cases requiring temporal reasoning or knowledge of outpatient-prescribing conventions.
Why it matters
The findings underscore the challenges in reliably detecting inconsistencies in electronic health records (EHRs) using LLMs. The context-dependent nature of detection suggests that current models may struggle with certain types of inconsistencies, which could have implications for clinical decision-making and patient safety. Establishing a graded ontology for classification may enhance future analyses and improve the reliability of automated systems.
Paper Resources
Source Excerpt
Objective: To characterize the kinds of internal documentation inconsistencies a general-domain (LLM) can surface from real-world discharge summaries, and to identify recurring failure modes that limit reliability at scale. Materials and Methods: We applied a two-stage LLM pipeline---open-ended candidate identification (Gemini 2. 5 Pro) followed by context-grounded verification (Gemini 2. 5 Flash)---to 3,000 randomly sampled MIMIC-IV-Note discharge summaries. A subset of the
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.