EarlyDx: An Admission-Anchored Benchmark for Open-Ended Generation of Evidence-Supported ED-Encounter Diagnoses
Quick Answer
EarlyDx introduces a new benchmark for emergency department diagnoses using 154,834 encounters from MIMIC-IV, focusing on rapid, admission-time evidence rather than discharge diagnoses.
Quick Take
Current models struggle with inference, achieving only 3-31% recall without training, and even post-training only reach 56%, highlighting significant gaps in performance.
Key Points
- EarlyDx benchmark uses 154,834 emergency encounters for diagnosis prediction.
- Focuses on admission-time evidence, unlike traditional discharge-based benchmarks.
- Zero-shot models recover only 3-31% of necessary diagnoses.
- Post-training improves inference-dependent recall to 56%, but gaps remain.
- No evaluated system matches clinician sensitivity and precision in time-critical conditions.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Clinical diagnosis at hospital admission must be made rapidly from limited, incomplete evidence. Existing diagnosis-prediction benchmarks are poorly suited to this setting: they restrict prediction to closed code sets, exclude free-text notes, and supervise with discharge diagnoses that incorporate the full inpatient course. We introduce EarlyDx, a large-scale benchmark for open-ended early diagnosis, built from 154,834 emergency department encounters in MIMIC-IV. Each encounter is restricted to records available at admission time $t_0$ and supervised by the diagnoses recorded during the ED encounter rather than at discharge. An LLM auditor further verifies every free-text label as supported, partially supported, or unsupported by that evidence; the primary evaluation scores only fully supported labels. Under a semantic LLM-as-judge protocol, no evaluated system --- frontier general, medical-specialized, or in-domain post-trained --- synthesizes admission-time evidence reliably. Zero-shot models score largely by extraction, recovering only 3-31% of diagnoses that must be inferred rather than read from the record; post-training raises inference-dependent recall to 56%, but a sizeable margin remains, and on time-critical conditions no system attains a clinician's balance of sensitivity and precision. We release the full construction and evaluation pipeline at here.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.28788 [cs.AI] |
| (or arXiv:2607.28788v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.28788 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiahui Li [view email]
[v1]
Thu, 30 Jul 2026 19:27:36 UTC (2,420 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.