LongMedBench: Benchmarking Medical Agents for Long-Horizon Clinical Decision-Making
Quick Answer
LongMedBench introduces a benchmark for evaluating long-horizon clinical decision-making in medical agents, utilizing EHR data from 335 patients.
Quick Take
LongMedBench introduces a benchmark for evaluating long-horizon clinical decision-making in medical agents, utilizing EHR data from 335 patients. It assesses agents through a taxonomy of fact-based QA, temporal reasoning, and decision-making, revealing challenges in implicit time inference despite recent LLM advancements.
Key Points
- LongMedBench integrates MIMIC-IV data for long-horizon clinical interactions.
- The benchmark includes 335 patients with an average of 19.72 visits each.
- Evaluation taxonomy consists of fact-based QA, temporal reasoning, and decision-making.
- Recent LLMs excel with explicit timestamps but struggle with implicit time inference.
- systems enhance information retrieval but decision-making performance varies.
Paper Resources
📖 Reader Mode
~2 min readAbstract:In this work, we introduce LongMedBench, a real-world EHR-based benchmark for long-horizon clinical decision-making. Prior evaluations of LLM-based medical agents have largely emphasized short-context knowledge QA and tool use. However, real-world medical care is inherently longitudinal, and clinicians must aggregate evidence across repeated visits, tests, and evolving treatments. Therefore, long-horizon interaction is essential for realistic assessment. LongMedBench is constructed via a reproducible pipeline that integrates MIMIC-IV admission records and clinical notes into time-series event streams and long-context memory datasets, enabling long-horizon, multi-session interactions between agents and a clinical environment. It comprises 335 patients, with 19.72 inpatient visits per patient on average and 44.91 medical events per visit. Guided by the long-horizon decision process, we propose an evaluation taxonomy with three suites: fact-based QA, temporal reasoning, and long-horizon decision-making. This taxonomy measures how agents understand and leverage historical patient information over extended horizons. Our experiments show that while recent LLMs can make good use of explicit timestamps, they have challenges in implicit time inference; The RAG and agent memory system can improve the performance of information retrieval tasks, but the performance of decision-making tasks is highly dependent on the model's immediate context.
| Comments: | Submitted manuscript prior to peer review in MICCAI 2026 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.09322 [cs.AI] |
| (or arXiv:2607.09322v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.09322 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zihan Xu [view email]
[v1]
Fri, 10 Jul 2026 12:04:31 UTC (3,344 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Adversarial Social Epistemology for Assemblies of Humans and Large Language Models
The paper introduces Adversarial Social Epistemology (ASE) to analyze how agents manipulate trust in public communications, highlighting mechanisms that undermine the reliability of testimony and inference. It critiques existing frameworks like epistemic bubbles and misinformation diffusion, proposing a new language for understanding trust breaches and auditing inferential chains in densely interactive environments involving humans and large language models.