PM-Bench: Evaluating Prospective Memory in LLM Agents
Quick Answer
PM-Bench is a new benchmark designed to evaluate prospective memory in LLM agents, revealing that even the best model, GPT-5.4, achieves only a 65.1% F1 score.
Quick Take
PM-Bench is a new benchmark designed to evaluate prospective memory in LLM agents, revealing that even the best model, GPT-5.4, achieves only a 65.1% F1 score. The benchmark assesses how well agents maintain and execute user intentions over a simulated week, highlighting the challenges in improving prospective memory across different models. This tool aims to help diagnose failures and develop interventions for reliable prospective behavior.
Key Points
- PM-Bench evaluates LLM agents' ability to maintain and execute intentions.
- The benchmark simulates a seven-day week with ongoing activities.
- GPT-5.4 achieved the highest F1 score of 65.1% among eight models.
- No single strategy effectively improves prospective memory across all models.
- PM-Bench serves as a testbed for diagnosing failures and developing interventions.
Paper Resources
📖 Reader Mode
~2 min readAbstract:A significant challenge in agentic AI is prospective memory: the ability to execute an intention at a specific future cue or state while other activities are ongoing. We introduce PM-Bench, a text-based benchmark for measuring prospective memory capabilities in modern LLM agents. Inspired by the Virtual Week paradigm from cognitive science, PM-Bench evaluates how well LLM agents maintain user intentions, execute delayed intentions, and monitor latent environment changes. Over the course of a simulated seven-day week, agents must continue an ongoing activity while deciding whether any deferred task is due. We compare eight state-of-the-art LLMs on PM-Bench under eight different agent configurations. PM-Bench proves challenging across all settings: the best method, a GPT-5.4 agent, reaches only 65.1\% F1 score under our evaluation. Furthermore, no single strategy for improving prospective memory dominates across models. We release PM-Bench as a controlled testbed for diagnosing these failures and developing training or inference-time interventions that support reliable prospective behavior.
| Comments: | Published as a conference paper at COLM 2026 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.12385 [cs.AI] |
| (or arXiv:2607.12385v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.12385 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Genglin Liu [view email]
[v1]
Tue, 14 Jul 2026 05:57:32 UTC (10,678 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Emerging Paradigm of Geospatial Foundation Models: From Pre-Training to Agentic Reasoning
The paper introduces Geospatial Foundation Models (GeoFMs), AI/ML models pre-trained on extensive geospatial datasets, enabling domain experts to fine-tune them for specific tasks. This paradigm shift democratizes access to advanced AI/ML while ensuring security, and proposes a framework for cost-effective adaptation strategies. The vision of Agentic Geospatial Reasoning is also presented, where Large Language Models orchestrate GeoFMs to automate complex analytical workflows.