PM-Bench: Evaluating Prospective Memory in LLM Agents
Quick Answer
PM-Bench is a new benchmark designed to evaluate prospective memory in LLM agents, revealing that even the best model, GPT-5.4, achieves only a 65.1% F1 score.
Quick Take
The benchmark assesses how well agents maintain and execute user intentions over a simulated week, highlighting the challenges in improving prospective memory across different models. This tool aims to help diagnose failures and develop interventions for reliable prospective behavior.
Key Points
- PM-Bench evaluates agents' ability to maintain and execute intentions.
- The benchmark simulates a seven-day week with ongoing activities.
- GPT-5.4 achieved the highest F1 score of 65.1% among eight models.
- No single strategy effectively improves prospective memory across all models.
- PM-Bench serves as a testbed for diagnosing failures and developing interventions.
Paper Resources
📖 Reader Mode
~2 min readAbstract:A significant challenge in agentic AI is prospective memory: the ability to execute an intention at a specific future cue or state while other activities are ongoing. We introduce PM-Bench, a text-based benchmark for measuring prospective memory capabilities in modern LLM agents. Inspired by the Virtual Week paradigm from cognitive science, PM-Bench evaluates how well LLM agents maintain user intentions, execute delayed intentions, and monitor latent environment changes. Over the course of a simulated seven-day week, agents must continue an ongoing activity while deciding whether any deferred task is due. We compare eight state-of-the-art LLMs on PM-Bench under eight different agent configurations. PM-Bench proves challenging across all settings: the best method, a GPT-5.4 agent, reaches only 65.1\% F1 score under our evaluation. Furthermore, no single strategy for improving prospective memory dominates across models. We release PM-Bench as a controlled testbed for diagnosing these failures and developing training or inference-time interventions that support reliable prospective behavior.
| Comments: | Published as a conference paper at COLM 2026 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.12385 [cs.AI] |
| (or arXiv:2607.12385v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.12385 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Genglin Liu [view email]
[v1]
Tue, 14 Jul 2026 05:57:32 UTC (10,678 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.