MemTrace: Probing What Final Accuracy Misses in Long-Term Memory
Quick Answer
MemTrace introduces a new benchmark for evaluating long-term memory in LLMs by focusing on knowledge points rather than question rows.
Quick Take
MemTrace introduces a new benchmark for evaluating long-term memory in LLMs by focusing on knowledge points rather than question rows. It reveals that pooled accuracy can obscure failures in evidence use, highlighting the need for improved evidence utilization over mere retrieval enhancements.
Key Points
- MemTrace measures memory across three dimensions: age, question type, and evidence condition.
- Evaluated 13 memory-system configurations, revealing different failures despite similar accuracy.
- Evidence use is the main bottleneck, with retrievable evidence failing 10 times more often than missing.
- Improving long-term memory requires better evidence utilization, not just increased storage.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 17328v1 Announce Type: new Abstract: LLM agents increasingly maintain long-term memory of user facts across sessions. Yet such memory is usually evaluated by aggregating accuracy over question rows or episodes. Because this approach scores question rows independently, even when several questions probe the same fact, it cannot show how that fact behaves as conditions change.
We introduce MemTrace, a benchmark whose unit of measurement is the knowledge point: a single typed fact about the user, rather than an individual question. MemTrace probes each fact along three controlled dimensions: memory age, defined by how many sessions ago the fact appeared in the history; question type, covering current state, earlier state, and trajectory of change; and evidence condition, covering present, missing, and contradicted-by-false-premise settings.
Evaluating 13 memory-system configurations across four paradigms, we find that similar pooled accuracy hides different failures: recovering a fact's current and earlier states does not imply tracking how it changed, and safe abstention does not imply correcting a false premise. The dominant bottleneck is evidence use, not retrieval: when systems fail, the evidence was retrievable 10 times more often than it was missing.
These results suggest that improving long-term memory requires better use of reachable evidence, not simply more storage or retrieval.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Arbor: Tree Search as a Cognition Layer for Autonomous Agents
Arbor introduces a multi-agent framework utilizing structured tree search for optimizing LLM inference, achieving up to 193% throughput-latency improvement compared to vendor-optimized systems. It employs an Orchestrator and Critic agent for stability and coordination, demonstrating hardware-agnostic performance with minimal variance.