MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale
Quick Answer
MemArena introduces a benchmark for on-device personal memory assistants, evaluating five open-weight models.
Quick Take
Key findings reveal that memory-backend choice significantly impacts content accuracy, with Memobase outperforming MemSearch by +32.5 pp. Additionally, permission-aware access universally fails, and search latency is manageable on edge devices.
Key Points
- MemArena benchmark uses 50 agents over 15 days, generating 10.3M dialog-text tokens.
- Memobase outperforms MemSearch with a +32.5 pp gain in content accuracy.
- Oracle retrieval leaks heavily, while other backends are overly cautious.
- Search latency is moderate, adding 87/7/48 ms on Spark GB10 edge node.
- Code and MASim simulator will be released upon acceptance.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Edge-deployed personal memory assistants must handle private interpersonal conversations on-device with open-weight models. Yet, existing memory benchmarks often under-test the combination of activity-dense interaction, ego-centric perspective, and coherent multi-session worlds. MemArena fills these gaps with a single-world conversational benchmark built with its MASim agent simulator, for 50 agents over 15 days (10.3M dialog-text tokens, 24.1K text-only ego-observed tokens/agent/day). With the interaction history, it co-generates ground truth over six recall, reasoning, and trustworthiness evaluation dimensions. We evaluate five open-weight readers with Vanilla context, BM25-RAG, Oracle retrieval, Memobase, and MemSearch as memory backends. Three results stand out: (1) Memory-backend choice matters more for content accuracy: At Qwen3-0.6B, Memobase-to-MemSearch gains +32.5/+19.2 pp, exceeding MemSearch reader scaling (+10.6/+6.8 pp). (2) Permission-aware access fails universally, with Oracle leaking heavily and other backends too timid to disclose. (3) Search latency bites only at very small reader: on a Spark GB10 edge node, memory-search adds a moderate and fixed 87/7/48 ms (BM25-RAG/Memobase/MemSearch) that composes a small part of TTFT for most reader-backend combinations. Code, the MASim simulator, and the MemArena-L benchmark will be released upon acceptance.
| Comments: | 48 pages, 6 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2608.02613 [cs.CL] |
| (or arXiv:2608.02613v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.02613 arXiv-issued DOI via DataCite |
Submission history
From: Jiadong Zhang [view email]
[v1]
Wed, 20 May 2026 17:07:32 UTC (1,663 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.