What Training Data Teaches RL Memory Agents: An Empirical Study of Curriculum Effects in Memory-Augmented QA
Quick Answer
This study reveals that the training curriculum significantly influences the specialization of reinforcement learning memory agents, with a mixed-benchmark approach yielding the highest F1 scores.
Quick Take
This study reveals that the training curriculum significantly influences the specialization of reinforcement learning memory agents, with a mixed-benchmark approach yielding the highest F1 scores. Notably, training on a narrow out-of-domain dataset enhances temporal reasoning skills, despite overall lower performance. Additionally, adapting GRPO for single-GPU setups highlights the need for filtering noise and using continuous reward functions.
Key Points
- Curriculum composition is a key factor in memory agent specialization.
- Mixed-benchmark training achieved the highest F1 scores across evaluations.
- Out-of-domain training improved targeted skills like temporal reasoning.
- Single-number benchmarks may underreport curriculum effects significantly.
- Adapting GRPO for single-GPU requires noise filtering and continuous rewards.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 23067v1 Announce Type: new Abstract: Reinforcement learning (RL) has emerged as a viable recipe for training LLM agents to reason over external memory banks in multi-session dialogue. Existing work trains exclusively on a single benchmark, leaving open how the composition of training data shapes the skills a memory agent acquires.
We present a controlled empirical study that holds architecture, RL algorithm, and all hyperparameters fixed and varies only the training curriculum across three conditions: in-domain (LoCoMo), mixed-benchmark (LoCoMo + LongMemEval), and out-of-domain (LongMemEval only). Across two benchmarks and ten question types, curriculum composition acts as a fine-grained lever on specialization rather than a uniform scaling factor on performance. The mixed curriculum yields the strongest overall F1 on both evaluation sets.
Training on a narrow out-of-domain set transfers a targeted skill - temporal reasoning - despite weak aggregate performance. Per-type differences substantially exceed aggregate differences, indicating that single-number benchmark comparisons systematically underreport curriculum effects.
We further report two practical lessons from adapting GRPO to a single-GPU regime: cross-benchmark mixing requires filtering format-specific noise from memory banks to preserve training signal, and binary exact-match reward produces no learning signal at the small group sizes (G = 4) required on one GPU, motivating continuous reward functions in this regime.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.
