Deployment-Time Memorization in Foundation-Model Agents
Quick Answer
This paper shows that Foundation-model agents, such as Gemma 3 12B and GPT-4o-mini, utilize deployment-time memorization, balancing personalization recall and extraction risk.
Quick Take
Key findings reveal that summarization reduces adversarial extraction by up to 76%, but deletion fidelity suffers, with 20% recoverability of deleted data unless comprehensive purging is applied.
Key Points
- Summarization reduces adversarial extraction by 76% on Gemma 3 12B.
- Personalization recall remains nearly intact despite aggressive summarization.
- Increasing retrieval breadth (k) post-compression does not restore leakage.
- 20% of deleted information remains recoverable without full-pipeline purge.
- Forgetting Residue Score (FRS) quantifies recoverability of deleted data.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 10062v1 Announce Type: new Abstract: Foundation-model agents are increasingly long-lived systems that remember users across interactions, making memorization an explicit deployment-time function rather than solely a property of model weights. Existing work addresses parametric memorization or audits fixed memory configurations, but does not characterize how memory-design choices jointly shape personalization utility, extraction risk, and deletion fidelity.
We study this surface as deployment-time memorization, formulating as a privacy-utility frontier measured by Personalization Recall (PR) and Adversarial Extraction Rate (AER), and sweeping three memory-design knobs: summarization aggressiveness, retrieval breadth (k), and deletion mode. We further introduce the Forgetting Residue Score (FRS) to quantify whether deleted information remains recoverable from derived memory tiers.
On LongMemEval, key-fact summarization reduces canary extraction by 76% on Gemma 3 12B and 64% on GPT-4o-mini while preserving nearly all personalization recall; critically, once content is compressed away, increasing k no longer restores leakage. The same compression, however, induces a deletion-fidelity failure: raw-only deletion leaves derived summary copies recoverable in approximately 20% of instances, and only full-pipeline purge or tombstone redaction drives worst-tier residue to zero.
Together, these results establish that persistent agent memory must be evaluated as a first-class memorization mechanism -- assessed by what it helps agents recall, what it makes extractable, and what it can truly erase.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.