Memory in the Loop: In-Process Retrieval as ExtendedWorking Memory for Language Agents
Quick Answer
This study reveals that integrating in-process memory retrieval significantly enhances language agent performance, reducing latency to ~100us compared to networked stores.
Quick Take
This study reveals that integrating in-process memory retrieval significantly enhances language agent performance, reducing latency to ~100us compared to networked stores. Across four GPT-5-class models, recall improved from 0/5 to 3.6-4.8/5, demonstrating that a fast memory store acts as extended working memory rather than a mere tool.
Key Points
- In-process memory retrieval reduces latency to ~100us, improving efficiency.
- Recall rates for GPT-5-class models increased from 0/5 to 3.6-4.8/5.
- The study highlights embedding as the main bottleneck, taking ~200-400ms over the network.
- Every memory write was successful, with no facts lost during runs.
- Pairing in-process storage with local embedding achieves ~40us operation time.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Language agents run a loop - observe, reason, act - but the memory they reason over sits outside it: a store queried at most once per turn. We study the regime where memory moves inside the loop, read and written on every step. The obstacle has always been latency: networked stores answer in tens to hundreds of milliseconds, and in-loop retrieval can inflate end-to-end latency by up to 83x when retrieval is expensive. Prior work manages that cost rather than questioning it: serving-layer scheduling hides it, "memory-first" designs ration retrieval to once per turn. We argue latency is a property of where the store lives, not the in-loop pattern: an in-process store answers in ~100us, three orders of magnitude below the network regime, and at that speed the per-step tax collapses. By the extended-mind thesis's parity principle, a store fast enough to be constantly and directly available becomes extended working memory, not a tool the agent merely consults. The premise is causal: holding a fixed per-turn memory-latency budget and varying only the store's answer speed, redundant actions rise monotonically with latency - 0.0 of 12 at in-process speed, 7.2 of 12 at a 110ms cloud round trip (gpt-5-nano, gpt-5-mini; exact permutation p=0.0079). We demonstrate the regime end-to-end: across four GPT-5-class models under a bounded window, recall improves from 0/5 to 3.6-4.8/5 with in-loop memory, store ops at p50 80-165us - though an instructed restate-every-reply baseline also solves it perfectly, at a token cost that grows with the working set. The store never lost a fact in any run (244 of 244 writes kept); every miss traces to the agent's read policy, not the store. Our measurements also relocate the bottleneck: the dominant per-step cost is embedding (~200-400ms over the network); pairing the in-process store with a small local embedder returns the complete operation to a measured ~40us.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.05690 [cs.AI] |
| (or arXiv:2607.05690v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.05690 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Carlo Lipizzi [view email]
[v1]
Mon, 6 Jul 2026 23:16:11 UTC (589 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Onnes: A Physics-Grounded LLM Simulator for Cryogenic Fault Diagnosis in Quantum Computing Infrastructure
Onnes is a physics-grounded digital twin simulator for dilution refrigerators, enhancing cryogenic fault diagnosis in quantum computing. It achieves a classification accuracy of 99.0% using few-shot demonstrations, matching a supervised ML classifier, while maintaining a low false alarm rate of 6.4% on real hardware.