Memory-Conditioned Tool Calling for Camera-First Visual Agents
Quick Answer
This study investigates the impact of a three-layer personal visual memory on tool selection in camera-first visual agents.
Quick Take
This study investigates the impact of a three-layer personal visual memory on tool selection in camera-first visual agents. Results show that removing this memory reduces tool-query relevance by 11.2% and overall utility by 9.7%, highlighting the importance of memory conditioning in enhancing user-aligned multi-tool lookups.
Key Points
- The study uses 800 images paired with synthetic memory blocks for controlled ablation.
- Removing the full three-layer memory block reduces tool-query relevance by 0.47 points.
- End-to-end utility decreased by 0.082 points after memory removal.
- The design includes a conflict-aware write-back to refresh user models.
- Focus is on image-only intake without multi-session write-back from live histories.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Recognition tells an agent what is in an image; personal memory affects what is worth looking up next. In a camera-first setting the user can send only an image, so the agent must form the lookups. We study whether personal visual memory improves agent-side tool choice and tool arguments, and thereby more user-aligned multi-tool lookups. The design uses a three-layer personal visual memory (profile, short-term focus, observations) that is loaded on each turn to condition an LLM tool-calling loop under camera-first intake, and includes conflict-aware write-back intended to refresh the user model for later captures. On 800 images paired with synthetic memory blocks constructed for controlled ablation, removing the full three-layer memory block reduces tool-query relevance by 0.47 points absolute (4.21 -> 3.74 on a 5-point scale; 11.2% relative) and end-to-end utility by 0.082 absolute (0.842 -> 0.760; 9.7% relative). These results measure memory conditioning of tool policy under image-only intake with fixed synthetic blocks, not multi-session write-back from live user histories.
| Comments: | 13 pages, 3 figures, 4 tables. Equal contribution: Xiaofan Wu, Xi Zeng. Corresponding author: xiaofan@chance.vision |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Human-Computer Interaction (cs.HC); Information Retrieval (cs.IR) |
| ACM classes: | I.2.10; I.2.7; I.2.11; H.3.3 |
| Cite as: | arXiv:2607.09822 [cs.CV] |
| (or arXiv:2607.09822v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2607.09822 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiaofan Wu [view email]
[v1]
Fri, 10 Jul 2026 08:40:33 UTC (16 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.