Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions
Quick Answer
The POLAR framework enhances personalized assistance in embodied multimodal large language model agents by organizing user interactions into a multimodal knowledge graph, improving task execution through memory retrieval.
Quick Take
The POLAR framework enhances personalized assistance in embodied multimodal large language model agents by organizing user interactions into a multimodal knowledge graph, improving task execution through memory retrieval. Evaluation shows significant performance gains, particularly in reasoning and context tracking across interactions.
Key Points
- POLAR organizes prior interactions into a multimodal knowledge graph for personalized context.
- The framework uses semantic and episodic memory to enhance task execution.
- Evaluation across multiple MLLM backbones shows consistent performance improvements.
- Significant gains are noted in multi-hop inference and user-specific context tracking.
- Memory retrieval is crucial for interpreting requests and guiding embodied tasks.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 26256v1 Announce Type: new Abstract: Multimodal large language model (MLLM)-based embodied agents have shown strong potential for solving complex tasks in physical environments. However, personalized assistance requires more than following generic instruction or recognizing object categories. In real-world scenarios, the intended target is often specified only implicitly through prior interactions, requiring agents to leverage personalized context accumulated over time.
In this work, we propose POLAR, a multiomodal memory-augmented framework for personalized embodied agents over long-term user interactions. POLAR organizes prior interactions into a multimodal knowledge graph that captures semantic memory for personalized context and visual concepts, and episodic memory for embodied experiences such as agent trajectories. To execute embodied tasks, POLAR retrieves relevant memories to interpret the current request and guide task execution.
We evaluate POLAR across multiple MLLM backbones and diverse evaluation scenarios to study the role of memory in long-term personalization. Results show that the proposed memory mechanism consistently improves performance by enabling more effective use of information accumulated over prior interactions. The gains are especially pronounced when the agents are required to reason across multiple interactions, perform multi-hop inference, or tracking updates in user-specific context over time.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Adversarial Social Epistemology for Assemblies of Humans and Large Language Models
The paper introduces Adversarial Social Epistemology (ASE) to analyze how agents manipulate trust in public communications, highlighting mechanisms that undermine the reliability of testimony and inference. It critiques existing frameworks like epistemic bubbles and misinformation diffusion, proposing a new language for understanding trust breaches and auditing inferential chains in densely interactive environments involving humans and large language models.