MMShopBench: A Real-Log Benchmark for Multimodal, Multi-Turn Shopping Agents
Quick Answer
MMShopBench is a new benchmark for multimodal, multi-turn shopping agents, addressing the limitations of existing text-only benchmarks.
Quick Take
It features real shopping logs with annotated purchase intents, enabling agents to infer requirements from images and dialogues. Initial evaluations show that fine-tuning open-source models can significantly reduce performance gaps with proprietary models.
Key Points
- MMShopBench is the first real-log benchmark for multimodal shopping agents.
- It includes manually annotated shopping logs with purchase intent and product requirements.
- Agents must use images and dialogues to retrieve and verify candidate products.
- Fine-tuning open-source models can significantly improve performance against proprietary models.
- An offline shopping sandbox ensures reproducibility of experiments.
DeepSignal Analysis
What happened
MMShopBench is introduced as a benchmark for multimodal, multi-turn shopping agents, utilizing real shopping logs with annotated purchase intents. This benchmark aims to address the limitations of existing text-only benchmarks, allowing agents to infer requirements from both images and dialogues. Initial evaluations indicate that fine-tuning open-source models can significantly reduce performance gaps with proprietary models.
Key evidence
- MMShopBench is built from cleaned and annotated shopping logs, providing ground-truth annotations for purchase intents and product requirements.
- The benchmark requires agents to infer requirements from user images and multi-turn dialogue, and to verify candidate products against these requirements.
- Initial evaluations show that fine-tuning open-source models can substantially narrow the performance gap with leading proprietary models.
Why it matters
The introduction of MMShopBench is significant as it fills a gap in existing benchmarks that primarily focus on text-only interactions. By incorporating real-world shopping scenarios that involve both images and dialogue, it enables more accurate training and evaluation of shopping agents. This could lead to improved AI shopping assistants that better understand and fulfill complex user needs, ultimately enhancing the online shopping experience.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Online shoppers increasingly turn to AI shopping assistants, using images and multi-turn dialogue to express and refine product needs that are difficult to articulate in text alone. However, existing benchmarks largely rely on text-only or synthetic requests, underrepresenting complex real-world shopping requirements jointly expressed through images and language. We introduce MMShopBench, the first real-log benchmark for multimodal, multi-turn shopping agents. Built from carefully cleaned and manually annotated shopping logs, MMShopBench provides ground-truth annotations of each request's purchase intent and mandatory product requirements. Agents must infer these requirements jointly from user images and multi-turn dialogue, retrieve candidate products through image and text search, and verify that each candidate satisfies all requirements using its product images and structured attributes. We evaluate representative open-source and proprietary models using an evidence-grounded multimodal protocol and construct a companion training set for fine-tuning an open-source model. To ensure reproducible experimentation, we build an offline shopping sandbox, where fine-tuning substantially narrows the performance gap between our open-source model and leading proprietary models, demonstrating the effectiveness of our training data.
| Comments: | 16 pages, 6 figures, including appendix |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.29002 [cs.AI] |
| (or arXiv:2607.29002v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.29002 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ze-Ying Hao [view email]
[v1]
Fri, 31 Jul 2026 04:00:15 UTC (4,480 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.