MMShopBench: A Real-Log Benchmark for Multimodal, Multi-Turn Shopping Agents
Quick Answer
MMShopBench is a new benchmark for multimodal, multi-turn shopping agents, addressing the limitations of existing text-only benchmarks.
Quick Take
It features real shopping logs with annotated purchase intents, enabling agents to infer requirements from images and dialogues. Initial evaluations show that fine-tuning open-source models can significantly reduce performance gaps with proprietary models.
Key Points
- MMShopBench is the first real-log benchmark for multimodal shopping agents.
- It includes manually annotated shopping logs with purchase intent and product requirements.
- Agents must use images and dialogues to retrieve and verify candidate products.
- Fine-tuning open-source models can significantly improve performance against proprietary models.
- An offline shopping sandbox ensures reproducibility of experiments.
DeepSignal Analysis
What happened
MMShopBench is introduced as a benchmark for multimodal, multi-turn shopping agents, utilizing real shopping logs with annotated purchase intents. This benchmark aims to address the limitations of existing text-only benchmarks, allowing agents to infer requirements from both images and dialogues. Initial evaluations indicate that fine-tuning open-source models can significantly reduce performance gaps with proprietary models.
Key evidence
- MMShopBench is built from cleaned and annotated shopping logs, providing ground-truth annotations for purchase intents and product requirements.
- The benchmark requires agents to infer requirements from user images and multi-turn dialogue, and to verify candidate products against these requirements.
- Initial evaluations show that fine-tuning open-source models can substantially narrow the performance gap with leading proprietary models.
Why it matters
The introduction of MMShopBench is significant as it fills a gap in existing benchmarks that primarily focus on text-only interactions. By incorporating real-world shopping scenarios that involve both images and dialogue, it enables more accurate training and evaluation of shopping agents. This could lead to improved AI shopping assistants that better understand and fulfill complex user needs, ultimately enhancing the online shopping experience.
Paper Resources
Source Excerpt
Online shoppers increasingly turn to AI shopping assistants, using images and multi-turn dialogue to express and refine product needs that are difficult to articulate in text alone. However, existing benchmarks largely rely on text-only or synthetic requests, underrepresenting complex real-world shopping requirements jointly expressed through images and language. We introduce MMShopBench, the first real-log benchmark for multimodal, multi-turn shopping agents. Built from carefully cleaned and ma
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.