MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations
Quick Answer
MerchantBench introduces a 365-day simulation for evaluating LLM agents in e-commerce, revealing that even the best LLMs achieve only 27.3% of the net assets of human participants.
Quick Take
This benchmark highlights the critical need for long-term coherence in AI decision-making within complex environments.
Key Points
- MerchantBench simulates 365 days of e-commerce operations using 98,843 product records.
- Eight were evaluated across two frameworks, revealing significant performance gaps.
- The best LLM configuration achieved only 27.3% of human participants' final net assets.
- The benchmark emphasizes the importance of long-term coherence in AI decision-making.
- Agents must adapt to delayed feedback and interdependent decisions throughout the simulation.
DeepSignal Analysis
What happened
MerchantBench is a new benchmarking tool designed to evaluate large language model (LLM) agents in e-commerce over a simulated 365-day period. The study found that the best LLMs achieved only 27.3% of the net assets accumulated by human participants, indicating a significant performance gap.
Key evidence
- MerchantBench simulates a 365-day e-commerce environment using 98,843 real product records and 26 tools for agent interaction.
- The evaluation involved eight LLMs tested under two frameworks across 48 runs, each lasting 365 simulated days.
- The best-performing LLM configuration reached only 27.3% of the mean final net assets of human participants.
Why it matters
This research underscores the limitations of current LLMs in maintaining long-term coherence in decision-making, particularly in complex environments like e-commerce. The findings suggest that while LLMs can perform specific tasks, they struggle with sustained, coherent strategies over time, which is crucial for real-world applications.
What to watch
Paper Resources
📖 Reader Mode
~2 min readAuthors:Qiming Shi, Yulong Tao, Linbo Jin, Zhaolu Kang, Yibo Dou, Jiawen Zhu, Tianjun Pan, Shaokang Fu, Chengyu Wang, Siyue Li, Yaping Cheng, Di Weng, Chengfu Huo
Abstract:Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. Evaluating this capacity requires a persistent environment in which actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects. Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions over Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation. We introduce MerchantBench, a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. MerchantBench couples promptly observable Upstream Supplier Events with delayed Downstream Order Outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. We evaluate eight LLMs under two agent frameworks in 48 runs, each spanning 365 simulated days. Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3\% of the mean final net assets achieved by human participants.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.28956 [cs.AI] |
| (or arXiv:2607.28956v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.28956 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qiming Shi [view email]
[v1]
Fri, 31 Jul 2026 02:20:24 UTC (2,640 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.