STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair Oracle
Quick Answer
The STOCKTAKE benchmark evaluates LLM agents' performance in supply-chain tasks, revealing a knowing-doing gap where models like Claude Sonnet 5 and GPT-5.4 detect 84-88% of hidden failures but show skill scores ranging from 0.62 to -0.23.
Quick Take
Notably, 34-43% of diagnosed stress weeks still end in stockout, highlighting the dual nature of failure in these agents.
Key Points
- STOCKTAKE is a 26-week supply-chain replenishment benchmark for agents.
- Models detect 84-88% of hidden failures but vary in skill scores from 0.62 to -0.23.
- 34-43% of correctly diagnosed stress weeks still result in stockouts.
- The benchmark separates state estimation from control performance.
- Under-response and over-response are both identified as failure factors.
Paper Resources
📖 Reader Mode
~2 min readAbstract:LLM agents are increasingly evaluated on multi-week decision tasks in which the state that drives cost is never directly observed. On such tasks the final cost cannot say why an agent failed: it may have misread the world, or read it correctly and still failed to act (the knowing-doing gap). Existing evaluations cannot separate these two failures; their reference policies either read privileged information the agent never sees, or are missing altogether. We introduce STOCKTAKE, a 26-week supply-chain replenishment benchmark built as a factored partially observable Markov decision process with six hidden factor processes, designed so that a fair reference policy is computable: an exact Bayes filter per factor drives a rollout policy on the identical observation stream the agent receives. Scoring each run between a symptom-blind base-stock floor (0) and this oracle (1) yields a skill score, and grading each week's written rationale yields a stated-belief detection lag and a knowing-doing rate, so state estimation and control are measured separately. On fifty seeds with curated stress profiles, Claude Sonnet 5, GPT-5.4, DeepSeek-V4-Pro, and Grok 4.5 detect 84-88% of hidden failures, typically within a week of onset, yet span skill scores from 0.62 to -0.23: two of the four end below the symptom-blind floor while naming factors slightly faster than the two that beat it. The failure has two faces. Where stress persists, 34-43% of correctly diagnosed stress weeks still end in stockout for every model, a rate that partly reflects the severity of the weeks models notice. That rate also runs opposite to skill: the two models under the floor stock out least on diagnosed weeks, so under-response is only one face of the gap, and their traces point to the other, responses whose cost exceeds what they protect. STOCKTAKE measures both directions of that failure.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.13618 [cs.AI] |
| (or arXiv:2607.13618v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.13618 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sagar Deb [view email]
[v1]
Wed, 15 Jul 2026 09:08:27 UTC (82 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.