STOCKTAKE: Measuring the Gap Between Perception and Action in LLM Agents with a Fair Oracle
Quick Answer
The STOCKTAKE benchmark evaluates LLM agents' performance in supply-chain tasks, revealing a knowing-doing gap where models like Claude Sonnet 5 and GPT-5.4 detect 84-88% of hidden failures but show skill scores ranging from 0.62 to -0.23.
Quick Take
The STOCKTAKE benchmark evaluates LLM agents' performance in supply-chain tasks, revealing a knowing-doing gap where models like Claude Sonnet 5 and GPT-5.4 detect 84-88% of hidden failures but show skill scores ranging from 0.62 to -0.23. Notably, 34-43% of diagnosed stress weeks still end in stockout, highlighting the dual nature of failure in these agents.
Key Points
- STOCKTAKE is a 26-week supply-chain replenishment benchmark for LLM agents.
- Models detect 84-88% of hidden failures but vary in skill scores from 0.62 to -0.23.
- 34-43% of correctly diagnosed stress weeks still result in stockouts.
- The benchmark separates state estimation from control performance.
- Under-response and over-response are both identified as failure factors.
Paper Resources
📖 Reader Mode
~2 min readAbstract:LLM agents are increasingly evaluated on multi-week decision tasks in which the state that drives cost is never directly observed. On such tasks the final cost cannot say why an agent failed: it may have misread the world, or read it correctly and still failed to act (the knowing-doing gap). Existing evaluations cannot separate these two failures; their reference policies either read privileged information the agent never sees, or are missing altogether. We introduce STOCKTAKE, a 26-week supply-chain replenishment benchmark built as a factored partially observable Markov decision process with six hidden factor processes, designed so that a fair reference policy is computable: an exact Bayes filter per factor drives a rollout policy on the identical observation stream the agent receives. Scoring each run between a symptom-blind base-stock floor (0) and this oracle (1) yields a skill score, and grading each week's written rationale yields a stated-belief detection lag and a knowing-doing rate, so state estimation and control are measured separately. On fifty seeds with curated stress profiles, Claude Sonnet 5, GPT-5.4, DeepSeek-V4-Pro, and Grok 4.5 detect 84-88% of hidden failures, typically within a week of onset, yet span skill scores from 0.62 to -0.23: two of the four end below the symptom-blind floor while naming factors slightly faster than the two that beat it. The failure has two faces. Where stress persists, 34-43% of correctly diagnosed stress weeks still end in stockout for every model, a rate that partly reflects the severity of the weeks models notice. That rate also runs opposite to skill: the two models under the floor stock out least on diagnosed weeks, so under-response is only one face of the gap, and their traces point to the other, responses whose cost exceeds what they protect. STOCKTAKE measures both directions of that failure.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.13618 [cs.AI] |
| (or arXiv:2607.13618v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.13618 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sagar Deb [view email]
[v1]
Wed, 15 Jul 2026 09:08:27 UTC (82 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Automatic Ordinary Differential Equations Discovery For Biological Systems Using Large Language Model Powered Agentic System
The MEDA system utilizes large language models and symbolic regression to autonomously discover ordinary differential equations for biological systems, achieving strong structural recovery and biologically plausible models. It outperforms existing methods by integrating domain knowledge and mechanistic constraints, demonstrating effective retrieval and extrapolation capabilities.