CEO-Bench: Can Agents Play the Long Game?
Quick Answer
CEO-Bench evaluates AI agents' abilities in long-term, complex tasks by simulating startup operations over 500 days.
Quick Take
Only Claude Opus 4.8 and GPT-5.5 manage to exceed the $1M starting balance, highlighting significant challenges in sustained profitability and adaptability for current models.
Key Points
- CEO-Bench tests agents on long-term decision-making in a simulated startup environment.
- Agents face challenges like uncertainty, noisy data, and dynamic environments.
- Claude Opus 4.8 and GPT-5.5 are the only models to exceed the $1M starting balance.
- Most state-of-the-art models struggle with sustained profitability in this benchmark.
- The benchmark aims to measure intelligence for adaptive progress over time.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Language model agents are becoming proficient executors at isolated, short-horizon tasks such as software engineering and customer service. Yet real-world challenges require a combination of sophisticated skills that remain largely untested in agents: (1) navigating long horizons amid uncertainty; (2) acquiring information in noisy environments; (3) adapting to a changing world; (4) orchestrating multiple moving parts toward a coherent goal. We introduce CEO-Bench, which evaluates these capabilities together by simulating a representative real-world task: operating a startup for 500 days. An agent manages pricing, marketing, budgeting, and many other aspects of a fictional company through a programmable Python interface, operating in the same environment and facing the same challenges as a human CEO. Success demands analyzing noisy, interconnected business databases, translating signals into sound strategy, and coordinating many decisions with programming. The strongest agents write sophisticated code that simulates customer cohorts to forecast future cash and mines negotiation history to uncover hidden customer preferences. Even so, most state-of-the-art models struggle in this environment. Only Claude Opus 4.8 and GPT-5.5 finish above the $1M starting balance, and neither consistently turns a profit. CEO-Bench takes a first step toward measuring the intelligence required to drive sustained, adaptive progress over time.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE) |
| Cite as: | arXiv:2606.18543 [cs.AI] |
| (or arXiv:2606.18543v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.18543 arXiv-issued DOI via DataCite |
Submission history
From: Haozhe Chen [view email]
[v1]
Tue, 16 Jun 2026 23:37:52 UTC (1,399 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.