Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
Quick Answer
The Long-Horizon-Terminal-Bench introduces a new benchmark for evaluating AI agents on 46 long-horizon tasks, emphasizing dense rewards and intermediate progress.
Quick Take
The Long-Horizon- introduces a new benchmark for evaluating AI agents on 46 long-horizon tasks, emphasizing dense rewards and intermediate progress. Testing 15 models revealed a mean pass rate of only 4.3% at a partial-reward threshold of 0.95, highlighting significant room for improvement in long-horizon planning and execution.
Key Points
- Long-Horizon-Terminal-Bench includes 46 tasks across nine categories.
- Tasks require hundreds of episodes and minutes to hours of execution.
- Agents averaged 9.9M tokens and 85.3 minutes per task.
- Strongest model achieved only 15.2% pass rate at 0.95 reward threshold.
- Benchmark aims to improve evaluation of long-horizon terminal agents.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ninghao Liu, Lichao Sun, Haotao Mi, LeoweiLiang
Abstract:AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.
| Comments: | 17 pages |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.08964 [cs.AI] |
| (or arXiv:2607.08964v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.08964 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zongxia Li [view email]
[v1]
Thu, 9 Jul 2026 21:56:37 UTC (5,056 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Adversarial Social Epistemology for Assemblies of Humans and Large Language Models
The paper introduces Adversarial Social Epistemology (ASE) to analyze how agents manipulate trust in public communications, highlighting mechanisms that undermine the reliability of testimony and inference. It critiques existing frameworks like epistemic bubbles and misinformation diffusion, proposing a new language for understanding trust breaches and auditing inferential chains in densely interactive environments involving humans and large language models.