How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks
Quick Answer
This study analyzes public LLM agent benchmarks, revealing that a partial evaluation is valid only if it supports the original benchmark's conclusions, with required task fractions varying significantly across benchmarks.
Quick Take
This study analyzes public LLM agent benchmarks, revealing that a partial evaluation is valid only if it supports the original benchmark's conclusions, with required task fractions varying significantly across benchmarks. For instance, AppWorld meets all targets at 15%, tau-bench at 25%, while Verified requires 90%. The findings emphasize the need for detailed reporting in partial evaluations.
Key Points
- Partial evaluations must support original benchmark decisions and cover required task groups.
- AppWorld meets targets at 15%, tau-bench at 25%, SWE-bench Verified at 90%.
- SWE-bench Lite fails to meet all targets at 95% under primary coverage rules.
- Reports should clarify performance deltas and unresolved comparisons.
- Task fraction requirements vary sharply across different benchmarks.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Agent benchmarks often compare two agents after all tasks have run, but costly evaluations make partial runs tempting. A task fraction alone does not show whether a partial run supports the same pairwise conclusion as the completed benchmark. We study this question by replaying completed public task-level records from SWE-bench, AppWorld, and tau-bench. A partial budget counts as enough only when it supports the completed benchmark's decision, covers required task groups, and leaves no more than a target fraction of comparisons unresolved. The required task fraction varies sharply. At the strict 0 percentage point threshold on a 5 percentage point budget grid, AppWorld first meets all targets at 15 percent, tau-bench at 25 percent, and SWE-bench Verified at 90 percent; SWE-bench Lite does not meet all targets by 95 percent under the primary coverage rule. Partial-evaluation reports should state how much one agent must outperform another, how tasks are selected, what coverage rule is required, what decision rule is used, and how many comparisons may remain unresolved.
| Comments: | KDD 2026 Workshop Agentic AI Evaluation and Trustworthiness |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.12338 [cs.AI] |
| (or arXiv:2607.12338v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.12338 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Wei-Jung Huang [view email]
[v1]
Tue, 14 Jul 2026 04:36:41 UTC (219 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Emerging Paradigm of Geospatial Foundation Models: From Pre-Training to Agentic Reasoning
The paper introduces Geospatial Foundation Models (GeoFMs), AI/ML models pre-trained on extensive geospatial datasets, enabling domain experts to fine-tune them for specific tasks. This paradigm shift democratizes access to advanced AI/ML while ensuring security, and proposes a framework for cost-effective adaptation strategies. The vision of Agentic Geospatial Reasoning is also presented, where Large Language Models orchestrate GeoFMs to automate complex analytical workflows.