Dissecting model behavior through agent trajectories
Quick Answer
The paper identifies the 'intent-execution' gap in AI agents, emphasizing its significance alongside harness design.
Quick Take
The 'Simple Strands Agent' (SSA) demonstrates improved performance on benchmarks like SWE-Pro and -2, analyzing 138k trajectories to uncover model-specific problem-solving behaviors.
Key Points
- SSA reproduces or improves pass@1 performance on SWE-Pro and Terminal-Bench-2 benchmarks.
- The intent-execution gap affects how models translate capabilities into agent performance.
- SSA analyzes 138k trajectories to reveal differences in problem-solving behaviors among models.
- Finer metrics like edit frequency and testing activity provide insights into model effort allocation.
- The study emphasizes the importance of harness-model alignment in AI agent design.
Paper Resources
Source Excerpt
arXiv:2606. 17454v1 Announce Type: new Abstract: AI agent performance is not just a modeling problem, it is fundamentally a systems problem. The advanced capabilities of models are realized through agent harnesses. Therefore, a gap between model assumptions and harness behavior can easily prevent the model's full capabilities from translating into agent performance. We formalize this as the `intent-execution' gap: the mismatch between what the model intends and what the harness executes, and vice versa.
We argue that minimizing this intent-execution gap is as important as other aspects of harness design such as tools and execution loops. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.