Stop Comparing LLM Agents Without Disclosing the Harness
Quick Take
The paper argues that the execution harness significantly influences LLM agent performance, often more than the model itself. It introduces the Binding Constraint Thesis, highlighting that performance differences stem from harness configuration rather than model choice, suggesting current evaluation methods misattribute gains. A new harness-aware evaluation framework is proposed to improve transparency in comparisons.
Key Points
- The Binding Constraint Thesis states harness configuration drives performance variance more than model choice.
- Control-theoretic analysis shows small harness changes can yield larger performance shifts than model swaps.
- Published benchmarks indicate harness-induced variance can exceed model-induced variance significantly.
- A harness-aware evaluation framework is proposed to enhance transparency in agent comparisons.
- Without harness disclosure, long-horizon agent comparisons may be misleading.
Article Content
From source RSS / original summaryarXiv:2605. 23950v1 Announce Type: new Abstract: This position paper argues that, for long-horizon tasks evaluated across models with comparable frontier capability, the agent execution harness, namely the infrastructure layer that governs context construction, tool interaction, orchestration, and verification around a language model, is often a stronger determinant of agent performance than the model it wraps.
We formalize and defend the Binding Constraint Thesis: in this regime, performance variance is governed more by harness configuration than by model choice, and current evaluation protocols therefore systematically misattribute harness-level gains to model improvements. We support this thesis along three lines.
First, a control-theoretic formalization treats the harness as the controller of a closed-loop dynamical system and the LLM as the stochastic policy it governs, which explains why small harness changes can produce performance shifts that exceed those obtained by substituting one model for another. Second, published benchmarks, industry deployments, and a controlled variance decomposition show that harness-induced variance can substantially exceed model-induced variance, including cases of model ranking reversal.
Third, we propose a harness-aware evaluation framework with a disclosure standard and a variance decomposition protocol. Until harness specifications are disclosed, leaderboard comparisons for long-horizon agents should be treated as incomplete and potentially misleading.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Importance of Out-of-Band Metadata for Safe Autonomous Agents: The Redpanda Agentic Data Plane
The Redpanda Agentic Data Plane (ADP) introduces out-of-band metadata channels to enhance the safety of autonomous AI agents, ensuring secure data access and tamper-proof audit trails. This architecture mitigates risks associated with unpredictable AI behavior by enforcing governance throughout the agent lifecycle, demonstrated in a multi-agent trading system with strict data scoping and approval thresholds.