The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
Quick Answer
The study evaluates Qwen 3.6 Plus and MiniMax M2.5 across three harnesses, revealing that harness choice can induce a 40x difference in tokens per task solved, while pass-rate differences remain minimal.
Quick Take
This highlights the importance of considering harness-model pairs for accurate coding agent evaluations, as they significantly affect real-world costs and performance metrics.
Key Points
- Harness choice can lead to a 40x difference in tokens per solved task.
- Pass-rate differences between models remain between 0-8 percentage points.
- Failure fingerprints indicate harness-level biases that are model-independent.
- Selecting harness-model pairs by pass rate is recommended for better evaluations.
- The study provides anonymized configs and raw trial logs for further analysis.
DeepSignal Analysis
What happened
A study evaluated the performance of coding agents Qwen 3.6 Plus and MiniMax M2.5 across three different harnesses. The findings indicated that the choice of harness could lead to a significant difference in tokens required per task solved, with variations up to 40 times, while pass rates showed minimal differences.
Key evidence
- The study assessed Qwen 3.6 Plus and MiniMax M2.5 using three open-source harnesses: Goose, OpenCode, and OpenHands-SDK.
- Harness choice resulted in up to a 40x difference in tokens per solved task, while pass-rate differences were only 0-8 percentage points.
- Failure patterns were consistent across models, indicating biases at the harness level that were largely independent of the models used.
Why it matters
This research emphasizes the need for a more nuanced evaluation of coding agents, as traditional metrics like model name and pass rate do not capture the full picture. The significant impact of harness choice on performance metrics suggests that evaluations could misrepresent a model's capabilities if harness effects are not considered. This has implications for real-world applications, where costs and performance can vary dramatically based on the selected harness.
Paper Resources
Source Excerpt
Public leaderboards for coding agents typically rank systems by model name and pass rate, while the surrounding harness (the scaffold that issues tools, manages context, and decides when to stop) is often under-specified. Model-to-model comparison is valid when the harness is fixed; when it varies, performance and efficiency conflate model and scaffold effects. We evaluate Qwen 3. 6 Plus and MiniMax M2. 5 across three open-source harnesses (Goose, OpenCode, OpenHands-SDK) on a stratified 50-task s
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →AINTMA: Agentic AI Architecture for Autonomous Test Management with Generative Intelligence, Secure Cloud Communication and Adaptive Quality Analytics
AINTMA, an autonomous test management architecture utilizing six specialized AI agents, achieves 88.4% test prioritization accuracy and reduces defect escape rates from 8.3% to 2.1%. The system demonstrates a 340% ROI within nine months, showcasing the potential of agentic AI in enhancing software quality management in cloud environments.