Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents
Quick Answer
This paper shows that Harness self-evolution in LLM agents shows that models like Qwen3.5-9B can produce effective updates similar to Claude Opus 4.6, despite varying base capabilities.
Quick Take
However, weak-tier models struggle to benefit from these updates, suggesting a focus on enhancing task-solving abilities over evolutionary capabilities.
Key Points
- Harness-updating shows flat performance across different model tiers.
- Qwen3.5-9B updates yield gains comparable to Claude Opus 4.6.
- Weak-tier models benefit little from updates due to activation failures.
- Mid-tier models gain the most from updated harnesses.
- Focus on task-solving capabilities rather than harness evolution in training.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Minhua Lin, Juncheng Wu, Zijun Wang, Zhan Shi, Yisi Sang, Bing He, Zewen Liu, Tianxin Wei, Zongyu Wu, Zhiwei Zhang, Dakuo Wang, Xiang Zhang, Benoit Dumoulin, Cihang Xie, Yuyin Zhou, Suhang Wang, Hanqing Lu
Abstract:LLM agents are increasingly deployed as systems built around editable external harnesses, including prompts, skills, memories and tools, that shape task execution without changing model parameters. Harness self-evolution adapts such agents by updating these harnesses from execution evidence. Yet it remains unclear whether a model's base capability in task-solving predicts its capabilities in harness self-evolution: which models produce useful harness updates, and which actually benefit from them? We analyze two harness self-evolution capabilities: (i) harness-updating, the capability to produce useful persistent harness updates from execution evidence; (ii) harness-benefit, the capability to benefit from updated harnesses during task solving. Our analysis reveals two findings. First, harness-updating is flat in base capability: models from different capability tiers produce harness updates that lead to surprisingly similar gains; even Qwen3.5-9B's updates yield gains comparable to those of Claude Opus~4.6. Second, harness-benefit is non-monotonic in base capability: weak-tier models benefit little from updated harnesses, mid-tier models benefit most, and strong-tier models benefit less than mid-tier. We trace low gains at the weak tier to two failure modes: weak-tier models may fail to activate relevant harness artifacts, or activate them but fail to follow them faithfully. These findings suggest investing capability budget in the task-solving agent rather than the evolver, and targeting harness invocation and long-horizon instruction following in agent training. Our source code is publicly available at this https URL.
| Comments: | 24 pages, 9 figures, 12 tables |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.30621 [cs.AI] |
| (or arXiv:2605.30621v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.30621 arXiv-issued DOI via DataCite |
Submission history
From: Minhua Lin [view email]
[v1]
Thu, 28 May 2026 22:16:14 UTC (1,619 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.