Why Solve It Twice? Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering
Quick Answer
This paper shows that HASTE, a hierarchical multi-agent system for ML engineering, organizes knowledge into three tiers, achieving a 100% medal rate with tiered loading compared to 62.5% with flat loading.
Quick Take
In 22 Kaggle competitions, it reached a 77.3% medal rate using Claude Sonnet 4.6, demonstrating that better knowledge organization can enhance performance while reducing compute costs.
Key Points
- HASTE organizes knowledge into global, domain, and competition-specific tiers.
- Tiered loading maintained a 159-skill inventory across 8 competitions.
- Warm-start runs used 52% fewer refinement iterations than cold-start runs.
- Agent's proposed changes acceptance rose from 42% to 85% with 50+ skills.
- HASTE achieved a 77.3% medal rate on the MLE-Bench Lite benchmark.
Paper Resources
📖 Reader Mode
~2 min readAbstract:ML engineering agents waste compute rediscovering known techniques because every competition is a cold start. We present HASTE, a hierarchical multi-agent system that organizes cross-competition knowledge into three scope tiers (global, domain, and competition-specific), each coupled to a matching agent level. An orchestrator coordinates domain specialists and promotes learning between tiers via LLM-driven abstraction. A controlled ablation provides evidence for scoped loading: holding a 159-skill inventory constant across 8 competitions, tiered loading achieves a 100% medal rate while flat loading reaches only 62.5%, the same medal rate as loading no skills, and consumes 2x the output tokens. On the full MLE-Bench Lite benchmark (22 Kaggle competitions), HASTE reaches a medal rate of 77.3% using Claude Sonnet 4.6 at 12h per competition; this is a single-seed campaign result, and multi-seed replication is the priority follow-up. In a cold-start run, the system begins with no accumulated skills. In warm-start runs, it reloads skills learned from earlier competitions, using only global and domain-level skills for transfer across competitions. Warm starts use 52% fewer refinement iterations, and the fraction of proposed changes kept by the agent rises from 42% at low inventory to 85% once 50+ skills are available. These results suggest that better knowledge organization can partly substitute for model strength and compute budget in ML-engineering agents.
| Comments: | 19 pages. Accepted to the 5th Workshop on Deep Learning for Code (DL4C), ICML 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2606.30911 [cs.AI] |
| (or arXiv:2606.30911v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.30911 arXiv-issued DOI via DataCite |
Submission history
From: Yongbin Kim [view email]
[v1]
Mon, 29 Jun 2026 20:59:14 UTC (76 KB)
[v2]
Wed, 1 Jul 2026 17:04:50 UTC (76 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.