Why Solve It Twice? Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering
Quick Answer
This paper shows that HASTE, a hierarchical multi-agent system for ML engineering, organizes knowledge into three tiers, achieving a 100% medal rate with tiered loading compared to 62.5% with flat loading.
Quick Take
HASTE, a hierarchical for ML engineering, organizes knowledge into three tiers, achieving a 100% medal rate with tiered loading compared to 62.5% with flat loading. In 22 Kaggle competitions, it reached a 77.3% medal rate using Claude Sonnet 4.6, demonstrating that better knowledge organization can enhance performance while reducing compute costs.
Key Points
- HASTE organizes knowledge into global, domain, and competition-specific tiers.
- Tiered loading maintained a 159-skill inventory across 8 competitions.
- Warm-start runs used 52% fewer refinement iterations than cold-start runs.
- Agent's proposed changes acceptance rose from 42% to 85% with 50+ skills.
- HASTE achieved a 77.3% medal rate on the MLE-Bench Lite benchmark.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 30911v1 Announce Type: new Abstract: ML engineering agents waste compute rediscovering known techniques because every competition is a cold start. We present HASTE, a hierarchical that organizes cross-competition knowledge into three scope tiers (global, domain, and competition-specific), each coupled to a matching agent level. An orchestrator coordinates domain specialists and promotes learning between tiers via LLM-driven abstraction.
A controlled ablation provides evidence for scoped loading: holding a 159-skill inventory constant across 8 competitions, tiered loading achieves a 100% medal rate while flat loading reaches only 62. 5%, the same medal rate as loading no skills, and consumes 2x the output tokens. On the full MLE-Bench Lite benchmark (22 Kaggle competitions), HASTE reaches a medal rate of 77. 3% using Claude Sonnet 4. 6 at 12h per competition. In a cold-start run, the system begins with no accumulated skills.
In warm-start runs, it reloads skills learned from earlier competitions, using only global and domain-level skills for transfer across competitions. Warm starts use 52% fewer refinement iterations, and the fraction of proposed changes kept by the agent rises from 42% at low inventory to 85% once 50+ skills are available. These results suggest that better knowledge organization can partly substitute for model strength and compute budget in ML-engineering agents.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Verification Horizon: No Silver Bullet for Coding Agent Rewards
As coding agents evolve, verifying solutions becomes more challenging than generating them, necessitating a focus on scalable, faithful, and robust verification methods. The study reveals that no fixed reward function can sustain effectiveness as model capabilities advance, emphasizing the need for verification to evolve alongside solution generation.