Skills on the Fly: Test-Time Adaptive Skill Synthesis for LLM Agents
1d ago
Quick Take
SkillTTA enables LLM agents to adaptively synthesize task-specific skills at test time.
Key Points
- Retrieves relevant training trajectories for task-specific skill synthesis.
- Outperforms static synthesis methods in multiple benchmarks.
- Utilizes failed trajectories to improve skill generation.
📖 Reader Mode
~2 min readAbstract:LLM agents benefit from reusable skills, yet test-time tasks often require guidance more specific than a static skill library can provide. We propose \emph{SkillTTA}, a Test-Time Adaptive Skill Synthesis method that retrieves a small set of training trajectories relevant to the current task and synthesizes them into a temporary, task-specific textual skill. The solver model is kept fixed, so adaptation happens entirely through generated context rather than parameter updates. We evaluate the method on SpreadsheetBench, ALFWorld, and BigCodeBench. Compared with static trajectory-to-skill synthesis using GPT-5.5, task-specific skills improve SpreadsheetBench Pass@1 from 0.397 to 0.505 and BigCodeBench Pass@1 from 0.517 to 0.651. On ALFWorld, the method matches a heavier memory-learning baseline within four points of success rate while producing the shortest successful trajectories among reported methods. Ablations on SpreadsheetBench further show that synthesized skills outperform raw trajectory prompting, that top-$k$ retrieval should stay small, and that failed trajectories are especially useful because they expose recurring evaluator-facing mistakes.
| Comments: | 10 pages, 4 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.16986 [cs.CL] |
| (or arXiv:2605.16986v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.16986 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jingxing Wang [view email]
[v1]
Sat, 16 May 2026 13:14:15 UTC (446 KB)
— Originally published at arxiv.org
More from arXiv cs.CL
See more →arXiv cs.CL·Leyao Wang, Yanan He, Peng Chen, Asaf Yehudai, Yixin Liu, Rex Ying, Michal Shmueli-Scheuer, Arman Cohan
17h ago FeaturedOriginal
Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
AI Summary
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.