SPEAR: Code-Augmented Agentic Prompt Optimization
Quick Answer
This paper shows that SPEAR, an advanced prompt optimizer, autonomously utilizes tools like Python sandbox for error analysis, achieving superior performance in LLM tasks with a $b1$ of 0.857 compared to 0.359 in tool-selection.
Quick Take
SPEAR, an advanced prompt optimizer, autonomously utilizes tools like Python sandbox for error analysis, achieving superior performance in LLM tasks with a $b1$ of 0.857 compared to 0.359 in tool-selection. It excels in industrial benchmarks, averaging 0.938 accuracy on BBH-7, outperforming GEPA and TextGrad significantly.
Key Points
- SPEAR employs a Python sandbox for structural error analysis in prompt optimization.
- Achieved $b1$ of 0.857 in tool-selection tasks, outperforming existing methods.
- Outperformed GEPA (0.628) and TextGrad (0.484) with 0.938 accuracy on BBH-7.
- Auto-rollback and guard metrics ensure continuous improvement in optimization.
- The Python tool significantly enhances performance on complex judge tasks.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 26275v1 Announce Type: new Abstract: Automatic prompt engineering (APE) rewrites prompts to improve downstream task performance, but existing APE loops treat the optimizer itself as a fixed pipeline. We port the code-as-action paradigm of CodeAct (Wang et al. , 2024a) to APE and propose SPEAR (Sandboxed Prompt Engineer with Active Roll-back), a free-form agentic optimizer with four tools -- evaluate, python, set_prompt, finish -- that decides autonomously how and when to use them.
The distinctive tool is the Python sandbox: the optimizer writes and executes arbitrary Python on the current evaluation DataFrame, performing structural error analysis (confusion matrices, error clustering, per group metrics) the agent itself authors. Two guardrails turn the long-horizon agent into a monotone-improving optimizer: auto-rollback on metric regression, and an optional guard metric floor.
We evaluate on three industrial LLM-as-judge suites (13 judge tasks across recruiter-intake, conversational-memory, and query-refinement systems) plus seven BBH tasks and GSM8K. SPEAR wins every industrial task on the primary metric ($\kappa$ 0. 857 vs 0. 359 on tool-selection; F1-macro 0. 815 vs 0. 763 on filter-relevance; $\kappa$ 0. 254 vs 0. 218 on the hardest extraction dimension). On BBH-7 SPEAR averages 0. 938 accuracy vs GEPA 0. 628 and TextGrad 0. 484.
Ablations show the Python tool is the largest single lever on complex judge tasks ($\Delta \approx +0. 79\kappa$ on the 5-class tool-selection judge, $\Delta \approx +0. 35\kappa$ on the hardest extraction dimension when removed); its irreplaceable contribution is class-pair confusion aggregation that a long-context LLM cannot extract reliably from the raw eval DataFrame.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.