Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1
Quick Answer
This paper shows that The DeepSeek V3.2 model achieves 67.25% pass@2 on ARC-AGI-1 tasks at $0.62 per task, significantly improving the baseline by 52 points without specific fine-tuning.
Quick Take
The DeepSeek V3.2 model achieves 67.25% pass@2 on -1 tasks at $0.62 per task, significantly improving the baseline by 52 points without specific fine-tuning. This architecture utilizes an Explorer-Definer Pipeline and a Reflective Orchestrator to enhance pattern discovery and transformation synthesis, demonstrating that broader generation is key to further improvements.
Key Points
- Explorer-Definer Pipeline separates pattern discovery from transformation synthesis.
- Reflective Orchestrator autonomously explores new transformations upon hypothesis failure.
- Achieved 57.50% pass@2 at $0.25 per task without benchmark-specific training.
- Unbiased analysis indicates the pipeline is generation-bound, not selection-bound.
- Removal of the think tool reduces pass@2 by 5.75 percentage points.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Recent progress on ARC-AGI-1 from disclosed architectures has come broadly from two regimes: heavy test-time compute over frontier models (evolutionary search, exhaustive sampling, extended chain-of-thought), or benchmark-specific training in which small models are fine-tuned on ARC data, often with task-specialized architectures. We study a third regime: an open-weight model in non-thinking mode (DeepSeek V3.2) under a strict budget, with no ARC-specific fine-tuning. We study what is recoverable through architecture alone, building agentic harnesses that decompose pattern-discovery and program-synthesis stages explicitly. First, we introduce an Explorer-Definer Pipeline that separates pattern discovery from executable transformation synthesis, implemented as a two-stage agent pipeline. Next, we present the Reflective Orchestrator, which augments the pipeline with autonomous exploration of new transformations when previous hypotheses fail on training pairs. On the ARC-AGI-1 public 400-task evaluation set, the pipeline reaches 57.50% pass@2 at \$0.25 per task, and the orchestrator reaches 67.25% pass@2 at \$0.62 per task. Together these architectures lift a 15.50% one-shot baseline by ~52 points without benchmark-specific training or heavy test-time compute. Furthermore, the orchestrator-driven lift tests a falsifiable diagnostic the pipeline produces; unbiased pass@k analysis suggests the pipeline is generation-bound, not selection-bound (selection via training-pair accuracy captures ~95% of the candidate ceiling) and predicts that significant improvement requires broader generation, not better ranking. The orchestrator implements this prediction via adaptive re-exploration and confirms it (unbiased pass@1 lift +9.81 pp, matching selection-mediated pass@2 lift). An additional pipeline ablation identifies its think tool as a significant component, with removal reducing pass@2 by 5.75 pp.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.06764 [cs.AI] |
| (or arXiv:2607.06764v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.06764 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kabir Moghe [view email]
[v1]
Tue, 7 Jul 2026 19:49:35 UTC (1,897 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Onnes: A Physics-Grounded LLM Simulator for Cryogenic Fault Diagnosis in Quantum Computing Infrastructure
Onnes is a physics-grounded digital twin simulator for dilution refrigerators, enhancing cryogenic fault diagnosis in quantum computing. It achieves a classification accuracy of 99.0% using few-shot demonstrations, matching a supervised ML classifier, while maintaining a low false alarm rate of 6.4% on real hardware.