How enabling two settings tripled our scores on the ARC-AGI-3 benchmark
Quick Answer
Enabling retained reasoning and compaction in the Responses API allowed GPT-5.6 Sol to triple its scores on the ARC-AGI-3 benchmark, achieving 38.3% compared to 13.3% with the official harness.
Quick Take
This change also reduced output tokens by six times, enhancing performance significantly in 2D puzzle games.
Key Points
- GPT-5.6 Sol scored 38.3% on -3 with new settings, tripling its previous score.
- Retained reasoning allowed the model to remember past actions and thoughts effectively.
- Compaction replaced rolling truncation, preserving learned information over longer tasks.
- Output tokens were reduced by 6x while achieving higher scores, improving efficiency.
- The ARC-AGI-3 benchmark measures AI learning and reasoning in 2D puzzle games.
Source Excerpt
How two API settings improved GPT-5. 6 performance on -3, boosting scores and efficiency by retaining reasoning and enabling compaction.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from OpenAI Blog
See more →Scientific computing in the age of agentic AI
AI agents are transforming scientific computing by streamlining software development, enabling researchers to focus on discovery. Projects using Codex and Claude Code report accelerated development and improved maintenance, though challenges in validating AI outputs remain. Long-term stewardship of research software is crucial to ensure reliability and reproducibility.