RL Post-Training Builds Compositional Reasoning Strategies
Quick Answer
This study investigates whether reinforcement learning (RL) post-training can compose primitive skills into higher-level strategies.
Quick Take
This study investigates whether reinforcement learning (RL) post-training can compose primitive skills into higher-level strategies. Using a Transformer model, RL demonstrated improved problem-solving capabilities in a rewrite-grammar environment, outperforming pretraining methods by reorganizing primitive competencies into reusable structures. The findings highlight that RL's effectiveness stems from its selective exploration of valid strategies rather than sheer volume.
Key Points
- RL post-training enhances a Transformer model's ability to solve complex tasks.
- The study utilized a fully observable rewrite-grammar environment for evaluation.
- RL reorganizes primitive skills into sequential and parallel compositions.
- Rejection fine-tuning produced many invalid rewrites, while RL focused on valid structures.
- Pretraining must organize primitive competencies for RL to build effective strategies.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Does RL post-training merely amplify primitive skills already latent in a base model, or can it compose primitive skills into new higher-level strategies? We study this question in a fully observable rewrite-grammar environment where the pretraining distribution is known and every generated rewrite can be audited. A Transformer is pretrained on primitive symbol-rewrite chains and post-trained on a Trace-based reasoning task with only a binary final-answer reward. RL solves held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets, while rejection fine-tuning improves early but plateaus. Trace analysis shows that RL reorganizes primitive competence through a phased compositional mechanism: it first strengthens primitive reductions, then discovers valid composed procedures. These include sequential compositions, which collapse ordered chains of primitive contractions, and parallel compositions, which combine independent primitive contractions in a single step. The composed procedures are not isolated samples; they are reused and consolidated into a stable repertoire. Comparing RL with rejection fine-tuning shows that the key difference is not exploration volume but selectivity: RFT produces many shortcut-like rewrites, much of them invalid, whereas RL concentrates exploration into valid reusable structure. Pretraining ablations show that the emergence of compositional strategies is gated not by primitive exposure alone, but by whether pretraining organizes primitive competence into reduction procedures that RL can later compress. The base model provides weak procedural ingredients; RL builds them into reliable higher-level strategies.
| Comments: | 8 pages, 6 figures. Accepted to the 2nd Workshop on Compositional Learning at ICML 2026, Seoul, South Korea |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.07646 [cs.AI] |
| (or arXiv:2607.07646v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.07646 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nishil Patel [view email]
[v1]
Wed, 8 Jul 2026 17:04:42 UTC (1,054 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Onnes: A Physics-Grounded LLM Simulator for Cryogenic Fault Diagnosis in Quantum Computing Infrastructure
Onnes is a physics-grounded digital twin simulator for dilution refrigerators, enhancing cryogenic fault diagnosis in quantum computing. It achieves a classification accuracy of 99.0% using few-shot demonstrations, matching a supervised ML classifier, while maintaining a low false alarm rate of 6.4% on real hardware.