Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models
Quick Answer
This paper shows that Reinforcement learning (RL) models outperform supervised fine-tuned (SFT) models in mathematical reasoning due to superior internal representations.
Quick Take
Linear probes indicate RL models have more structured representations, while hierarchical architecture shows deeper layers are more critical. Token allocation variability suggests adaptive compute allocation is influenced by training pipelines rather than model type alone.
Key Points
- RL models achieve higher accuracy in predicting answer correctness than SFT models.
- Mean ablation studies show RL models develop a hierarchical architecture.
- Deeper layers in RL models become progressively more critical for performance.
- Token-count variability indicates adaptive compute allocation across different models.
- Variability in token allocation suggests influence from overall training pipelines.
Paper Resources
Source Excerpt
Large reasoning models trained via reinforcement learning (RL) have been increasingly shown to outperform their supervised fine-tuned (SFT) counterparts on mathematical reasoning tasks; Yet the mechanistic basis for this advantage remains unclear. We therefore ask, what internal representational differences enable RL models' superior performance? Our work presents two converging lines of evidence: First, linear probes trained on layer-wise hidden states reveal that RL models tend to achieve high
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.