Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learning
Quick Answer
The paper introduces DiRL, a Direction-Aware Reinforcement Learning framework that enhances exploration in large language models by distinguishing between reasoning and memorization.
Quick Take
By focusing on reasoning-aligned exploration, DiRL shows significant improvements in mathematical and general reasoning benchmarks compared to existing methods. This approach integrates with (GRPO) and effectively suppresses memorization-driven variations.
Key Points
- DiRL anchors exploration to a reasoning-memorization direction extracted from model representations.
- The framework constructs direction-weighted gradient features for rollout updates.
- DiRL amplifies reasoning-aligned exploration while suppressing memorization variations.
- Extensive experiments show DiRL's effectiveness over various exploration methods.
- DiRL integrates seamlessly with standard Group Relative Policy Optimization (GRPO).
Paper Resources
Source Excerpt
arXiv:2606. 10346v1 Announce Type: new Abstract: Reinforcement learning has become a key paradigm for eliciting reasoning abilities in , where exploration is crucial for discovering effective solution trajectories. Existing exploration methods typically encourage diversity in semantic or gradient spaces, without distinguishing what drives this diversity. A trajectory may appear novel because it follows a new reasoning process, or because it varies memorized patterns and shortcuts.
Rewarding both cases equally may steer exploration toward memorization rather than genuine reasoning improvement. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →AINTMA: Agentic AI Architecture for Autonomous Test Management with Generative Intelligence, Secure Cloud Communication and Adaptive Quality Analytics
AINTMA, an autonomous test management architecture utilizing six specialized AI agents, achieves 88.4% test prioritization accuracy and reduces defect escape rates from 8.3% to 2.1%. The system demonstrates a 340% ROI within nine months, showcasing the potential of agentic AI in enhancing software quality management in cloud environments.