Accelerate RL rollouts by up to 50% with distribution-aware speculative decoding
Quick Answer
Together AI's Distribution-aware Speculative Decoding (DAS) framework accelerates RL rollouts by over 50% without altering model outputs, addressing the rollout bottleneck in reinforcement learning.
Quick Take
This improvement is crucial for large models like DeepSeek-R1, which experience significant delays during the rollout phase, consuming 70% of total training time.
Key Points
- DAS reduces rollout time by over 50% on math reasoning and code generation tasks.
- 70% of total RL training time is spent on the rollout phase, causing inefficiencies.
- The framework uses an adaptive suffix tree drafter to stay aligned with evolving model weights.
- Length-aware scheduling minimizes GPU idle time by balancing long and short requests.
- No degradation in reward quality was observed during the rollout time reduction.
Source Excerpt
Rollout is the silent bottleneck in RL post-training. DAS fixes it with adaptive speculative decoding — up to 50% faster, zero degradation in reward quality.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from Together AI
See more →
Open, convenient and predictable: Introducing Provisioned Throughput
Together AI introduces Provisioned Throughput, offering guaranteed inference capacity for MiniMax M3 and GLM-5.2 at $0.05 per PTU per minute, achieving costs up to 90% lower than Claude Opus 4.8. This new model provides predictable pricing and a 99% uptime SLA, catering to companies transitioning to open weight models for production workloads.
