Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism
Quick Answer
The proposed PAT method enhances synchronous RLHF training by dynamically adapting tensor parallelism, achieving up to 34.6% reduction in generation latency and 27.2% in end-to-end training iteration latency on models like LLaMA3.1-8B and Qwen3-14B.
Quick Take
The proposed PAT method enhances synchronous RLHF training by dynamically adapting tensor parallelism, achieving up to 34.6% reduction in generation latency and 27.2% in end-to-end training iteration latency on models like LLaMA3.1-8B and Qwen3-14B.
Key Points
- PAT adapts tensor parallelism during RLHF generation to optimize GPU utilization.
- The method uses a predictor-guided approach for efficient reconfiguration.
- Implementation on SGLang and integration with VeRL framework demonstrated significant performance gains.
- Evaluations showed substantial latency reductions compared to the original VeRL setup.
- Key techniques include cost-model-based choices for KV-cache and in-place weight resharding.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 23945v1 Announce Type: new Abstract: Reinforcement Learning from Human Feedback (RLHF) has become a key post-training paradigm for improving model quality. However, the synchronous three-stage RLHF pipeline is often bottlenecked by the generation stage, where response-length skew causes the effective batch size to shrink rapidly during decoding, leaving GPUs underutilized while a few long responses remain unfinished.
Mainstream frameworks employ a static tensor parallelism (TP) configuration that cannot adapt to changing batch characteristics, leaving substantial performance headroom unexplored. We propose PAT, an adaptive TP method that dynamically reconfigures TP during the generation stage of each RLHF iteration. PAT introduces two key techniques.
First, a predictor-guided online reconfiguration method decides both the reconfiguration point and the target TP configuration based on offline profiling, triggering reconfiguration only when the predicted latency benefit outweighs the reconfiguration overhead.
Second, a lightweight online reconfiguration mechanism updates only the states and layouts affected by TP changes: it adapts unfinished decoding states through a cost-model-based choice between KV-cache migration and recomputation, performs in-place weight resharding, and reuses cached communication groups. We implement PAT on top of SGLang and integrate it with the VeRL framework. Evaluations on LLaMA3. 1-8B and Qwen3-14B using DeepScaleR show that PAT reduces generation latency by up to 34.
6% and end-to-end RLHF training iteration latency by up to 27. 2% compared to the original VeRL setup.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Adversarial Social Epistemology for Assemblies of Humans and Large Language Models
The paper introduces Adversarial Social Epistemology (ASE) to analyze how agents manipulate trust in public communications, highlighting mechanisms that undermine the reliability of testimony and inference. It critiques existing frameworks like epistemic bubbles and misinformation diffusion, proposing a new language for understanding trust breaches and auditing inferential chains in densely interactive environments involving humans and large language models.