JetFlow: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting
Quick Answer
JetFlow introduces a novel speculative decoding framework that enhances autoregressive LLMs by achieving up to 9.64x speedup on MATH-500 and 4.58x on conversational tasks, outperforming existing methods.
Quick Take
By integrating causal conditioning with efficient drafting, JetFlow maximizes draft budgets for longer accepted prefixes and improved performance on dense and MoE Qwen3 models.
Key Points
- JetFlow combines one-forward drafting efficiency with branch-wise causal conditioning.
- Achieves 9.64x speedup on MATH-500 and 4.58x on conversational workloads.
- Outperforms bidirectional-head and tree-based SD baselines across various benchmarks.
- Utilizes fused hidden states from a frozen target model for improved candidate tree scoring.
- Code and models available at https://github.com/hao-ai-lab/JetFlow.
Paper Resources
Source Excerpt
Speculative decoding (SD) accelerates autoregressive (LLMs) by drafting multiple tokens and verifying them in parallel, but it faces a scaling limitation: increasing the draft budget improves speed only when acceptance remains high and drafting overhead stays low. This ceiling has been difficult to break because prior head-based SD methods face a causality-efficiency dilemma. Autoregressive drafters produce path-conditioned candidates that are effective for tree speculative
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.