DiffusionGemma Technical Report
Quick Answer
DiffusionGemma is a novel open-weight language model that utilizes discrete diffusion for rapid text generation, achieving around 1,500 tokens per second on an NVIDIA H100 GPU.
Quick Take
By fine-tuning the Gemma 4 model with 3.8B activated parameters, it overcomes the sequential decoding limitations of traditional autoregressive models, generating 20 tokens per forward pass and maintaining multimodal input support.
Key Points
- Generates 1,500 tokens per second on a single NVIDIA H100 GPU.
- Fine-tuned from the Gemma 4 model with 3.8B activated parameters.
- Achieves 20 tokens per forward pass, surpassing traditional AR models.
- Utilizes a two-stage training pipeline for efficiency and quality.
- Supports multimodal inputs and retains autoregressive generation capabilities.
DeepSignal Analysis
What happened
DiffusionGemma is a new open-weight language model that employs discrete diffusion for fast text generation, achieving approximately 1,500 tokens per second on an NVIDIA H100 GPU. It fine-tunes the Gemma 4 model with 3.8 billion activated parameters, allowing it to generate 20 tokens per forward pass.
Key evidence
- DiffusionGemma generates text at around 1,500 tokens per second on a single NVIDIA H100 GPU, significantly outpacing traditional autoregressive models.
- The model fine-tunes the Gemma 4 model, which has 3.8 billion activated parameters and 25.2 billion total parameters, using less than 10% of the original training token budget.
- It maintains multimodal input support and can perform autoregressive generation with only minor performance degradation, indicating potential for hybrid decoding methods.
Why it matters
The introduction of DiffusionGemma represents a significant advancement in language model technology, particularly in overcoming the limitations of sequential decoding found in traditional autoregressive models. Its ability to generate text rapidly while retaining multimodal capabilities could enhance applications in various fields, including natural language processing and AI-driven content generation. This model sets a new benchmark for balancing generation speed and model capability.
Paper Resources
Source Excerpt
We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) . Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3. 8B activated and 25. 2
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.
