DiffusionGemma: How Google's New Open LLM Hits 1,000 Tokens/sec and Changes Inference Economics

TL;DR: Google released DiffusionGemma, an open Apache 2.0 diffusion-based LLM that generates text up to 4x faster than autoregressive models, hitting 1,000+ tokens/sec on a single H100 and fitting in 18 GB VRAM. It trades some accuracy for speed. Here is what that means in practice. What DiffusionGemma Actually Is Google DeepMind released DiffusionGemma , the first production-grade open-weight model that applies discrete diffusion to text generation. The same family of techniques behind image generators like Stable Diffusion, now applied to language. Instead of predicting one token at a time left-to-right, DiffusionGemma fills a 256-token block with noise and iteratively refines the entire block across multiple denoising passes until confidence thresholds are met. It commits roughly 15-20 tokens per forward pass on average, not one. This is a fundamentally different compute pattern from everything shipping in production today. The Numbers Metric Value Tokens/sec (H100, FP8, low batch) 1,100+ Tokens/sec (RTX 5090) 700+ Total parameters 25.2B (marketed as 26B) Active parameters at inference 3.8B MoE expert config 8 active / 128 total VRAM required (quantized) 18 GB Canvas (block) size 256 tokens Tokens committed per forward pass ~15-20 Max denoising steps 48 Context window 256K tokens License Apache 2.0 For context: comparable autoregressive models on the same H100 generate roughly 200-250 tokens/sec. DiffusionGemma is up to 4x faster on throughput. The jump comes from shifting the decode bottleneck from memory bandwidth to compute. Why the Architecture Matters DiffusionGemma is a 26B Mixture of Experts (MoE) model built on the Gemma 4 backbone, but it replaces the autoregressive decoder with a diffusion head . How a single generation works: The model initializes a 256-token block with random placeholder tokens It runs up to 48 denoising steps, refining all tokens simultaneously with bidirectional attention (every token attends to every other token in the block) Token