Speculative Decoding and MTP: Why Guessing Is Free

I saw "MTP round-trip" on a checklist for a Megatron conversion pipeline and had no idea what it meant. Two acronyms, one hyphen, apparently important enough that someone had listed it as a thing to verify. Working out what it meant took me somewhere I didn't expect. The interesting part turned out not to be MTP at all — it was the reason speculative decoding works in the first place, which rests on a fact about hardware that I had backwards. TL;DR Generating text is slow because it's sequential: one full forward pass per token. But a forward pass over five tokens costs about the same as over one. Generation is bottlenecked by moving weights, not by arithmetic. Speculative decoding exploits that: something cheap drafts k tokens, the big model verifies all of them in one pass. It is exact , not an approximation. Same output distribution as normal decoding. MTP (Multi-Token Prediction) is one way to produce those drafts — a small module trained into the model itself. MTP has two separate lives: a training-time auxiliary loss you can throw away, and an inference-time draft head you can't. Why generating text is slow To produce token N+1, the model needs token N. There's no way around that ordering — it's what "language model" means. So generating 100 tokens means 100 full passes through the network. For a model like GLM-5.2, that's 78 layers, 100 times over. The obvious conclusion is that generation is 100 times as expensive as reading the prompt. The obvious conclusion is wrong, and the way it's wrong is the whole point. The part that got me A forward pass processing one token and a forward pass processing five tokens take roughly the same wall-clock time. I had assumed compute scaled with tokens. It doesn't, because compute isn't the bottleneck. Every forward pass has to read the model's weights out of memory and into the compute units. That's hundreds of gigabytes moving across a memory bus, and it happens whether you're processing one token or fifty . The actual ar