标签：#transformers

Last week, a Baseten inference engineer who goes by @waterloo_intern published a technical blog post titled "22,580: From GPT-2 to Kimi K3, Explained." It hit 2.4 million views in days. He didn't write a press release. He wrote runnable PyTorch code — starting from GPT-2's attention block, stepping through every architectural change, explaining one problem and one cost per iteration. It's the best transformer lineage explanation I've seen. I devoured his post, then cross-checked the key claims against 5 original papers. Here's the full picture. The 22,580x Number In February 2019, OpenAI released GPT-2 — 124M parameters. Seven years later, Moonshot AI open-sourced Kimi K3 — 2.8T parameters. You could fit 22,580 GPT-2s inside one Kimi K3 . But this isn't a "throw more compute at it" story. It's a story about how we store, update, and retrieve memory . Starting Point: GPT-2 class Block ( nn . Module ): def forward ( self , x ): x = x + self . attn ( self . ln_1 ( x )) x = x + self . mlp ( self . ln_2 ( x )) return x Every time the model generates a new token, it recomputes Q, K, V projections for all historical tokens, then runs an O(N²) softmax attention. K and V from tokens 1 through N-1? Thrown away. Token N+1 arrives? Recompute everything. That's why KV Cache was invented. KV Cache: Store It, Don't Recompute Simple idea: cache the already-computed keys and values. For the next token, new Q only needs one dot product against the cached K. Problem solved — but a new one created. KV cache grows linearly with sequence length. At 1M tokens × d_model × layers, that's dozens of GB of VRAM. Every decoding step reads all of it from HBM. The bottleneck isn't compute. It's memory bandwidth. This is the key to understanding every improvement that follows. Linear Attention: Fixed-Size Memory Can we compress O(N²D) into O(ND²)? The idea: replace softmax with a feature map. # Standard softmax (must materialize N×N first) attention = softmax(QKᵀ / √d) × V # Linear attention (fold

An interactive guide to the architecture behind modern language models. Instead of predicting the next word, this Transformer predicts the next move in a game of fading Tic-Tac-Toe—making every step of the model easy to visualize and understand. Play the game, inspect every matrix multiplication, and watch tokens flow through the network in real time. What's covered Tokenization and embeddings Learned positional encoding Self-attention (Q, K, V) Multi-head attention Causal masking and softmax Residual connections and layer normalization MLP (feed-forward network) Unembedding and sampling Model ablations (no positional encoding, no causal mask, no MLP, no residual stream) Includes interactive visualizations for every stage of the Transformer pipeline - from input tokens to the final prediction. https://sbondaryev.dev/articles/transformer