标签：#transform

Last week, a Baseten inference engineer who goes by @waterloo_intern published a technical blog post titled "22,580: From GPT-2 to Kimi K3, Explained." It hit 2.4 million views in days. He didn't write a press release. He wrote runnable PyTorch code — starting from GPT-2's attention block, stepping through every architectural change, explaining one problem and one cost per iteration. It's the best transformer lineage explanation I've seen. I devoured his post, then cross-checked the key claims against 5 original papers. Here's the full picture. The 22,580x Number In February 2019, OpenAI released GPT-2 — 124M parameters. Seven years later, Moonshot AI open-sourced Kimi K3 — 2.8T parameters. You could fit 22,580 GPT-2s inside one Kimi K3 . But this isn't a "throw more compute at it" story. It's a story about how we store, update, and retrieve memory . Starting Point: GPT-2 class Block ( nn . Module ): def forward ( self , x ): x = x + self . attn ( self . ln_1 ( x )) x = x + self . mlp ( self . ln_2 ( x )) return x Every time the model generates a new token, it recomputes Q, K, V projections for all historical tokens, then runs an O(N²) softmax attention. K and V from tokens 1 through N-1? Thrown away. Token N+1 arrives? Recompute everything. That's why KV Cache was invented. KV Cache: Store It, Don't Recompute Simple idea: cache the already-computed keys and values. For the next token, new Q only needs one dot product against the cached K. Problem solved — but a new one created. KV cache grows linearly with sequence length. At 1M tokens × d_model × layers, that's dozens of GB of VRAM. Every decoding step reads all of it from HBM. The bottleneck isn't compute. It's memory bandwidth. This is the key to understanding every improvement that follows. Linear Attention: Fixed-Size Memory Can we compress O(N²D) into O(ND²)? The idea: replace softmax with a feature map. # Standard softmax (must materialize N×N first) attention = softmax(QKᵀ / √d) × V # Linear attention (fold

An interactive guide to the architecture behind modern language models. Instead of predicting the next word, this Transformer predicts the next move in a game of fading Tic-Tac-Toe—making every step of the model easy to visualize and understand. Play the game, inspect every matrix multiplication, and watch tokens flow through the network in real time. What's covered Tokenization and embeddings Learned positional encoding Self-attention (Q, K, V) Multi-head attention Causal masking and softmax Residual connections and layer normalization MLP (feed-forward network) Unembedding and sampling Model ablations (no positional encoding, no causal mask, no MLP, no residual stream) Includes interactive visualizations for every stage of the Transformer pipeline - from input tokens to the final prediction. https://sbondaryev.dev/articles/transformer

Un’area interna che sembra una lavagna di ragionamento: non è coscienza, ma è un indizio forte su come emergono controllo e pianificazione nei transformer. Negli ultimi anni ci siamo abituati a pensare ai modelli linguistici come a enormi “scatole nere”: un prompt entra, un testo esce, e nel mezzo c’è un mare di matrici difficili da ispezionare. Ma c’è una novità interessante: alcune analisi suggeriscono l’esistenza di una piccola regione interna, relativamente organizzata, che funziona come uno spazio di lavoro per concetti . Un posto dove il modello “tiene a mente” qualcosa prima di produrre la risposta. È un’idea che fa scattare subito l’associazione più pericolosa (e più abusata) del momento: coscienza . In realtà, il punto non è stabilire se un LLM sia cosciente; il punto è molto più concreto e utile per chi sviluppa: se esiste un’area interna che concentra il ragionamento controllabile , allora possiamo capire meglio cosa guida certe risposte e come intervenire su errori, allucinazioni e comportamenti indesiderati. J-Space: una “lavagna” interna per il ragionamento L’idea chiave è questa: dentro il modello emergerebbe un piccolo insieme di pattern neurali “coerenti” (chiamiamoli J-Space ) che si comporta come una lavagna. Su questa lavagna compaiono concetti (non necessariamente parole che verranno stampate). Questi concetti influenzano la catena di ragionamento . Molte altre abilità—fluency, grammatica, stile, completamento locale—sembrano invece scorrere “automaticamente” altrove. Se questa separazione regge, spiega un fenomeno che tutti abbiamo osservato: modelli capaci di scrivere in modo impeccabile, ma fragili nel ragionamento o incoerenti quando devono mantenere vincoli. Il test più interessante: sostituire un concetto e vedere il ragionamento obbedire Un esperimento illuminante consiste nell’individuare un concetto attivo nello spazio di lavoro e sostituirlo con un altro, senza cambiare né prompt né output manualmente. Esempio (semplificato): Domanda:

The translateZ() function moves an element closer to or farther from the user. translateZ() originally handwritten and published with love on CSS-Tricks . You should really get the newsletter as well.

The translateY() function shifts an element vertically by the specified amount. translateY() originally handwritten and published with love on CSS-Tricks . You should really get the newsletter as well.

The translateX() function shifts an element horizontally by the specified amount. translateX() originally handwritten and published with love on CSS-Tricks . You should really get the newsletter as well.

The CSS translate() function shifts an element from its default position on a 2-dimensional plane translate() originally handwritten and published with love on CSS-Tricks . You should really get the newsletter as well.

The rotateX() function rotates an element around the x-axis in a three-dimensional space rotateX() originally handwritten and published with love on CSS-Tricks . You should really get the newsletter as well.

The rotateY() function rotates an element around its vertical y-axis. rotateY() originally handwritten and published with love on CSS-Tricks . You should really get the newsletter as well.

The rotateZ() function rotates an element around its z-axis, so clockwise or counterclockwise. rotateZ() originally handwritten and published with love on CSS-Tricks . You should really get the newsletter as well.

The rotate() function spins an element either clockwise or counterclockwise in a 2D plane. rotate() originally handwritten and published with love on CSS-Tricks . You should really get the newsletter as well.