How a Transformer Plays Tic-Tac-Toe

An interactive guide to the architecture behind modern language models. Instead of predicting the next word, this Transformer predicts the next move in a game of fading Tic-Tac-Toe—making every step of the model easy to visualize and understand. Play the game, inspect every matrix multiplication, and watch tokens flow through the network in real time. What's covered Tokenization and embeddings Learned positional encoding Self-attention (Q, K, V) Multi-head attention Causal masking and softmax Residual connections and layer normalization MLP (feed-forward network) Unembedding and sampling Model ablations (no positional encoding, no causal mask, no MLP, no residual stream) Includes interactive visualizations for every stage of the Transformer pipeline - from input tokens to the final prediction. https://sbondaryev.dev/articles/transformer