Chain of Thought — why 'think step by step' actually works

📺 Prefer to watch? 90-second YouTube Short · 💬 Telegram Originally published on software-engineer-blog.com . You already know the trick: add "think step by step" to your prompt and the model's answer gets better. Almost nobody explains why — and the real reason has nothing to do with motivation or effort. Mental model: A transformer spends a fixed stack of layers per token, so adding reasoning tokens doesn't make the model smarter — it buys it more compute passes and an external scratchpad to read from. The Problem: Fixed Compute per Token Here's the floor. When a transformer generates a token, it runs through the same neural network layers every time. The stack depth is fixed at model-creation time. Whether you ask it "2+2" or "Sara has 3 packs of 8 markers, gives 5 to each of 4 friends, how many left?", the model gets the same amount of layered computation to produce each output token. That compute budget never grows with problem difficulty. Now imagine you ask for just the answer: "Sara has 3 packs of 8 markers, gives 5 to each of 4 friends, how many left? Answer only the number." The model has to solve a three-step problem (multiply 3 × 8 = 24, multiply 4 × 5 = 20, subtract 24 − 20 = 4) in a single forward pass. It needs to hold "24" and "20" somewhere while computing the final step. But it's only got one forward pass, one set of layer outputs, and nowhere internal to stash intermediate values. So it guesses. It might say 19. It didn't get the math wrong because it's bad at math. It got it wrong because you handed it the wrong compute budget for the job. The Mechanism: Three Small Shifts Now ask the same question and let it write the steps: "Sara has 3 packs of 8 markers, gives 5 to each of 4 friends, how many left? Think step by step." Three mechanical things happen: 1. The model becomes a loop. Every token the model emits is appended to the input context and fed back in on the next forward pass. So if it writes "First, 3 × 8 = 24", that token sequence gets rea