Adam and AdamW: The Optimizer That Made Modern LLM Training Possible

Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product. Most people learn neural networks by staring at the model. Weights. Attention. MLPs. LayerNorm. Tokenizers. Context windows. But when you actually train an LLM, there is another piece of machinery making billions of decisions every second: the optimizer. A 70-billion-parameter model does not "learn" because gradient descent tells it which direction is better. It learns because an optimizer turns an enormous, noisy stream of gradients into parameter updates that are small enough not to explode, large enough to make progress, and adaptive enough that different parameters can move at radically different effective rates. For the last decade, the dominant answer has largely been some form of Adam , and increasingly AdamW . The interesting part is that Adam is not some mysterious LLM-specific invention. The original Adam paper was submitted in December 2014 by Diederik Kingma and Jimmy Ba, before the Transformer, before GPT, and before the modern LLM era. Kingma was working on scalable machine learning and generative models; Ba was then a PhD student working with Geoffrey Hinton at Toronto. Three years later, the Transformer paper used Adam directly in its training recipe. Then came AdamW, which fixed a subtle but important problem in how regularization interacted with adaptive optimization. By 2025, Adam was sufficiently influential to receive an ICLR Test of Time award. So what exactly is Adam doing? And why is AdamW usually what you actually want when training a Transformer? 1. First, forget Adam: what problem is the optimizer solving? Suppose your neural network has parameters theta = [theta_1, theta_2, ..., theta_N] and your training batch produces a loss L . Backpropagation gives you g = dL/dtheta The simplest possibl