The Anatomy of Catastrophic Forgetting

We train a model on handwritten digit classification. 99% accuracy . Then we train the same model on a new task — say, fashion item recognition. We go back and test it on digits. 34% accuracy . It has completely forgotten. Not gradually, not partially — almost entirely. What Just Happened? We trained a CNN on MNIST digits — 99.2% accuracy . After fine‑tuning on Fashion MNIST, it reached 91.1% accuracy . But when re‑evaluated on MNIST, accuracy collapsed to 33.9% . This collapse is catastrophic forgetting : the model’s weights shifted to optimize for the new task, erasing the old solution. Why did training on more data make the model worse at something it already knew? MNIST is handwritten digits (0–9). Fashion MNIST is clothing items like shirts and shoes. Both are 28×28 grayscale images, but the tasks are distinct. Why Does It Happen? The core issue is that the model relies on the same set of weights for both tasks. There is no separation or dedicated memory; every parameter is shared . When training shifts from Task A ( MNIST digits ) to Task B ( Fashion MNIST ), gradient descent simply minimizes the loss on the data it sees at that moment. It has no awareness that Task A ever existed. In the loss landscape, imagine two parabolic bowls: one for Task A and one for Task B. The optimum for Task A lies at θ A ∗ ​ , while Task B's optimum is at θ B ∗ ​ . As training on Task B progresses, the weights θ move towards θ B ∗ ​ . This movement inevitably raises the loss for Task A because its minimum is left behind. The root cause is the shared weight space. Gradient descent is a stateless optimizer; it only follows the current gradient signal. Since the minima for Task A and Task B are far apart, there is no single configuration of θ that satisfies both tasks simultaneously. This is why catastrophic forgetting occurs. Weight space can be visualized as an N-dimensional space, where each axis corresponds to one parameter. Every point in this space represents a full set of wei