Your CNN's Advantage Is One Assumption — and I Measured What Happens When It Breaks

A small convolutional network beats a plain flatten-and-feed-it-forward network by 7.0 points on CIFAR-10. That's convolutions, pooling, normalisation and skip connections doing honest work. Then I shuffled the rows of every image, destroying no information at all, and that 7.0-point margin fell to 0.3 . Same architecture. Same data, in a strict sense I'll defend in a moment. Almost the entire advantage, gone. The experiment Take one fixed permutation of the 32 row indices. Apply it to every image in the training set and every image in the test set — the same permutation, every time. import torch g = torch . Generator (). manual_seed ( 1234 ) row_perm = torch . randperm ( 32 , generator = g ) def shuffle_rows ( x ): # x: (C, H, W) return x [:, row_perm , :] print ( row_perm [: 8 ]. tolist ()) # [15, 9, 8, 1, 4, 12, 30, 7] That's the whole intervention. Then train two models twice each — once on natural images, once on shuffled ones: Model Params Natural rows Shuffled rows Flatten → 512 → 10 (MLP) 1,578,506 51.4% 51.7% Small CNN 94,538 58.4% 52.0% CNN's margin +7.0 pts +0.3 pts The baseline is a real fully-connected network, not a single linear layer — Flatten → Linear(3072, 512) → ReLU → Linear(512, 10) . It has the capacity to learn anything the CNN can; what it lacks is any reason to look at pixels near each other. Two things in that table are worth sitting with. The CNN wins the natural case with sixteen times fewer parameters — that's the prior paying for itself. And in the shuffled case it doesn't just lose its lead; it drops 6.4 points in absolute terms, down to roughly where the linear model already was. "You destroyed the data" — no, and this is the important part This is the objection everyone raises, so let's take it seriously, because the experiment is worthless if the objection holds. A fixed permutation is a bijection . Nothing is added, nothing is removed, nothing is averaged or blurred: img = torch . arange ( 3 * 32 * 32 , dtype = torch . float32 ). r