Revisiting the Efficient Channel Attention paper (2019, 12k citations) - the central hypothesis isn't quite right [D]

ECA was positioned as a successor to SE . The idea behind ECA is quite simple. Unlike SE which reduces the channel means into a smaller hidden layer, it directly uses a 1d convolution kernel on the channel means themselves, avoiding the need for dimensionality reduction. The results are undeniable: ECA is a clear improvement over SE. The authors claim that cross-channel interaction is a key ingredient. But on a conceptual level, the design of ECA doesn't make much sense. Let's take a step back. Why do we use convolutions in the first place? Convolutions are fundamentally designed for data with an underlying topology (e.g. space or time). They assume locality (adjacent elements interact) and translation invariance (the same kernel applies everywhere). Sliding a kernel across a 2D image works because coordinates have meaning, and the statistical properties of an image are largely stationary across the frame. This isn't perfectly true - which is why modern CNNs have moved towards dynamic convolutions - but it's still good enough to be useful. If you randomly permuted the pixels in an image, a convolution would be meaningless. Now consider tabular data. Suppose we have 32 channels e.g. [cost, weight, material, colour, volume, speed, ...]. Using a CNN architecture for this kind of data is clearly inappropriate. A 1d kernel of width 3 would be moved across the channels, so that [cost, weight, material] was input and also [ weight, material, colour] was input and so on, and have to somehow output something meaningful. ECA is doing exactly this type of computation. ECA does a 1d convolution over the channel dimension. It is a cursed convolution because tabular data does not have a topology to suit it. In practice, if you did use a CNN on tabular data, I would expect better than random performance because neural networks are ridiculously good at fitting to the dataset given their constraints and would reorganise the channel order (using the initial 1x1 projection layer) to s