Preprocessing images, centering each pixel

Thanks for correcting my statement about correlations. So for a full RGB image representation, would your matrix be n x (p*k) where k is the total number of pixels?

What I meant to say was that the inherent smoothness/consistency between neighboring pixels would be lost for each individual image (ie the normalization would make it look much less like an image). It seems like it would be harming things like rotation/shift invariance since the normalization assumes each pixel globally represents the same feature in the dataset. I can understand this centering being more useful in non-image tasks than image related ones.

From just the quote you provided, I don't know if they're operating on individual pixels (ignoring neighbors) or using the neighborhood around each pixel during training/classification. If you are operating on independent pixels, then for an M x N RGB image, I would represent X as a k x p matrix (where k = M * N and p = 3 ). You could just as easily use the transpose of that representation. If you are using neighborhoods, then you probably would want to keep an M x N x p representation.

I'd be careful about using the term "normalization" to refer to mean subtraction because normalization typically involves scaling (stretching or contracting) of the data values, which is not happening with mean subtraction. From an algorithmic perspective, there is no loss of smoothness or consistency due to mean subtraction (vector differences between pixels in an image will remain unchanged). Of course, the image would visually appear different if rendered with means subtraction.

If you use one image (or set of images) for classification and then try to classify pixels in a new image, you don't want your algorithm to perform poorly due to a mean offset between the training data and new data (e.g., due to the overall scene being brighter in the new image). By subtracting the RGB means separately from each image, you can mitigate that effect (to the extent that it can be represented by an additive offset).

I think we might be referring to different types of centering. To simplify things, let's refer to 300x300 grayscale images. For a dataset of 10000 images, my datastructure would be 10000x90000, would I center the columns of this matrix by subtracting each row by the same 90000 element vector?

If you are trying to reproduce the quoted experiment, given the example you just stated, it isn't clear whether you would subtract a scalar from your 10000x90000 matrix (mean pixel value subtraction) or subtract a common length-90000 vector from all rows of the matrix (mean image subtraction). I would need to know more about what the author did in his/her experiment (details are missing from the quote).

Both row-wise and column-wise normalisation (and I guess by extension subtracting the mean, which is normalisation without the scaling step) can be useful.

If each row is an example, as in your description, then normalising each row boils down to a form of brightness/contrast normalisation in the case of image data. I suppose by leaving out the scaling step it affects the brightness only.

Normalising each column in the matrix amounts to 'feature-wise' (or in this case pixel-wise) normalisation, which I think is particularly useful for models trained with gradient descent, since that tends to work better if the all input features have the same scale.

Of course if the scaling step is left out, I suppose this only affects the bias terms in such models (which can also be useful, since you save time not having to learn the right biases first).