How is divisive normalization a "poor man's whitening"?

In one of his talks, Yann LeCun stated that the divisive (and subtractive) normalization that he does in his convolutional neural nets is like a poor-man's whitening?

I don't see the parallels here. Whitening, W, is linear. In particular,

W(2 x) = 2 W(x)

On the other hand, divisive normalization, D, is non-linear. In particular,

D(2 x) = D(x)

(I'm assuming 0-mean, where needed, here, and that x is sufficiently far from 0 to neglect any "regularization")

Whitening and divisive normalization seem like two very different things. Why is one a poor man's version of the other?

asked May 15 '13 at 18:00

Max
476●16●27●29

One Answer:

oldest newest most voted

Whitening is used in machine learning to refer to making the data points have 0 mean and unit covariance. Then, divisive and subtractive normalization makes them have zero mean and unit variance in each direction, which is an approximation to full whitening.

answered May 15 '13 at 18:25

Alexandre Passos ♦
25541●54●278●421

D(x) = x / norm(x) which leads to unit-length, but it doesn't guarantee unit variance in each component. Perhaps you are thinking about feature scaling instead?

(May 15 '13 at 18:44) Max

I was, sorry. I thought what I described is what Yann did.

(May 15 '13 at 21:18) Alexandre Passos ♦

Subscription:

Once you sign in you will be able to subscribe for any updates here

Tags:

neural-networks ×186
deep-learning ×128
pca ×44
computer-vision ×22

Asked: May 15 '13 at 18:00

Seen: 1,322 times

Last updated: May 15 '13 at 21:18

How is divisive normalization a "poor man's whitening"?

Subscription:

Related questions