regularization in PCA/ZCA whitening

0

When applying PCA/ZCA whitening to contrast-normalized image patches, people seem to use regularization, in which the whitening matrix is chosen to be

W = U * diag(sqrt(1/(D + epsilon))) * U'

(where U and D are the eigensystem of the data covariance matrix).

In the code I've seen, epsilon = 0.1.

Is there a principled and automatic way of choosing epsilon (other than using labeled data and cross-validation) ?

Edit: the expression for W clarified

asked May 21 '12 at 21:42

Oleg%20Trott's gravatar image

Oleg Trott
246●8●10●16

edited May 23 '12 at 14:51

One Answer:
oldest newest most voted
0

You can regularize amount of variance of the data to keep, e.g. 90%, using the cumulative sum of the sorted eigenvalues. If you choose wisely, you avoid division by zero and do not need an epsilon. However, this just makes your threshold a bit more interpretable, you'll have to see what it does for your task. E.g. in the "algorithms for hyper parameter optimization" paper, James Bergstra et al also crossvalidate the ZCA energy.

answered May 22 '12 at 10:45

Hannes%20S's gravatar image

Hannes S
86●2●2●9

  1. But how would you choose 90% vs, say, 70%? (other than using labeled data and cross-validation)

  2. In the above expression for W, ignoring small principal components would be equivalent to formally setting small eigenvalues (d's) to inf. To me, this seems to create a disturbing discontinuity.

(May 23 '12 at 15:02) Oleg Trott
Your answer
toggle preview

learn more | privacy | contact

powered by OSQA

User submitted content is under Creative Commons: Attribution - Share Alike; Other things copyright (C) 2010, MetaOptimize LLC.