|
Hi, I am trying constrain an empirical covariance matrix to be positive definite - I calculate the covariance from the data, but occasionally I don't have enough data to make the covariance matrix positive definite (or even invertible...). What I do now to constrain the matrix to be PD is I calculate the eigenvectors matrix V, the eigenvalues matrix D - and set every entry of D which is smaller than epsilon (say, 1e-6) to epsilon. Then the covariance matrix is simply sig=V'DV. Then, to make sure the matrix is symmetric I do: sig = (sig + sig')/2; The problem is that occasionally this causes sig to STILL have negative eigenvalues, even though I explictly constrain them to be positive. What am I doing wrong? are there more established ways to regularize covariance matrices? Thanks! |
|
Have a look at the documentation for the covariance learning part of the scikit-learn: http://scikit-learn.org/dev/modules/covariance.html I don't have time right now to summarize the theory behind this, but hopefully the documentation and examples give pointers and key elements. Don't hesitate to improve it :). Thanks, looks great - I'll read it through.
(Dec 05 '11 at 03:23)
zdanielz
This is exactly what I was looking for - thanks again!
(Dec 05 '11 at 08:56)
zdanielz
|
|
If what you have is not enough data to accurately estimate your covariance matrix what you really should be doing is some form of factor analysis, which is modelling your data with a Gaussian with a covariance matrix restricted to be the sum of a diagonal term and a low-rank term, both of which are easy to estimate in your setting. If I understand correctly, a factor analyzer assumes the data resides on a low dimensional subspace, and the rest of the dimensions are "noise" of some sort - this is not the case I have, my data is not on a subspace of the data space, I just don't have enough data... In any case, in practice, adding a diagonal term to the low rank covariance matrix is what I do in any case (moreover after reading the reference Gael pointed me to...).
(Dec 06 '11 at 09:18)
zdanielz
This is just one interpretation of factor analysis. Another is that you can reliably estimate a subspace where your data is known to vary (the low-rank part of the covariance matrix) and the individual variance of each component (the diagonal part). If your data is not perfectly modeled by the low-rank part the diagonal part will capture most of the variance. Varying the rank of the low-rank part can be done to test whether this approximation is harmful or not (set a bunch of points aside, estimate factor analyzers with many different ranks, compute the held-out likelihood of the points set aside, choose the rank which maximizes that or the first rank at which this stops growing). If it is not harmful then the data is just not enough to estimate more than X axes of variation, which is ok.
(Dec 06 '11 at 09:22)
Alexandre Passos ♦
Thanks, I'll see you how this works my stuff here...
(Dec 06 '11 at 09:36)
zdanielz
|
V' D V should always be symmetric.
I think the usual way of regularizing covariances is adding a constant to the diagonal, which is roughly equivalent to a prior.
I agree that V'DV should always be symmetric, but sometimes due to numerical issues (I suppose) it isn't. that's why I do the (sig+sig')/2 part. But then, as it seems, some of the eigenvalues turn negative. Adding a constant to the diagonal helps, but suffers from similar problems... I'll try it again.
just curious, but why do you want to do this?
I have relatively high dimensional data, and I don't have enough data to accurately estimate the covariance matrix...