|
I am trying to estimate the sparse inverse covariance matrix in very high dimensional data, I mean with million variables. Up to now all the papers like this that I have found, they are limited to few thousands. All of them like graphical lasso, non parnormal, they use the estimated covariance matrix and then use the gaussian likelihood function which they optimize to find the precision matrix, which actually encodes the graph structure In my case, I have million nodes. So if I try to estimate the covariance matrix at the beginning, that is a dense matrix of 1 million x 1 million. I will run out of memory with that I wanted to know if any methods have been devised that actually deal with this issue. |
|
The closest that you will get to is the PC-DAG method by Rutimann and Bulhmann: http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.ejs/1259677088 @Gael But in their paper, they have experiments with p = 120-400 which is much lesser than I need. I have a graph of 1 million nodes. I need something of that sort
(May 20 '13 at 12:41)
Jason Tyler
|
|
I'm not aware of any implementation that does this, but you could potentially perform one million lasso regressions of each variables on all other variables (excluding itself), and pack the regression coefficients into a precision matrix (accounting for the fact that one variable will be missing there and there's a diagonal in the final precision matrix). This approach is taken by Meinshausen and Buhlmann in "High-dimensional graphs and variable selection with the Lasso", Annals of Statistics, 2006. You might also need to fix issues that arise when one regression selects a particular entry as zero and another as non-zero, like making it zero if any one is zero etc. Not sure whether the resulting precision matrix is positive definite, their paper is very dense... |
