|
I'm looking for a dimensionality reduction tool for visualization: I have N data points, each of which has a k-dimensional vector of feature values. What are some good techniques and tools to reduce the feature dimensionality down to 2 features, so that all N data points can be nicely drawn on a map? |
|
If k is not very high (say less than 10) or highly correlated and simple projection on the first 2 components of a PCA might do. Otherwise you can try Multi Dimensional Scaling (MDS) or t-distributed Stochastic Neighbors Embedding( t-SNE). If you expect your data to live on a single fully connected manifold, Locally Linear Embedding (or Hessian-LLE) might work well. 1
t-SNE should be tried first because there is code on the web for it (http://homepage.tudelft.nl/19j49/t-SNE.html) and it seems to dominate the other methods you mention in my experience
(May 07 '11 at 02:19)
gdahl ♦
There is code for the other methods in every machine learning library ;)
(May 11 '11 at 10:53)
Andreas Mueller
|
|
The fastest, ugliest and simplest way I can think of for that kind of problem would be Self Organized Maps. Self Organized Maps are designed to do exactly what you want, they do a projection from an N-Dimensional Space to a 2D map, while preserving the topology of your data, so nearby vectors are still grouped together. It is rather easy to implement and there are a lot of good libraries written for it. Adding to this, there was a paper from Hinton's group not too long ago along these lines. They basically used a combination of a DBN and PCA, as I recall. I wish I could say which paper it was, who the author is, or even when it was published ... it's kind of a blur with how many I've sifted through.
(May 13 '11 at 14:23)
Brian Vandenberg
|
|
I'd try everything ogrisel suggested and also possibly isomap (when you expect your data to live on some manifold - but otherwise: how would you project it down anyway?) You can also try Kernel-PCA or any other dimensionality reduction method like (kernel) ICA or Factor Analysis. About t-SNE: This is from Hinton's lab and I think there is some magic involved. Hinton talked about it at last NIPS and used phrases like "... and then I let the probabilities sum to 4. The unnormalized probabilities summed to two. I tried to normalize it but then it didn't work as well so I went in the other direction." There is a fairly detailed discussion of t-SNE in this thread. 2
Hehehe I think he just says that because he knows it will annoy the folks at NIPS. Having probabilities that sum up to 4 in the early stages of the optimization is not at all required to get good results with t-SNE. Admittedly, there are quite a few tricks involved in getting into good local minima of the t-SNE cost function (none of which you have to bother about much when using the online implementations).
(May 04 '11 at 10:51)
Laurens van der Maaten
Could be. I have never tried it myself, he just gave the impression that there is lots of magic.
(May 11 '11 at 10:52)
Andreas Mueller
|
|
This isn't exactly what you asked for, but depending on what you need you might try Parallel Coordinates. They can be useful for identifying regions in feature space where points clump together, especially if you can represent different groups (classes, clusters) with different colors. There are weaknesses, of course (for one, the ordering and scaling of the axes can impact how easy/difficult it is to draw conclusions) but I believe it is a pretty popular technique for visualizing high-dimensional data. EagerEyes has another good overview. Hope this helps. |
|
As an alternative to SOM, see the Generative Topographic Maps. What is wrong with generative topographic maps? There is a free implementation contained in the toolbox Netlab for Matlab.
(Jun 17 '11 at 09:29)
Nikos G
|
|
Hi Guys, Just to parasite to this thread and ask a quick question, in that scenario, let' say we have a 100 snippets (each with around 100 words), how do we determine the top k (say 10) important terms from that collection? Regards, Andy. Tf-idf (http://en.wikipedia.org/wiki/Tf%E2%80%93idf) maybe? Why don't you open a separate question for this?
(Jun 08 '11 at 16:47)
Frank
|
|
Hi Guys, Just to parasite to this thread and ask a quick question, in that scenario, let' say we have a 100 snippets (each with around 100 words), how do we determine the top k (say 10) important terms from that collection? Regards, Andy. Please post a new question. It helps everyone.
(Jun 09 '11 at 00:27)
Robert Layton
|
Try PCA first, before anything else. If your data is simpler than you think, this might to a very good job of mapping it. It's also about 2 lines in matlab, or any machine learning library.
But someone here said that PCA is only good if I have about <10-dimensional feature vectors. Mine are pretty large and can be in the order of thousands or a million.
If they are highly correlated this might still give you an interesting viz. If you have a milion features, you need a PCA lib that supports sparse inputs such as this implementation.
PCA gives you the fraction of variance for the first n eigenvectors, e.g. % total variance: [ 64.6 81.7 89.4 ... ], so look at this curve for your data. (1000 raw? features sounds noisy though.)