|
I'm trying to understand practical difference between a (denoising, sparse, contractive) Autoencoder and a Restriced Boltzmann Machine. One thing I don't quite get. It is said that RBMs are generative, you can 'run them backward' if you will. More formally you can sample from them. AEs are not said to have this property but I can't imagine why not. The AE, too, learns some representation of the input data? The end state of the two approaches are the same right? Both have a weight matrix and units with some activation function. |
|
AEs are a discriminative model. They are training to predict the conditional distribution P(x|x') where x is the clean data output, and x' is the input data. RBMs are graphical models which encode the joint distribution P(x,h) over the data and the latent variables. Because the joint distribution is over x we can sample from it by marginalizing out (or conditioning on) the hidden variables h. The difference is very similar to the differences between generative and discriminative models in machine learning more broadly. Recently, however, people at U. Montreal have been working on a version of AEs from which you can draw samples of x. That work in progress is here: Generative Autoencoders The Bengio tutorial on deep learning also discusses RBMs compared with AEs I believe. |
|
Actually recent work (Alain and Bengio 2013) shows that autoencoders can be interpreted as generative models. Instead of estimating the density directly, the reconstruction error of the autoencoders is an approximation to the gradient of the original density function. It is possible to sample from the density induced by an autoencoder, even. |
