|
I have started to read articles about deep Boltzmann machine, and I wonder can free energy of deep Boltzmann machine be computed efficiently like in RBM(linear with respect to the number of hidden units). It seems the answer is No because now hidden units are dependent given visible units, but still can someone clarify this? |
|
You are correct. The free energy of deep Boltzmann machines (conditioned on the visible variables) is intractable for the same reasons that the partition function of an RBM is. If you have a three layer DBM and fix the visible units to a certain value, the two layers of hidden units form an RBM in which none of the variables are observed and all depend on each other. It is not possible to compute the marginals of these units either. This is why in most papers about DBMs, a mean-field approximation is used to approximate the free energy with an upperbound (often presented as a lowerbound on the likelihood by including the partition function and reversing the sign) and the gradient of this bound is used as a substitute for the gradient of the free energy. You can think of the free energy as a conditional log partition function. Thank you very much for your answer!
(Jul 11 '13 at 06:57)
Midas
|
|
The free energy over all even layers is tractable. The free energy over all odd layers is tractable. The free energy over just one layer is not tractable, unless the DBM has <= 2 layers. @IanGoodfellow 'unless DBM has <= 2 layers' Do you mean visible and hidden layers or two hiden layers?
(Jul 11 '13 at 16:29)
Midas
|
|
Hello everybody, I have a question somehow related to this. I'm trying to do collaborative filtering with Deep Boltzmann Machines, inspired by Hinton's paper on RBM on the topic : Restricted Boltzmann Machines for Collaborative Filtering. I'm trying to build a similar prediction using DBMs instead. The thing is, that prediction highly rely on the model's free energy computation ; also, the mean-field inference solution to tackle that problem in a multi-layer case seem intractable to me (at least, from a computation viewpoint, when compared to a simple RBM) : computing the lower bound for a 2-hidden layer model isn't easy, and besides requires using Monte-Carlo methods can greatly slow calculus. Does anybody has solution for this ? I am working on Python 2.7. |
