|
Obviously not, but bear with me. The variational lower bound says that, for any Q, log P(x) > - KL(Q||P), right? So what if I set Q=P? Then, obviously, the KL divergence will be zero and hence the log-likelihood has to be lower-bounded by zero? You can get at something similar if you go the long way, and write down the usual derivation of the variational lower bound using jensen's inequality: log P(x) = log int dz P(x|z) P(z) = log int dz Q(x,z)P(x|z)P(z)/Q(x,z) > int dz Q(x,z) log(P(x|z)P(z)/Q(x,z)); but if you choose q = P the log will always be zero. Where is the mistake? |
|
First of all it seems that in your "long way" derivation you misapply Jensen's inequality: int dz Q(x,z) is not 1. A correct derivation is like this: log P(x) = log int dz P(x|z) P(z) = log int dz Q(z)P(x|z)P(z)/Q(z) > int dz Q(z) log(P(x|z)P(z)/Q(z)) So the variational lower bound is int dz Q(z) log(P(x|z)P(z)/Q(z)) Note that in order to have this lower bound equal to zero you should set Q(z) = P(x,z) but this can not be the case (x is missing from the left part) and I am not sure what are you trying to prove :-) Thanks, I understand it now :-)
(Apr 01 '11 at 06:04)
Alexandre Passos ♦
|
|
This may just be making the same observation you are (P=Q being the same distribution KL diverges to zero), but does this help you at all? link text If nothing else there is some contact info at the top maybe that will lead to a better informed answer.
This answer is marked "community wiki".
I'm not sure. That KL goes to zero for Q=P is well known, and IIRC the EM bound is different from the variational bound in that the EM bound is not just a negated KL divergence, while the variational bound is.
(Mar 31 '11 at 14:02)
Alexandre Passos ♦
|
|
How do you get log int dz P(x|z) Q(z)P(z)/Q(z) > int dz P(x|z) Q(z) log(P(z)/Q(z)) ? If you applying Jensen inequality, you have to take the P(x|z) into the log also. Typo, I'm fixing it now.
(Mar 31 '11 at 18:40)
Alexandre Passos ♦
|
