Hi everyone,

I'm a beginner to learn the LDA Algorithm. But I can't understand how to expand equation (14)

alt text

to equation (15)

alt text

in the original paper Latent Dirichlet Allocation(Blei et al., 2003).

Just take the last expectation E_q[log q(z)] of (14) for example, in my understanding,

alt text

But how can I get the expectation like 5th line of equation (15)?

Best regards, Richard

asked Nov 01 '11 at 00:02

Richard's gravatar image

Richard
36117

edited Nov 01 '11 at 02:29

which equation is 14 ? and which is 15 ? (there is not 5th line here... only 4..)

(Nov 01 '11 at 01:58) Dov

Sorry for missing the link of the original paper...Add it now...

What I want to know is how to expand the expectation from equation (14) to equation (15). Thanks a lot!

(Nov 01 '11 at 02:14) Richard

One Answer:

Each of the terms in equation 14 expands to one line in equation 15. To compute these expectations you just write down the expansions of the formulas for the probability densities needed and use regular algebraic identities. For example, for the first case, E[log P(theta|alpha)], note that the density of theta is a Dirichlet, so it becomes

E[log ((prod_i=1^k Gama(alpha_k))/Gamma(sum_i=1^k alpha_k)) 
          prod_i=1^k theta_k^(alpha_k-1 ))]

Expanding the logarithms you get

E[sum_i=1^k log Gama(alpha_k)  - log Gamma(sum_i=1^k alpha_k)  
    + sum_i=1^k  (alpha_k-1 ) log theta_k]

Using linearity of expectation then you get the first two terms in line 15. To get the last term, look up on wikipedia for the Dirichlet distribution the expectation of log theta_i, which is Digamma(alpha_i) - Digamma(sum_k alpha_k). Using the fact that the variational parameters for theta are named gamma then you have the final expression. For the other four lines in eq (15) you use essentially the same logic.

answered Nov 01 '11 at 08:14

Alexandre%20Passos's gravatar image

Alexandre Passos ♦
1899744214335

edited Nov 01 '11 at 08:15

Thank you very much! It is very helpful!

May I ask one more question that what's the different between the expectation operator "E_q" in equation (14) and standard expectation operator "E"?

(Nov 01 '11 at 09:03) Richard

E_q means an expectation over the variational distribution q, while E just means expectation and the distribution is implicit. In other words, E_q[f(x)] = int dq q(x) log f(x)

(Nov 01 '11 at 09:06) Alexandre Passos ♦

Many thanks!

(Nov 01 '11 at 09:16) Richard
Your answer
toggle preview

powered by OSQA

User submitted content is under Creative Commons: Attribution - Share Alike; Other things copyright (C) 2010, MetaOptimize LLC.