Evaluation of topic models through posterior predictive checks

This is an original answer by David Mimno posted here

Hi Ira,

Yes, the method should be applicable to a wide range of models, and produce comparable results across models.

There are two elements to the paper, a general method for Bayesian model checking and a specific way to measure violations of topic model assumptions. The first part, posterior predictive checks with realized discrepancies, is a way of generating confidence intervals for arbitrary functions. This type of PPC is applicable to any Bayesian latent variable model.

The second part, empirical mutual information, is the specific function that we evaluated with a PPC. When we use a topic model, we're asserting that each document only controls the topics it contains, and that the specific choice of words is left entirely to the topics. That is, the document that contains a token and the word type of that token are statistically independent conditioned on the token's topic. We measure empirically whether that's actually true, by calculating the mutual information between the "document" variables and the "word type" variables given the "topic" variables.

Any model that includes a topic indicator variable for each token can be evaluated in this way. You just have to make a table with three columns (document, topic, word id) and one row for every token. Most Bayesian hierarchical models fit this criterion.

-David