Questions about prior distributions

Just to touch on the general issue of inference, introducing a Dirichlet prior can be viewed as adding extra data to the problem.

For instance, suppose you observed k heads out of n tosses, your prior is Dirichlet(a,b), what's the probability the next toss is heads? Bayesian answer is (k+a)/(n+a+b). This is the same as Maximum Likelihood estimate if you observed additional a heads and additional b tails. As n increases with a,b fixed, relative contribution of a,b becomes small. In the limit of a,b going to 0, Bayesian Estimate and Maximum Likelihood estimates coincide.

You probably have a bug in your inference algorithm. Are you using sampling or variational? Regardless, when you're initializing the topics, you should take the prior into account. If you just assign words to topics uniformly you risk taking a very long time until your model settles down to the correct posterior proportion, and this might never happen with variational.

A quick hack I use when I'm using assymmetric priors (that is, with one alpha_i different from the others) is to assign words not to uniform(Ntopics) but to Discrete(alpha/sum(alpha)) (or even Discrete(Dirichlet(alpha)), which is more correct, or even Discrete(Dirichlet(alpha+current_counts)), but this can bias your model in uninteresting ways sometimes, specially if the alphas are small).

Noel Welsh is correct, and the dirichlet distribution is not defined for alpha=0, but it doesn't lose its meaning or any of its properties if you just assume that there is probability one that a component with prior alpha_i = 0 will have posterior counts 0.

To see why the distribution forever stays empty at the component with alpha_i = 0 you have to look at the sampling/variational equations. The probability of adding a word to a topic, when gibbs sampling, is proportional to (count_topic_i_in_document + alpha_i)(count_word_in_topic_i + beta_i)/(all_words_in_topic_i + V*beta_i). So if alpha_i is 0 and count_topic_i_in_document is zero as well, this probability is zero and no word will ever be assigned to that topic. In variational this is less clear, but I don't know of any vb implementation out there that lets you use assymmetric alphas.

There are at least two questions here: what role does a prior distribution play in Bayesian inference, and why is your particular implementation of LDA behaving in the way it does.

Addressing the second question to start with, the Dirichlet distribution is only defined for alpha > 0. The Gamma function, from which the normalising constant is calculated, is not defined at zero as you can see from the graph on the linked page. My guess is that you're seeing a numeric issue in your code.

Now onto the first question. One view is that yes, the prior is just a way of performing smoothing. Others argue that there are always a prior, but it just might not be explicit. From a purely engineering point of view you need a prior to compute a posterior. I find Bayesian methods appealing as they 1) quantify uncertainty in the true model, which is useful for decision making, and 2) provide a simple unifying framework for inference.