|
In a variational Logistic regression approach, we approximate a logistic sigmoid function to a Gaussian Distribution by means of its lower bound. It can be solved using an EM algorithm, given that we work with a fixed prior and known initial parameters (which we usually do not know). My question is: What is the merit of this approach? since common logistic regression has a closed solution and is faster to implement than an EM approach. What can we learn from the data by using the variational approach rather than the straight forward approach. Thanks |
|
The variational approach fits a Bayesian logistic regression model. This means that you do not only get the maximum likelihood solution for the weights, you get a distribution over weights. And even more: you can integrate out the weights and make predictions considering "every possible weight-vector". If you think this is a good thing probably depends on your opinion on Bayesian methods. In practice, this amounts to "automatic regularization" - where you hope that you can either integrate out hyper parameters, hope to have some idea what you should use, or use non-informative priors. |
|
Adding to Andreas Muller's answer, the main advantage of variational logistic regression for me is that it is easier to adapt to more complex models as a black box. So you can, for example, do variational inference on a supervised LDA model with logistic regression using the topics as features, and it will all fall out nicely, while trating the logistic function analytically would be a fair amount of work. You mean since a variational approach allows us to model a sigmoid as a normal, and thus we can use Conjugate properties?
(May 31 '11 at 01:57)
Leon Palafox ♦
|
Can you give some source where this method is described?
Sure Thing, it's in Bishop's chapter on Variational Inference (10) sub-chapter 6 , variational logistic regression (10.6)
Are you sure that common logistic regression has a closed form solution?
Logistic regression does not have a closed form solution.
Indeed, it was my mistake, still, MAP Logistic regression is way simpler than variational logistic regression