|
For a given learning scenario, are there any thumb rules to figure out whether to use bagging or boosting? I understand that one reduces variance and the other bias, which might be a preliminary criteria to guide the choice of the particular ensemble method to be used. However, I was wondering whether people have come across other empirical factors which might lead to favoring one over the other !! |
|
Usually what you want is boosting, I think, that generally improves the performance of your classifier (by adding one layer of depth to the model). Bagging seems to be more useful when you want to derive confidence intervals and/or probability estimates from classifiers that don't naturally give you these informations, not necessarily to improve performance as such. |
|
In general, boosting tends to learn faster than bagging as it concentrates more in the misclassified cases. Another difference is that the weak learners used in boosting can be weak classifiers (e.g., small decision trees) but for bagging the weak learners actually need to be reasonably strong (e.g., large decision trees). Boosting however is a bit more sensitive to errors in training data (classification noise). For some early experimental comparison, see this paper. But then there exists more robust variants of boosting that are less sensitive to classification noise. You provided excellent links. Thank you.
(Sep 15 '10 at 17:01)
Lucian Sasu
|
|
I lean more towards bagging than boosting myself. A key issue for me is how noisy is the data I'm working with. Usually, the noise is non-trivial and bagging is a safer choice than vanilla boosting. (I don't personally have experience w/ the robust boosting variants.) My advice is to try both if it is not too hard. Just be sure to a) use enough base models to reach the asymptotic performance, and b) use early stopping on validation data to prevent boosting from overfitting. Also, I know the conventional wisdom is to use weak models with boosting, but I've seen empirical evidence suggesting that you can do better with more powerful models (e.g., big decision trees). 1
On a more technical level, both bagging and boosting can reduce both bias and variance. Performance improvements from bagging are mostly from variance reduction, whereas improvements from boosting are more from bias reduction than variance reduction. I recommend Bauer & Kohavi's paper "An empirical comparison of voting classification algorithms: bagging, boosting, and variants" for those interested.
(Sep 15 '10 at 13:24)
Art Munson
|
|
When in doubt, measure. If you don't have a theoretical basis for choosing one other the other, compare them using some test data and use the one that scored better. |
