|
Can Ensemble Learning Methods be merged with support vector machine regression to improve the accuracy of a predictive model? |
|
Many good answers. I will add that when the base learning algorithm is stable, and has low variance, traditional bagging tends not to do much --- there is no algorithmic variance to reduce. Stable algorithms include naive Bayes and linear SVMs. In these cases you should try attribute bagging. In attribute bagging you randomize a stable base learning algorithm by using a random subset of the features, for each base model. This has been shown for KNN (Bay, Combining nearest neighbor classifiers through multiple feature subsets, ICML 1998.), and I expect it will also work for other stable learning algorithms. I think of this as feature bagging, where instead of (or in addition to) drawing a sample of examples for a particular bag, you draw a sample of features (Bryll et al, Attribute Bagging: Improving Accuracy of Classifier Ensembles by Using Random Feature Subsets. Pattern Recognition 36(6):1291-1302, 2003.). Interestingly random forests (usually the best classifier to be thrown at any problem) is also based on feature sampling, but it does so much more agressively.
(Aug 25 '11 at 20:27)
Alexandre Passos ♦
Good observation. Random forests incorporate feature randomization at a deep level. It would be hard to similarly randomize features at such a fine granularity for most learning algorithms. Maybe in neural networks one could randomly select a subset of inputs from the previous layer, for each hidden unit / output unit? I have not seen this tried, but it seems plausible.
(Aug 25 '11 at 20:34)
Art Munson
|
|
Here's a paper that explores the general question of ensemble selection, and includes experiments using SVMs. I think methods like bagging have more effect if you use SVMs with non-linear kernels. http://www.niculescu-mizil.org/papers/shotgun.icml04.revised.rev2.pdf In "Elements of statistical learning", they mention that bagging has no effect on linear models, which kind of makes sense.
(Aug 21 '11 at 06:53)
Mathieu Blondel
|
|
It depends on your real losss function, but usually you can use ensembles of whatever you want. If your real loss is the squared loss, you can just train lots of support vector machines (on different subsets of the data or with different kernels) and average them; if your loss is |x-f(x)| you could take the median value of the regressors' outputs. You can also fit one regressor to the data directly and another on its residuals; this has also been found to improve performance. You can also combine support vector regressors with other sorts of regressors in an ensemble model. I don't understand, however, why this is even a question. 1
To be fair, since SVMs deterministically choose the max-margin hyperplane (which tends to be pretty robust), typical ensemble methods like bagging won't necessarily do a lot; they'll be mostly redundant.
(Jul 18 '11 at 13:52)
Jacob Jensen
|
