|
I know this is not a programming forum but I'm trying to implement the AdaBoost algorithm with linear SVMs as weak learners. I modified the SVM such that it can take weights into account. I also checked that the implementation of the weighted SVM works 100% correct. So now I wanted to implement AdaBoost. This is the (Matlab) code:
I cannot see any error in my code and I think I the implementation corresponds correctly to the AdaBoost algorithm. But it does not work. This is a sample plot I get (first six leant SVMs) See a plot here: AdaBoost - Plot of first 6 weak classifiers The cyan colored line denotes the theta vector that describes the orientation of the separation plane (black), as one expects it is perpendicular. The plot doesn't look good in my opinion... EDIT: Finally I come to the conclusion that my algorithm is not wrong at all. I've tested the dataset against WEKA using libSVM and the results are more or less equally poor. Then I tested the same dataset with DecisionStump and I hat an AUC of 94% versus 54% using libSVM or my implementation. Although I think I used a rather simple 2D toy dataset (see the plots of my adaboost implementation) I'm quite suprised that adaboost does not handle it appropriatly. Therefore I tested the algorithm on the banana dataset and it did not work for libSVM nor DecisionStump. THEREFORE: I come to the conclusion that AdaBoost is not a very good algorithm. |
|
I have implemented Adaboost using the decision stump weak learner with Matlab. Maybe you can have a look at my code. https://github.com/xiejuncs/CS534-Machine-Learning-Programming-Assignment-Source-Code/blob/master/Bagging-Boosting/trainadaboost.m. Thanks for the answer. I tested your implementation with the banana dataset. After deactivating the normal bagging method (because it always crashed) I noticed that the adaboot error on both training and testing data remained at 1. I made shure that the format of the data is [ feature1, feature2, label] for the train and test data, as the data from the spect.mat file. Maybee you should check this as well.
(Jul 06 '12 at 05:43)
Tom
Hi, Tom. After checking the banana data set and my code, I think that the reason is the decision stump algorithm only deal with the binary feature. If the feature is not binary, for example, the banana case, you need to discretize the continuous feature into binary feature. Obviously, my code does not tackle this. Very sorry for the bug.
(Jul 06 '12 at 15:06)
Jun
No problem. Thanks for your help anyway :)
(Jul 09 '12 at 06:43)
Tom
|
|
hi. Is wsvm your written function by yourself? would you mind put the codes here? |
Does anybody know a source where i can download an implementation that does not use decision trees as a weak classifier? Logistic regression would be nice.
The norm() function returns the 2-norm of a vector. I assume you want to replace the line that says G = G / norm(G) with G = G / norm(G, 1), which will then make G a vector that sums to 1.
Okay, thanks for the tip with the norm. But the correction does not have an effect. The algorithm still does not work.