0
2

I know this is not a programming forum but I'm trying to implement the AdaBoost algorithm with linear SVMs as weak learners. I modified the SVM such that it can take weights into account. I also checked that the implementation of the weighted SVM works 100% correct.

So now I wanted to implement AdaBoost. This is the (Matlab) code:

    function models = adaboost( X, Y )
        N = length(X);
        G = 1/N * ones(N,1); %The weights
        M = 20; % Number of boosting iterations 
        models = cell( M, 4 ); % will keep track of learned parameters
        for m=1:M
            Cs = [.01,.1,.5,1,10]; %The cost parameters of the linear SVM
            best_error = -1;
            C_opt = -1;
            theta_opt = -1;
            b_opt = -1;
            %Learn the best parameter
            for cc=1:length(Cs)
                C = Cs(cc);
                [theta,b,S] = WSVM.wsvm( X, Y, C, G ); %So this here is a weighted SVM
                YPred = WSVM.classify( X, theta, b );
                %err = 1/sum(G) * sum( G.*(Y ~= YPred) ); %alternative as proposed in Bishop - Pattern recognition page 658
                err = sum(.5*G.*abs( Y - YPred ))/sum(G); %another way of error calculation but essentially equal to line above
                if err < best_error | best_error < 0
                    C_opt = C;
                    theta_opt = theta;
                    b_opt = b;
                    best_error = err;
                end
            end

            %Calculate the error again for the best parameter
            YPred = WSVM.classify( X, theta_opt, b_opt );
            err = sum(.5*G.*abs( Y - YPred ))/sum(G);
            alpha = log( (1-err)/err );

            %Update weights for the next round
            %G = G.*exp( alpha .* (YPred ~= Y) );
            G = G.*exp( - alpha.*Y.*YPred );
            G = G/norm(G);

            %Store model parameters
            models{ m, 1 } = theta_opt;
            models{ m, 2 } = b_opt;
            models{ m, 3 } = C_opt;
            models{ m, 4 } = alpha;
        end
end

I cannot see any error in my code and I think I the implementation corresponds correctly to the AdaBoost algorithm. But it does not work. This is a sample plot I get (first six leant SVMs)

See a plot here: AdaBoost - Plot of first 6 weak classifiers

The cyan colored line denotes the theta vector that describes the orientation of the separation plane (black), as one expects it is perpendicular.

The plot doesn't look good in my opinion...

EDIT:

Finally I come to the conclusion that my algorithm is not wrong at all. I've tested the dataset against WEKA using libSVM and the results are more or less equally poor. Then I tested the same dataset with DecisionStump and I hat an AUC of 94% versus 54% using libSVM or my implementation. Although I think I used a rather simple 2D toy dataset (see the plots of my adaboost implementation) I'm quite suprised that adaboost does not handle it appropriatly. Therefore I tested the algorithm on the banana dataset and it did not work for libSVM nor DecisionStump.

THEREFORE:

I come to the conclusion that AdaBoost is not a very good algorithm.

asked Jul 05 '12 at 11:10

Tom's gravatar image

Tom
71101214

edited Jul 11 '12 at 05:24

Does anybody know a source where i can download an implementation that does not use decision trees as a weak classifier? Logistic regression would be nice.

(Jul 06 '12 at 04:21) Tom

The norm() function returns the 2-norm of a vector. I assume you want to replace the line that says G = G / norm(G) with G = G / norm(G, 1), which will then make G a vector that sums to 1.

(Jul 06 '12 at 11:01) Alexandre Passos ♦

Okay, thanks for the tip with the norm. But the correction does not have an effect. The algorithm still does not work.

(Jul 09 '12 at 06:42) Tom

3 Answers:

I have implemented Adaboost using the decision stump weak learner with Matlab. Maybe you can have a look at my code. https://github.com/xiejuncs/CS534-Machine-Learning-Programming-Assignment-Source-Code/blob/master/Bagging-Boosting/trainadaboost.m.

answered Jul 05 '12 at 13:39

Jun's gravatar image

Jun
16225

Thanks for the answer. I tested your implementation with the banana dataset. After deactivating the normal bagging method (because it always crashed) I noticed that the adaboot error on both training and testing data remained at 1. I made shure that the format of the data is [ feature1, feature2, label] for the train and test data, as the data from the spect.mat file. Maybee you should check this as well.

(Jul 06 '12 at 05:43) Tom

Hi, Tom. After checking the banana data set and my code, I think that the reason is the decision stump algorithm only deal with the binary feature. If the feature is not binary, for example, the banana case, you need to discretize the continuous feature into binary feature. Obviously, my code does not tackle this. Very sorry for the bug.

(Jul 06 '12 at 15:06) Jun

No problem. Thanks for your help anyway :)

(Jul 09 '12 at 06:43) Tom

... how about just using weka? It has adaboost and will let you choose the classifier e.g. SVM, Logistic regression etc.

answered Jul 06 '12 at 15:51

amair's gravatar image

amair
2452312

edited Jul 06 '12 at 15:53

Yeah, that actually was a good tip. So I downloaded WEKA and had a look at the source code.

(Jul 10 '12 at 08:09) Tom

hi. Is wsvm your written function by yourself? would you mind put the codes here?

answered Aug 11 '13 at 03:33

parmin's gravatar image

parmin
1

Your answer
toggle preview

powered by OSQA

User submitted content is under Creative Commons: Attribution - Share Alike; Other things copyright (C) 2010, MetaOptimize LLC.