|
When tuning the parameters of an SVM (e.g. using cross-validation), what are in general good values to try for the cost, C? |
|
Anything from 10^-5 to 10^5, logarithmically spaced (that is, try 10^-5, 10^-4, 10^-3...10^3,10^4,10^5). It really depends on (1) the size of your dataset, (2) the margin in your dataset, (3) the expected amount of noise in your data. In general larger values of C make the optimization problem harder (they will make the optimizer take longer to find a solution) as C determines how strongly convex the problem actually is and lowers its condition number (intuitively 1/C is added to all eigenvalues of the hessian, and for all positive a, b (where a>b), and c it is true that (a/b) >= ((a+c)/(b+c))). Thanks, both!
(Feb 09 '12 at 10:07)
Vam
|
|
Try them all, in about the same time it takes to solve for one value of C! Hard to believe, but this is possible: Hastie et al. The Entire Regularization Path for the Support Vector Machine, JMLR 2004. Once upon a time Trevor Hastie had a sample implementation in R that you might still be able to find. If this works so well, why is no-one using it? Not that I doubt it, but this is already quite old work. It should have made it into one of the standard SVM solvers by now.
(Feb 09 '12 at 11:12)
Andreas Mueller
I can only speculate, and I'm not an SVM expert by any means. My tenuous understanding is that it requires solving the optimization problem in dual space. From my perspective, much of the more recent work on fast solvers is solving the optimization problem in primal space. Also, it is not clear how this would be adapted for online settings. The other consideration is that the R demo code did not handle large data sets, at least when I tried it circa 2007. This was a deal killer for me since I did not have time to spend troubleshooting or writing a more robust implementation. So I used SVM-light instead and took advantage of a small cluster we had to do a standard grid search.
(Feb 09 '12 at 11:33)
Art Munson
|
Try exponential search, i.e. powers of 10 from -5 to 5.
Which is the same as saying: there is no way to know ;) You need to have a script doing cross-validation any way. There is one that comes with LibSVM. Very easy in python also: http://scikit-learn.org/dev/auto_examples/svm/plot_svm_parameters_selection.html