|
Line search is a subroutine of descent-direction-based convex optimization techniques. Stephen Boyd argues in his lectures that the overhead of doing line search accurately is not worth it, and we should use something simple like some backtracking search. Such backtracking search does not employ the derivatives of the objective, only its values. In logistic regression (and general discriminative parameter learning in factor graphs), the overhead of computing the derivatives of the objective is not much more than computing the objective (once you have the partition function). Therefore, it seems that it might be worth using a line search technique that uses the derivative (such as DBrent, from Numerical Recipes). Does anyone have any experience with using these more sophisticated line searchers? Is it worth the work to implement it? |
|
The work to implement them is usually just copy-porting some code from numerical recipes (unless you want to release it as open-source, which is not allowed by the license in the book). There are some backtracking line searches that use the gradient as well. David MacKay's optimizer, for example, does a slightly fancier line search. I'm not sure, however, if that actually matters in practice for logistic regression. |
