optimization techniques

A rule of thumb I tend to use: - if your cost function is convex, use elaborate batch methods such as L-BFGS. When doing so, initialize it to a good starting point by doing a pass of SGD first? - if your cost function is not convex and you have a small dataset: also use a batch method - if your cost function is not convex and you have a large dataset: use a stochastic method such as SGD or its variants.

Additional details:

• Except for tiny problems, full Newton methods are almost always a bad idea.
• Coordinate descent is nice when you can exactly compute the optimum. I've never used it otherwise, which does not say it is not useful.
• Conjugate gradient only helps when you do not have any better batch algorithm.
• Stochastic methods are a bit trickier to make work than batch methods, especially in the choice of the learning rate. It is more and more accepted that there is no need to make the stepsize decay to achieve good performance on the test set.