Stochastic Gradient Descent with Numerical Derivatives

Main problem is that you need lots of evaluations to get a stable gradient. If you had N parameters, doing it correctly would require 2N evaluations of the loss. You can of course approximate this, but it is still a lot more computation than doing it analytically.

If you are using models of which the gradient is tedious to derive and implement, I whole heartedly recommend tools like autodiff or theano, which do this for you. It safes so much time.

If you don't want to switch, using a numerical gradient in order to check your implementation of he analytical gradient is great help as well.

You are much better off just writing code by hand to compute the derivatives. The numerical finite difference derivatives should just be used to check your other code. Since you mentioned that you would like to try automatic differentiation, I can only assume that it is possible to write the code for the derivatives.

Using numerical derivatives will be unstable numerically and thus inaccurate and it will almost certainly be much slower.

another way to smooth out the 'wild' gradients is to use mini-batches. simply build up a buffer of m examples, for some small m. Compute the gradient over this small set, update your hypothesis, and repeat. This is usually a very simple change to one's code.