Number of samples training a CRF

Hi,

In fact, my distribution is a model of the real world, and test data is real data. What sounds strange for me is the increase in the success untill using 1000 samples, which is a high number of samples, and then go down, each time worse while using more samples.

It sounds then like you are either not training your model properly or using an evaluation metric which is not statistically consistent. For example if you are using a bad optimization algorithm it might mean that it takes longer to train with more than 1k examples and hence you end up not training that model so well. What is your test-time evaluation metric?

To measure the success I'm using the F metric (http://en.wikipedia.org/wiki/Precision_and_recall).

To train, i.e., to optimize, I'm relying on a Quasi-Newton with Limited-Memory BFGS implemented in the UGM library (http://www.di.ens.fr/~mschmidt/Software/UGM.html), and the function to optimize is the sum of the negative log likelihood of the training samples.

I've tested with 1000 training samples, and the optimization takes about 100 steps, but with 7000 it takes only 60. Is there an explanation for that? I can't figure it out...

It sounds like the convergence criteria for LBFGS is wrong, and its stopping too early.

Here's a test you can do: pick a very small learning rate and just do batch gradient descent on your objective, a learning rate small enough that eveyr step strictly decreases the objective value, and run for somethng like 1000 iteration. See which value you can achieve with this method, and which value you achieve with lbfgs. My bet is that lbfgs will achieve a worse value (higher negative log likelihood), and this would explain the behavior you're seeing.