Bakeoff to compare language models?

I have two objections:

1. A language model is supposed to be generative, and not discriminative, and this is clearly a discriminative setting (if we're only after the argmax que can use margin methods and other non-probabilistic methods to estimate things that won't necessarily be densities). Also this doesn't take in consideration the fact that there is more than one answer for this sort of question; it's more honest to frame this as a ranking problem (there is a true order between the sentences and you're trying to predict that) than a classification problem, but this makes it harder to evaluate (kendall's tau? NDCG? ERR?). More clearly, if we have access to the loss of choosing the wrong translation we can optimize that (say, with an M³N) instead of building a consistent generative model. This might be better suited for this task, but it's not an actual languahe model (in the same sense that naive bayes is a language model but logistic regression isn't).
2. You might be able to solve this task better without a real language model (this is related to the ranking thing above). If we can see that all the candidate sentences are similar in some ways (since they're presumably samples from a decoding model) we can probably hack something that produces inconsistent probabilities (like prefers sentence A to B in some contexts but B to A in others) that does better than a straight language model for this task (assuming we have access to training data)

But overall I like this idea. Maybe one could couple that with speech recognition and text compression (i.e., test-set perplexity) to get a fair evaluation.

Also there is the issue that BLEU might bias towards n-gram-based models.

Maybe the ideal way to represent this bakeoff should be code-focused: a corpus format is specified, together with an interface for querying the log-probability of a sentence, and maybe a toy subset of the real data is made public, so people can be sure their code works correctly. Then the evaluation process consists of people submitting the code and something like MLCOMP actually runs the pipeline with the corpus, the development sentences, and the test sentences and reports the results on the development sentences (and before a conference workshop or something like that the real test results are revealed and people get to present their methodology). What do you think?