What is the difference between conditional and discriminative models?

In a probabilistic setting, training a generative model typically means that you estimate a joint probability: P(x, y), whereas a discriminative model means that you estimate a conditional probability: P(y | x).

Used more loosely, a generative model is one in which the error term is computed with respect to some unsupervised criterion over the input, e.g. reconstruction error. A discriminative model is one tuned for a supervised task of interest.

Ng + Jordan (2001) "On Discriminative vs. Generative classifiers", among other works, demonstrate that generative models converge faster, but discriminative models achieve lower error. Hence, one can imagine a training regime in which one first quickly fits a generative model, then fine tunes the generative parameters under a discriminative criterion.

In fact, that is the very training strategy of most deep architectures. However, in a deep setting, the generative then discriminative training approach is used not for speed, but to avoid local minima in the optimization. In particular, modeling the input carries a stronger error signal than modeling the output (because the input is typically much high dimension than the output), which makes it less likely for the optimizer to fall in bad local minima. After the optimizer has found a good local minimum using the generative criterion, it then fine tunes against the discriminative criterion to improve its generalization.

A probabilistic model is places a distribution on a group of random variables. It is generative if some of those random variables which are explicitly parametrized are input random variables. Usually, the input variables condition on the label variables in such models.

Strictly speaking, being discriminative is a criterion for selecting model parameters which for the specific case of conditional likelihood models means selecting parameters to maximize P(y|x); obviously, this criterion would be indifferent to the setting of parameters associated with P(x), since you condition on them. However, discriminative is more broad since non-probabilistic approaches, think SVM and related margin techniques, have discriminative objectives without having any explicit probabilistic elements.

Tom Minka has a short writeup that discusses discriminative models vs. training.

As stated by others, discriminative models will focus on estimating P(Y|X) without first trying to estimate P(X, Y). This is the case for logistic regression w.r.t. naive bayes classifier, or CRF w.r.t. HMM for instance.

Furthermore most discriminative models don't even try to model P(Y|X) but just f(x) = argmax_y P(Y=y|X=x) .

As far as I'm concerned, the discriminative model is data-driven. It needs only a small amount of data and corresponding labels and there's no need of priors about P(x) and P(y|x). This implies the entropy is maximized. While the generative model is model-driven, which needs the extra priors. And if there's no prior, we need to approximate it using a vast amounts of data. So the former should be faster and more convenient, while the latter more proper in some degree.