How to measure distance between discrete probability distributions?

There's a large number of ways to measure distance between probability distributions. I'd like to get a sense of what the good ones are and why. Here's a couple I know how to justify

1. KL(p,.) where p is the true (empirical) probability distribution. There are many ways to justify minimizing this measure -- ie, the minimizer is asymptotically consistent, efficient, achieves lowest expected coding length and lowest asymptotic error for optimal hypothesis tests. KL(p,.) minimizers are easy to analyze theoretically in framework of M-estimators, also there are efficient algorithms to compute KL(p,.) and it's gradient for high dimensional distributions. Related measure, Chernoff Information shares some of the "optimal hypothesis test" justification.

2. L-inf distance in the space of log-odds. This measure or a monotonic transformation of it has come up under many names (dynamic range, L-inf quotient metric, A Distance Measure for Bounding Probabilistic Belief Change, Hilbert's projective metric). The fact that people keep reinventing this measure probably has something to do with the fact that this is a unique (up to monotonic transformation) measure under which any two positive vectors are brought "closer together" by positive matrix multiplication. Since both exact and approximate inference in general graphical models can be written in terms of matrix multiplications where our matrices are joint probability tables, this measure gives a useful tool to get correlations and accuracy bounds.

3. Any distance function based on empirical risk. For instance, pseudo-likelihood objective gives us empirical risk of modeling P(X given all other variables), hence KL-divergence based on pseudo-likelihood partitioning is relevant

Some of the ones I'd like more justifications for are KL(.,p), Hellinger distance, more generally Ali-Sivey distances, Bregman divergences, geodesic distance