Comparing two Linear Models

I indirectly posed this question to Tony Jebara (through a student that is taking his class). His answer was to either 1) look at the cosine similarity between w1 and w2, or 2) use the p-norm of the distance vector w1-w2. Both very sensible. But neither touch on the variance estimation -- probably because the question posed by the student didn't address this aspect of the question :).

The two citations look interesting, I'll need to do some reading before I have a clear sense to use them for this task.

I don't think you'd want to just put equal variance gaussians over the classifiers, for the reasons you describe. But if you interpret L2 regularization as using a zero-mean constant-variance gaussian prior over the weight vector, the weight vector is the maximum likelihood estimate of the posterior mean. You should also have access to the variance though. Then you can do a series of t-tests to see which weights differ significantly. no? it seems like this is straight-forward, but i don't have a citation of anyone doing it. So i was hoping to sanity check it here. (Though it might be a simplified version or special case of what's included in the papers you added above).

I'm looking at two classifiers trained to predict the same label, using the same features, drawn from different training data. if you have test data, it's pretty easy to see if the two classifiers are behaving the same way. Compare accuracy, use McNemar's test, etc. But this is essentially just a monte carlo exploration of the feature space. The random samples are the test data. Generally the test data is small, so you're only exploring a limited part of the feature space, and you don't get any explanatory power about which features/dimensions are responsible for the differences in performance. So i was thinking it'd be better to look at the models themselves.

I guess the data vs model comparison depends on the dimensionality?

ie does your data lie on a subspace of your input vector. if so then a cosine similarity would give you lower similarity than you might actually observe? ie you have to project the weight vectors first ...

Well you can't really compare the similarity measure based on, say, mcnemar's test or another error based measure that doesn't introduce another model (data) and the model space distance of decision boundaries (models). So "lower similarity" here is really just a "lower [but incomparable] similar score", no?

Well I think that's my point...What do you want to measure with your similarity score?

To me the cosine approach is equivalent to doing a Monte carlo on randomly 'uniform/spherical gaussian'? generated data ( ie exploring the whole feature space) and measuring the classification differences

whereas if you only use the test data to compare the two models, you are implicitly saying that the observed data does not occupy the full feature space...(and/or is not uniformly distributed)

Personally the test data approach makes more sense, and I was suggesting you could do something similar using the cosine approach by first projecting the weight vectors onto the space spanned by your training data... what I feel you are missing out is the distribution of your data ... To take some trivial examples if your features are two dimensional but only one feature changes in the data then the weight on the second feature is irrelevant.

So what I believe you need to incorporate is the mean and covariance matrix of your input data - but I don't know how!?

OK, i see where you're coming from. Right, the measures between the decision boundaries ends up considering all regions in the feature space as equally relevant, while evaluation on test data samples from the "plausible" areas in the feature space.

My thinking to avoid the test data evaluation is that I don't trust that it has sufficient coverage to thoroughly capture the generalization of the classifier and I was looking for a more comprehensive description. But you bring up a good point about not really caring about all feature space areas equally.

I wouldn't want to try to approximate the mean and covariance of the input data -- I don't think it's gaussian. But measure theory might have something to say about this...