|
I have an algorithm which returns what it believes to be the best ranking of 20 numbered nodes from a network (ranked elements of a set). I want to compare these results to the analogous output from another algorithm. I was using kendal-tau wiki articlebefore. Can I use ROC curves, and AUC in this situation? If so, how since we do not have a binary classification problem, the problem has multiple dimensions. Or is some type of summation scheme possible? Are there any other tests that are good to use? |
|
Kendal-tau is a type of rank correlation, as you can see, alternatives include Spearman and Goodman/Kruskal. You could possibly turn this into a binary classification problem (recall -> concordant pairs, false alarm -> discordant pairs), but it seems this would just be a convoluted version of one of the above mentioned metrics. |
|
Depending on the requirements, you might want to look into methods that consider differences near the top of the ranking to be more significant than differences in the lower ranks. This can give a more realistic estimate of differences in utility/relevance (if the results are for human consumption). e.g. Rank-Biased Overlap: http://blog.codalism.com/?p=1317 |
Any reason why you're looking at applying ROC-AUC?
@Jonathan Purnell, because ROC is a metric which is nicer as you can look at a graph as well as the AUC value. Maybe it doesn't make sense to use ROC given your answer?
ROC doesn't quite make sense since this isn't a binary classification problem, as you noted. For a nice graph, I'd suggest either a visualization of the two rankings (since there's only 20 nodes) or a histogram of the change in rank for each node.