|
Hey all, So I'm looking into variations on Pyramid Scoring for summarization evaluation. Essentially the idea of pyramid scoring is that for summarization, a few pieces of information are very relevant, a few more are quite relevant, and a lot are only mildly important. The scoring notion is to construct a set of N reference summaries, and then establish a weight for each piece of information (SCU or Summary Content Unit using the original paper's vocabulary) equal to the number of reference summaries that contain the SCU. I've found a few variants on the original formulation of pyramid scores. In DUC-05, Becky Passonneau and others looked at a variant on the original normalization. And in TAC-08, pyramid scoring was used only in the calculation of recall, and was combined with a length based measure approximating precision. I'm curious if anyone else has run into different measures that are based on Pyramid scoring. |
