I want to compare my semi-supervised text classification algorithm with the GE-FL algorithm in “Learning from labeled features using generalized expectation criteria” (http://people.cs.umass.edu/~mccallum/papers/druck08sigir.pdf).

I want to evaluate both the algorithms on the basis of Macro F1 and human labeling efforts.

The GE-FL algorithm is semi-supervised. But the labeled features are discovered using complete labeled data set.
In my algorithm labeled features are discovered without using labeled data set.

How should we compare human efforts required in my algorithm to the efforts required in the GE-FL algorithm?

Consider following example:
For some data set, Macro F1 using GE-FL algorithm is 0.89 and the number of labeled features are 40.
For the same data set, Macro F1 using my algorithm is 0.88 and the number of labeled features are 30.
Which algorithm is performing better in this case?

asked Sep 17 '13 at 07:30

swapnilhingmire's gravatar image

swapnilhingmire
31151516


One Answer:

I believe labeled features were discovered from the full dataset in some GE papers solely because getting humans to label data was too expensive for most experiments.

Hence if you want a fair comparison I suggest you either give both algorithms human features or features extracted from the labeled data in similar ways.

Regarding your example it's hard to tell from a single point. What I'd do is plot a curve of F1 versus number of labeled features, and see which algorithm outperforms the other.

answered Sep 17 '13 at 09:43

Alexandre%20Passos's gravatar image

Alexandre Passos ♦
2554154278421

Thanks Alexandre! My next doubt is what are the actual labeling efforts involved in the GE-FL algorithm when the features are labeled by the oracle? (# features or # training documents)

(Sep 18 '13 at 06:04) swapnilhingmire
Your answer
toggle preview

powered by OSQA

User submitted content is under Creative Commons: Attribution - Share Alike; Other things copyright (C) 2010, MetaOptimize LLC.