|
As a learning exercise I'm building my own twitter client. The simple idea is that my active binary classifier should learn the tweets I like. The main point of the project was to play with semi supervised stream based approaches but my tinkering has raised a design question that I hadn't seen in any previous classification problem I've looked at. I bootstrapped the learner by labelling a few hundred tweets; the positive cases for those I did want to read and the negative cases for those I didn't want to read. Once bootstrapped it's been effectively filtering a stream of incoming tweets by only showing me the ones it's being classifying positive. I'm able to give it ongoing feedback by continuing the labelling by marking the ones I did not want to read. My concern is that while during bootstrapping I was labelling both positive and negative cases but now I'm only labelling instances the classifier thought were the postive cases that it got wrong. This is only part of the picture; what about the negative cases it's getting wrong that it should be showing me? I'm not sure the best approach to weaving some of these in? just including the occasional one that it's classified as negative as a form of sanity check somehow feels a bit clumsy. Is there a name for this type of problem situation? I'm using vowpal wabbit (in the case there is something specific for this). Cheers Mat |
|
Active learning seems exactly like what you need: it's a principled way to select what examples to label from a large pool of unlabeled examples. In the case of SVM active learning, the most popular approach to pick up what instance to label next consists in selecting the example which is the closest to the current hyperplane, i.e. the example for which the classifier was the least confident (x with smallest margin f(x) = w . x). In the early active learning algorithms, people would retrain the model entirely with a batch SVM but in the case of online algorithms, adding a new example to the current hypothesis (i.e., adding a new support vector) is very natural. Also, your problem seems to be very similar to the problem that in spam filtering, nobody wants to go to the spam box to annotate false spams. So, the spam filtering literature may be relevant here. Actually, I wouldn't be surprised if Gmail forcibly puts spams in my inbox in an active learning way from time to time. I think I should focus on the active approach more. The analogy with a spam filter is a good one, something I hadn't even considered. Thanks!
(Sep 12 '11 at 16:08)
mat kelcey
|
|
As a simple solution, you could set the algorithm to 'explore' a bit. In other words, every now and then, show a tweet it expects you to not like, just to see if its right. You could take these tweets from near the decision boundary as well, to adjust it slightly - i.e. pick tweets that the program thinks you won't like, but isn't sure. Yeah I was thinking of doing this but it somehow feels a bit clumsy. I've been looking at a bit at the active learning capabilities of vowpal wabbit and think that they might provide a targeted way of the model giving feedback on the things it's least confident with.
(Sep 12 '11 at 13:44)
mat kelcey
|
