Suppose I am interested in using crowdsourcing to annotate my data. What are the main services for this, and what are the pros/cons of each?

Amazon's Mechanical Turk appears to be most popular. But recently I've heard of CrowdFlower (which seems nice in that it includes Amazon MTurk's labor force and also provides more visualization and feedback of results). In your experience/opinion, which is more recommended? Or are there other even better services? I appreciate any tips on this!

asked Jul 23 '10 at 09:33

Kevin%20Duh's gravatar image

Kevin Duh

4 Answers:

I have used AMT extensively and only started using Crowdflower recently. That being said, I think that Crowdflower is better. It allows you to post jobs on many different sites (including AMT). The real selling point for Crowdflower is the integrated ability for incorporating "gold" answers (questions where you know the answer). It uses the gold answers to rate users, give feedback to users, and to identify problems with questions/instructions. Of course, you can do the same with AMT with a fancy iFrame application on your own server, but that's a pain (at least the way we did it).

Beyond that, the differences are trivial (both can handle the same CSV IO scheme). Crowdflower has a slightly nicer interface and AMT has a better command line API. Crowdflower also has a little higher overhead, but this is balanced by nice money saving features like automatically pausing your jobs if workers aren't getting any gold answers right.

answered Jul 26 '10 at 20:24

Jordan%20BoydGraber's gravatar image

Jordan BoydGraber

Thanks! Very helpful comparison!

(Jul 26 '10 at 21:26) Kevin Duh

I have started using MTurk to do some NLP tasks. One of the advantages of MTurk is the API and MIT Turkit which allows you to do quality assurance recursively and as a part of your project. Alternatively you can integrate their API with your own codes (they have a Python, JAVA wrapper and also a command line tool)

answered Jul 27 '10 at 07:48

Mark%20Alen's gravatar image

Mark Alen

I used MTurk for a collaborative drawing project, I feel once you get past understanding the system, it works rather well. Be sure to be overly descriptive, possibly, in the instructions. People will take longer reading the first time, but the quality overall is worth it (you can put a notice like "if this is your first time with this HIT, read this link..." or some such). I haven't seen many others, only in passing, and this possibly because MTurk has the highest viewership.

answered Jul 26 '10 at 01:00

th0ma5's gravatar image


I've not posted jobs to either, I have undertaken some work in MTurk to figure out the process (and earned $0.08 for 30 mins work). One of the two jobs I did was posted by CrowdFlower - they were looking for ideas on how people could use crowdsourcing (I kid you not).

answered Jul 25 '10 at 18:09

Ian%20Ozsvald's gravatar image

Ian Ozsvald

Your answer
toggle preview

powered by OSQA

User submitted content is under Creative Commons: Attribution - Share Alike; Other things copyright (C) 2010, MetaOptimize LLC.