The NPR podcast Planet Money (of which I am a fan) just released an episode that looks at Amazon’s Mechanical Turk (MTurk)- a website where people are paid small sums of money to complete online micro-tasks (like categorising photos or writing transcripts of recordings). The site is named after an 18th Century chess playing ‘robot’ that was actually controlled by a hidden person (see above). This is an elegant analogy for the modern MTurk platform where anonymous workers complete Human Intelligence Tasks (HITs) like a well oiled machine. If you want a slightly more in-depth and accessible introduction then I’d heartily recommend listening to Planet Money‘s short episode (available here).

However, while the episode does provide a useful introduction, as a researcher who has run a variety of studies on MTurk, I also have some specific (/pedantic) reservations with what their portrayal. In some ways the Planet Money reporters went to a lot of effort to dig into the topic: interviewing one of the founders of MTurk and embedding a ‘secret message’ in an MTurk task so that they could speak to some actual workers. Yet  I still found listening to the episode frustrating, due to some surprising misrepresentations that implied a low level of fact checking. I realise what follows will likely be too ‘insider baseball’ for most people, but for those who have some interest in how research on MTurk works, below are some of the biggest inaccuracies/pet peeves I had with the episode.

1. They call the MTurk workers Turks but the workers refer to themselves as Turkers or MTurkers.

This might seem nitpicky (and it is) but anyone who bothered to register for any of the multiple worker/requester forums, or even spoke to a worker and asked them, would quickly note that Turker is the standard terminology used by the workers. If you listen to the episode you will hear them use the incorrect phrase repeatedly, indeed they call up workers and immediately refer to them as ‘Turks’ (or in one cringe inducing spot ‘My Turk’), the workers either laugh at this, ignore it or sound a little confused but then continue. Not being aware of such basic terminology implies a rather superficial level of research.

2. The journalists say that the Amazon Mechanical Turk website is too complicated and confusing for them to use directly.

While not being a particularly user friendly site, I would estimate that to learn how to post a basic task on MTurk would require a computer literate person (at most) a few hours. The journalists of Planet Money might have better things to do with their time but it seems like spending a few hours on the site might have given them a better understanding of how it all works, rather than just getting someone else to do it for them (in the end they used a consultant).

3. In their task, they ask MTurkers to provide personal details/contact information via a ‘secret message’.

Asking for personal contact information from workers is explicitly prohibited in Amazon’s terms of service:

You may not use Amazon Mechanical Turk for illegal or objectionable activities. Here are some examples of prohibited activities:

• collecting personal identifiable information

You could (I think legitimately) argue that by making the contact provision entirely optional and irrelevant to payment that they did not break the terms of service. Indeed, for full disclosure, this is also the approach my research group takes; so for example, if we ask for people’s email address, say for sending detailed debriefings, we always present it as optional and make sure this is clearly indicated. But regardless, it would have been nice of the journalists to at least acknowledge the potential issues with their approach, rather than to ignore the fact that embedding ‘secret messages’ asking for personal details on a site designed to insure worker anonymity, is a little bit shady.

4. The journalists give the impression that it is very difficult to contact MTurk workers.

While it is difficult, due to the terms of service, to contact the anonymous workers you hire to have private chats, it is definitely not difficult to find and contact MTurk workers in general. There are a number of freely accessible forums, where workers are more than happy to share their thoughts and opinions on MTurk and provide feedback/advice to requesters (see MTurk forum for one example). They do mention these forums at the very end of the podcast, so the journalists were aware of them… but that’s my point; if logging onto a forum is all that is required to speak to workers, the whole ‘secret message’ thing seems a bit unnecessary and misleading. I guess that’s journalistic license. There are also some academic papers on the demographics of MTurkers (see here and here) for anyone interested in how representative the interview subjects for the podcast are.

5. They claim that all workers are equally able to apply for all HITs.

When discussing competition one of the hosts explicitly states: “When any 5 cent job is posted, anyone who signs up for the service and has access to the internet can go after the same job [as experienced workers]”. This is wrong. Requesters can set requirements for their HITs, such as ‘must have completed 1,000 HITs (jobs)’, ‘must have a HIT approval rate of 99%’ or ‘must have a Masters qualification’ (an award provided by Amazon for high quality workers). They can also assign their own qualifications, and provide tasks that operate as qualifying tests in order to get access to higher paying HITs. So MTurk is not the pure level field of competition that Planet Money presents.

Despite these inaccuracies, where the podcast was most successful was in humanising the workers on MTurk and highlighting the general level of insecurity and low pay they typically endure. I enjoyed hearing from the workers, as I do when we run studies on MTurk, and I agreed with the sentiment from the MTurk consultant that better regulations to protect workers need to be in place. Turkopticon, which allows workers to rate requesters and then view their ratings when selecting HITs via a browser plug-in, is a good start. But official efforts, such as establishing minimum pay rates and a more transparent internal feedback system, are sorely needed.

It is also worth noting that while MTurkers are a varied bunch and do seem more representative than the standard university student convenience sample, there is an inevitable skew, specifically to those who are internet savvy and not entirely economically stable (a position with which I have much personal sympathy). But regardless, I still think MTurk is still a very valuable research environment. However, the current lax regulations means that it is incumbent on researchers/businesses to self-police and insure that they are paying a fair amount to their ‘workers’. For businesses primarily concerned with saving money this might be a hard sell, but for researchers the turkers should be treated more like research participants than some an anonymous survey completing work force.