Planet Money’s MTurk episode

The NPR podcast Planet Money (of which I am a fan) just released an episode that looks at Amazon’s Mechanical Turk (MTurk)- a website where people are paid small sums of money to complete online micro-tasks (like categorising photos or writing transcripts of recordings). The site is named after an 18th Century chess playing ‘robot’ that was actually controlled by a hidden person (see above). This is an elegant analogy for the modern MTurk platform where anonymous workers complete Human Intelligence Tasks (HITs) like a well oiled machine. If you want a slightly more in-depth and accessible introduction then I’d heartily recommend listening to Planet Money‘s short episode (available here).

However, while the episode does provide a useful introduction, as a researcher who has run a variety of studies on MTurk, I also have some specific (/pedantic) reservations with what their portrayal. In some ways the Planet Money reporters went to a lot of effort to dig into the topic: interviewing one of the founders of MTurk and embedding a ‘secret message’ in an MTurk task so that they could speak to some actual workers. Yet  I still found listening to the episode frustrating, due to some surprising misrepresentations that implied a low level of fact checking. I realise what follows will likely be too ‘insider baseball’ for most people, but for those who have some interest in how research on MTurk works, below are some of the biggest inaccuracies/pet peeves I had with the episode.

1. They call the MTurk workers Turks but the workers refer to themselves as Turkers or MTurkers.

This might seem nitpicky (and it is) but anyone who bothered to register for any of the multiple worker/requester forums, or even spoke to a worker and asked them, would quickly note that Turker is the standard terminology used by the workers. If you listen to the episode you will hear them use the incorrect phrase repeatedly, indeed they call up workers and immediately refer to them as ‘Turks’ (or in one cringe inducing spot ‘My Turk’), the workers either laugh at this, ignore it or sound a little confused but then continue. Not being aware of such basic terminology implies a rather superficial level of research.

2. The journalists say that the Amazon Mechanical Turk website is too complicated and confusing for them to use directly.

While not being a particularly user friendly site, I would estimate that to learn how to post a basic task on MTurk would require a computer literate person (at most) a few hours. The journalists of Planet Money might have better things to do with their time but it seems like spending a few hours on the site might have given them a better understanding of how it all works, rather than just getting someone else to do it for them (in the end they used a consultant).

3. In their task, they ask MTurkers to provide personal details/contact information via a ‘secret message’.

Asking for personal contact information from workers is explicitly prohibited in Amazon’s terms of service:

You may not use Amazon Mechanical Turk for illegal or objectionable activities. Here are some examples of prohibited activities:

  • collecting personal identifiable information

You could (I think legitimately) argue that by making the contact provision entirely optional and irrelevant to payment that they did not break the terms of service. Indeed, for full disclosure, this is also the approach my research group takes; so for example, if we ask for people’s email address, say for sending detailed debriefings, we always present it as optional and make sure this is clearly indicated. But regardless, it would have been nice of the journalists to at least acknowledge the potential issues with their approach, rather than to ignore the fact that embedding ‘secret messages’ asking for personal details on a site designed to insure worker anonymity, is a little bit shady.

4. The journalists give the impression that it is very difficult to contact MTurk workers.

While it is difficult, due to the terms of service, to contact the anonymous workers you hire to have private chats, it is definitely not difficult to find and contact MTurk workers in general. There are a number of freely accessible forums, where workers are more than happy to share their thoughts and opinions on MTurk and provide feedback/advice to requesters (see MTurk forum for one example). They do mention these forums at the very end of the podcast, so the journalists were aware of them… but that’s my point; if logging onto a forum is all that is required to speak to workers, the whole ‘secret message’ thing seems a bit unnecessary and misleading. I guess that’s journalistic license. There are also some academic papers on the demographics of MTurkers (see here and here) for anyone interested in how representative the interview subjects for the podcast are.

5. They claim that all workers are equally able to apply for all HITs.

When discussing competition one of the hosts explicitly states: “When any 5 cent job is posted, anyone who signs up for the service and has access to the internet can go after the same job [as experienced workers]”. This is wrong. Requesters can set requirements for their HITs, such as ‘must have completed 1,000 HITs (jobs)’, ‘must have a HIT approval rate of 99%’ or ‘must have a Masters qualification’ (an award provided by Amazon for high quality workers). They can also assign their own qualifications, and provide tasks that operate as qualifying tests in order to get access to higher paying HITs. So MTurk is not the pure level field of competition that Planet Money presents.

Despite these inaccuracies, where the podcast was most successful was in humanising the workers on MTurk and highlighting the general level of insecurity and low pay they typically endure. I enjoyed hearing from the workers, as I do when we run studies on MTurk, and I agreed with the sentiment from the MTurk consultant that better regulations to protect workers need to be in place. Turkopticon, which allows workers to rate requesters and then view their ratings when selecting HITs via a browser plug-in, is a good start. But official efforts, such as establishing minimum pay rates and a more transparent internal feedback system, are sorely needed.

It is also worth noting that while MTurkers are a varied bunch and do seem more representative than the standard university student convenience sample, there is an inevitable skew, specifically to those who are internet savvy and not entirely economically stable (a position with which I have much personal sympathy). But regardless, I still think MTurk is still a very valuable research environment. However, the current lax regulations means that it is incumbent on researchers/businesses to self-police and insure that they are paying a fair amount to their ‘workers’. For businesses primarily concerned with saving money this might be a hard sell, but for researchers the turkers should be treated more like research participants than some an anonymous survey completing work force.

Advertisements

5 comments

  1. I am a worker (and part-time Requester) that posts on many of the forums. I am mostly found at Mturkforum. There is a poster who was interviewed by one of the journalists and he stated he felt as though they had a “pre-conceived thesis” while questioning him. http://mturkforum.com/showthread.php?32703-Can-t-Find-A-Savory-Amount-of-HITs-2-1-Super-Hand-Egg-Sunday!&p=1130186&viewfull=1#post1130186

    I also pointed out that Zoe Chace solicited information on the Mturk Subreddit and never even responded to emails (not to mention the information wasn’t included in the program)
    http://mturkforum.com/showthread.php?32703-Can-t-Find-A-Savory-Amount-of-HITs-2-1-Super-Hand-Egg-Sunday!&p=1130185&viewfull=1#post1130185

    I gave her a source that to some full-time workers including a turker with well over a million HITs completed. You’d think that point of view would have been included in the episode but alas it was not. It seems as though the same narrative is given ad nauseam and almost encourages new requester to treat the platform as a digital sweatshop. Amazon severely neglects the platform and does not do anything to protect workers from requesters that scam. While I’m not personally on board with the setting of static minimum wages on the platform, I do think workers need the due consideration that is afforded to Requesters against Turkers. It’s an inequitable power balance that needs to shift to move the platform forward. If the power schism did not exist, things like Turkopticon would not have developed out of necessity.

    Thanks for the insight and well-informed post from a requester’s perspective.

    Like

  2. Thanks for the message/info Mike. I understand the need for journalists to craft a simple narrative but it does sound a bit strange that they wouldn’t follow up on the kind of information you and others sent them, even just to acknowledge they received it.

    The entire episode lowered my estimation of the show to be honest, as it made me realise that there were probably similar misrepresentations on other topics that I just don’t pick up on.

    Not getting the terminology right and/or making use of the forums seems to indicate a pretty low standard of fact checking. I mean if they had asked one worker or even just read a few forum threads they would have quickly noted, oh no one is using the term Turk to refer to themselves here.

    I also agree that Amazon severely neglects the platform, from the requester side, even simple things like automatically providing qualifications are missing and require programming to add.

    As a basic intro to someone who has never heard of MTurk the show was ok… but they could have easily made the same show and avoid the inaccuracies I mentioned. Having some very experienced workers would have also provided an interesting POV.

    Anyway, thanks again for the extra info!

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s