Using Mechanical Turk and Volunteer Science

  1. Link your MTurk account to Volunteer Science
  2. Create a Turk-ready study
  3. Post the study to Volunteer Science
  4. Create a HIT in the Mechanical Turk Sandbox
  5. Test the HIT
  6. Make the HIT live
  7. Monitor and Pay Turkers
  8. Post more HITs
  9. Get the Data
  10. Best Practices
  11. Frequently Asked Questions

We will walk through the process of running turkers on Volunteer Science. Volunteer Science interacts with Mechanical Turk in the same way as if you were using your own local code to automatically interact with Mechanical Turk. You must set up your MTurk Researcher Account to use the Mturk API. Here are Amazon's Instructions

Go to the Turk Management page on Volunteer Science and select "Attach AMT Account". Fill out the form by providing a name for your account and select the team(s) that can access the account. Then enter your Access ID, which looks like 1506923870, and your Access Key, which looks like A13ASDF781HL1219CB83. If the accounts do not link, verify that your account is set up to use the MTurk API.

Create a Turk-ready study

Any study on Volunteer Science can utilize our native integration with Mechanical Turk. You just need to make sure you call the function payAMT(pay,bonus) correctly. Our payAMT(pay,bonus) function automates the payment process so you don't have to go through your results to verify their quality and pay turkers manually. In your study, you should decide when turkers should be paid and when they should not. For example, you can keep track of the number of questions a turker answers correctly in an attention check. When the study ends, if they have answered enough questions correctly, you automatically accept the HIT by calling payAMT(true) or, if they did not answer enough questions correctly, you can reject the HIT by calling payAMT(false). No matter what you choose, you should make it clear in the instructions for your HIT what is and isn't acceptable.

Oftentimes, researchers want to pay workers bonuses equal to some money they earned during a study. For example, you might want to offer a Turker real money for participating in a Bargaining Game. To do this, you need to keep track of the bonus amount during the study and then make it part of your payAMT() call. For example, say in you have a multi-person study in which you keep track of bonuses using this array: turk_bonuses = ['0':2.25, '1':1.50, '2':3.50, '3':0.50, '4':1.75] and where the indexes 1, 2, 3, and 4 are the myid values for each turker. You can pay workers the correct bonus for them by calling payAMT(true,turk_bonuses[myid]).

Post your study to Volunteer Science

Once your study is ready for Mechanical turk, you must post the study to Volunteer Science like any other study. See instructions here.

Create a HIT in the Mechanical Turk Sandbox

Once your study is posted on Volunteer Science, on the administration page next to your study you'll see a link to "Make AMT HIT." Click that link and you'll go to our form where you can put in the details for your HIT, inlcuding the MTurk account you want to use, the information Turkers will see like the title, description, keywords, and reward amount; and Mturk-specific metadata like the number of HITs to create, how long Turkers have to take the HIT, and how long the task should be available to turkers. Fill out the form with the details you need and press the "Add New HIT" button at the bottom. This will take you to the status page for your account. From there, you need to find your HIT by name and click the "Activate" link to actually create the HITs in the Mechanical Turk Worker Sandbox. (Please allow several minutes for your HITs to show up on Mechnical Turk.) Only you will be able to test your HIT in the sandbox and no money will actually be paid on HITs posted to the sandbox.

There are several items unique to Volunteer Science that you should understand.

Sandbox checkbox: There is a checkbox next to "Sandbox." When you are initially creating your HIT, make sure this is checked. If this is not checked, the HIT will be posted live to Mechanical Turk and you will be responsible for any costs incurred.

Max bonus: max bonus is a variable only available on Volunteer Science. It's purpose is to prevent you from paying too much on accident. For example, if you forget a decimal and make the bonus $99 instead of $0.99, max bonus will kick in and prevent you from making those bonus payments. Set Max Bonus to the maximum amount of money a turker could conceivably earn in your study. If you find that some workers are hitting the Max Bonus, you will be able to go into your Mechanical Turk account on Amazon and pay them the correct amount.

Frame Height: How large, in pixels should the webpage be where turkers are taking your study. If your study is larger than the frame, a scroll bar will appear in users windows.

Qualifications/Requirements: You can issue Turkers qualifications when they complete a study and then require that they do or do not have the qualification to take other studies. If you want to create your own qualificaiton and requirement or already have your own and want to import them into Volunteer Science, please reach out to us.

Test the HIT

Once the HIT has been created in the workers sandbox, you can take the HIT as if you were a turker on Mechanical Turk Worker Sandbox. You can view your study as a preview and as a simulated Turker from the sandbox. You may tweak your study code at any time to fix any bugs or change the study. However, once the HIT is created you cannot change the account it is under, the study it is using, or its title, description, keywords, bonus, max bonus, or duration. In order to change these, you must click "Edit" next to the HIT on the status page in Volunteer Science and click "clone." When you clone a HIT, you must change at least one character in the title or the description or Amazon will not accept the new HIT.

Make the HIT live!

Once you've tested the HIT in the sandbox, have validated the payments are correct and the study is ready, return to the HIT information in your AMT Status page. Go to the Turk Management page and select the right MTurk account. This will show you the HITs you've created. Your HIT should be in the Sandbox table at the bottom. Click "Edit" in the right side column of the table beside your HIT. Click "Clone." That will take you to a new version of your HIT with the exact same information. Uncheck the Sandbox checkbox. Verify all of your HIT's information and click "Add New HIT" button at the bottom. This will take you to the status page for your account. From there, you need to find your HIT by name and click the "Activate" link to actually create the HITs.

Monitor and Pay Turkers

Once your HIT is live, participants will begin accepting the HIT and then completing them. In the status page table, you'll see the following columns:

  • Available: Number of HITs for Turkers to take
  • Need to Pay: The subcolumn on the left is the number of HITs accepted but not completed, the subcolumn in the middle is the number of HITs completed but unpaid. The right subcolumn is the amount of money owed to turkers.
  • Pay: When you click this link, you pay the Turkers who have completed your HIT. If you do not click this, turkers will automatically be paid according to the ApprovalDelayHours variable in your HIT.
  • Paid: The number of Turkers paid and how much you have paid in total.
  • Problem: The number of problems in paying turkers and the amount owed.
  • Completed:
  • PPM/AvMin columns: The Pay per Minute and average number of minutes it takes for turkers to complete your study.

Post more HITs

If you want to launch more HITs, go to your HIT in the the Turk Management page and select "edit." Increase either the MaxHitsPerWorker or MinUniqueWorkers to the number of new HITs you want. For example, if MaxHitsPerWorker is 1 and MinUniqueWorers is 5 and you want to add 10 new hits, you should change MinUniqueWorkers to 15. Click the "Submit" button and then, to the left of the HIT on the management page, click "More." Note, it may take up to 5 minutes for the new HITs to be reflected in the table.

Get the Data

Once you've completed data collection, you can export your data on the study page. See instructions here. You can opt to download only data from workers from Mechanical turk by selecting the "Only" radio button next to "AMT" in the data export page.

Best Practices

  • Start small! Do a few test runs with 5-10 HITs to make sure everything is working well.
  • Recruiting groups via Mechanial Turk: if you want to recruit workers to participate in multiperson studies you need to have workers start hits at the same time. To do that, you can launch many HITs at the same time. Two things to note however, Mechanical Turk is slow to show users the last 5-10 HITs available, so you need to add extra HITs. Second, workers put HITs in their queue, meaning they will accept the HIT but not take it right away. Again, this means you should post more HITs than you'll need. We recommend posting at least 15 HITs to get groups and then two or three times your group size to sustain the groups. For example, if you want to run groups of five, we recommend launching 25-30 HITs at a time and adding 10-15 more as the HITs not yet completed (not those that have been accepted) get below 10-15.

Frequently Asked Questions

  1. I want to/have created by own Qualificaiton, how do I use it through Volunteer Science?
    2.

    We do allow you to create and use your own qualifications; however at the moment you must contact us to make them available

  2. How do I award a Qualification for taking one of my studies?
    3.

    You can award a qualification automatically by selecting the qualification from the "Grant on Complete" box when you create your HIT.

  3. I want to use MTurk's premium qualificaitons, how do I know if they are accurate?
    4.

    You don't. You should independently verify any data that is essential to your study when using a paid subject pool.

Facebook | Twitter | Subscribe | Blog | Contact