Mechanical Amazon Turk Terminology

Mechanical Amazon Turk Terminology

This article describes the concepts and terminology you need to understand to use Amazon Mechanical Turk (mTurk) effectively. They are presented in the order you will most like encounter them.

Requesters

A Requester is a person (or company or organization) who asks questions to Amazon Mechanical Turk. As a Requester, he use a software application to interact with the Amazon Mechanical Turk Service to submit questions, retrieve answers, and perform other automated tasks. A requester can use the Requester Console (http://requester.mturk.amazon.com/) to check the status of his questions, and manage his account.

To Workers, he is known as the creator of his HITs, and as the creator and maintainer of his Qualification types. Workers see his name, as specified with his Amazon.com account, on the Amazon Mechanical Turk web site.

A requester perform actions with the Amazon Mechanical Turk Service by using an AWS Access Key ID and AWS Secret Key to cryptographically sign each request. To obtain an AWS Access Key ID and AWS Secret Key, go to http://aws.amazon.com/mturk and sign in with your Amazon.com account e-mail address and password.

Workers

A Worker  is a person who answers questions for Amazon Mechanical Turk. A Worker uses the Amazon Mechanical Turk web site to find questions, submit answers, and manage his or her account.
To Requesters, a Worker is known as the submitter of a HIT assignment, and as a user requesting a Qualification. Requesters see the Worker’s account ID (an alphanumeric string assigned by the system) included with assignment data and Qualification requests.
Qualifications represent the Worker’s reputation and abilities. A Worker’s Qualifications are matched against a HIT’s Qualification requirements to allow or disallow the Worker to accept the HIT. A Worker’s Qualifications cannot be accessed directly by other users.

Human Intelligence Tasks (HITs)

Each question a requester’s application asks is a Human Intelligence Task, or HIT. A HIT contains all of the information a Worker needs to answer the question, including information about how the question is shown to the Worker and what kinds of answers would be considered valid.

Each HIT has a reward, an amount of money requesters pay to the Worker that successfully completes the HIT.
He can request that more than one Worker ought to complete a HIT by specifying a Max Assignments property for the HIT. When a Worker finds a HIT to complete, the Worker accepts the HIT. mTurk  creates an assignment to track the completion of the task and store the answer the Worker submits.

Amazon Mechanical Turk reserves the assignment while the Worker is actively working on it, so no other Worker can accept it or submit results. If the Worker fails to complete the assignment before the deadline the requester specified (the Worker abandons the HIT), or if the Worker chooses not to complete it after accepting it (the Worker returns the HIT), the assignment is once again made available for other Workers to accept.

A HIT can have multiple assignments. This is useful for gathering multiple answers to a single question for comparison, or for collecting multiple opinions. A Worker can only accept a HIT once, so a HIT with multiple assignments is guaranteed to be performed by multiple Workers.

Requester can specify the maximum number of assignments that any Worker can accept for his HITs. He can set two types of limits:

The maximum number of assignments any Worker can accept for a specific HIT type he has created
The maximum number of assignments any Worker can accept for all your HITs that don’t otherwise have a HIT-type-specific limit already assigned

Approval and Payment

Once a HIT has all of the answers that were requested, or an expiration date a requester specified has passed, his application retrieves the assignments with the answer data. If an assignment’s answer satisfies the question, he approves the assignment. He may reject the assignment if the HIT was not completed successfully.

Amazon Mechanical Turk automatically processes payment of the reward to the Worker once the assignment is approved. The reward is transferred from Requester’s Amazon.com account to the Worker’s Amazon.com account. He can deposit or withdraw funds from his Amazon Mechanical Turk account at any time using the Requester web site.

Qualifications and Quality Control

Requester can manage which Workers can accept a particular HIT using Qualifications. A Qualification is an attribute assigned by him to a Worker. It includes a name and a number value. A HIT can include Qualification requirements that a Worker must meet before they are allowed to accept the HIT. Each Qualification Requirement describes an expression that a score or metric about the Worker must match for the Worker to be considered “qualified” to complete the HIT.

He create a Qualification type to represent a Worker’s skill or ability. A Worker discovers his Qualification type either by browsing HITs that require it, or by browsing Qualification types directly. The Worker requests a Qualification of the type, and you grant the request with a value.

A Qualification type may include a Qualification test. A Qualification test is a set of questions, similar to a HIT, that the Worker must answer to request the Qualification. Requester can grant the request manually by evaluating the Worker’s test answers, or he can include an answer key for the test when they create the Qualification type. For Qualification types with a test and an answer key, Amazon Mechanical Turk processes Qualification requests automatically, and sets Qualification values as specified by the answer key.

Amazon Mechanical Turk provides several system Qualifications that represent a Worker’s account history. The values are updated continuously as the Worker uses the system. A HIT may include Qualification requirements based on these system Qualifications.