Amazon Mechanical Turk vs UserTesting: What are the differences?

Introduction

Below are the key differences between Amazon Mechanical Turk and UserTesting.

1. Pricing and Payments: Amazon Mechanical Turk (AMT) offers a flexible pricing structure, allowing requesters to set their own prices for completed tasks. Requesters must pay fees to Amazon for using the platform, in addition to paying the workers. On the other hand, UserTesting has a fixed pricing model, with packages starting at a specific price per participant. Payment to workers on AMT is usually in the form of Amazon.com gift cards or cash, while UserTesting pays participants directly via PayPal.

2. Task Variety and Complexity: AMT provides a wide range of tasks, including data validation, categorization, content moderation, surveys, and more. The tasks can vary in complexity, from simple tasks that take a few seconds to complete to more complex tasks that require specific skills or expertise. In contrast, UserTesting focuses primarily on user experience testing, including website and app usability testing, prototype testing, and UI/UX feedback. The tasks on UserTesting are generally focused on evaluating the user experience and usability aspects of a product.

3. Worker Pool and Quality Control: AMT has a large pool of workers, often referred to as "turkers," who are available to complete tasks. The worker pool comprises a diverse range of individuals, including both experienced workers and those who are new to the platform. While AMT provides some tools for quality control, such as qualifications and reviews, maintaining consistent task quality can be a challenge due to the large worker pool and varying skill levels. UserTesting, on the other hand, has a curated panel of testers who are selected based on specific demographics and criteria. This allows UserTesting to ensure higher quality feedback and user testing results.

4. Task Turnaround Time: Due to the large worker pool and potentially lower payment rates, tasks on AMT can be completed relatively quickly. Depending on the complexity of the task and the payment offered, results can be obtained within minutes or hours. UserTesting, on the other hand, typically requires a minimum of 24 hours to recruit and receive feedback from participants. This longer turnaround time may be necessary to ensure participants have sufficient time to complete the assigned tasks and provide comprehensive feedback.

5. Interaction with Participants: AMT tasks are often designed as discrete tasks that do not typically require direct interaction between the requester and the worker. Communication between the requester and the worker is limited to specific instructions provided in the task. In contrast, UserTesting allows requesters to interact with participants through live conversations, interviews, and follow-up questions. This level of interaction enables UserTesting to gather more qualitative insights and clarifications directly from the participants.

6. Expertise and Support: Amazon Mechanical Turk is a self-serve platform, offering limited support to both requesters and workers. While they provide documentation and resources, users are expected to have a certain level of expertise to effectively use the platform. UserTesting, on the other hand, provides more hands-on support to both requesters and participants. They offer guidance and assistance in setting up tests, recruiting participants, analyzing results, and conducting user research.

In summary, Amazon Mechanical Turk provides a broader range of tasks, a larger worker pool, and quicker task completion, but may have lower overall quality control. UserTesting focuses specifically on user experience testing, employs a curated panel of testers, offers more interaction options, and provides more support throughout the testing process.