Comparing Amazon’s Mechanical Turk Platform to Conventional Data Collection Methods in the Health and Medical Research Literature

Abstract

Background

The goal of this article is to conduct an assessment of the peer-reviewed primary literature with study objectives to analyze Amazon.com’s Mechanical Turk (MTurk) as a research tool in a health services research and medical context.

Methods

Searches of Google Scholar and PubMed databases were conducted in February 2017. We screened article titles and abstracts to identify relevant articles that compare data from MTurk samples in a health and medical context to another sample, expert opinion, or other gold standard. Full-text manuscript reviews were conducted for the 35 articles that met the study criteria.

Results

The vast majority of the studies supported the use of MTurk for a variety of academic purposes.

Discussion

The literature overwhelmingly concludes that MTurk is an efficient, reliable, cost-effective tool for generating sample responses that are largely comparable to those collected via more conventional means. Caveats include survey responses may not be generalizable to the US population.

Amazon.com’s Mechanical Turk (MTurk) is an online, web-based platform that started in 2005 as a service to allow researchers to “crowdsource” labor-intensive tasks for workers registered on the site to complete for compensation.^{1, 2} MTurk has rapidly become a source of subjects for experimental research and survey data for academic work, as its representativeness, speed, and low cost appeal to researchers.^{2, 3} Researchers post links to surveys and experiments and use MTurk to crowdsource the survey, collect the data, and compensate workers.⁴ A Google Scholar search of “Amazon Mechanical Turk” revealed 15,000 results published between 2006 and 2014³ and 17,400 results by mid-2017. MTurk is the largest online crowdsourcing platform,⁴ with about one-third of the tasks related to academic tasks.⁵ The growing popularity of MTurk has led to questions about its soundness as a subject pool; MTurk is the most studied nonprobability sample available to researchers.³

The MTurk pool of potential workers is vast, diverse, and inexpensive. MTurk has 500,000 registered users³ with 15,000 individual US workers at any given time.⁶ MTurkers have been paid as little as $0.05 to complete 10- to 15-min tasks.⁴ Researchers can collect data from large enough samples to generate significant statistical power at one-tenth of the cost of traditional methods.⁴ The MTurk population is more representative of the population at large than other online surveys and produces reliable results.^{2, 3, 6–11}

There is a rapidly growing literature exploring the generalizability of MTurk responses to other data collection methods. Data obtained via MTurk surveys and experiments are at least as reliable as those obtained via traditional methods, are attractive for conducting internally and externally valid experiments, and the advantages outweigh the disadvantages.^{3, 8–15} However, the benefits and drawbacks to using MTurk in the health and medical literature are largely unexplored beyond a taxonomy of how MTurk has been used in health and medical research.¹⁶ This article is the first synthesis to assess the peer-reviewed literature that has a study objective to analyze MTurk as research tool in a health services research and medical context and uses MTurk for part or all the results. The results from this synthesis can guide academic researchers as they explore the strengths and weaknesses of employing MTurk as an academic research platform.

METHODS

A literature search was performed for articles published between 2005 and mid-February 2017 using Google Scholar and PubMed databases. Searches for variations of the terms Mechanical Turk, MTurk, health, healthcare, clinic*, and medic yielded an initial total of 331 non-duplicative articles. Two reviewers (TH and KM) screened the articles first by title review, eliminating those that did not pertain to health as defined by the World Health Organization,¹⁷ leaving 181 articles. After abstract and full-text review, 35 articles were included in the final analysis.

RESULTS

The 35 articles that met the criteria of primary peer-reviewed article, MTurk used in part or all of the results, and an objective of the study was to analyze MTurk as a research tool in a health services research and medical context are described briefly in Table 1.^{12, 18–51} A number of strengths of using MTurk in an academic health services setting were identified in the literature. The studies were overwhelmingly supportive of the economical, cost-effective nature of MTurk.^{18, 19, 23, 25, 26, 29–34, 37, 38, 40, 41, 45–49, 51} Additional strengths include the time-saving component of using MTurk, reliability, and high quality. Accurate,³⁴ effective,^{29, 30, 51} performance comparable to quality of medical experts,^{18, 26, 33, 34, 39, 41, 43, 48} high verification,⁴² reliable,^{27, 31} objective,³² statistically equivalent to data from other samples,^{12, 22, 24, 38, 49, 50} diverse,^{19, 21, 47, 49} and viable,^{28, 36} high quality,^{35, 42, 45} among other strengths, were consistent conclusions in the literature.

Table 1.

Summary of Research Findings

Research study	Objective	Description
Aghdasi, Bly, White, Hannaford, Moe, and Lendvay (2015)	To assess the ability of a large group of laypersons using a crowdsourcing tool (MTurk) to grade a surgical procedure (cricothyrotomy) performed on a simulator	When compared to expert head and neck surgeons, the crowdsourced group took significantly less time to complete the analyses (10 h vs. 60 days), and the assessment of complex surgical performance by laypersons matched those of the experts (correlation coefficient 0.833). CSATS using MTurk workers provides an efficient, accurate, and inexpensive method of evaluating surgical performance, even when applied to complex procedures
Arch and Carr (2016)	To assess the presence of cancer survivors on MTurk and the feasibility of using it as an efficient, cost-effective, and reliable platform for psycho-oncology research	MTurk was shown to be a successful tool for recruiting young adult cancer survivors in particular, and the participants recruited were more geographically, medically, and socio-demographically diverse groups of cancer survivors than with many other available psycho-oncology recruitment sources. It was cost-effective, time-efficient, and the data obtained were reliable
Arditte, Çek, Shaw, and Timpano (2015)	To examine the similarities and differences of clinical phenomena reported by MTurk workers and the general US population	Results suggested that MTurk workers were significantly more likely to report clinical symptoms associated with social anxiety and depression than traditional community or epidemiological samples. Psychopathology researchers are encouraged to exercise caution when generalizing MTurk findings to the larger US population
Bardos, Friedenthal, Spiegelman, and Williams (2016)	To evaluate, using MTurk, a cloud-based workforce to assess patients’ perspectives of health care	The authors were able to effectively and efficiently collect survey data from a large national pool, including quantitative and qualitative data about patients’ knowledge and experience with miscarriage. These quality data were collected from over 1000 respondents over a 3-day window for under $300 USD (4 to 69 times less than other survey methods)
Boynton and Richman (2014)	To illustrate the utility of MTurk to recruit a diverse sample of adults for participation in an online daily diary study of alcohol use	Multilevel models of daily alcohol data derived from the MTurk sample replicated findings commonly reported in daily diary studies of alcohol use
Brady, Villanti, Pearson, Kirchner, Gupta, and Shah (2014)	To develop and validate the use of MTurk as a method of fundus photograph grading	MTurk workers were able to quickly and correctly categorize retinal images of diabetic patients into normal and abnormal, though there remains a need to improve MTurkers’ ability to correctly rate the degree and severity of retinopathy. MTurk offers a novel and inexpensive means to reduce skilled grader burdens and increase screening for diabetic retinopathy
Briones and Benham (2016)	To compare data from MTurk to data on psychological stress and sleep quality obtained from a college student sample	The data obtained from MTurk were statistically equivalent to data from the conventional undergraduate college sample
Brown and Allison (2014)	To compare MTurk workers’ evaluations of abstracts on nutrition and obesity to authors’ expert expectations	MTurkers reached consensus in 96% of reviewed abstracts, and over 99% of their ratings of obesogenicity of foods were complete and usable. Crowdsourcing on MTurk can be an economical and timesaving approach to evaluate large bodies of published literature
Chen, White, Kowalewski, Aggarwal, Lintott, Comstock, Kuksenok, Aragon, Holst, and Lendvay (2013)	To test a web-based grading tool using crowdsourcing on MTurk and Facebook	Surgery-naive MTurk workers were able to assess robotic suturing performance equivalent to experienced faculty surgeons in a shorter timeframe
Deal, Lendvay, Haque, Brand, Comstock, Warren and Alseidi (2016)	To assess the feasibility and reliability of CSATS for evaluation of technical skills in general surgery	Qualitative feedback from both crowd workers and faculty experts was “remarkably similar.” Use of MTurk is a reliable basic tool for the standardization of surgical technique evaluation
Gardner, Brown, and Boice (2012)	To investigate MTurk as a research tool for measuring body size estimation and dissatisfaction	MTurker’s assessment of their body size based on anthropometric data were similar to the results of three previous studies using traditional samples. MTurk can serve as a viable method for collecting data on perception and attitudes of body image quickly and inexpensively
Good, Nanis, Wu, and Su (2015)	To compare MTurk workers’ disease mention annotations to classifications by the “expert-crafted gold standard”	Overall, the MTurk workers generally performed at a high level for the task relative to the gold standard annotation. The MTurk worker population was able to effectively process biomedical text
Harber and Leroy (2015)	To illustrate the utility of crowdsourcing on MTurk for occupational health surveillance	MTurk was used to recruit and obtain information from employed individuals with asthma, who answered questions about the interaction of their asthma and work. Crowdsourcing methods are extremely effective and have potential for occupational health surveillance tools because of their efficiency, effectiveness, and financial viability
Harris, Mart, Moreland-Russell, and Caburnay (2015)	To evaluate the ability of crowdsourcing on MTurk to classify Twitter postings containing diabetes information into 9 topic categories, relative to classifications by experts	Classification by MTurkers relative to experts was reliable at the good or excellent level. MTurk may be a reliable, quick, and economical way for researchers to code large amounts of data gathered from social media
Hipp, Manteiga, Burgess, Stylianou, and Pless (2016)	To validate the use of MTurkers to interpret images from webcams to explore the effects of built environment changes on active transportation	Classification by MTurkers relative to classifications developed by trained research assistants resulted in an objective, cost-effective alternative to traditional methods
Holst, Kowalewski, White, Brand, Harper, Sorensen, Truong, Simpson, Tanaka, Smith, and Lendvay (2015)	To evaluate the ability of crowd workers to provide valid performance scores of surgical skills in live tissue when compared to the gold standard of assessment	Crowdsourcing basic surgical skills compares favorably to assessments by expert surgeons, and it provides larger volume feedback in a shorter period of time than expert assessors. However, further research is needed to link CSATS to clinical outcomes to confidently presume that non-medically trained workers can accurately assess surgical skills
Khare, Burger, Aberdeen, Tresner-Kirsch, Corrales, Hirchman, and Lu (2015)	To compare drug indication cataloging of FDA drug labels by MTurk workers to the annotation by domain experts	The MTurk workforce’s judgments on cataloging drug indications from FDA drug labels achieved an aggregated accuracy of 96%. Employing MTurkers results in significant cost and time saving while reaching accuracy comparable to that of domain experts
Kim and Hodgins (2016)	To evaluate the validity and internal and retest reliability of data obtained from addiction populations on MTurk	Self-reported data for alcohol and gambling populations are of high quality, though caution is warranted because of significant differences in the cannabis sample
Kuang, Agro, Stoddard, Bray, and Zeng-Treitler (2015)	To explore the application of online crowdsourcing for health informatics research, specifically the testing of medical pictographs	The MTurk group scored significantly higher on depicting pictographs with discharge instructions than the traditional in-person sample. Crowdsourcing is a viable complement to traditional in-person surveys, but it cannot replace them
Lee, Lee, Keane, and Tufail (2016)	To evaluate the feasibility of using MTurk as a platform for performing manual segmentation of macular optical coherence tomography (OCT) images	Using MTurk workers to manually perform OCT segmentation generated thousands of data points in a timely and cost-effective manner, and it showed a high degree of interrater reliability. Though this study shows promise of the novelty of using crowdsourced workers for OCT segmentation, it remains unknown whether the accuracy of the data is comparable to evaluations by trained ophthalmologic experts
Lloyd, Yen, Pietrobon, Wiener, Ross, Kokorowski, Nelson, and Routh (2014)	To assess the feasibility of using MTurk online platform to elicit utility values for performing cost-utility analyses of multiple treatment choices	Mturk participants were not representative of the US population, but were more representative than other traditional convenience samples, such as undergraduate campuses. Response rates using MTurk were within acceptable range for survey research and utility values were similar to previous studies. However, these data were not validated against a standard sample using face-to-face interview techniques, and the authors cannot recommend widespread adoption until validated
Maclean and Heer (2013)	To compare MTurkers’ ability to identify medically relevant terms in patient-authored text to annotations by registered nurses	The inter-rater reliability scores for the MTurk and a group of 30 registered nurses on medical word identification tasks were nearly identical, MTurkers performance is comparable in quality to those given by medical experts, and they are an acceptable approximation for expert judgment
Mitry, Peto, Hayat, Blows, Morgan, Khaw, and Foster (2015)	To evaluate the performance and repeatability of crowdsourcing the classification of normal and glaucomatous discs from optic disc images	MTurk workers were able to categorize optic disc images with high sensitivity (83–88%), but poor specificity (35–43%). Crowdsourcing represents a cost-effective image analysis method with good repeatability
Mitry, Zutis, Peto, Hayat, Khaw, Morgan, Moncur, Trucco, and Foster (2016)	To develop a novel online tool to facilitate large-scale annotation of digital retinal images and assess the accuracy of MTurk worker grading using the tool compared to expert classification	Annotation of abnormalities retinal fundus photograph images by ophthalmologically naive MTurk workers is comparable to expert annotation, and the highest agreement with expert annotation was achieved in workers that underwent compulsory training. MTurk has the potential to deliver timely, accurate, and cost-effective image retinal analysis
Mortensen, Musen, and Noy (2013)	To compare MTurkers’ ability to verify correct and incorrect biomedical relationships to the correct answers determined by experts	A method developed to verify ontology relations applied to crowdsourced MTurk workers verified 86% of the relations. High-performance, cost effective strategies can be deployed via an MTurk workforce
Powers, Boonjindasup, Pinsky, Dorsey, Maddox, Su, Gettman, Sundaram, Castle, Lee, and Lee (2016)	To compare the assessment of surgeons’ technical performance of renal artery and vein dissection during robotic partial nephrectomy done by crowdsourced workers and expert surgeon graders	Crowdsourced ratings on MTurk were highly correlated with surgical content experts’ assessments. Crowdsourcing provides a rapid, cost-effective, scalable alternative or adjunct to surgical expert ratings
Santiago-Rivas, Schnur, and Jandorf (2016)	To explore the use of MTurk crowdsourcing for cluster analysis of the assessment of sun protection beliefs relative to the clustering by experts	The authors conclude that the results of their study provide a potential alternative approach to developing future sun protection initiatives in the population
Schleider and Weisz (2015)	To test the feasibility of MTurk as a platform to obtain reports from parents on their family function and youth mental health	Parents on MTurk provided high-quality data, and the authors conclude that MTurk was successful in achieving enrollment goals and comparable to other studies using different samples in attrition, race/ethnicity, and enrollment of single parents
Shao, Guan, Clark, Liu, Santelices, Cortes, and Merchant (2015)	To explore the yields, speed, and costs of recruitment and participant diversity in a world-wide, internet-based study of HIV/AIDS and HIV testing knowledge	MTurk yielded the most participants, recruited participants at the fastest rate, had the highest completion-to-enrollment ratios, and lowest cost per completion for English-speaking platforms relative to other platforms like Google and Facebook. However, international MTurk respondents tend to be well-educated participants from South and Southeast Asia, so the results may not be demographically reflective of the global population
Shapiro, Chandler, and Mueller (2013)	To assess the utility of using Amazon’s MTurk for conducting research on psychopathology	The prevalence of depression, general anxiety, and trauma exposure among MTurk workers matched or exceeded prevalence in the general population, allowing researchers to access participants with the full range of symptoms as they would in the general population
Turner, Kirchhoff, and Capurro (2012)	To examine crowdsourcing as a method to gather feedback on the design of health promotion messages for oral health	Crowdsourcing has the potential to reach more diverse populations than convenience sampling, while substantially reducing the time and cost of gathering participant feedback
White, Kowalewski, Dockter, Comstock, Hannaford, Lendvay (2015)	To test whether crowdsourced layworkers could discriminate robotic surgical skill levels of two dry-laboratory surgical tasks in agreement with expert faculty surgeons	Evidence shows that crowdsourced workers’ assessments are largely in agreement with expert evaluators and are less costly. Limitations of generalizability beyond dry-laboratory setting (potential issues with generalizability to actual human surgery performance) and robotic surgery (i.e., to laparoscopic surgery, open surgeries, etc.)
Wu, Hultman, Diegidio, Hermiz, Garimella, Crutchfield, and Lee (2017)	To test a conjoint analysis of attributes favored by patients seeking an esthetic surgeon, comparing data gathered from MTurk and a pilot study of a university-wide community	Results from a conjoint analysis of desired attributes of an esthetic surgeon were similar among an anonymous university survey and an MTurk survey. The authors conclude that MTurk benefits include broad, diverse, anonymous participant pools, low-cost, rapid data collection, and high completion rate
Wymbs and Dawson (2015)	To determine the utility of MTurk for studying adults with ADHD by screening workers for ADHD diagnostic histories, symptoms, and diagnostic comorbidities	Upon comparison with DSM-V’s and CDC’s current estimates for the national prevalence of ADHD in childhood and adulthood, diagnostic prevalence, demographic correlates, symptom profiles, and internalizing comorbidities are consistent with studies of “offline” populations. MTurk offers an efficient and inexpensive way to gather large quantities of clinically relevant data from adults with ADHD
Yu, Willis, Sun, and Wang (2013)	To compare Mturkers’ interpretations of medical pictograms for pharmaceuticals to those of two judges, and of a panel in another study that used an open-ended, in-person panel	Crowdsourcing via MTurk can be used as an effective and inexpensive approach for participatory evaluation of medical pictograms. The authors noted that misinterpretations of pictograms made by MTurkers were a result of concepts that were difficult to depict graphically and thus the workers exposed design problems, rather than lack of skill

The weaknesses are dominated by the identified strengths, but important to note. Four studies^{20, 36–38} noted three caveats: (1) researchers should exercise caution when generalizing MTurk findings to the US population;^{20, 36} (2) despite a high degree of inter-rater reliability in the MTurk sample, it is unknown whether the accuracy of the data is comparable to evaluations by trained ophthalmologic experts;³⁷ (3) the data were not validated against a sample using face-to-face interview techniques.³⁸ The literature overwhelming concludes that MTurk is an efficient, reliable, cost-effective tool for a variety of tasks with results comparable to those collected via more conventional means. However, results from surveys on MTurk should not be generalized to the US population.

Compliance with Ethical Standards

Conflict of Interest

The authors declare no conflicts of interest.