Behavioral scientists often think in abstract terms about the people who participate in research. University students form a ‘subject pool’. People recruited outside the university are a ‘community sample’. And, people who complete studies online are part of an ‘online panel’ or a ‘crowdsourcing platform’. This language obscures something important: participants in human subjects research are people, shaped by their unique experiences and shared cultural events. Since early 2020, coronavirus disease 2019 (COVID-19) has been one of these events.
How might research participants change in response to COVID-19? One possibility raised by Lourenco and Tasimi [1] is that online participants may become less diverse. Because tens of millions of people have recently become unemployed [2], it is reasonable to wonder whether people who are struggling to pay the bills will be able to maintain an internet connection–let alone find the time and energy to participate in research. If people are forced to choose between essential things like food and nonessential things like the internet, the diversity of people in online platforms like Amazon Mechanical Turk (MTurk) may decrease, leaving researchers with a narrower group from which to sample. As Lourenco and Tasimi note, decreasing diversity should concern all researchers at a time when more research is moving online, because less diversity may affect the replicability of studies.
Fortunately, on MTurk, we do not see evidence of this so far. Demographic data showing who used MTurk before COVID-19 [3,4] compares favorably with data from today. Table 1 shows the race, income, and gender of people on MTurk from January 2019 through mid-May 2020. The data–which are from CloudResearch’s Metrics tool and are publicly available at https://metrics.cloudresearch.com/–are largely consistent. Despite small changes here and there, the race, income, and gender of people on MTurk have remained remarkably constant.
Table 1.
Selected Participant Demographics on Mechanical Turk across 18 Monthsa
| 2019 |
2020 |
||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Variable | January | February | March | April | May | June | July | August | September | October | November | December | January | February | March | April | May |
| Race | |||||||||||||||||
| White | 69 | 70 | 69 | 69 | 69 | 69 | 69 | 69 | 71 | 69 | 70 | 68 | 70 | 70 | 69 | 70 | 70 |
| Hispanic | 7 | 7 | 8 | 8 | 8 | 8 | 8 | 7 | 7 | 8 | 7 | 7 | 7 | 7 | 7 | 7 | 7 |
| Black | 10 | 9 | 9 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 10 | 11 | 11 | 10 | 10 | 10 |
| Asian | 11 | 11 | 11 | 10 | 11 | 11 | 10 | 11 | 10 | 11 | 10 | 12 | 10 | 10 | 11 | 11 | 11 |
| Other | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 2 | 3 | 2 | 3 | 2 | 3 | 3 | 2 | 2 |
| Gender | |||||||||||||||||
| Men | 45 | 45 | 46 | 44 | 45 | 44 | 43 | 44 | 43 | 43 | 43 | 45 | 44 | 44 | 44 | 44 | 45 |
| Women | 55 | 55 | 54 | 56 | 55 | 56 | 57 | 56 | 57 | 57 | 57 | 55 | 56 | 56 | 56 | 56 | 55 |
| Income | |||||||||||||||||
| <$10 000 | 5 | 5 | 5 | 5 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 5 | 6 | 6 | 6 | 6 |
| $10 000–$19 999 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 7 | 7 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 7 |
| $20 000–$29 999 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 11 | 11 | 11 | 11 |
| $30 000–$39 999 | 13 | 12 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 13 | 12 | 12 |
| $40 000–$49 999 | 12 | 11 | 11 | 11 | 11 | 11 | 12 | 12 | 11 | 11 | 12 | 11 | 11 | 11 | 11 | 10 | 11 |
| $50 000–$59 999 | 11 | 11 | 11 | 11 | 11 | 11 | 11 | 12 | 12 | 12 | 11 | 11 | 12 | 12 | 11 | 11 | 11 |
| $60 000–$69 999 | 9 | 9 | 9 | 9 | 8 | 8 | 8 | 8 | 9 | 8 | 9 | 8 | 9 | 8 | 9 | 8 | 8 |
| $70 000–$79 999 | 8 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 | 9 |
| $80 000–$89 999 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 6 | 6 | 5 | 5 |
| $90 000–$99 999 | 6 | 6 | 6 | 6 | 6 | 6 | 5 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 | 6 |
| $100 000–$124 999 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 8 | 9 | 9 |
| $125 000–$149 999 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 5 | 5 | 5 | 5 |
| $150 000–$174 999 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 3 |
| $175 000–$199 999 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| $200 000–$224 999 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| $225 000–$249 999 | 0.4 | 0.4 | 0.4 | 0.4 | 0 | 0.4 | 0.4 | 0.4 | 0.4 | 0.4 | 0.3 | 0.3 | 0.4 | 0.4 | 0.4 | 0.5 | 0.5 |
| >$250 000 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Numbers are percentages of unique participants within each month from the CloudResearch database. A description of this database and how well it captures the Mechanical Turk population appears in Robinson et al. [7] and Moss et al. [8]. To examine the percentage of studies completed by people within each demographic group each month, visit https://metrics.cloudresearch.com/
Of course, demographic consistency thus far is no guarantee changes will not occur. For a few reasons, however, we are cautiously optimistic about the stability of participant demographics on platforms like MTurk. First, most people who use MTurk do so from home. According to one survey, less than 2% of people connect to MTurk in public spaces (https://www.cloudresearch.com/resources/blog/trends-of-mturk-workers/), meaning an inability to access the internet in places like a coffee shop or library is unlikely to change who is on MTurk.
Next, while a staggering number of people are currently unemployed, people who lost their job and previously used MTurk differ from those who lost their job and do not use MTurk in at least one important way: users of MTurk have access to an online platform that enables them to earn money from home. Although MTurk was not intended as a source of full-time employment, it provides an opportunity for people to supplement their income. This opportunity may be all the more important to people who have recently become unemployed and may rely on MTurk to temporarily make ends meet or pay for relatively small bills like an internet connection, something most people agree is currently important [5].
Finally, we are optimistic about the demographic consistency of people on platforms like MTurk because, ultimately, what matters more than a change in the overall demographics of participants, which have historically been stable [6], is the demographic composition of participants within samples. Although within sample variability is overlooked by many researchers, the composition of participants within samples can vary substantially [6]. For example, while platform level data on MTurk indicate a near even gender split [4], the percentage of men can vary from 25% to as high as 75% across samples. Similar variation occurs for other demographics [6]. If participant demographics are important for generalizability and estimating the strength of effect sizes, then this issue deserves more attention. Fortunately, because online platforms are bigger, more diverse, and easier to manage than many traditional sources of participant recruitment, researchers can control the demographics of participants within studies using quotas. Most participant recruitment platforms make setting such quotas easy.
To close, even though we are relatively sanguine about the prospect of demographic consistency within online research platforms, we would like to echo a point made by Lourenco and Tasimi [1] about the new normal of conducting behavioral research. COVID-19 has changed how social scientists approach their work. As the pandemic progresses, more changes may follow. To conduct rigorous research, behavioral scientists need to be creative. Finding ways to reach people where they are, to push for a close of the digital divide, and to connect with people who are not currently part of online research platforms will strengthen the quality of behavioral research.
Disclaimer Statement
All authors are employed at Prime Research Solutions, the company that owns CloudResearch (formerly TurkPrime). One CloudResearch offering is its MTurk Toolkit which helps researchers run flexible social science studies on Mechanical Turk.
References
- 1.Lourenco S.F., Tasimi A. No participant left behind: conducting science during COVID-19. Trends Cogn. Sci. 2020;24:583–584. doi: 10.1016/j.tics.2020.05.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Schwartz N.D. The New York Times; 2020. How bad is unemployment? ‘Literally off the charts’. 8 May. [Google Scholar]
- 3.Difallah D. Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. 2018. Demographics and dynamics of Mechanical Turk workers; pp. 135–143. [Google Scholar]
- 4.Robinson J. Who are the Mechanical Turk workers? In: Litman L., Robinson J., editors. Conducting Online Research on Amazon Mechanical Turk and Beyond. Sage Academic Publishing; 2020. pp. 121–146. [Google Scholar]
- 5.Vogels E.A. Pew Research Center; 2020. 53% of Americans Say the Internet Has Been Essential during the COVID-19 Outbreak. [Google Scholar]
- 6.Litman L. Sampling Mechanical Turk workers: problems and solutions. In: Litman L., Robinson J., editors. Conducting Online Research on Amazon Mechanical Turk and Beyond. Sage Academic Publishing; 2020. pp. 148–172. [Google Scholar]
- 7.Robinson J. Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool. PLoS One. 2019;14 doi: 10.1371/journal.pone.0226394. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.A.J. Moss, et al., Is it ethical to use Mechanical Turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages. PsyArxiv Published online April 28, 2020. 10.31234/osf.io/jbc9d [DOI] [PubMed]