In The Lancet Psychiatry, Matthias Pierce and colleagues1, 2 identify the importance of sampling in studying mental health effects of COVID-19. We found that a mental health survey3 using a commercial panel (of approximately 20 000 people) overrepresented mentally unhealthy respondents by approximately 2·5 times. This overrepresentation occurred despite multiple measures to ensure representativeness: prespecified demographic and geographical sampling quotas; post-collection checks on the distribution of socioeconomic parameters; and adjustments for mismatches between clinical psychological scores and use of health-care services. Further random subsampling, before analysis, was required to correct for this sampling bias.
It seems that self-selected commercial survey panels in general might be biased towards mentally unhealthy or unhappy individuals. Commercial survey organisations operate through networks of subcontractors who hold customer contact lists. Individuals self-select to take part, for a small financial incentive. This might create bias towards people who are in difficult financial circumstances, and hence are under mental stress. The turnover in these self-selected panels is high.
It is now easy to target precise population segments using social media, but difficult to obtain random representative population samples. Political4 and personality5 representativeness have been tested. Surveys measuring mental health specifically can correct for bias during analysis.3 However, commercial surveys are also widely adopted in physical and social health research, and these might risk invalid results if they omit mental health measures.
Acknowledgments
We declare no competing interests.
References
- 1.Pierce M, McManus S, Jessop C. Says who? The significance of sampling in mental health surveys during COVID-19. Lancet Psychiatry. 2020;7:567–568. doi: 10.1016/S2215-0366(20)30237-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Pierce M, Hope H, Ford T. Mental health before and during the COVID-19 pandemic: a longitudinal probability sample survey of the UK population. Lancet Psychiatry. 2020 doi: 10.1016/S2215-0366(20)30308-4. published online July 21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Buckley R, Brough P, Hague L. Economic value of protected areas via visitor mental health. Nat Commun. 2019;10 doi: 10.1038/s41467-019-12631-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Boas TC, Christenson DP, Glick DM. Recruiting large online samples in the United States and India: Facebook, Mechanical Turk, and Qualtrics. Political Sci Res Methods. 2020;8:232–250. [Google Scholar]
- 5.Buchanan T. Personality biases in different types of internet samples can influence research outcomes. Comput Human Behav. 2018;86:235–244. [Google Scholar]