Says who? The significance of sampling in mental health surveys during COVID-19

The COVID-19 pandemic is expected to have profound and enduring effects on mental health but, until we have data, we will not know its form, extent, duration, or distribution. An appropriate public health response to mitigate and manage mental health sequelae is likely to require substantial diversion of resources. Such decisions must be underpinned by reliable information: policy makers, commissioners, and services need to know both the scale of need and who is most vulnerable. A position paper¹ in The Lancet Psychiatry highlights that “an immediate priority is collecting high-quality data on the mental health effects of the COVID-19 pandemic across the whole population and vulnerable groups”. This statement should be a clarion call for governments to fund, and for researchers to gather, timely, high-quality population mental health data that represent the true need arising from the pandemic.

Instead, the desire for quick information has driven the rapid propagation of online surveys using non-probability and convenience samples, some of which claim to be representative. Understandably, many are receiving widespread media attention. These early insights might be valuable, but we caution against relying on them to drive policy and resource because they are prone to substantial bias: acting on misleading information could be worse than having no information at all.

Survey sampling and design choices must be led by their purpose. If the survey is to generate quick ideas, consult on perspectives, or foster community engagement, rapid, low-cost convenience sampling is appropriate. However, to understand prevalence in a population, how survey respondents are recruited is crucially important. Non-probability samples are usually recruited by approaching membership lists, through service providers, existing large convenience panels, or from snowball recruitment using word-of-mouth, often via social media. Such samples attract volunteers who are already well engaged, interested in the topic, and who can access the internet.

Bias can affect any survey, but can be particularly problematic for social and mental health surveys in which those excluded are often most in need. Individuals with existing or severe mental illness are less likely to participate online than those without such conditions,² whereas half of people aged 75 and over, and many with mental illness (who represent a key COVID-19 risk group), are not regular internet users. Access to digital devices is also limited among the most vulnerable and deprived children. Most surveys weight their sample to match their target population by specific characteristics; however, these adjustments miss crucial elements of bias and cannot account for groups not included at all, particularly if the response rate is unknown.

A common misconception is that larger samples solve these biases. One Chinese study³ of mental health responses to the pandemic gathered an impressive 52 730 respondents; however, 65% were female, indicating a highly skewed sample in a population with significantly fewer women than men.⁴ We recommend for all surveys to detail their sampling strategy and to publish comparative statistics with the population they are sampled from so that informed judgments can be made about representativeness.

The value of a survey depends on its use of data. Non-probability sampling lacks a sound theoretical basis for statistical inference,⁵ which means basic descriptive analyses and explorations of potential associations are appropriate but measures of uncertainty (ie, confidence intervals around estimates of prevalence) are generally not valid. Moreover, the ability to compare the population's mental health before and after the COVID-19 pandemic is compromised if surveys do not use standardised measures that are reliable and stable over time and if pre-pandemic baseline data from the same population are not available.

The current crisis has compromised several established data sources: health registries that previously quantified mental illness prevalence⁶ have reported a reduction in patient contacts. National registries of mental illness and suicide will catch up but are a poor tool in the short term. Many official surveys have suspended the collection of data in response to physical distancing guidelines or transferred to remote interviews, which have affected comparability with previous waves⁷ and created new challenges, such as how to gather sensitive data on self-harm, suicidality, or intimate partner violence.⁸

We believe it is possible, and cost-effective, to generate high-quality evidence of mental health needs in the current crisis. We recommend using random sampling to reduce risk of bias, allow quantification of non-response, and permit valid statistical analysis. A major investigation into online survey panels⁹ concluded that “Researchers should avoid nonprobability online panels when...[the] objective is to accurately estimate population values.” When determining the prevalence of the mental health effects of COVID-19, investigators should use rigorous methods that sample from the whole population to reduce erroneous conclusions and potentially damaging actions. This approach might be more expensive but is essential to gain reliable insights into how to mitigate psychological risks during this and future pandemics. Cutting corners to provide quick, cheap answers will result in poorer quality evidence, poorer policy, and wasted resources in the longer term. We can and must do better.