Probability Samples Provide a Means of Benchmarking and Adjusting for Data Collected From Nonprobability Samples

I want to thank Keith et al. (p. 768) for an important exploration of the much-neglected problem of obtaining accurate estimates of the COVID-19 pandemic spread in real time. The authors compared a standard probability sample with a convenience sample (each with about a 1% finite population fraction) and administrative records to estimate seroprevalence of COVID-19 in Jefferson County, Kentucky. Jefferson County is essentially Louisville and the immediate suburbs, with a population of approximately 800 000.¹ They found little difference between the probability sample and the convenience sample with respect to either the distribution of covariates or the prevalence estimates after raking both samples to known sex, race, and geographic region, but substantial differences between the prevalence estimates from the sampling methods and the administrative record estimates.

The results provide important findings that in some ways match prior expectations and in other ways defy them. First, the sample-based estimates of prevalence are higher than prevalences obtained from the administrative records, and the authors assume that sample-based estimates are more accurate. The authors do not state exactly why they make this assumption, but it is presumably because administrative records require a reported positive test filed with Louisville Metro Public Health and Wellness and thus exclude nearly all cases that were asymptomatic, as well as cases that did not result in a visit to a physician or testing site. Assuming that the sample estimates are the “gold standard,” this leads to an underreporting by two factors or more, and perhaps even higher among minorities. This is not a novel finding,^2,3 but it confirms some previous literature in the area and provides a rough estimate of the magnitude by which administratively reported cases (which constitute the vast bulk of prevalence data in both the United States and around the world) should be multiplied to obtain an estimate of the true number of cases, at least during the 2020–2021 period.

The second finding—that the probability sample and the convenience sample prevalence estimates correspond—contradicts a nontrivial body of literature that suggests that even low-response-rate probability samples can yield more accurate results than convenience samples. Kennedy et al.⁴ found major quality concerns using on-line panel convenience samples with respect to the measurement of US political attitudes and recreational interests, especially for Black and Hispanic samples, although there was substantial variation in quality among vendors. In a major study that compared both random digit-dial (RDD) and address-based sampling-web (ABS-web) probability samples with six nonprobability samples with respect to a variety of demographic, health, economic, and transportation measures, MacInnis et al.⁵ found that the probability samples performed substantially better when benchmarked against high-quality census data obtained by the Current Population Survey, the Current Expenditure Survey, or National Center for Health Statistics surveys such as the National Health Interview Survey or the National Health and Nutrition Examination Survey. Furthermore, Groves and Peytcheva⁶ found that nonresponse rates were only weakly linked to nonresponse bias in a large meta-analysis. Further work by Tourangeau⁷ and by Brick and Tourangeau⁸ explained this finding by arguing that most nonresponse is due to missing completely at random⁹ factors (i.e., factors completely independent of any data being collected from participants) such as happenstance of contact time or study-level design features unrelated to sampled member characteristics, or to participant-level characteristics unrelated to survey variables. Thus, it is somewhat surprising that even a low-response-rate survey did not differ to some degree from a volunteer sample. Although the authors cite 30% as a “safe” cutoff for response rates,¹⁰ MacInnis and colleagues’ RDD and ABS-web probability samples had response rates of 15% and 2%, respectively, yet still dominated their convenience sample competitors with respect to bias. It may be that, with their response rates of 2% to 5% (depending on region of the county), Keith et al. have finally descended into volunteer territory, especially given that the data collection required the respondent to make an in-person visit to a separate clinic.

As a survey statistician, I became enormously frustrated that nearly a century of learning how to obtain accurate prevalence estimates from a population appeared to be all but forgotten in this public health crisis. Systems such as the Behavioral Risk Factor Surveillance System that had been put into place long ago to provide flexible, real-time data collection on “emerging public health problems”¹¹ were not up to the task, given the speed of infectious disease spread. As noted by Keith et al., there have been a few attempts to use traditional methods. For example, in a study somewhat similar to that of the authors, Menachemi et al.¹² conducted a study in Indiana with 68 statewide testing facilities, obtaining a considerably higher 24% response rate. Their resulting estimate of population prevalence early in the pandemic (2.8% at the end of April 2020) was far higher than the rate obtained from the number of confirmed cases at the Indiana State Department of Health (0.3% of the population); despite the higher response rate, racial minorities were severely undersampled (8% non-White vs 23% in the population, and 2% Hispanic vs 8% in the population; estimates were postratified to age, gender, and race distributions). Although there are no “gold standards” to assess prevalence measures, the resulting derived case-fatality rate was 0.58% at the time, consistent with the 0.66% Chinese fatality rate estimated at that time after careful adjustment for censoring and ascertainment bias.¹³ On the other hand, major nonprobability samples such as Delphi-Facebook¹⁴ were shown to perform poorly when estimating COVID-19 vaccine uptake compared with the probability sample obtained from Ipsos online KnowledgePanel,¹⁵ even though the latter had only a 10.5% response rate.¹⁶ Although hardly a complete literature review of a still evolving “autopsy” of the failure of the public health and medical community to grapple with this aspect of the COVID-19 pandemic response, it does suggest that a more nimble and survey science–informed response may have been helpful. Survey researchers could have perhaps been more creative in suggesting alternatives to standard methods (e.g., use of at-door drop boxes rather than requiring travel to remote sites), although undoubtedly many such suggestions would have foundered on blanket data collection shutdowns of survey-related research.

In sum, although Keith et al. provide an example in which probability sampling and convenience sampling gave similar results, I believe a broader overview still suggests the need for probability samples to provide a means of benchmarking and adjusting for data collected from nonprobability samples.^17,18 For an excellent example of this approach applied to prevalence estimates that leverage the previously mentioned Indiana study in combination with an Indianapolis-only probability sample, state-level Delphi-Facebook reports of symptoms, and administrative COVID-19 death data, see Dempsey.¹⁹ Dempsey develops a procedure that combines administrative case-count data, data from nonprobability samples, and data from random samples over time to estimate selection propensities based on key covariate information. These selection propensities are then combined with epidemiological forecast models to construct a doubly robust estimator that accounts for both measurement-error and selection bias to estimate population seropositivity.