Observed negative vaccine effectiveness could be the canary in the coal mine for biases in observational COVID-19 studies

Abstract

Since the emergence of the SARS-CoV-2 Omicron variant, multiple observational studies have reported negative vaccine effectiveness (VE) against infection, symptomatic infection, and even severity (hospitalization), potentially leading to an interpretation that vaccines were facilitating infection and disease. However, current observations of negative VE likely stem from the presence of various biases (e.g., exposure differences, testing differences). Although negative VE is more likely to arise when true biological efficacy is generally low and biases are large, positive VE measurements can also be subject to the same mechanisms of bias. In this perspective, we first outline the different mechanisms of bias that could lead to false-negative VE measurements and then discuss their ability to potentially influence other protection measurements. We conclude by discussing the use of suspected false-negative VE measurements as a signal to interrogate the estimates (quantitative bias analysis) and to discuss potential biases when communicating real-world immunity research.

Observational studies are essential for measuring the effects of vaccination in real-world settings [1]. At the end of 2021, observational studies measuring vaccine effectiveness found negative vaccine effectiveness (VE) against infection [2,3] for the SARS-CoV-2 Omicron variant. These negative VE measurements attracted media attention and generated widespread concern about the possible harmful effects of COVID-19 vaccines [4,5]. Although these initial studies occurred during Omicron's early emergence (when reported cases may be nonrepresentative), subsequent studies also have found negative VE measurements against symptomatic infection [6], [7], [8], [9], [10] and against hospital admission [11]. Negative vaccine efficacy is biologically plausible [12]; however, before negative VE measurements can be interpreted as negative biological efficacy, it is important to first consider, examine, and communicate the plausibility and likelihood that negative measurements instead stem from biases, such as confounding and selection bias. Such biases are known to affect VE estimates from retrospective, population-based observational studies that largely rely on linked health-administrative data [13,14]. The key aims of this perspective are to (i) summarize patterns of negative VE and their potential meaning for immunity research (paragraphs 2 and 6); (ii) discuss sources (or mechanisms) of bias related to negative VE using a proposed bias classification framework (paragraphs 3-5; Figure 1 ); and (iii) highlight methods, study designs, and reporting guidelines designed to help communicate and address biases in observational studies (paragraphs 8-10).

Figure 1.

Bias classification framework for observed negative VE against infection. Sources (or mechanisms) of bias capable of producing false-negative VE against infection either affect (1) true levels of infection by vaccination status or (2) observed levels of infection by vaccination status. An example of a mechanism of bias for (1) is differences in exposure (arrows), where higher exposure for vaccinated individuals compared to unvaccinated individuals produces higher infection levels among vaccinated. This bias persists when all infected and uninfected vaccinated and unvaccinated are observed (dark blue and red). In contrast, an example of a mechanism of bias for (2) is differences in testing by vaccination status (due to testing behavior or testing access) where testing differences only result in the perception of higher infection levels. This bias occurs because the unobserved uninfected and infected (light blue and light orange) are excluded from VE measurements. VE, vaccine effectiveness.

There are general patterns across current VE studies that suggest biases are likely the cause of negative VE measurements. First, observed negative VE has not been consistently found for Omicron across VE studies with some studies reporting measurements of negative VE [2,3,[6], [7], [8], [9], [10] and others reporting only positive measurements [15,16]. Second, even within studies that reported negative VE, the observed negative VE only occurred in some instances. For example, in one study, researchers observed a negative VE for those individuals aged 18-59 years but found positive VE measurements for those aged 60 years and older [10]. This pattern of co-existing negative and positive estimates within a single study has also occurred for different vaccination dosages [2,6] and for different times since vaccination [3,9]. Finally, observations of negative VE typically have been found in scenarios where biases could more easily cause an estimate to appear falsely negative. For example, negative VE was often reported for those groups with fewer doses and with less recent vaccinations [3,7,8] when true biological efficacy is likely lower. Here, the same degree of bias that could cause a VE measurement to appear negative may only have resulted in a positive but underestimated VE measurement when true biological efficacy was higher (e.g., with three doses or more recent vaccinations).

Different sources of bias produce negative VE by influencing either (1) the true levels of infection or symptomatic infection by vaccination status (e.g., by differences in exposure) or (2) the observed levels of infection or symptomatic infection by vaccination status (e.g., by differences in testing) (Figure 1). Although the biases related to (1) cause true underlying (symptomatic) infection differences that exist even given perfect sampling, biases related to (2) cause a perception of differences that do not reflect the real (symptomatic) infection levels present in the population. Although both categories of bias are important to discuss in the broader context of VE studies, their relevance for a specific study will depend on the type of VE being measured. The biases related to (1) and (2) can both influence VE measurements of infection and symptomatic infection; the biases related to (2) are less likely to play a role when measuring VE against severe outcomes (e.g., intensive care unit admission, intubation, and death) because testing for these outcomes is unlikely to differ across vaccinated and unvaccinated individuals [14].

The mechanisms of bias that influence the true levels of infection can produce false-negative VE when they lead to vaccinated individuals becoming infected at higher rates than their unvaccinated counterparts. These biases are related to the uncontrolled and often unknown differences in contact, exposure, susceptibility, and immunity. For example, higher contact rates among vaccinated compared with unvaccinated populations (e.g., due to vaccine mandates) can cause higher infection levels in vaccinated populations and therefore can produce observed negative VE when the true biological efficacy is low but still positive [17]. Similarly, if vaccinated individuals experience higher network-level exposure risk (e.g., essential workers), higher susceptibility (e.g., individuals with compromised immune systems), or lower previous infection (e.g., high VE for previous variants), the ratio of infected vaccinated to infected unvaccinated is increased. A larger ratio means VE can be underestimated if the analyses are unable to account for the previously mentioned relationships [14]. Finally, for test-negative designs, the factors that influence infection levels of the control cases also have the potential to lead to false-negative VE measurements. For example, the test-negative controls may include persons with other vaccine preventable infections. If the probabilities of vaccination for COVID-19 and vaccination for other infections are positively correlated, the vaccination of the other infection can act as a confounder, potentially causing COVID-19 VE to be underestimated [1].

The mechanisms of bias related to differential observations by vaccination (and sometimes infection) status can also create the perception of negative VE. The ability to observe infections may be influenced by top-down testing policies (e.g., clinical or employment-based criteria for who can access a test), which can vary across jurisdictions and institutions (e.g., different hospitals may institute different testing policies). They can also be influenced by individual testing-behaviors, which can be shaped by experiences, such as living in households with individuals at greater risk of severity (e.g., older adults and/or persons with compromised immune systems). VE can be underestimated (and negative VE observed) when vaccinated individuals have either more access to testing or are more likely to seek testing [18] . Although a test-negative design can help to correct for some sources of bias (e.g., selection bias due to differences in health care seeking behavior [19,20]), this correction depends on having strict criteria for both enrollment and testing [21]. Similarly, although measuring VE against symptomatic infection may be perceived as being less prone to bias than VE against infection, testing biases can still persist, for example, when testing-behaviors also vary with symptoms [14]. Overall, multiple sources of bias can converge to influence VE estimates with variations in testing, further shaping the direction and magnitude of these other biases [22].

False-negative VE measurements occur when the biases that drive measurements downward become large enough to overcome a vaccine's true positive benefit (i.e., its true biological efficacy) [17]. This reason is likely why, as outlined previously, negative VE estimates have been often observed in scenarios when biological efficacy is expected to be lower [3,7,8]. Omicron has been found to confer an overall lower biological efficacy in reducing susceptibility with vaccination [23,24], which means there have been more opportunities for the previously mentioned mechanisms of bias to potentially cause false-negative VE measurements. Although observed negative VE has been mostly limited to the Omicron period (with some examples from pre-Omicron [25], [32]), the same mechanisms of bias could have been present for any variant and time period but did not lead to an observed negative VE.

Observational studies related to COVID-19 immunity include examining the roles of previous infection and/or hybrid-immunity with vaccination [26], [27]. The measurements of the protective effect of previous infection and hybrid-immunity are also susceptible to the same and similar mechanisms of bias as VE measurements. For example, studies estimating the effectiveness of previous infection may be subject to a selection bias related to a previous infection (e.g., those without a previous infection being less likely to test than those with a history of previous infections [27]); studies estimating effectiveness of hybrid-immunity may also additionally be subject to a selection bias related to vaccination (e.g., those who are unvaccinated being less likely to test than those who are vaccinated). These immunity estimates can also be influenced by misclassification bias or by unknown differences in risk-averse behaviors if they are unaccounted for in the analysis [26]. Therefore, previous infection and hybrid-immunity studies also require careful assessments of potential biases that could affect the protection measurements.

We posit that the findings of the negative VE can act like a canary in a coal mine, signaling that mechanisms of biases likely remain at play in the study design or analytic approaches. An observational study design subject to fewer biases is the prospective cohort design [29] (e.g., SARS-CoV-2 Immunity and Reinfection Evaluation study), which can include systematic and repeated measurements of exposure, potential confounders, and outcomes. Although more resource-intensive than a retrospective cohort design, systematic SARS-CoV-2 testing, infection, and symptom data can reduce the bias related to both differential testing and unknown previous infections [14]. Beyond signaling the potential of bias, observing negative VE further signals the importance of quantitative bias analyses as part of retrospective (and prospective) observational VE studies.

Quantitative bias analysis (QBA) involves a process of systematically examining and testing for the potential impact of systematic errors. The goal is to help estimate the potential direction and magnitude of biases and to quantify the uncertainties around these biases [28]. QBA generally involves visualizing the causal relationships between variables using causal directed acyclic graphs [28], which can be used to conceptualize a priori the various possible mechanisms of bias that could create false-negative VE measurements. When feasible, the collection or use of external data could inform QBA by helping determine the influence of each mechanism of bias on the VE measurements. For example, contact and testing surveys [18,30] could help elucidate whether contact, exposure, or testing vary by vaccination status and if so, provide an estimated strength of the relationships.

The communication surrounding the interpretation of negative VE measurements is also important. The Strengthening the Reporting of Observational Studies in Epidemiology guidelines provide recommendations about detailing which potential biases were considered and controlled for, which biases could remain (e.g., as visualized with directed acyclic graphs), and of those remaining, which could have influenced estimates (assessed via sensitivity analysis) [31]. A recent VE study also explicitly highlighted the presence of a suspected bias in their summary/abstract before outlining its potential causes in their discussion [32]. Future communication could benefit from interpreting the finding of one or more negative VE measurements, including what it means when interpreting other VE estimates in the same study.

The success of a vaccine campaign is influenced by the quality of and trust in real-world evaluations by observational studies [33]. The arrival of negative VE created a media sensation and brought up valid concerns about the potential use of vaccination as our primary control measure for COVID-19 epidemics. Although discussing the plausibility of true negative biological efficacy is important, we posit that emphasis should first be placed on the existence, causes, and implications of false-negative VE measurements. Ignoring the existence of false-negative VE measurements and their causes could inadvertently undermine strategies of future vaccination programs and increase vaccine mistrust. By investing in understanding and addressing false-negative VE, we will not only improve our abilities to interpret existing and future VE studies but also create opportunities to develop new frameworks and methods that can generally advance how we conduct real-world immunity research.

Declarations of competing interest

The authors have no competing interests to declare.