Use of Recently Vaccinated Individuals to Detect Bias in Test-Negative Case–Control Studies of COVID-19 Vaccine Effectiveness

Abstract

Postauthorization observational studies play a key role in understanding COVID-19 vaccine effectiveness following the demonstration of efficacy in clinical trials. Although bias due to confounding, selection bias, and misclassification can be mitigated through careful study design, unmeasured confounding is likely to remain in these observational studies. Phase III trials of COVID-19 vaccines have shown that protection from vaccination does not occur immediately, meaning that COVID-19 risk should be similar in recently vaccinated and unvaccinated individuals, in the absence of confounding or other bias. Several studies have used the estimated effectiveness among recently vaccinated individuals as a negative control exposure to detect bias in vaccine effectiveness estimates. In this paper, we introduce a theoretical framework to describe the interpretation of such a bias indicator in test-negative studies, and outline strong assumptions that would allow vaccine effectiveness among recently vaccinated individuals to serve as a negative control exposure.

The test-negative case–control study is a common observational study design for estimating vaccine effectiveness, including vaccines against influenza,^1,2 rotavirus,^3,4 and other infectious diseases.⁵ With worldwide roll-out of vaccines against COVID-19 in progress, this study design has been a key tool for assessing the direct effect of vaccination on individuals in real-world settings, within groups not well represented in clinical trials, effectiveness against outcomes other than primary trial outcomes, and effectiveness against variants of concern.

Valid estimates of vaccine effectiveness are obtained from test-negative case–control studies when (i) vaccination has no effect on incidence of the test-negative condition, (ii) misclassification of disease etiology is minimized, and (iii) bias from other sources is minimized.^6,7 Previous work^8,9 has quantified the bias arising from differences in exposure, susceptibility, and healthcare-seeking between unvaccinated and vaccinated populations (manifesting as confounding or selection bias), misclassification of test-positive or test-negative individuals, and differential buildup of naturally acquired immunity over time among the vaccinated and unvaccinated. Such bias is likely to be exacerbated in many situations in which COVID-19 vaccines are being evaluated, with priority for vaccine given to individuals at highest risk of severe disease, individual utilization of vaccine being associated with perceptions of risk or prior exposure,^10–12 and high spatiotemporal heterogeneity in vaccine coverage, infection risk, and access to testing.^13–15

Test-negative controls are an example of a negative control outcome^16,17 used to reduce bias in observational studies, but additional negative control outcomes or exposures can uncover remaining biases. A test-negative study of influenza vaccination in seniors used hospitalization before and after the influenza season as negative control outcomes to detect bias in influenza vaccine effectiveness estimates.¹⁸ Ideally, a negative control outcome used as a bias indicator is an outcome that shares as many common causes as possible with the pathogen of interest, except for vaccination. An association between vaccination and the negative control outcome suggests differences in disease risk between vaccinated and unvaccinated individuals that is unrelated to vaccine effectiveness. Under various assumptions, negative control outcomes have been used to reduce or correct bias.^19,20 A negative control exposure such as vaccination for an unrelated pathogen, not causing the outcome but sharing unmeasured confounders with the exposure of interest, can be used in a similar way.^21,22

As immune response to COVID-19 vaccines takes time to develop, and individuals can be infected before vaccination but develop symptoms within the incubation period, protection is not expected to manifest immediately following vaccination.^23,24 Therefore, for observational studies of COVID-19 vaccines, recent vaccination has been used as a negative control exposure to detect bias.^25–29 Figure 1 shows a directed acyclic graph to represent the potential role of recent vaccination as a negative control exposure; it may share common sources of unmeasured confounding with full vaccination and be causally unrelated to the odds of acquiring or testing positive for COVID-19. In this case, an association between recent vaccination and COVID-19 represents the magnitude of unmeasured confounding. Here, we describe how time-variant differences between vaccinated and unvaccinated individuals, as well as changes in risk over time within vaccinated individuals, would manifest among recently vaccinated individuals and discuss the utility of this group as a bias indicator. In addition, we outline the assumptions necessary for using recent vaccination to correct for bias in vaccine effectiveness in later time periods.

FIGURE 1.

DAG to represent the use of recent vaccination as a negative control exposure in observational studies of vaccine effectiveness. The effect of interest is between A (full vaccination) and Y (COVID-19, or some COVID-19-related outcome). Unmeasured confounding (U) could introduce bias to this estimate. Recent vaccination (Z), which shares some of the same unmeasured confounders (U affects Z), but has no clinical effect on COVID-19 (the arrow from Z to Y is struck out), could serve as a negative control exposure. DAG, directed acyclic graph.

Methods

Theoretical Framework

We follow the framework from Lewnard et al⁸ for a test-negative case–control study. Specifically, we assume that a clinical syndrome of interest (e.g., acute respiratory illness or infection [ARI]) could arise from SARS-CoV-2 infection (test-positive, +) or from infection from causes unrelated to SARS-CoV-2 (test-negative control, –). Individuals undergo constant hazard λ_i of infection, where subscript i denotes the test-positive or test-negative outcome (λ₊ or λ_–), and develop ARI upon infection with probability π_i. Upon developing ARI, individuals seek treatment with probability μ_v, where subscript v represents vaccination status. To allow for differential risk of infection among vaccinated and unvaccinated individuals independent of vaccination (representing unmeasured confounding U in Figure 1), we define the parameter α_vi to be the hazard ratio for infection with SARS-CoV-2 (+) and non-SARS-CoV-2 pathogens (–) among this group relative to the general population.

We extend the framework to model a vaccination campaign, in which a proportion v of a population of size P are vaccinated (V), with the remaining population unvaccinated (U). For simplicity, we consider a single-dose vaccine, which provides no protection within T_P days of vaccination, and has efficacy given by $VE=\phi (1-\theta )$ thereafter, where $\phi$ is the proportion who respond to vaccination and $\theta$ is the hazard ratio of SARS-CoV-2 infection among those who respond. Therefore, among those who will eventually be vaccinated, at different times individuals may have not received the vaccine (pending vaccination, P), recently vaccinated (R), or fully vaccinated (F). The timescale t is defined relative to the start of the vaccination campaign, t = 0. For simplicity, we assume that all individuals are vaccinated at the same time T_V, so that they are protected at time T_V + T_P. Allowing each individual to have different vaccination times produces identical results if vaccination time is not associated with SARS-CoV-2 infection risk (eAppendix; http://links.lww.com/EDE/B913). Table 1 displays a list of model parameters used in this framework.

TABLE 1.

Table of Parameters and Definitions, Adapted from Lewnard et al⁸

Parameter	Definition
${\lambda }_i$	Force of infection for SARS-CoV-2 (+) or non-SARS-CoV-2 pathogens (–)
${\pi }_i$	Probability of ARI given infection for SARS-CoV-2 (+) or non-SARS-CoV-2 pathogens (–)
${\mu }_v$	Probability of seeking treatment given ARI, among individuals who are unvaccinated (v = U), pending vaccination (v = P), recently vaccinated (v = R), or fully vaccinated (v = F)
${\alpha }_{vi}$	Hazard ratio for infection with SARS-CoV-2 (+) or non-SARS-CoV-2 pathogens (–) (relative to population average) due to factors other than vaccine-derived protection, among individuals who are unvaccinated (v = U), pending vaccination (v = P), recently vaccinated (v = R), or fully vaccinated (v = F)
P	Total size of population
v	Proportion of population who are vaccinated
TP	Time from vaccination to full protection from vaccine
φ	Proportion of individuals responding to vaccine
θ	Hazard ratio for infection resulting from vaccine-derived protection (among responders)
TV	Time of vaccination, relative to start of vaccination campaign

For simplicity, we assume that the study is conducted in a setting in which both the prevalence of SARS-CoV-2 infection and of ARI from other causes are low, to minimize misclassification of cases due to ARI of other etiologies occurring in individuals infected with SARS-CoV-2.⁹ If this assumption were not met, misclassification would introduce bias, which would affect the interpretation of vaccine effectiveness and of a negative control exposure. However, such bias is expected to be small unless the clinical syndrome is nonspecific or prevalence of the clinical syndrome is high (>0.1).⁹ We follow the method of Lewnard et al⁸ demonstrating that, in the presence of differential exposure and susceptibility between vaccinated and unvaccinated individuals, the odds ratio (OR) of vaccination comparing test-positive to test-negative individuals estimates the following quantity:

$$1-OR\left(t\right)= 1-\frac{{\alpha }_{U-}}{{\alpha }_{V-}}\left(\frac{\left(1-\phi \right)\left(1-e^{-{\alpha }_{V+}{\lambda }_{+}t}\right)+\phi \left(1-e^{-{\alpha }_{V+}\theta {\lambda }_{+}t}\right)}{1-e^{-{\alpha }_{U+}{\lambda }_{+}t}}\right)$$

RESULTS

Interpretation of the Odds Ratio Comparing Recently Vaccinated to Unvaccinated (Bias Indicator)

If we assume the vaccine has no biologic effect on the risk of being a case among those recently vaccinated, the cumulative incidence of the test-positive and test-negative conditions among recently vaccinated individuals arises from person-time between T_V and T_V+T_P among those who are vaccinated.

$$C_{R+}\left(t\right)={\pi }_{+}{\mu }_R\left(1-e^{-{\alpha }_{R+}{\lambda }_{+}{T}_P}\right)vP$$

$$C_{R-}\left(t\right)={\pi }_{-}{\mu }_R{\alpha }_{R-}{\lambda }_{-}{T}_{P}vP$$

The cumulative incidence of the test-positive and test-negative conditions among unvaccinated individuals arises from person-time among those who are not vaccinated at the time of analysis, and among those who are pending vaccination. In particular,

$$C_{U+}\left(t\right)={\pi }_{+}{\mu }_U\left(1-e^{-{\alpha }_{U+}{\lambda }_{+}t}\right)\left(1-v\right)P+{\pi }_{+}{\mu }_P\left(1-e^{-{\alpha }_{P+}{\lambda }_{+}{T}_V}\right)vP$$

$$C_{U-}\left(t\right)={\alpha }_{U-}{\lambda }_{-}{\pi }_{-}{\mu }_U\left(1-v\right)Pt+{\alpha }_{P-}{\lambda }_{-}{\pi }_{-}{\mu }_{P}vP{T}_V$$

The estimated vaccine direct effect among individuals recently vaccinated, compared with those not vaccinated, could serve as an indicator of bias.

$$\begin{array}{l}O{R}_U^R\left(t\right)= \frac{[{\mu }_U{\alpha }_{U-}(1-v)t+{\mu }_P{\alpha }_{P-}v{T}_V]}{[{\mu }_U(1-e^{-{\alpha }_{U+}{\lambda }_{+}t})\left(1-v\right)+{\mu }_P(1-e^{-{\alpha }_{P+}{\lambda }_{+}{T}_V})v]}\\ \frac{(1-e^{-{\alpha }_{R+}{\lambda }_{+}{T}_P})}{{\alpha }_{R-}{T}_P}\approx \frac{{\alpha }_{R+}}{{\alpha }_{R-}}\frac{\left[{\mu }_U{\alpha }_{U-}\left(1-v\right)t+{\mu }_P{\alpha }_{P-}v{T}_V\right]}{\left[{\mu }_U{\alpha }_{U+}\left(1-v\right)t+{\mu }_P{\alpha }_{P+}v{T}_V\right]} \end{array}$$ (1)

when t is close to T_V, and T_V and T_P are small. Therefore, over a short time scale following initiation of the vaccine campaign, the odds ratio comparing recently vaccinated individuals to unvaccinated individuals estimates the relative susceptibility to SARS-CoV-2 infection compared with infections with another etiology among individuals eligible for and having recently received vaccination, compared with those who are not vaccinated. The bias indicator is dependent on the proportion of individuals who are pending vaccination among unvaccinated individuals. Individuals awaiting vaccination might be more similar in their characteristics to vaccinated individuals compared with unvaccinated individuals, affecting the magnitude of the bias indicator. In addition, the composition of the unvaccinated group might change over the course of a vaccination campaign, leading to dynamic changes in the bias indicator.

Interpretation of the Odds Ratio With Recently Vaccinated Persons as a Reference Group (Bias-correction)

Now assume that differences in exposure, susceptibility, and healthcare-seeking over time in vaccinated individuals are negligible (i.e. ${\alpha }_{R-}={\alpha }_{F-}={\alpha }_{V-},$ ${\alpha }_{R+}={\alpha }_{F+}={\alpha }_{V+},$ and ${\mu }_R={\mu }_F={\mu }_V$). Then,

$$\begin{array}{l}1-O{R}_R^F\left(t\right)= 1-\left(\frac{\left(1-\phi \right)\left(1-e^{-{\alpha }_{V+}{\lambda }_{+}\left(t-T_P-T_V\right)}\right)+\phi \left(1-e^{-{\alpha }_{V+}\theta {\lambda }_{+}\left(t-T_P-T_V\right)}\right)}{1-e^{-{\alpha }_{V+}{\lambda }_{+}{T}_P}}\right)\\ \frac{T_P}{\left(t-T_P-T_V\right)}. \end{array}$$ (2)

The eAppendix (http://links.lww.com/EDE/B913) shows a full derivation of this expression). When t is close to the T_V, and T_P is small, equation 2 reduces to:

$$1-O{R}_R^F\left(t\right)=\phi (1-\theta ),$$

providing an unbiased estimate of vaccine efficacy.

Illustration of Bias Indicator and Bias-correction Method

The magnitude and direction of the bias indicator (equation 1) is displayed in Figure 2 (left column), and the effect of applying the bias-correction method (equation 2) to the vaccine effectiveness estimate (right column). In the left column, the solid line represents the bias indicator estimated over time since vaccination. In the right column, the solid line represents the vaccine effectiveness estimate in the presence of bias, and the dotted line represents the bias-corrected estimate. The efficacy of the vaccin e is 70%.

FIGURE 2.

Bias indicators (left column) and biased (solid) and bias-corrected (dotted) estimates of vaccine effectiveness (right column) over time since vaccination. We assume higher risk of SARS-CoV-2 among vaccinated individuals that is time-invariant (first row), higher risk of SARS-CoV-2 in recently vaccinated individuals (second row), higher risk of SARS-CoV-2 among fully vaccinated individuals (third row), a small effect of vaccination on disease risk among recently vaccinated individuals (fourth row), and reduced probability of seeking testing among recently and fully vaccinated individuals (fifth row). The bias indicator should be 1 if vaccinated and unvaccinated individuals have the same underlying risk of testing positive for COVID-19 (dashed line, left column). The true vaccine effectiveness is 70% (dashed line, right column).

In the first row, the relative hazard of SARS-CoV-2 comparing vaccinated to unvaccinated, from factors unrelated to vaccination, is α_P+ = α_R+ =α_F+=1.25 (i.e., vaccinated individuals have higher infection risk due to factors other than vaccination). In this case, the bias indicator is above one, the uncorrected method underestimates vaccine effectiveness, and the bias-corrected method returns a valid estimate in the period immediately following the vaccine campaign. As immunity builds up in the unvaccinated population, the estimated vaccine effectiveness decreases over time.⁸ In the second row, the relative hazard of SARS-CoV-2 is higher among those who are recently vaccinated, so that α_R+ =1.25 (i.e., individuals may change their behavior immediately following vaccination if they believe they are protected from infection). In this case, the bias indicator is above one but the uncorrected estimate is unbiased, and the bias-correction method introduces bias. In the third row, both individuals pending vaccination and recently vaccinated have lower risk than those fully vaccinated and unvaccinated (representing relaxation of other risk mitigation practices by those who are fully immunized). In this case, counteracting biases cause the bias indicator to be close to one, but the bias-correction method overestimates vaccine effectiveness because recently vaccinated individuals are at lower risk than unvaccinated individuals. In the fourth, the vaccine efficacy is 10% among recently vaccinated individuals. In this case, the bias indicator is slightly above one, and the bias-corrected method underestimates vaccine effectiveness as the reference period includes time in which individuals were protected by vaccination. Finally, in the bottom row, recently and fully vaccinated individuals are less likely to seek care for moderate symptoms. In this case, the bias indicator is above one, the uncorrected method underestimates vaccine effectiveness (as severe cases are overrepresented among vaccinated individuals), and the bias-correction method returns a valid estimate of vaccine effectiveness immediately following the vaccine campaign, that decreases over time. Code to reproduce these figures is provided in the Supplementary Material; http://links.lww.com/EDE/B913.

DISCUSSION

Comparison of recently vaccinated and unvaccinated individuals in a test-negative case–control study can be interpreted, under certain assumptions and over a short time scale following initiation of the vaccination campaign, as the relative difference in infection risk between vaccinated and unvaccinated groups due to factors other than vaccination. However, due to possible time-varying changes in risk, the bias indicator will often not fulfill the properties of a negative control exposure as outlined in Figure 1. Use of recently vaccinated individuals as unexposed individuals to remove unmeasured confounding relies on the strong assumption that differences between vaccinated and unvaccinated individuals are time-invariant and is not guaranteed to reduce bias.

The interpretation of the bias indicator and the validity of the bias-correction method rely on two key assumptions: that all parameters relating to differences in exposure, susceptibility, and test-seeking between fully vaccinated and recently vaccinated individuals are time-invariant, and that the definition of recently vaccinated is chosen such that vaccination has not had time to affect the risk of infection or the chosen clinical outcome. One can imagine situations in which the first assumption does not hold. For example, individuals who have a scheduled vaccine appointment may take extra measures to protect themselves from risk in the time immediately before or following vaccination, and conversely take fewer precautions once they believe they are protected by vaccination (α_F–>α_R– and α_F+>α_R+). In addition, some observational studies^26,28 have observed “deferral bias”, in which individuals feeling sick choose not to get vaccinated and subsequently test-positive for SARS-CoV-2, leading to apparent protection from vaccination among recently vaccinated individuals (α_PI>α_RI). Finally, COVID-19 vaccination campaigns have been conducted so that high-risk individuals are initially targeted or are early adopters.²⁷ In such situations, there is more potential for differences between those who vaccinated earlier and later. The likelihood of being fully vaccinated among cases and controls may therefore be confounded by predictors of early vaccination. Even if controlling for risk group through matching or stratification, those who receive the vaccine earlier within eligibility groups may have different risk of disease.

For studies not restricted to severe symptoms, differences in the distribution of moderate and severe disease, and differences in test-seeking behavior, between fully vaccinated and recently vaccinated individuals, can lead to further bias if not accounted for.³⁰ Individuals who have been recently vaccinated may be less likely to seek testing for moderate symptoms, believing them to be side effects of vaccination. On the other hand, a fully vaccinated individual might be less likely to seek testing for moderate symptoms once they believe themselves to be protected against SARS-CoV-2 infection. Therefore, the bias indicator may represent a mixture of time-invariant and time-varying bias that could be in either direction, and consequently the proposed bias-correction method is not guaranteed to reduce bias.

In addition, it is not clear for every COVID-19 vaccine what time period should define a recently vaccinated individual who has yet to experience clinical protection. Consistent with results from phase III trials of COVID-19 vaccines, vaccine protection is not immediate. Differences in COVID-19 risk were observed starting 10–12 days following first dose in the trials of the BNT162b2 mRNA²³ and the mRNA-1273 vaccines,²⁴ and 28 days following first dose in the interim analysis of the ChAdOx1 vaccine.³¹ A natural choice would be some quantile of the incubation period of SARS-CoV-2 (e.g., 11.5 days³²), as infections seen in this period would likely have been acquired before vaccination. If the clinical syndrome of interest is hospitalization, a longer time window would be appropriate, representing the time from infection to hospitalization. However, it is likely that protection builds up during this period, contrary to the assumptions of the model, meaning that longer time windows will include periods in which the vaccine is partially protective. In addition, in populations with moderate seroprevalence, individuals who have had prior infection may experience protection from a single dose of vaccine earlier than expected based on trial results.³³ In addition, although this has not been demonstrated for any COVID-19 vaccine, some vaccines are known to elicit nonspecific immune responses,³⁴ which could lead to some vaccine effect in the days immediately following vaccination and an underestimate of vaccine effectiveness among fully vaccinated individuals when correcting for bias. The time window should therefore be chosen to be as short as possible to minimize the possibility of bias. However, a shorter window leads to lower prevalence of recent vaccination, increasing the standard error of $O{R}_U^R\left(t\right)$. Selection of the time window for definition of recent vaccination thus constitutes a trade-off between bias and variance.

From this discussion, it is clear that the bias-correction method presented here is not guaranteed to eliminate or even reduce bias, relying as it does on strong, unverifiable assumptions, and consequently we do not recommend it in general. On the other hand, the bias indicator is easy to estimate and can provide useful context in which to interpret estimates of vaccine effectiveness from an observational study. Table 2 outlines suggested interpretations and recommendations based on observed estimates of the bias indicator. In particular, a bias indicator close to one provides evidence that unmeasured confounding is low, although it could be consistent with time-varying risks that act in different directions (as in Figure 2, third row). A bias indicator different from one indicates either that there is unmeasured confounding or selection bias (as in Figure 2, first and fifth row), or that recently vaccinated individuals have different COVID-19 risk than other types of individuals (as in Figure 2, second row). In this case, further adjustment for bias should be attempted, and secondary analyses using smaller time windows should be performed to understand change in risk over time within recently vaccinated individuals. We caution that further adjustment should involve structural confounders (e.g., U in Figure 1), and that achieving a bias indicator of one should not be the goal of this adjustment, given that the bias indicator is not guaranteed to be one in the absence of unmeasured confounding. If the bias indicator remains far from one, the discussion should address this analysis and offer possible explanations as well as caution in interpreting estimates of vaccine effectiveness.

TABLE 2.

Interpretations of Bias Indicator Results and Associated Recommendations

Result	Interpretation	Recommendation
Bias indicator close to 1	• Consistent with negligible unmeasured confounding between vaccinated and unvaccinated; or	• Vaccine effectiveness estimate could be unbiased
	• Consistent with different sources of unmeasured confounding acting in different directions	• Include secondary analysis dividing time window into smaller windows if sample size allows; examine trend in bias indicator over time
Bias indicator not close to 1, with low precision (wide 95% confidence interval)	• Lack of precision due to small sample size of recently vaccinated	• Increase sample size if feasible;
		• Include secondary analysis with longer time window to define recent vaccination
Bias indicator not close to 1, with high precision (narrow 95% confidence interval)	• Consistent with unmeasured confounding between vaccinated and unvaccinated; or	• Include covariates in model to reduce confounding
	• Different COVID-19 risk among recently vaccinated individuals only	• Include secondary analysis dividing time window into smaller windows if sample size allows
		• Express caution in interpretation of vaccine effectiveness

Several large studies have found bias indicators that differ from one. Lopez-Bernal et al³⁵ found increased odds of COVID-19 up to 9 days following receipt of a single dose of BNT162b2, that reduced over the course of the campaign. The authors interpreted this finding as indicating that high-risk individuals were more targeted for vaccination early in the vaccine campaign in the United Kingdom, which would imply that vaccine effectiveness was underestimated in this study. Chung et al³⁶ similarly found increased odds of COVID-19 7–13 days following a single dose of mRNA vaccines, but not after 0–6 days. The authors suggested an increase in risk among those believing they were protected by the vaccine (time-varying), or that vaccinated individuals were at higher baseline risk, although the lack of effectiveness from 0 to 6 days suggests that unmeasured confounding is small. Further studies found bias indicators that did not vary from one, and used these results as evidence for lack of unmeasured confounding in vaccine effectiveness estimates.^37–39

These considerations apply to other observational designs, notably the cohort study.^25,26 Use of recently vaccinated individuals as the unexposed group would necessitate similar assumptions about time-invarying risk of infection and testing behavior following vaccination. For all such designs, this discussion clarifies the usefulness of the bias indicator and its limitations as a true negative control exposure. Although under certain assumptions bias can be minimized, these assumptions are likely to be unverifiable from the available data. Data detailing time to onset of immunogenicity with established correlates of protection would be of value to inform design of studies comparing risk in different time periods after vaccination as a bias-correction strategy.