Abstract
Background:
Observational studies are used for estimating vaccine effectiveness under real-world conditions. The practical performance of two common approaches – cohort and test-negative designs – need to be compared for Covid-19 vaccines.
Methods:
We compared the cohort and test-negative designs to estimate the effectiveness of the BNT162b2 vaccine against Covid-19 outcomes using nationwide data from the U.S. Department of Veterans Affairs. Specifically, we (1) explicitly emulated a target trial using follow-up data and evaluated the potential for confounding using negative controls and benchmarking to a randomized trial, (2) performed case–control sampling of the cohort to confirm empirically that the same estimate is obtained, (3) further restricted the sampling to person–days with a test, and (4) implemented additional features of a test-negative design. We also compared their performance in limited datasets.
Results:
Estimated BNT162b2 vaccine effectiveness was similar under all four designs. Empirical results suggested limited residual confounding by health care-seeking behavior. Analyses in limited datasets showed evidence of residual confounding, with estimates biased downward in the cohort design and upward in the test-negative design.
Conclusions:
Vaccine effectiveness estimates under a cohort design with explicit target trial emulation and a test-negative design were similar when using rich information from the VA healthcare system, but diverged in opposite directions when using a limited dataset. In settings like ours with sufficient information on confounders and other key variables, the cohort design with explicit target trial emulation may be preferable as a principled approach that allows estimation of absolute risks and facilitates interpretation of effect estimates.
Keywords: causal inference, cohort design, Covid-19, electronic health records, target trial, vaccine effectiveness, test-negative design
Introduction
Observational studies are widely used to estimate vaccine effectiveness under real-world conditions. Two study designs are commonly used: cohort design and test-negative design. A cohort design with explicit emulation of a target trial prevents design-induced bias1,2 and yields estimates of both absolute risks and relative risk, but these estimates may be affected by residual confounding, especially by health care-seeking behaviors. The test-negative design3 yields only estimates of relative measures,4 but may reduce confounding because it restricts the study population to those who seek health care and get tested for the infection of interest.5,6 However, the test-negative design does not explicitly emulate a target trial and thus its estimates may be affected by design-induced bias. For example, conditioning on receiving a test during the follow-up period is a form of post-baseline stratification that may result in selection bias.5,7
Here we describe a methodologic framework to connect the cohort design and the test-negative design and use both designs to estimate the effectiveness of the BNT162b2 vaccine against Covid-19 outcomes in the largest integrated health care system in the United States. We start by specifying a target trial of vaccine effectiveness. Then, we use the observational data to explicitly emulate the target trial under a cohort design, and progressively modify the design until transformation into a test-negative design that no longer emulates a target trial. We then compare the performance of the cohort design and test-negative design when using (i) all available variables in the data and (ii) only the subset of the variables that is often available when using test-negative designs. We conclude by providing recommendations for the implementation of these designs in real world data.
Specification of the target trial
The preferred approach to estimate vaccine effectiveness is to conduct a randomized trial in which individuals are assigned to either vaccination or no vaccination. Here we specify the (hypothetical) pragmatic trial that would answer the causal question of interest – the target trial.
The key components of the protocol of the target trial are summarized in Table 1. Eligible individuals between January 4 and May 27 2021 would be randomly assigned to either (1) immediate vaccination with a first dose of the BNT162b2 vaccine with a second dose scheduled 21 days later, or (2) no vaccination at any time over follow-up. The two outcomes of interest are polymerase-chain-reaction [PCR]-confirmed SARS-CoV-2 infection and symptomatic Covid-19. Individuals would be followed until the outcome, 22 weeks after assignment, or July 1 2021, whichever occurred first.
Table 1.
Specification and Emulation of a Target Trial Evaluating the Effectiveness of the BNT162b2 Vaccine Using Observational Data from Veterans Health Administration Electronic Health Records (January 4–July 1 2021).
| Target trial specification | Target trial emulation | |
|---|---|---|
| Eligibility criteria |
|
Same as for the target trial, except:
|
| Treatment strategies |
|
Same as for the target trial. Vaccination was identified using records from the VA Corporate Data Warehouse (i.e., records in the ‘Immunization’ domain, procedures recorded in the ‘Inpatient’ and ‘Outpatient’ domains, and records in the VA Covid-19 Shared Data Resource), linked with Medicare databases. There was strict adherence to vaccine deployment protocols in this population. We confirmed that the frequency of testing was similar in the vaccinated and unvaccinated groups. |
| Assignment procedures | Individuals are randomly assigned to a strategy at baseline within strata defined by calendar date (5-day bins), age (5-year bins), sex (male or female), race (White, Black, other or unknown), urbanicity of residence (urban or not urban), geographic location (coded as 18 categories of Veterans Integrated Services Network), smoking status (never, former, or current), body-mass index (<18.5, 18.5–25, 25–30, or ≥30 kg/m2), number of SARS-CoV-2 PCR tests before baseline (0 or 1), and number of influenza vaccinations over the previous 5 years (0, 1–2, 3–4, or ≥5), Individuals will be aware of the assigned treatment strategy. | We assumed random assignment after matching individuals who were vaccinated with the BNT162b2 vaccine in a 1:1 ratio to eligible unvaccinated individuals, using the same factors used for stratified randomization as in the target trial. |
| Follow-up period | For each person, follow-up starts on the day of assignment (time zero) and ends on the day of the outcome of interest, 22 weeks after baseline, or the administrative end of follow-up (July 1 2021), whichever occurs first. | Same as for the target trial. |
| Outcome | Documented SARS-CoV-2 infection (PCR-confirmed) Symptomatic Covid-19 (defined as at least one of the following symptoms documented within the VA health care system within 4 days before or after a positive PCR test: fever, chills, cough, shortness of breath or difficulty breathing, sore throat, loss of taste or smell, headache, myalgia, diarrhea, and vomiting) |
Same as for the target trial. |
| Causal contrast | Intention-to-treat effect, per-protocol effect | Observational analogue of per-protocol effect |
| Analysis plan | Intention-to-treat analysis: Cumulative incidence (risk) curves and estimates of 22-week risk, risk differences, risk ratios, and incidence rate ratios comparing the BNT162b2-vaccinated vs unvaccinated groups. Per-protocol analysis: Same except that individuals in the no vaccine group are censored if/when they received the vaccine and prognostic factors are adjusted for. |
Same per-protocol analysis as for the target trial, except that adjustment is achieved by censoring both members of the matched pair when its unvaccinated member received a Covid-19 vaccine. |
Abbreviations: PCR, polymerase chain reaction; VA, Department of Veterans Affairs.
Both the intention-to-treat effect and the per-protocol effect would be of interest. In the intention-to-treat analysis, risks (cumulative incidences) would be calculated using the Kaplan–Meier estimator,8 and 22-week risks of each outcome would be calculated and compared between the vaccine groups via differences and ratios. The per-protocol analysis would be identical except that unvaccinated individuals would be censored if and when they received a Covid-19 vaccine, and the analysis would be adjusted for prognostic factors measured at baseline if they became imbalanced between the groups after censoring.
For comparison with previous studies, risks would also be estimated between day 28 (7 days after the second scheduled dose) and the end of follow-up. Excluding this early follow-up period, when immunity is gradually building,9 was standard practice in randomized trials but may induce selection bias.10 To facilitate comparison with estimates from the randomized trial and other designs that we consider below, incidence rates (number of cases per person–days of follow-up) would also be calculated in each group and vaccine effectiveness would be defined as (1 – incidence rate ratio) x 100%. Nonparametric bootstrapping with 500 samples would be used to calculate percentile-based 95% confidence intervals for all estimates.
The protocol of the target trial also needs to specify the frequency with which participants would be tested for infection because, by definition, individuals can be determined to have the outcome only on days when a test is performed. That is, the protocol needs to specify a joint intervention strategy on both vaccination and testing. Ideally, individuals would be tested every day so that infections would be detected as soon as possible (Figure 1). Because daily testing may be infeasible, the protocol of a pragmatic trial would establish weekly testing or any other testing strategy (e.g., testing whenever an individual reports symptoms of infection) that is the same for the vaccinated and unvaccinated groups. For each participant, all days before a positive test would be considered days free of the outcome, regardless of whether a test was conducted (with a negative result) or not.
Figure 1.
Study designs when outcome ascertainment requires a test as in studies of vaccine effectiveness for documented infection. Boxes denote person–days contributing to the analysis, C covariate ascertainment, E eligibility criteria assessment, A vaccination status ascertainment, T- negative PCR test, T+ positive PCR test.
We now use the observational data to (1) explicitly emulate the target trial under a cohort design, (2) perform case–control sampling of the cohort to confirm empirically that the same estimate is obtained, and, last, sequentially implement modifications that transform the case–control design into a test-negative design by (3) further restricting the sampling to person–days with a test, and (4) assessing eligibility, vaccination status, and covariates at the time of testing. Throughout this process, we describe how the modifications associated with the test-negative design result in deviations from the target trial. Figure 1 illustrates the progression of designs considered in this study.
Emulation of the target trial under a cohort design
We emulated the above target trial using nationwide health care databases from the United States Department of Veterans Affairs, the largest integrated health care system in the country, as described previously.11 (Additional details on the target trial emulation are provided in Table 1, and detailed definitions of all study variables are provided in eTable 1).
Because vaccination was not randomly assigned in the real world, we assumed that vaccination occurred approximately at random within levels of measured prognostic factors: calendar date (5-day bins), age (5-year bins), sex (male or female), race (White, Black, other, or unknown), urban residence (yes or no), geographic location (coded as one of 18 categories of the Veterans Integrated Services Network), smoking status (never, former, or current), body-mass index (<18.5, 18.5–25, 25–30, or ≥30 kg/m2), number of SARS-CoV-2 PCR tests previously received (0 or 1), and number of influenza vaccinations over the previous five years (0, 1–2, 3–4, or ≥5). We then matched eligible vaccinated and unvaccinated persons with the same values of these variables in a 1:1 ratio.11 The matching factors are potential confounders associated with the probability of receiving a vaccine and risk of the outcomes. (Additional details on the matching algorithm are provided in eMethods 1.)
If no prognostic factors are imbalanced between groups, then this matching appropriately emulates randomization in a target trial conducted in individuals with the same distribution of baseline characteristics as the vaccinated persons. We evaluated balance of measured factors between matched groups via standardized mean differences, with a difference of 0.1 or less considered acceptable.12 To assess the potential for unmeasured confounding (e.g., by health care-seeking behavior, as depicted in Figure 2a), we evaluated documented SARS-CoV-2 infection during the first 10 days of follow-up as a negative control outcome that is not affected by vaccination, but for which the effect of vaccination is expected to be confounded similarly to the main outcome.13 Due to the absence of a prespecified testing strategy in the real world, we also checked whether the frequency of testing over follow-up was similar in the vaccinated and unvaccinated groups.
Figure 2.
Causal directed acyclic graphs (DAGs) under (a) a cohort design, (b) case–control sampling, and (c) case–control sampling restricted to person–days with a test. A denotes vaccination status, T receipt of a PCR test, Y SARS-CoV-2 infection, Y* documented SARS-CoV-2 infection (positive PCR test), U unmeasured confounders (e.g., health care-seeking behavior), S selection of cases and controls. Boxes around nodes denote stratification/conditioning. For simplicity, all DAGs omit measured covariates (matching factors) and censoring of matched pairs, and the DAG in panel (c) omits matching on test dates.
The analysis proceeded as for the target trial, except (1) to estimate the per-protocol effect, we censored follow-up of a matched pair if and when its unvaccinated member received a Covid-19 vaccine, (2) we estimated the risk between day 28 and the end of follow-up using matched pairs in which both members were still at risk at the beginning of the period, and (3) we repeated the analysis sequentially starting on each day between January 4 and May 27 2021.
In sensitivity analyses, we defined documented SARS-CoV-2 infection as a positive PCR or antigen test (identified via the VA Covid-19 National Surveillance Tool)14, and restricted membership in the vaccinated group to individuals who received a BNT162b2 vaccine inside the VA health care system (to further restrict the population to those who may be more likely to seek health care within the VA system).
Analyses were performed with SAS software, version 8.3 (SAS Institute), SAS PROC IML on Linux Operating System, and R software, version 4.1.2 (R Foundation for Statistical Computing).
Results
Of 571,307 vaccinated individuals who were eligible for the cohort analysis (Figure 3), 546,810 were matched with unvaccinated individuals (see eTable 2 for comparison with the unmatched population). Measured variables were well-balanced between the vaccinated and unvaccinated groups (Table 2, eFigure 1). We found a nearly identical risk pattern of documented SARS-CoV-2 infection in the first 10 days of follow-up in both groups (Figure 4), which suggests little unmeasured confounding. The frequency of negative PCR tests during the follow-up was similar in the vaccinated and unvaccinated groups, except that vaccinated individuals received fewer tests during the first week after their first dose and more tests around the second dose (eFigure 2), which suggests that differences in risk between groups cannot be attributed to differences in testing patterns.
Figure 3.
Selection of Individuals for the Emulation of a Target Trial Evaluating the Effectiveness of the BNT162b2 Vaccine under a Cohort Design, U.S. Veterans Health Administration (January 4–July 1 2021). VA denotes Department of Veterans Affairs.
Table 2.
Baseline Characteristics of Matched Individuals in the Cohort Analysis for Evaluating the Effectiveness of the BNT162b2 Vaccine, U.S. Veterans Health Administration (January 4–July 1 2021).
| Vaccinated (n = 546,810) | Unvaccinated (n = 546,810) | |
|---|---|---|
| Age group, years (%) | ||
| 18–39 | 32281 (5.9) | 32488 (5.9) |
| 40–49 | 38275 (7.0) | 38435 (7.0) |
| 50–59 | 74969 (13.7) | 75487 (13.8) |
| 60–69 | 128975 (23.6) | 130273 (23.8) |
| 70–79 | 210171 (38.4) | 207917 (38.0) |
| ≥80 | 62139 (11.4) | 62210 (11.4) |
| Male sex (%) | 502520 (91.9) | 502520 (91.9) |
| Race (%) | ||
| White | 392248 (71.7) | 392248 (71.7) |
| Black | 117515 (21.5) | 117515 (21.5) |
| Other | 13093 (2.4) | 13093 (2.4) |
| Unknown | 23954 (4.4) | 23954 (4.4) |
| Ethnicity (%) | ||
| Not Hispanic | 490620 (89.7) | 490569 (89.7) |
| Hispanic | 38528 (7.0) | 38304 (7.0) |
| Unknown | 17662 (3.2) | 17937 (3.3) |
| Urban residence (%) | 407362 (74.5) | 407362 (74.5) |
| Smoking Status (%) | ||
| Never | 199682 (36.5) | 199682 (36.5) |
| Current | 167059 (30.6) | 167059 (30.6) |
| Former | 180069 (32.9) | 180069 (32.9) |
| Body-mass index, kg/m² - mean (SD) | 30.3 (6.0) | 30.3 (6.1) |
| Comorbidities (%) | ||
| Cancer | 66003 (12.1) | 61459 (11.2) |
| Chronic lung disease | 79647 (14.6) | 83134 (15.2) |
| Cardiovascular disease | 133992 (24.5) | 135741 (24.8) |
| Hypertension | 333592 (61.0) | 334791 (61.2) |
| Diabetes | 176085 (32.2) | 180868 (33.1) |
| Chronic kidney disease | 48418 (8.9) | 49729 (9.1) |
| Chronic liver disease | 16762 (3.1) | 16542 (3.0) |
| Obesity | 255614 (46.7) | 255614 (46.7) |
| Dementia | 9354 (1.7) | 11184 (2.0) |
| Substance use disorder | 33926 (6.2) | 35189 (6.4) |
| Immunocompromised state | 31225 (5.7) | 28670 (5.2) |
| Primary care visits in the past 5 years (%) | ||
| 1–9 | 100363 (18.4) | 91232 (16.7) |
| 10–19 | 197086 (36.0) | 193392 (35.4) |
| 20–29 | 124367 (22.7) | 126129 (23.1) |
| ≥30 | 124994 (22.9) | 136057 (24.9) |
| Hospital admissions in the past 5 years (%) | ||
| 0 | 416224 (76.1) | 420121 (76.8) |
| 1–4 | 79855 (14.6) | 70498 (12.9) |
| ≥5 | 50731 (9.3) | 56191 (10.3) |
| Emergency room visits in the past 5 years (%) | ||
| 0 | 272286 (49.8) | 276220 (50.5) |
| 1–2 | 129431 (23.7) | 125171 (22.9) |
| 3–4 | 59282 (10.8) | 57355 (10.5) |
| ≥5 | 85811 (15.7) | 88064 (16.1) |
| Influenza vaccinations in the past 5 years (%) | ||
| 0 | 74190 (13.6) | 74190 (13.6) |
| 1–2 | 107187 (19.6) | 107187 (19.6) |
| 3–4 | 177800 (32.5) | 177800 (32.5) |
| ≥5 | 187633 (34.3) | 187633 (34.3) |
| PCR tests in the past (%) | ||
| 0 | 427327 (78.1) | 427327 (78.1) |
| 1–2 | 101365 (18.5) | 101900 (18.6) |
| 3–4 | 14134 (2.6) | 13544 (2.5) |
| ≥5 | 3984 (0.7) | 4039 (0.7) |
Abbreviations: PCR, polymerase chain reaction.
Figure 4.
Negative Control Outcome: Cumulative Incidence of Documented SARS-CoV-2 infections in the First 10 Days After the First Dose by Vaccination Status under a Cohort Design, U.S. Veterans Health Administration (January 4–July 1 2021).
Over 22 weeks, 2,808 SARS-CoV-2 infections were documented, of which 1,308 were detected as symptomatic within the VA health care system. Among individuals who received a first dose of the vaccine and had at least 21 days of follow-up, 97% received a second dose of the vaccine (69% on day 21, 90% before day 24 and 93% before day 28). Data for 61% of the unvaccinated individuals and their matched pairs were censored when the unvaccinated received a vaccine.
Figure 5 shows the cumulative incidence curves. The estimated 22-week risk ratios (95% CI) for the BNT162b2 vaccine vs. no vaccine were 0.30 (0.28 to 0.35) for documented SARS-CoV-2 infection and 0.26 (0.22 to 0.30) for symptomatic Covid-19 (eTable 3). In the period starting on day 28, estimated vaccine effectiveness was 87.2% (85.4% to 91.8%) for symptomatic Covid-19 (Table 3); as a benchmark, the estimates also based on incidence rate ratios for symptomatic Covid-19 from a randomized trial were 95.0% (90.3% to 97.6%) up to 14 weeks after the second dose9 and 91.3% (89.0% to 93.2%) up to 6 months (~24 weeks) after the second dose.15 (See eMethods 2 for a discussion of differences between risk ratio and rate ratio estimates.) Estimates were similar under all sensitivity analyses (eTable 4, eFigure 3).
Figure 5.
Cumulative Incidence of Covid-19 Outcomes by Vaccination Status under a Cohort Design, U.S. Veterans Health Administration (January 4–July 1 2021).
Documented SARS-CoV-2 infection
Symptomatic Covid-19
Table 3.
Estimated Vaccine Effectiveness (VE) a of the BNT162b2 Vaccine under Different Designs, U.S. Veterans Health Administration (January 4–July 1 2021).
| From day 0 | From day 28 | |||
|---|---|---|---|---|
| Cases | VE, % (95% CI) | Cases | VE, % (95% CI) | |
| Documented SARS-CoV-2 infection | ||||
| Target trial emulation – Cohort | 2808 | 63.6 (59.5, 65.9) | 1292 | 80.4 (77.1, 82.7) |
| Target trial emulation – Case–control sampling | 2808 | 63.5 (59.3, 65.7) | 1292 | 80.6 (77.0, 82.8) |
| Case–control sampling restricted to test days | 2798 | 59.6 (54.2, 62.9) | 1290 | 80.4 (76.6, 83.2) |
| Test-negative design b | 14159 | 66.3 (63.9, 68.5) | 13407 | 80.2 (78.2, 82.0) |
|
| ||||
| Symptomatic Covid-19 | ||||
| Target trial emulation – Cohort | 1308 | 66.3 (61.9, 70.9) | 504 | 87.2 (85.4, 91.8) |
| Target trial emulation – Case–control sampling | 1308 | 66.0 (61.6, 70.7) | 504 | 87.2 (85.2, 91.7) |
| Case–control sampling restricted to test days | 1104 | 61.9 (56.4, 69.9) | 455 | 87.0 (84.3, 92.6) |
| Test-negative design b | 6696 | 67.8 (64.2, 71.1) | 6308 | 86.8 (84.2, 89.0) |
All analyses adjusted for calendar date, age, sex, race, urban residence, geographic location, smoking status, body-mass index, number of SARS-CoV-2 PCR tests previously received, and number of influenza vaccinations over the previous five years.
As the test-negative design does not explicitly specify the period of follow-up, we considered estimates from day 0 as “receipt of the first dose of BNT162b2 between January 4 2021 and the test date” and estimates from day 28 as “receipt the first dose of BNT162b2 between January 4 2021 and 28 days before the test date.”
Case–control sampling of the above cohorts
We performed risk-set case–control sampling16 of each of the above sequential cohorts (after matching and censoring). Cases were all individuals with a positive PCR test over the study period. For each case, we randomly selected 1000 controls who were under follow-up and without a positive PCR test on the case’s test date (Figure 2b). The odds ratio from this case–control sampling is an unbiased estimator of the incidence rate ratio in the target trial.16 However, the case–control sampling generally precludes the estimation of absolute risk. We used conditional logistic regression to estimate odds ratios, and defined vaccine effectiveness as (1 – odds ratio) x 100%. We repeated the case–control sampling in each of the 500 bootstrap samples from the cohort design to calculate percentile-based 95% confidence intervals for all estimates.
Results
As expected, vaccine effectiveness estimates were the same as those from the cohort (Table 3).
In the analyses that follow, we sequentially implement modifications that transform this case–control design into a test-negative design.
Case–control sampling restricted to person–days with a test
Our analyses above strongly suggest little residual confounding (e.g., by health care-seeking behavior), because there was similar testing frequency for the vaccinated and unvaccinated groups, comparable distributions of measured risk factors, nearly identical risk patterns for the negative control outcome, and effect estimates were close to those of a randomized trial. In other settings, however, there may be concerns about unmeasured confounding because those who seek health care and get tested for the infection may be different from those who are not tested. A proposed approach to tackle this problem is to restrict the analysis to individuals who received a test during the follow-up. However, this approach is the equivalent of a post-baseline restriction in the target trial, and thus may result in selection bias. The magnitude of the bias increases with the amount of residual confounding and the association of testing with vaccination status and with the outcome of interest (Figure 2c).17–19
To implement this approach, we repeated the case–control sampling described above but selecting matched controls only from person–days with a negative test near the case’s test date20–23 (in a 5-day bin), in a ratio of 4 controls:1 case. For symptomatic Covid-19, we only considered person–days with a test and with symptoms recorded within 4 days of the test date. As above, we used conditional logistic regression to estimate odds ratios, and defined vaccine effectiveness as (1 – odds ratio) x 100%. The odds ratio from this design can be interpreted as the rate ratio from a target trial with daily testing if both groups were tested daily or adequate adjustment could be made for prognostic factors associated with differences in testing strategies.
Results
In the period starting on day 28, the estimated vaccine effectiveness was 80.4% (76.6% to 83.2%) for documented SARS-CoV-2 infection, and 87.0% (84.3% to 92.6%) for symptomatic Covid-19 (Table 3). These estimates were similar to those obtained under the previous designs.
Test-negative design
The test-negative design resembles a case–control sampling restricted to person–days with a test. However, the test-negative design deviates from the emulation of a target trial because it uses the time of testing, rather than the start of follow-up of the target trial, to determine eligibility, define vaccination status, and assess covariates for confounding adjustment. This is equivalent to adjusting for post-baseline variables in the target trial and thus may induce selection bias.24,25
To implement a test-negative design in our data, we applied the same eligibility criteria as in the cohort analysis (except “having no health care interaction or SARS-CoV-2 tests within the past seven days”), but we assessed eligibility on the test day rather than at time zero of follow-up. We then matched cases to controls in a 1:4 ratio on test date (5-day bins, same as above). For analyses of symptomatic Covid-19, we restricted cases and controls to individuals who had symptoms around the time of testing. We classified individuals as vaccinated if they received the first dose of BNT162b2 between January 4 2021, and 28 days prior to the test date, and unvaccinated if they had not received any dose of vaccine before the test date. In an attempt to estimate vaccine effectiveness that also includes the first 28 days of follow-up, we conducted separate analyses in which individuals were instead classified as vaccinated if they received the first dose of BNT162b2 between January 4 2021, and the test date (i.e., these analyses included individuals vaccinated in the 28 days prior to the test date). Multivariable conditional logistic regression was used to estimate odds ratios and 95% confidence intervals, conditional on the matched sets and adjusted for covariates (same as the matching factors in the cohort design, except calendar date at time zero) measured at the time of testing. We estimated vaccine effectiveness as (1 – odds ratio) x 100%.
Finally, in sensitivity analyses for our test-negative designs, we (1) excluded individuals with a history of documented SARS-CoV-2 infection within 90 days (vs. ever) before the test date, (2) removed the eligibility criterion of receiving care at a station eligible to administer the BNT162b2 vaccine, (3) redefined controls as individuals with exclusively negative tests during the follow-up, (4) assessed eligibility criteria and covariates on January 4 2021, instead of on the test date, (5) restricted each individual to be matched as a control at most once, (6) additionally adjusted for symptoms recorded in the database within 4 days of the test date (via conditional logistic regression), (7) conducted a test-negative analysis within each of the sequential cohorts used to emulate the target trial, and (8) only considered vaccines administered inside the VA health care system.
Results
Of 1,176,837 SARS-CoV-2 PCR tests documented in the VA health care system between January 4 and July 1 2021, 363,677 tests met the eligibility criteria (Figure 6). After excluding tests within 28 days of vaccination, 13,407 test-positive cases were matched to 53,628 test-negative controls. The matched controls had a lower proportion of vaccinated individuals than the initially eligible controls (eTable 5). Compared with matched controls, matched cases included a higher proportion of younger individuals, never smokers, documented symptoms around the test date, and a lower proportion of individuals with comorbidities (except for obesity) and markers of high health care utilization (Table 4). 4.9% of cases and 16.5% of controls were vaccinated.
Figure 6.
Selection of Tests to Evaluate the Effectiveness of the BNT162b2 Vaccine under a Test-Negative Design, U.S. Veterans Health Administration (January 4–July 1 2021). VA denotes Department of Veterans Affairs.
Table 4.
Characteristicsa of the Matched Study Participants under a Test-Negative Design, U.S. Veterans Health Administration (January 4–July 1 2021).
| Cases (n = 13,407) |
Controls (n = 53,628) |
|
|---|---|---|
| BNT162b2 vaccinated b (%) | 651 (4.9) | 8831 (16.5) |
| Age group, years (%) | ||
| 18–39 | 1749 (13.0) | 4922 (9.2) |
| 40–49 | 1581 (11.8) | 5008 (9.3) |
| 50–59 | 2740 (20.4) | 9444 (17.6) |
| 60–69 | 3183 (23.7) | 14676 (27.4) |
| 70–79 | 3411 (25.4) | 16178 (30.2) |
| ≥80 | 743 (5.5) | 3400 (6.3) |
| Male sex (%) | 11838 (88.3) | 47316 (88.2) |
| Race (%) | ||
| White | 9083 (67.7) | 36397 (67.9) |
| Black | 3277 (24.4) | 13732 (25.6) |
| Other | 390 (2.9) | 1399 (2.6) |
| Unknown | 657 (4.9) | 2100 (3.9) |
| Ethnicity (%) | ||
| Not Hispanic | 11738 (87.6) | 47462 (88.5) |
| Hispanic | 1280 (9.5) | 4547 (8.5) |
| Unknown | 389 (2.9) | 1619 (3.0) |
| Month of test (%) | ||
| January 2021 | 6799 (50.7) | 27123 (50.6) |
| February 2021 | 2206 (16.5) | 8863 (16.5) |
| March 2021 | 1352 (10.1) | 5476 (10.2) |
| April 2021 | 1416 (10.6) | 5639 (10.5) |
| May 2021 | 983 (7.3) | 3940 (7.3) |
| June 2021 | 626 (4.7) | 2507 (4.7) |
| July 2021 | 25 (0.2) | 80 (0.1) |
| Urban residence (%) | 9609 (71.7) | 39963 (74.5) |
| Smoking Status (%) | ||
| Never | 5533 (41.3) | 16890 (31.5) |
| Current | 4128 (30.8) | 23530 (43.9) |
| Former | 3746 (27.9) | 13208 (24.6) |
| Body-mass index, kg/m² - mean (SD) | 31.5 (6.5) | 30.0 (6.4) |
| Comorbidities (%) | ||
| Cancer | 1664 (12.4) | 9734 (18.2) |
| Chronic lung disease | 2289 (17.1) | 12346 (23.0) |
| Cardiovascular disease | 2885 (21.5) | 15353 (28.6) |
| Hypertension | 7760 (57.9) | 33659 (62.8) |
| Diabetes | 4484 (33.4) | 18005 (33.6) |
| Chronic kidney disease | 1336 (10.0) | 7003 (13.1) |
| Chronic liver disease | 545 (4.1) | 3773 (7.0) |
| Obesity | 7330 (54.7) | 24463 (45.6) |
| Dementia | 226 (1.7) | 1415 (2.6) |
| Substance use disorder | 1242 (9.3) | 8278 (15.4) |
| Immunocompromised state | 1200 (9.0) | 6089 (11.4) |
| Primary care visits in the past 5 years (%) | ||
| 1–9 visits | 1267 (9.5) | 3909 (7.3) |
| 10–19 | 3976 (29.7) | 12553 (23.4) |
| 20–29 | 3400 (25.4) | 13154 (24.5) |
| ≥30 | 4764 (35.5) | 24012 (44.8) |
| Hospital admissions in the past 5 years (%) | ||
| 0 | 8593 (64.1) | 27813 (51.9) |
| 1–4 | 2329 (17.4) | 10395 (19.4) |
| ≥5 | 2485 (18.5) | 15420 (28.8) |
| Emergency room visits in the past 5 years (%) | ||
| 0 | 3194 (23.8) | 11553 (21.5) |
| 1–2 | 3177 (23.7) | 12337 (23.0) |
| 3–4 | 2111 (15.7) | 8272 (15.4) |
| ≥5 | 4925 (36.7) | 21466 (40.0) |
| Influenza vaccinations in the past 5 years (%) | ||
| 0 | 3530 (26.3) | 11042 (20.6) |
| 1–2 | 2853 (21.3) | 10720 (20.0) |
| 3–4 | 3503 (26.1) | 14936 (27.9) |
| ≥5 | 3521 (26.3) | 16930 (31.6) |
| PCR tests in the past (%) | ||
| 0 | 7664 (57.2) | 19616 (36.6) |
| 1–2 | 4455 (33.2) | 20274 (37.8) |
| 3–4 | 873 (6.5) | 6942 (12.9) |
| ≥5 | 415 (3.1) | 6796 (12.7) |
| Documented symptoms within 4 days (%) | 6981 (52.1) | 8895 (16.6) |
Abbreviations: PCR, polymerase chain reaction.
Characteristics were assessed as of the test date, unless otherwise noted.
Received the first dose of BNT162b2 between January 4 2021 and 28 days prior to the test date
For documented SARS-CoV-2 infection, the estimated vaccine effectiveness (95% CI) was 80.2% (78.2% to 82.0%) (Table 3), and ranged from 78.4% to 82.6% in the specified sensitivity analyses (eTables 6–7). Estimated vaccine effectiveness when allowing any vaccination before the test date (i.e., first dose of BNT162b2 received between January 4 2021 and the test date) ranged from 60.6% to 69.3% in sensitivity analyses (eTables 6–7).
For symptomatic Covid-19, the estimated vaccine effectiveness was 86.8% (84.2% to 89.0%) comparing full BNT162b2 vaccination vs. no vaccination (Table 3), and ranged from 85.4% to 88.9% in sensitivity analyses. Estimated vaccine effectiveness when allowing any vaccination before the test date ranged from 60.2% to 70.6% (eTable 6).
Analyses in limited datasets (using only a subset of variables)
In this study, we used health care databases that capture rich information on demographic factors, medical records, and markers of health care utilization, which allowed us to apply strict eligibility criteria and closely match individuals according to key confounders. Accordingly, we found empirical evidence for limited residual confounding in our analysis.
In settings of limited data where fewer variables are available, test-negative designs may be particularly appealing, as a way to reduce (potentially unmeasured) confounding by health care-seeking behavior.6 Indeed, many test-negative designs26–29 apply fewer eligibility criteria and adjust for fewer covariates than in our analysis. To compare the performance of cohort and test-negative designs in limited data settings, we implemented both designs in a limited dataset, with the following commonly available variables: calendar date, history of prior SARS-CoV-2 infection and vaccination, basic demographics (age, sex, race, geographic location), vaccination status, documented SARS-CoV-2 infection, and symptoms. (See eMethods 3 for design protocols.)
Results
Of 1,558,808 vaccinated individuals who were eligible for the cohort analysis in the limited dataset (eFigure 4), 1,546,157 were matched with unvaccinated individuals. Empirical assessment of residual confounding showed that the vaccinated group included a higher proportion of individuals with urban residence, recorded comorbidities and markers of health care utilization at baseline, and received more PCR tests during the follow up; however, the risks for the negative control outcome were similar in the two groups (eFigure 5). Estimated vaccine effectiveness was 52.3% (48.7% to 54.3%) for documented SARS-CoV-2 infection and 55.1% (51.2% to 59.2%) for symptomatic Covid-19 (Table 5, eFigure 6). In the period starting on day 28, estimated vaccine effectiveness was 72.0% (68.9% to 74.4%) for documented SARS-CoV-2 infection and 82.5% (79.3% to 85.5%) for symptomatic Covid-19. These estimates of effectiveness were lower than those obtained in the primary cohort analysis (in the full dataset) and in randomized trials.
Table 5.
Estimated Vaccine Effectiveness (VE) of the BNT162b2 Vaccine under Designs in Limited Datasets, U.S. Veterans Health Administration (January 4–July 1 2021).
| From day 0 | From day 28 | |||
|---|---|---|---|---|
| Cases | VE, % (95% CI) | Cases | VE, % (95% CI) | |
| Documented SARS-CoV-2 infection | ||||
| Cohort Designs | ||||
| Target trial emulation (ref a) | 2808 | 63.6 (59.5, 65.9) | 1292 | 80.4 (77.1, 82.7) |
| In a limited dataset b | 5891 | 52.3 (48.7, 54.3) | 2775 | 72.0 (68.9, 74.4) |
| Test-Negative Designs | ||||
| Primary analysis (ref a) | 14159 | 66.3 (63.9, 68.5) | 13407 | 80.2 (78.2, 82.0) |
| In a limited dataset b,c | 28522 | 69.3 (67.8, 70.8) | 27098 | 83.4 (82.2, 84.5) |
| In a further limited dataset c,d | 35126 | 81.5 (80.7, 82.2) | 33586 | 88.9 (88.2, 89.5) |
|
| ||||
| Symptomatic Covid-19 | ||||
| Cohort Designs | ||||
| Target trial emulation (ref a) | 1308 | 66.3 (61.9, 70.9) | 504 | 87.2 (85.4, 91.8) |
| In a limited dataset b | 2582 | 55.1 (51.2, 59.2) | 964 | 82.5 (79.3, 85.5) |
| Test-Negative Designs | ||||
| Primary analysis (ref a) | 6696 | 67.8 (64.2, 71.1) | 6308 | 86.8 (84.2, 89.0) |
| In a limited dataset b,c | 13026 | 69.8 (67.5, 72.0) | 12278 | 88.6 (87.0, 90.0) |
| In a further limited dataset c,d | 17811 | 84.7 (83.7, 85.6) | 17047 | 94.0 (93.3, 94.6) |
Estimates repeated from Table 3 for comparison.
Analyses adjusted for calendar date (corresponding to test date in the test-negative design), age, sex, race, and geographic location.
As the test-negative design does not explicitly specify the period of follow-up, we considered estimates from day 0 as “receipt of the first dose of BNT162b2 between January 4 2021 and the test date” and estimates from day 28 as “receipt the first dose of BNT162b2 between January 4 2021 and 28 days before the test date.”
Analysis without covariate adjustment.
Under the test-negative design in the limited dataset, the estimated vaccine effectiveness (95% CI) for documented SARS-CoV-2 infection was 83.4% (82.2% to 84.5%). When allowing any vaccination before the test date, the estimated vaccine effectiveness was 69.3% (67.8% to 70.8%) (Table 5). For symptomatic Covid-19, the estimated vaccine effectiveness was 88.6% (87.0% to 90.0%). When allowing any vaccination before the test date, the estimated vaccine effectiveness was 69.8% (67.5% to 72.0%). These estimates were slightly higher than those obtained under test-negative designs in the full dataset previously considered. When the test-negative design was conducted in a further limited dataset (to omit matching on test dates and any covariate adjustment), the estimated vaccine effectiveness for documented SARS-CoV-2 infection and symptomatic Covid-19 was even higher (Table 5). These estimates were similar when additionally omitting the eligibility criterion of no previously documented SARS-CoV-2 infection (estimated vaccine effectiveness 87.0% [86.4% to 87.5%] for documented SARS-CoV-2 infection, 92.7% [91.9%, 93.3%] for symptomatic Covid-19).
Discussion
We provided a framework to compare a cohort design (that explicitly emulates a target trial) and test-negative design (that does not) to estimate vaccine effectiveness. When we implemented both designs using healthcare data with rich information on eligibility criteria and confounders, the estimated effectiveness of the BNT162b2 vaccine against Covid-19 outcomes was similar.
In the target trial, the strategy of interest is a joint intervention on vaccination and testing strategies. Therefore, bias in its emulation can arise because of differences in testing patterns between the vaccinated and unvaccinated groups, or residual confounding for either vaccination or testing. In our cohort analysis, however, both groups had similar testing patterns, comparable distributions of measured risk factors, and nearly identical risk patterns for the negative control outcome. The estimated vaccine effectiveness after day 28 was similar to, but slightly lower than, estimates from the randomized trial, which may be partially explained by the inclusion of a more vulnerable population (with respect to age and burden of comorbidities) and/or potentially undocumented infections at baseline or over follow-up (which may contribute to differences in disease susceptibility between the groups, e.g., by depleting the unvaccinated group of susceptible members).30,31
The limited residual confounding in our study, especially by health care-seeking behavior, may be due to the combination of our stringent eligibility criteria (e.g., recent users of VA health care system), close matching on markers of health care utilization (e.g., number of influenza vaccinations over the previous five years, number of PCR tests previously received), full health care coverage, and the integrated nature of the VA health care system. Additionally, during the study period (earlier in the pandemic), we expect that most people were on high alert and would seek health care and get tested due to symptoms or by requirement, as home testing was not as widely available, which may also help to explain a limited role of unmeasured confounding by health care-seeking behavior in this particular context. As a result, minimal selection bias was introduced when restricting to person–days with a test, and the estimated vaccine effectiveness was similar across all study designs considered. (In fact, a previous study showed that it may be possible to obtain unbiased estimates under a test-negative design, if all confounders are measured.18) However, the observed consistency in estimates across designs may not hold for a different population, data source, or time-period (i.e., later in the pandemic).
In the test-negative design, the effectiveness estimates do not correspond to a well-defined follow-up period and thus are not directly comparable with those from the randomized trials or the target trial emulation. Specifically, when estimating effectiveness from 28 days post-vaccination (for comparison with prior randomized trials), the target trial emulation excluded events in both groups during the first 28 days, but the test-negative design excluded events during the same period only in the vaccinated group. Yet the effectiveness estimates in the test-negative design were similar to those obtained under other designs, implying that the test-negative design itself did not introduce much additional bias in this particular context. This similarity in estimates could be attributed to the large effect of BNT162b2 after day 28, which may be more robust to design-induced biases. Accordingly, the test-negative designs in limited datasets (with only a subset of variables available for analysis) appeared to provide poorer adjustment in analyses beginning from day 0 (including the early period of little-to-no effectiveness) than in analyses beginning from day 28, as indicated by an estimated vaccine effectiveness further from that of other designs (in the full data) in analyses from day 0.
In the analyses that used limited data, bias is likely because these analyses do not adjust for individual characteristics (e.g., disease susceptibility) that predict both vaccination and infection or testing, and may include individuals ineligible to receive the vaccine (e.g., individuals who receive care at stations ineligible to administer the BNT162b2 vaccine, or non-users of the VA healthcare system). In our analyses in limited datasets, the estimated vaccine effectiveness was biased downward in the cohort design and biased upward in the test-negative design (even more so under the test-negative design in a further limited dataset that directly compared tests without any adjustment). However, the direction of bias in other applications is difficult to predict and may differ for the cohort design and test-negative design. See eMethods 4 for further discussion.
In summary, estimates of BNT162b2 vaccine effectiveness under both a cohort design with explicit target trial emulation and a test-negative design were similar, albeit slightly lower than those obtained from a previous randomized trial, when rich information from the VA healthcare system was used. However, when only a limited dataset was used, estimates from the cohort and test-negative designs diverged in opposite directions. In settings like ours with sufficient information on confounders and other key variables, a cohort design with explicit emulation of target trial may be preferable because it (1) enables estimation of absolute risks, (2) prevents design-induced bias due to misaligned timing of assessment for eligibility criteria, vaccination assignment, and covariates, and (3) allows for empirical assessment of potential residual confounding.
Supplementary Material
Acknowledgments:
We thank Dr. Roger W. Logan for advice on the analysis; Constance A. Hoag for management of the administrative and regulatory aspects of the project; Drs. Rachel Ramoni and Grant Huang for VA ORD CSP #2032 leadership; the VA Covid-19 Shared Data Resource team and Million Veteran Program (MVP) Data Core (MVP#000) for their contributions and support; the VA Health Services Research and Development Service, VA Information Resource Center (Project #SDR 02-237 and #98-004) for support for VA/CMS data; and the VA health care providers, employees, and volunteers for their dedication to caring for our veterans through this pandemic.
Funding:
This research was supported by the U.S. Department of Veterans Affairs (VA), Office of Research and Development (ORD), Cooperative Studies Program (CSP), CSP #2032, by resources and the use of facilities at the CSP Epidemiology Center at the VA Boston Healthcare System and VA Informatics and Computing Infrastructure (VINCI) (VA HSR RES 13-457), and by the use of data from the VA Covid-19 Shared Data Resource. G.L., A.L.M., J.M.R., M.A.H., and B.A.D. were supported by CSP #2032. B.A.D. was supported by a grant (R00 CA248335) from the National Institutes of Health. M.J.F.M. was supported by a grant (T32 GM140972) from the National Institute of General Medical Sciences Interdisciplinary Training Program for Biostatisticians. M.L. was supported by the Morris-Singer Fund and a grant (R01 GM139926) from National Institute of General Medical Sciences.
Footnotes
Conflicts of interest: None declared.
Code availability: Access to the computer code used in this research is available by request to the corresponding author.
Disclaimers: The contents of this article do not represent the views of the U.S. Department of Veterans Affairs or the U.S. Government. The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Department of Health and Human Services and its agencies, including the Biomedical Advanced Research and Development Authority and the Food and Drug Administration, as well as any other agency of the U.S. Government. Assumptions made within and interpretations from the analysis do not necessarily reflect the position of any U.S. Government entity.
Data availability:
The data that support the findings of this study are available from the VA. VA data are made freely available to researchers (behind the VA firewall) with an approved VA study protocol. More information is available at https://www.virec.research.va.gov or by contacting the VA Information Research Center (VIReC) at VIReC@va.gov.
References
- 1.Hernán MA, Robins JM. Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available. American Journal of Epidemiology 2016;183:758–64. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Dickerman BA, García-Albéniz X, Logan RW, Denaxas S, Hernán MA. Avoidable flaws in observational analyses: an application to statins and cancer. Nat Med 2019;25:1601–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Skowronski DM, Masaro C, Kwindt TL, et al. Estimating vaccine effectiveness against laboratory-confirmed influenza using a sentinel physician network: results from the 2005–2006 season of dual A and B vaccine mismatch in Canada. Vaccine 2007;25:2842–51. [DOI] [PubMed] [Google Scholar]
- 4.De Serres G, Skowronski DM, Wu XW, Ambrose CS. The test-negative design: validity, accuracy and precision of vaccine efficacy estimates compared to the gold standard of randomised placebo-controlled clinical trials. Euro Surveill 2013;18. [DOI] [PubMed] [Google Scholar]
- 5.Sullivan SG, Tchetgen Tchetgen EJ, Cowling BJ. Theoretical Basis of the Test-Negative Study Design for Assessment of Influenza Vaccine Effectiveness. Am J Epidemiol 2016;184:345–53. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Jackson ML, Nelson JC. The test-negative design for estimating influenza vaccine effectiveness. Vaccine 2013;31:2165–8. [DOI] [PubMed] [Google Scholar]
- 7.Lipsitch M, Jha A, Simonsen L. Observational studies and the difficult quest for causality: lessons from vaccine effectiveness and impact studies. Int J Epidemiol 2016;45:2060–74. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Kaplan EL, Meier P. Nonparametric estimation from incomplete observations. J Am Stat Assoc 1958;53:457–81. [Google Scholar]
- 9.Polack FP, Thomas SJ, Kitchin N, et al. Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine. N Engl J Med 2020;383:2603–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Dagan N, Barda N, Kepten E, et al. Supplementary Methods 4: BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting. N Engl J Med 2021;384:1412–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Dickerman BA, Gerlovin H, Madenci AL, et al. Comparative Effectiveness of BNT162b2 and mRNA-1273 Vaccines in U.S. Veterans. New England Journal of Medicine 2021;386:105–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Harder VS, Stuart EA, Anthony JC. Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychol Methods 2010;15:234–49. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Lipsitch M, Tchetgen Tchetgen E, Cohen T. Negative controls: a tool for detecting confounding and bias in observational studies. Epidemiology 2010;21:383–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Chapman A, Peterson K, Turano A, Box T, Wallace K, Jones M. A Natural Language Processing System for National COVID-19 Surveillance in the US Department of Veterans Affairs. Proceedings of the 1st Workshop on NLP for COVID-19 at Association for Computational Linguistics 2020. https://www.aclweb.org/anthology/2020.nlpcovid19-acl.10
- 15.Thomas SJ, Moreira ED Jr., Kitchin N, et al. Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine through 6 Months. N Engl J Med 2021;385:1761–73. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Miettinen O. Estimability and estimation in case-referent studies. Am J Epidemiol 1976;103:226–35. [DOI] [PubMed] [Google Scholar]
- 17.Li KQ, Shi X, Miao W, Tchetgen ET. Double Negative Control Inference in Test-Negative Design Studies of Vaccine Effectiveness. ArXiv 2022. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Schnitzer ME. Estimands and Estimation of COVID-19 Vaccine Effectiveness Under the Test-Negative Design: Connections to Causal Inference. Epidemiology 2022;33:325–33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Westreich D, Hudgens MG. Invited Commentary: Beware the Test-Negative Design. Am J Epidemiol 2016;184:354–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Bruxvoort KJ, Sy LS, Qian L, et al. Effectiveness of mRNA-1273 against delta, mu, and other emerging variants of SARS-CoV-2: test negative case–control study. BMJ (Clinical research ed) 2021;375:e068848-e. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Pilishvili T, Gierke R, Fleming-Dutra KE, et al. Effectiveness of mRNA Covid-19 Vaccine among U.S. Health Care Personnel. N Engl J Med 2021;385:e90. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Thiruvengadam R, Awasthi A, Medigeshi G, et al. Effectiveness of ChAdOx1 nCoV-19 vaccine against SARS-CoV-2 infection during the delta (B.1.617.2) variant surge in India: a test-negative, case–control study and a mechanistic study of post-vaccination immune responses. Lancet Infect Dis 2022;22:473–82. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Young-Xu Y, Korves C, Roberts J, et al. Coverage and Estimated Effectiveness of mRNA COVID-19 Vaccines Among US Veterans. JAMA Netw Open 2021;4:e2128391. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Greenland S Quantifying biases in causal models: classical confounding vs collider-stratification bias. Epidemiology 2003;14:300–6. [PubMed] [Google Scholar]
- 25.Hernán MA, Hernández-Díaz S, Robins JM. A structural approach to selection bias. Epidemiology 2004:615–25. [DOI] [PubMed] [Google Scholar]
- 26.Tang P, Hasan MR, Chemaitelly H, et al. BNT162b2 and mRNA-1273 COVID-19 vaccine effectiveness against the SARS-CoV-2 Delta variant in Qatar. Nat Med 2021;27:2136–43. [DOI] [PubMed] [Google Scholar]
- 27.Skowronski DM, Setayeshgar S, Zou M, et al. Single-dose mRNA Vaccine Effectiveness Against Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2), Including Alpha and Gamma Variants: A Test-negative Design in Adults 70 Years and Older in British Columbia, Canada. Clin Infect Dis 2022;74:1158–65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Carazo S, Talbot D, Boulianne N, et al. Single-Dose Messenger RNA Vaccine Effectiveness Against Severe Acute Respiratory Syndrome Coronavirus 2 in Healthcare Workers Extending 16 Weeks Postvaccination: A Test-Negative Design From Québec, Canada. Clin Infect Dis 2022;75:e805–e13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Chemaitelly H, Yassine HM, Benslimane FM, et al. mRNA-1273 COVID-19 vaccine effectiveness against the B.1.1.7 and B.1.351 variants and severe COVID-19 disease in Qatar. Nat Med 2021;27:1614–21. [DOI] [PubMed] [Google Scholar]
- 30.Kahn R, Schrag SJ, Verani JR, Lipsitch M. Identifying and Alleviating Bias Due to Differential Depletion of Susceptible People in Postmarketing Evaluations of COVID-19 Vaccines. Am J Epidemiol 2022;191:800–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Lipsitch M, Goldstein E, Ray GT, Fireman B. Depletion-of-susceptibles bias in influenza vaccine waning studies: how to ensure robust results. Epidemiol Infect 2019;147:e306. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The data that support the findings of this study are available from the VA. VA data are made freely available to researchers (behind the VA firewall) with an approved VA study protocol. More information is available at https://www.virec.research.va.gov or by contacting the VA Information Research Center (VIReC) at VIReC@va.gov.