An analytic approach considering two temporal mechanisms driving breakthrough viral infections after vaccination

Abstract

Real world data is an increasingly utilized resource for post-market monitoring of vaccines and provides insight into real world effectiveness. However, heterogeneous mechanisms may drive observed breakthrough infections among vaccinated individuals, such as waning vaccine-induced immunity or the emergence of a new strain against which the vaccine has reduced protection. Analyses of breakthrough infection incidence rates are typically predicated on a presumed temporal mechanism in their choice of an “analytic time zero” after which infection rates are modeled. In this work, we propose a test that utilizes a standard Cox proportional hazards framework to investigate two temporal mechanisms that can drive breakthrough infections of viral pathogens: waning immunity and the emergence of new strain. We explore the test’s performance in simulation studies and in an illustrative application to real world data. We additionally introduce subgroup differences in infection incidence and evaluate the impact of time zero misspecification on bias and coverage of model estimates. In this study we observe strong power and controlled type I error of the test to detect true waning immunity effects under various settings. Similar to previous studies, we find mitigated bias and greater coverage of estimates when the analytic time zero is correctly specified or accounted for.

1. Introduction

Vaccine effectiveness studies increasingly leverage real world data (RWD) to assess how a vaccine performs in real world settings. Analyzing vaccine effectiveness using RWD has offered insight into differential vaccine-induced immunity across diverse populations and subgroups as well as under varying schedules or series completion [1,2]. Moreover, RWD consists of longitudinal data captured over a longer time frame than clinical trials data, which may reveal rarer outcomes or interactions with time-varying covariates [3]. However, there are challenges in studying vaccine effectiveness using RWD. The precise timing of vaccine receipt and current vaccine status of a given individual may be incomplete in electronic health record (EHR) or administrative claims databases, which only reflect the individual’s healthcare encounters through particular clinics or insurers, introducing the risk for misclassification and potential bias [4,5]. Data elements such as medications, laboratory measurements, vital signs, and comorbidities may be documented at a more regular cadence among specific patient subgroups (e.g., sicker patients, patients with more comprehensive insurance coverage) so that their presence is not only informative but may act as a collider or confounder in the association of interest [6]. Furthermore, as in all observational studies, unobserved or observed patient characteristics may confound the association between vaccine and outcomes. Many prior vaccine effectiveness studies that utilized RWD have addressed the complexities above by focusing on a vaccinated-only population [7], utilizing data sources with consistent capture of the characteristics of interest [8], and employing standard regression methods to adjust for potential confounders [9].

In effectiveness studies of vaccinated-only populations, the focus is often on characterizing the incidence and relative incidence of breakthrough infections as in Goldberg et al. [1] and Coburn et al. [10]. In such studies, it is implicit that infections may be driven by different mechanisms, e.g., naturally declining immunity following vaccination or a dominant new strain. This uncertainty is accounted for by focusing on a narrow time window to hold calendar-time-varying mechanisms constant or by constructing patient cohorts based on the timing of vaccination. However, little work has been done to develop methods that explicitly test for the mechanism driving breakthrough infections or explore the performance of standard approaches under a variety of infection-generating signals.

In this work, we develop an analytic approach to assess and account for the breakthrough mechanism in effectiveness studies of vaccines targeting viral pathogens, focusing on two temporally driven mechanisms of vaccine failure: waning immunity and the emergence of a new viral strain. We start by showing how the assumed breakthrough mechanism affects the analytic design. We then propose a novel analytic framework for assessing the breakthrough mechanism. We leverage a standard time-to-event framework and introduce a vaccination offset variable to accommodate possible misspecification of time-zero. This framework allows for inference on a potential driving mechanism behind breakthrough events and estimation of subgroup differences in event rates. Our work is motivated by efforts to assess the effectiveness of COVID-19 vaccine during different waves of the pandemic, but is appropriate for any pathogen that exhibits cyclical variation in the presence of vaccinated populations (e.g., influenza virus). We evaluate the framework in a simulation study under a variety of settings and illustrate use to evaluate COVID-19 vaccine waning among patients from our institution’s health system.

2. Material and methods

2.1. Motivation

We consider a vaccinated population in which we observe breakthrough infections over time and are interested in assessing two questions: 1) which temporally driven mechanism is causing the breakthrough infections; and 2) accounting for the mechanism, are there differential rates of breakthrough infections in population subgroups? In addressing these questions, we posit that breakthrough infections can arise via one of two temporally driven mechanisms: 1) the vaccine effectiveness has waned due to naturally declining immunity following vaccination or 2) the vaccine is no longer effective against the strain of virus currently in circulation.

As we develop below, a natural framework to address these questions is a time-to-event framework. A time-to-event analysis requires the specification of a time origin or time zero, which is traditionally chosen to represent the time at which an individual is first at risk for the outcome. In studies of breakthrough infections in a vaccinated population, the date of vaccination is a common choice for time zero since individuals become at risk for the event no earlier than this date. However, if the hypothesized driving mechanism for an observed time period of breakthrough infections is a new viral strain, a time-to-event analysis should be indexed with respect to a calendar date corresponding to the emergence of the strain. As illustrated in Fig. 1, incorrectly defining time zero will lead to inaccurate estimates of incidence and relative incidence.

Fig. 1. Kaplan-Meier curves for breakthrough infections under two choices of time zero.

We show Kaplan-Meier survival estimates for 2 subgroups in a simulated dataset. The driving mechanism is a new strain. We obtain survival curves estimates specifying time zero as individuals’ vaccination dates or a calendar date corresponding to the emergence of the new strain.

As such, one of the analytic challenges in this work is to properly index a time-to-event model to account for an unknown breakthrough mechanism. We consider a setting in which breakthrough events are observed following a landmark calendar date, and may be driven by a newly emerged strain or waning vaccine-induced immunity (i.e., driving mechanism and correct time zero are uncertain). In this setting, we propose a straightforward approach to make inference on the two questions above by defining an explicit test for waning immunity as a driving mechanism.

2.2. Analytic framework

Under the two possible mechanisms above, we consider a time-to-event based analysis. We assume patient-level data, including date of vaccination and associated characteristics (e.g., demographic, socioeconomic, baseline clinical) x_i = (x_i1, x_i2, …, x_ip). We take the characteristics to be time invariant but the framework can be modified to accommodate time-varying features as well. We designate a landmark date L after which we are interested in assessing the risk of breakthrough infection, and we specify an administrative censoring date after which we are no longer interested in assessing risk. We additionally define a vaccination period start date and identify the set of people who are vaccinated between this date and the landmark date, and who have remained event free up through time L. For these individuals we generate an event indicator C_i where C_i= 1 indicates an observed breakthrough infection and C_i= 0 indicates censoring. We define T_i > 0 as the time from the landmark date to the patient’s event or censor date. We further define the variable z_iΔ > 0 as the days between the patient’s vaccination and the landmark date, as illustrated in Fig. 2.

Fig. 2.

Illustration of proposed framework.

We perform Cox proportional hazards (PH) regression based on hazard function shown in Eq. 1 which corresponds to a standard Cox PH regression with x_i and z_iΔ as independent variables. In this framework, our parameter of interest is β_Δ, which reflects the association between vaccination recency and hazard of infection. We test the null hypothesis shown in Eq. 1: i.e., the hypothesis that there is no association between recency of vaccination and hazard of infection.

$$h\left(t|x_i,z_{i\Delta }\right)=h_0\left(t\right)\times \text{exp}\left({\beta }_{\Delta }{z}_{i\Delta }+{\beta }_1{x}_{i1}+{\beta }_2{x}_{i2}+\dots {\beta }_p{x}_{ip}\right)$$ Equation 1

$$H_0:{\beta }_{\Delta }=0\text{vs}{H}_A:{\beta }_{\Delta }\ne 0$$

The rejection of this null suggests that waning immunity is contributing to the infections observed following the landmark date L. We note that the failure to reject the null does not confirm that breakthrough infections are due to a new strain or not due to waning immunity. In the framework above, we can simultaneously obtain estimates of β₁, β₂, ⋯, β_p, which allows for standard inference on subgroup differences and continuous effects of covariates.

2.3. Simulation study

We conducted a simulation study to 1) assess the power and type one error rates of the hypothesis test in Eq. 1, providing inference on our first question of interest; and 2) examine the bias of subgroup differences estimates β₁, β₂, ⋯, β_p obtained from the analytic framework above, corresponding to our second question of interest.

2.3.1. Assessing underlying breakthrough mechanism

We simulated data under two proposed breakthrough mechanisms. The first mechanism, M_W, generated breakthrough events due to naturally declining immunity following vaccination, and corresponded to a desire to reject the null hypothesis β_Δ = 0. Under M_W, we assumed the following:

• Immediately following vaccination, the individual had the lowest daily risk of infection a;

• The daily risk was constant at its lowest level for duration d days following vaccination;

• The daily risk then grew linearly at rate r until the individual’s daily risk returned to pre-vaccination levels b.

The second mechanism, M_S, was the emergence of a new strain that heightened an individual’s risk of infection and uncoupled risk from the date of vaccination. When infections were generated under M_S, we expected to fail to reject the null hypothesis. Under M_S, we specified:

• The patient’s daily risk of infection was at a constant level k prior to the new strain

• At the emergence of the new strain, the risk jumped to a constant level c greater than the pre-strain risk.

Each of these mechanisms corresponded to a different hazard function represented in Fig. S1.

To mimic real world data, the simulations were anchored around an arbitrary window of calendar time. We generated uniformly random vaccination dates across a 365-day period and set the landmark date L as the end of the period. We applied an administrative censoring date 365 days following the landmark date L. Individuals without an infection during this 365-day period were censored.

2.3.2. Assessing subgroup differences

We additionally evaluated the estimation of subgroup differences under both mechanisms. Using the notation in Eq. 1, we simulated a single binary variable x₁ indicating subgroup membership with possible values 0 or 1. No other categorical or continuous variables were generated, i.e., x₂,…,x_p = NULL. We simulated data in which the hazard was consistent across the two subgroups (β₁ = 0) and data in with different hazards between the subgroups (β₁ ≠ 0). We applied an unadjusted Cox PH regression to the former setting, with z_Δ as the only independent variable, and an adjusted Cox PH regression to the latter setting with z_Δ and the subgroup indicator x₁ as independent variables. In data with subgroup differences, we randomly assigned 50 % of all patients to a subgroup (captured in a binary indicator x₁) and let this subgroup modify patients’ hazard functions through a proportional term exp(β₁x₁). Thus, β₁ translated to the log hazard ratio between the two subgroups. We assessed the bias and coverage of the model-estimated $\widehat{{\beta }_1}$ with respect to the true β₁. We additionally assessed these measures from two naïve Cox PH regressions with only x₁ as an independent variable, one designating time zero as the landmark date and one designating time zero as the vaccination date.

2.3.3. Simulation settings

We simulated B = 1000 analytic datasets under different parameter settings corresponding to varying event (hazard) rates, immunity waning rates, sample sizes, and presence of subgroup differences (Table 1). Across all datasets, an overly large sample size was simulated and individuals who had an event prior to the landmark date were left-truncated as would occur in a real data analysis. A random sample from the remaining individuals was taken obtain the desired sample size. In datasets with subgroup differences, we set β₁ = 0.15. We calculated the power of the hypothesis test as (# p-values ≤ α)/B when the null hypothesis should be rejected (i.e., data simulated under M_W). We calculated the type I error of the hypothesis test as (# p-values ≤ α)/B for when the null hypothesis should not be rejected (i.e., data simulated under M_S).

Table 1.

Simulation settings.

	No subgroup differences	With subgroup differences
Waning immunity (M w )
Lowest hazard	a = 0.0001	a = 0.0001
Highest hazard	b = 0.0007	b = 0.0007
Duration at lowest hazard	d = (90, 180, 240 days)	d = (90, 180, 240 days)
Rate of declining immunity (increase in hazard per day)	r = (1e-06, 5e-06, 1e-05, 5e-05, 1e-04)	r = (1e-08, 1e-07, 1e-06, 1e-05)
New strain (M S )
Flat hazard	c = (1e-04, 5e-04, 1e-03, 5e-03, 1e-02)	c = (1e-04, 5e-04, 1e-03, 5e-03, 1e-02)
Method evaluation at each combination of the above settings
Analytic dataset sample size	N = (500, 1000, 10000,100,000)	N = (500, 1000, 10000,100,000)
Significance level for power and type I error	α = (0.01, 0.05, 0.10)	α = (0.01, 0.05, 0.10)

2.4. Real world data illustration

To illustrate the analytic approach, we applied the proposed above framework to a real-world data set and evaluated breakthrough COVID-19 infections among vaccinated individuals in Duke University Health System (DUHS). DUHS is a medium sized health in central North Carolina. Of relevance from a population health standpoint, it manages the largest network of clinics and the only two hospitals in Durham, North Carolina.

We identified individuals who were vaccinated between 1/1/21 (the beginning of vaccine administration) and 7/1/21 (prior to the emergence of the SARS-CoV-2 Delta variant). Vaccine information was collected through linkage of DUHS medical records with the North Carolina Immunization Registry. The outcome of interest was a positive COVID-19 test result recorded in the EHR in any type of encounter. To increase the likelihood of observing their test results in the EHR, we limited the cohort to individuals living in Durham County and individuals with at least one encounter in our health system in the prior two years. We abstracted outcomes in the time window from 7/1/21–12/1/21, corresponding to a period when the Delta variant was dominant and prior to the emergence of the Omicron variant. We extracted demographics (age, sex, race-ethnicity) and chronic health conditions (chronic obstructive pulmonary disease [COPD], renal disease, liver disease, hypertension, diabetes, asthma, cardiovascular disease [CVD], cancer) for each individual in the cohort (see Table S2 for ICD-10-based phenotyping of health conditions). We tested for the mechanism driving breakthrough infections and estimated subgroup differences across patient demographics and chronic diseases. We report the results of the test for vaccination offset variable as hazard ratios (HR) and 95 % confidence intervals (95 % CI) for subgroup differences.

All analyses were performed in R version 4.0.2. The simulation code is available in the Supplement.

3. Results

3.1. Illustration of vaccination offset variable and breakthrough infections under different mechanisms

We first explored breakthrough infections generated under the two simulation mechanisms in a Kaplan-Meier survival analysis (Fig. 3). Infections were simulated under the condition of no subgroup heterogeneity and time-to-infection was defined with respect to the calendar landmark date as described in the Methods section. Patients were grouped into cohorts based on their z_Δ variable, in 30-day bins spanning patients vaccinated within 30 days prior to the landmark date to patients vaccinated >180 days prior. In panel A, infections driven by naturally declining immunity show a fanning effect in which patients with the smallest z_Δ (vaccinated most recently) have the lowest infection rate (light red curve) and the cohort of patients with the largest z_Δ (longest time since vaccination) have the highest infection rate (dark red curve). In panel B, the incidence of breakthrough infections driven by a new strain does not appear to vary across cohorts. Thus, we observe a gradient of survival curves as z_Δ varies when declining immunity is driving infections, and no gradient when a new strain resets the hazard of infection.

Fig. 3. Kaplan-Meier survival curves for simulated breakthrough infection data under two different mechanisms.

This figure illustrates survival curve estimates for cohorts of patients stratified by z_Δ, i.e., their recency of vaccination at the landmark date. The different colored curves correspond to cohorts of patients based on their recency of vaccination. A) The true mechanism driving these simulated breakthrough infections is naturally declining immunity following vaccination. B) The true mechanism driving these simulated breakthrough infections is a new strain emerging.

3.2. Simulation study results

3.2.1. Test for vaccination offset variable

The type I error rate of the proposed test is shown in Table S1. The Type I error was controlled at the nominal levels under all settings with no apparent trends across varying simulation parameters. Error was controlled under simulated data with and without subgroup heterogeneity. The power of the test is visualized in Fig. 4, under no subgroup heterogeneity and an alpha of 0.05. We observed that the test achieves perfect or near perfect power under all settings when the sample size is 10,000 or larger. Power was slightly greater under a shorter duration of maximum immunity. We observed similar trends in power curves corresponding to data with subgroup heterogeneity (Fig. S2).

Fig. 4. Power of proposed test.

Power is calculated based on data simulated under naturally declining immunity. Power is shown on the y-axis and the waning immunity rate varies along the x-axis. Power was evaluated at 5 values of waning immunity rate; 3 different durations of maximum immunity shown in red, green, and blue curves; and 4 different sample sizes (4 sub-panels). These curves correspond to power under alpha = 0.05 and no subgroup heterogeneity.

3.2.2. Estimation of subgroup differences

We explored the bias of subgroup estimates under a signal arising from a new strain, shown in Fig. S3. The proposed method produced nearly identical estimates to those from a naïve Cox PH regression, suggesting that the inclusion of the vaccination offset variable does not significantly impact the performance of a standard survival analysis when the time zero is correctly specified. We calculated the bias of subgroup estimates under waning immunity along with their coverage as shown in Fig. 5. Under the slowest immunity decay rate settings, the proposed method and calendar time Cox PH regression methods performed similarly. As immunity waning rates increased, the proposed method produced less biased estimates than the naïve approach. Across all settings, the vaccination time Cox PH regression produced biased estimates. The coverage under the proposed method and the vaccination time Cox PH regression consistently hovered around 95 %, while the coverage under the calendar time Cox PH regression decreased sharply when immunity decay rates were higher.

Fig. 5. Bias and coverage of subgroup difference estimates under waning immunity.

Panel A) Mean bias is shown on the y-axis and the waning immunity rate varies along the x-axis. B) The percent of estimates whose 95 % confidence interval covered the true value (coverage) is shown on the y-axis and the waning immunity rate varies along the x-axis. Both metrics were calculated at 5 values of waning immunity rate; 4 different sample sizes (4 sub-panels); and a duration of maximum immunity of 180 days.

3.3. Real world data illustration results

The real-world data analysis included 96,158 vaccinated individuals with 509 breakthrough infections between 7/1/21–12/1/21. We visualized the incidence of breakthrough infections in Fig. 6, stratified by the recency of vaccination or 30-day bins of z_Δ. We constructed 6 cohorts of patients who were vaccinated within 1-month increments of calendar time ranging from January 2021 (maroon) to June 2021 (pale red). The significant overlap of these survival curves along with some fanning behavior exhibits similarities with the survival curves observed under both of the simulation study mechanisms (Fig. 3).

Fig. 6. Kaplan-Meier curves of breakthrough infections in real world data.

The breakthrough infection rates are illustrated in 6 cohorts of patients constructed based on the recency of their vaccination with respect to the landmark date of 7/1/21. This is equivalent to stratifying by the vaccination offset variable z_Δ,

We applied the proposed method to test whether waning immunity may be contributing to infections observed after emergence of the Delta variant. As shown in Table 2, the test was marginally significant with a p-value = 0.040 and the estimated HR (95 % CI) was 1.003 (1.000–1.005), suggesting that breakthrough infections in 7/1/21–12/1/21 are driven partially by naturally declining immunity. We additionally performed a sensitivity test by removing patients in the earliest waves of vaccinations, keeping patients who were vaccinated within 90 days prior to 7/1/21 (z_Δ ≤ 90). In this population, the recency of vaccination was no longer informative.

Table 2.

Test for vaccination offset variable in real world data. These results are obtained from a fully adjusted model including the vaccination offset variable as well as patient demographics and comorbidities.

Population	# events / # patients	Vaccination offset variable Hazard Ratio* (95 % CI)	P-value
All patients	509 / 96,158	1.003 (1.000, 1.005)	0.040
Sensitivity: Recently vaccinated patients (zΔ ≤ 90)	149 / 28,646	0.998 (0.990, 1.005)	0.507

^*This corresponds to an estimate of exp(β_Δ) in the model given in Eq. 1.

In Fig. 7, we visualize the HRs and 95 % CIs for all covariates included in the model fit to the full population. As a comparison, we also applied a standard Cox PH regression to the data. We observe that the proposed method and Cox PH methods produce similar estimates and similar inference with the exception of Hispanic patients. In alignment with the behavior observed in the Kaplan-Meier analysis and test for vaccination offset variable, the subgroup difference estimates demonstrate qualities that are a mix of the subgroup estimation observed under both M_W and M_S in the simulation study. Some estimates obtained under the proposed method may have smaller bias that those arising from the naïve approach, although we do not know the true subgroup differences in this population. HRs, 95 % CIs and p-values are tabulated in Table S3.

Fig. 7. Adjusted hazard ratios obtained from proposed method and naïve approach.

Adjusted hazard ratios and confidence intervals are shown for the proposed method (blue) and naïve Cox PH regression (red).

4. Discussion

In this study, we propose an analytic framework to test and account for the potential influence of waning vaccine-induced immunity in breakthrough infections. In simulations and real world data, we considered two temporal mechanisms that could drive the loss of vaccine effectiveness: waning immunity and the emergence of a variant that is less sensitive to an existing vaccine. Setting up the analysis based on calendar time, we make inference on the mechanism driving infections through a test for a mis-specified time zero: with strong evidence of a waning-immunity effect, events should be analysed with respect to individuals’ vaccination dates, not calendar time. We employed a Cox PH regression framework set in calendar time with the addition of a vaccination offset variable to account for the potential mis-specification of time zero. Overall, the approach has good power and type I error control under a variety of settings. We also observed lower bias in subgroup difference estimates under the proposed method compared to two standard Cox PH approaches when the calendar time zero was mis-specified for the mechanism. When simulated infections were driven by a new strain, the vaccination offset variable was not informative and produced estimates equivalent to a correctly specified Cox PH regression.

In an exploratory application to COVID-19 breakthrough infections in RWD, we observed a slight gradient in Kaplan-Meier survival curves based on the recency of an individual’s vaccination, a mixture of what we observed under the two simulation study mechanisms. The test of the vaccination offset variable suggested that waning immunity following vaccination had a marginal but significant effect on breakthrough infections observed after the delta variant emerged. Given this, it was not surprising that the fully adjusted subgroup HRs were similar to those obtained under naïve Cox PH regression, behavior observed in the simulation study under a dominant new strain signal.

The ability to monitor vaccine effectiveness is an important part of public health efforts to combat infectious diseases at the population level [11]. While randomized controlled trials are the gold standard for evaluating the efficacy of a vaccine in preventing infections for a given pathogen, the individuals who participate in these trials do not represent the population at large in terms of age, health conditions, and other factors that strongly influence vaccine effectiveness. Clinical trials are also unable to address changes in transmission dynamics, pathogen evolution, and human behavior that have a significant impact on infection risk [12]. In contrast, post-market monitoring using RWD can identify sub-groups that may be particularly at risk, reflect vaccine effectiveness in real world conditions (e.g., imperfect adherence to schedule or series), and allow for longer follow-up to observe breakthrough infections. Thus, RWD-based monitoring of vaccine effectiveness is critical to determining when and how to modify vaccine dosing schedule and to identify the need for updated vaccine products.

A number of different mechanisms can drive decreases in vaccine effectiveness. As was recently illustrated by the COVID-19 pandemic, viral pathogens can evolve rapidly, and may develop the ability to escape vaccine-induced immune responses [13]. Notably, vaccine deployment can also drive pathogen evolution that allows immune escape. Additionally, there are differences in the duration of protection elicited by different vaccines and against different pathogens. We explicitly evaluated the two temporal mechanisms in the present study, as they present specific analytic challenges that would influence our ability to make valid inference in a time-to-event framework.

Of note, several other non-temporal mechanisms can also contribute to breakthrough infections that are not addressed in the current study. Primary vaccine failure [14] is estimated to occur in 2–10 % of the population, primarily among individuals who are immunosuppressed, including the elderly, in whom the immune system is evolving to a state of immunosenescence [15]. These individuals are expected to be protected by herd immunity, though this effect requires that a high proportion of the population be vaccinated. Vaccine failure would likely be identified through subgroup analyses of immune suppressed populations. In contrast, prior immune experience in immunocompetent individuals could contribute to vaccine effectiveness, as has been observed among individuals with hybrid immunity to SARS-CoV-2 [16]. Such hybrid immunity is unlikely to be well captured within RWD. Vaccine failure could also be attributed to a failure of the product itself, such as through cold chain failure, where direct damage to the vaccine product results in failure. Such failures are likely to be identified through geographic analyses, as damaged products would most likely be distinguished by a geographic region with an excess number of infections. Finally, different pathogens and vaccine products may exhibit different mechanisms and degrees of waning effectiveness over time. For example, viruses exhibit much more rapid evolution compared to bacteria due to the nature of their genomes [17,18]. Thus, the mechanisms evaluated in the current study are likely to be most appropriate for vaccines against pandemic viral pathogens that are being transmitted through a high proportion of the population.

The limitations of using RWD for vaccine effectiveness monitoring are also increasingly recognized. Common RWD sources like EHR, insurance claims, and administrative databases each have limited catchment and may imperfectly represent vaccine status and any breakthrough infections, leading to issues of misclassification and missing outcomes [4]. For example, such data may not be available in low-income settings or in settings with limited healthcare resources. In addition, some vaccinations may be uniquely under-captured in EHR and healthcare system databases if they are frequently administered in pharmacies (e.g., influenza) or in public health departments (e.g., prenatal Tetanus, Diphtheria, and Pertussis [TDAP]). The population represented in the data source might be biased to include individuals who are healthier and younger (e.g., commercial claims database), older (e. g., Medicare claims), or who have more interaction with a health system due to illness or greater engagement in managing their health. Community infection rates are also not captured in most RWD sources and individuals may be vaccinated at varying times, leading to complex interplay between time-varying risk of infection due to exposure and time-varying vaccine-conferred protection. Researchers have developed strategies to address these limitations including merging data from multiple sources, analyzing populations that are most likely to be well observed in the data, assessing bias in the construction of the analysis cohort and warning about the generalizability of findings as appropriate, and anchoring the analysis to calendar time to control for temporal trends.

This work highlights the impact of a mis-specified time zero in a time-to-event framework. We examined one possible direction of mis-specification, in which the analytic time zero was set to be a calendar date when infections are driven by waning vaccine-induced immunity. We observed that a naïve approach that ignores this mis-specification results in biased estimates that worsen as the vaccine-induced immunity decays at a faster rate. Adjusting for the incorrect time zero through the proposed vaccination offset variable resulted in less biased estimates under most simulation settings. Other work has explored the topic of time zero specification and its impact on the estimation of covariate or subgroup effects. In epidemiologic cohort studies, researchers found that incorrectly specifying time zero with respect to study enrollment date rather than participant age led to biased estimates. [19,20] In the case of a mis-specified time-scale, Thiébault et al. [21] explored a similar model to ours with the appropriate time-scale included as a covariate. They found a significant reduction in bias under this model compared to a naïve unadjusted approach, as we also observed in our simulation study. However, in a subsequent study, Pencina et al. [22] found that bias from a mis-specified time-scale could be nearly eliminated through appropriate adjustment.

This study has some limitations. We evaluated the model’s performance under relatively simple mechanisms and did not vary community infection rates or explicitly model viruses with different levels of transmissibility. Our analysis considered whether breakthroughs were due to a new strain or waning immunity, when in reality, it could be a mixture of the two. Analytically, we assumed a piecewise linear curve in the declining immunity following vaccination, when biological models might suggest an exponential decay. We also assumed a piecewise constant hazard under the introduction of a new strain, when in reality, the takeover of a new strain and the accompanying change in hazard may be more gradual. We did not account for patients’ vaccine series in the simulations or real world data analysis, simply considering their first vaccine. A limitation of the RWD illustration is that it relied on capture of positive tests in the EHR of one healthcare system, and as a result, the absolute infection rates are lower than expected.

Overall, this work has analytic implications for how we use RWD to monitor vaccine effectiveness. By manipulating the time zero specification, we show that we can uncover the source of waning immunities and avoid biased effect estimates. Future work should consider more complex scenarios, such as waning due to multiple mechanisms.

Abbreviations:

RWD

real world data

EHR

electronic health records

pH

proportional hazards

DUHS

Duke University health system

HR

hazard ratio

CI

confidence interval

Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.vaccine.2025.127638.

Material and/or code availability

Simulated datasets, data-generating code, and all code used to evaluate the method on simulated and real data are available upon request.

Data availability

Data will be made available on request.