State-of-the-art learning COVID-19 vaccine effectiveness using LSTM

Abstract

The effect of COVID-19 vaccines in reducing hospitalization risks was studied using the Long Short-Term Memory (LSTM) model. We first devised a dynamic environment using an LSTM that characterizes the impact of COVID-19 vaccine administrations on COVID-19 infections in the real-world setting from May 2021 to April 2023. Then, we generated hypothetical subjects with various vaccination profiles (e.g., all subjects received or not received the booster vaccine, or all subjects had followed the vaccine policy) and predicted their counterfactual outcomes based on the LSTM to make inferences on the vaccine effectiveness and estimate the population-averaged risk of infection if there was full compliance for the vaccine policy. Our findings confirm that booster doses significantly reduced the risk of COVID-19 hospitalization while bivalent booster had similar or slightly better effectiveness than the monovalent booster. Additionally, our analysis highlights the importance of adhering to vaccine policies in effectively reducing the risk of hospitalizations. Our study contributes to understanding the dynamics of vaccine efficacy and supports informed decision-making in public health strategies.

1. Introduction

The COVID-19 pandemic, caused by the SARS-CoV-2 virus, has been a global health crisis since its emergence in 2019. Vaccination has been the most effective tool against the pandemic. Initial vaccines, primarily monovalent, were rapidly developed, focusing on the original viral strain. Prior research has demonstrated the efficacy of COVID-19 vaccines in reducing general COVID-19 infection rates and COVID-19 hospitalization. However, with the advent of new variants, these vaccines’ effectiveness has been challenged. The role of booster doses and bivalent vaccines has gained prominence in maintaining vaccine effectiveness, particularly against new variants.

Two statistical models are commonly used to evaluate vaccine efficacy, the Cox proportional hazards regression [1–3] and test-negative case-control method [4–7]. The Cox model includes a time-varying vaccination variable and estimates the vaccine effectiveness by the hazard ratio comparing the hazard of COVID-19 infection among vaccinated subjects versus the unvaccinated subjects (vaccinated patients were included in the unvaccinated group before they received the vaccine), adjusting for important confounders. The test-negative case-control method recruits subjects who had a COVID-19 test (real-time reverse transcription–polymerase chain reaction or antigen). Subjects with positive test results are identified as cases, while those who were tested as negative are identified as controls. Vaccine effectiveness is estimated from the odds ratio comparing the odds of vaccination among the cases versus controls, adjusting for important confounders. This method only includes subjects whose outcome data (general COVID-19 infection or COVID-19 hospitalization) is certain and mitigates the unmeasured confounding bias like differential healthcare-seeking behavior between vaccinated and unvaccinated subjects.

A limitation of the Cox model is its dependence on the proportional hazards assumption, which may not be true in the context of the dynamically changing vaccine status and infection status over time. To estimate the vaccine waning effect over time, existing methods include time-from-vaccination (splines or categories) in the Cox or logistic regression. However, these approaches primarily focus on one dose (e.g., a second dose or a booster dose) and they cannot be easily extended to study multiple doses. Moreover, existing approaches cannot capture the complex vaccination profiles in the real-world setting, such as varying numbers and timing of the doses. Fig. 1 shows all possible vaccination profiles in our data. For example, some subjects had four vaccine doses (some received monovalent boosters while others received bivalent boosters); some subjects just received the first two doses without getting any booster. Modeling different vaccination profiles with varying vaccination time is challenging using Cox or logistic regression. Moreover, these models face difficulties in handling a large number of variables, including the time-fixed patient demographic and clinical variables and time-varying variables, such as vaccination and infection status.

Fig. 1.

This plot shows the sample sizes for various combinations of COVID-19 vaccine doses in the study population. The height of the vertical bars at the top represent the number of individuals for a particular combination of doses. The height of the horizontal bars on the right represents the number of individuals for a particular dose, including monovalent (Dose 1–Dose 4) and bivalent vaccines.

To address these limitations, a more advanced analytical method is needed to handle complex vaccination profiles, non-linear effects of variables, and interactions between vaccination and infection patterns. Recent developments in deep learning-based methods, such as Recurrent Marginal Structural Networks (RMSN) [8], Counterfactual Recurrent Networks (CRN) [9] and G-Net [10], have been use to estimate the casual treatment effect. RMSN extends the Marginal Structural Model (MSM) framework by using recurrent neural network architectures, allowing for the adjustment of time-dependent confounders; CRN is a sequence-to-sequence model that employs domain adversarial training to construct balanced representations that can effectively make counterfactual predictions; G-Net is a sequential deep learning framework for G-computation, designed to estimate expected counterfactual outcomes and handle time-series data. However, these methods are specifically designed to estimate the effect of one treatment variable; in the vaccine efficacy study, it can be used to study the effect of one vaccine dose while controlling for complex time-varying confounding effects. Our goal is to develop a model that allows us to estimate the vaccine effectiveness under various scenarios while controlling for complex time-varying factors (such as time from prior infection, number of vaccine doses received, and waning effects of prior vaccines). For example, we want to evaluate the efficacy of booster doses and compare the effect of a bivalent booster versus a monovalent booster vaccine in reducing COVID-19 hospitalization. Furthermore, we want to estimate if the population infection rate can be significantly reduced if the population had followed the vaccination policy for the primary doses and two booster doses.

Our approach has two simple steps. Firstly, we train LSTM to learn the complex relationships between the outcome (general COVID-19 infection and COVID-19 hospitalization), vaccination variables, and other potential time-fixed and time-varying variables. This constructs an environment that reflects the real-world complexities of vaccine administration and its effects over time during this study period. Then we generate hypothetical data from this environment (i.e., the “trained” LSTM model) to predict the counterfactual outcomes under different vaccination scenarios (e.g. every subject in the population received the booster, or every subject in the population follows vaccination policy). To assess the vaccine effectiveness, we calculate the difference in the risk of COVID-19 hospitalization between vaccinated and unvaccinated data.

In our data, the rate of general COVID-19 infections is 28.1%, whereas COVID-19 hospitalizations is only 3.4%. To improve the prediction accuracy for the rare event of COVID-19 hospitalizations, we apply the focal loss function [11], a widely used method that ensures a balanced contribution of minority samples in deep learning models. Moreover, we used the Monte Carlo dropout approach, a method validated for offering reliable prediction errors [12–14], to estimate the uncertainty of vaccine effectiveness.

2. Our approach

2.1. Our data

We used electronic health record (EHR) data at the Corewell Health East (CHE, formerly known as Beaumont Health). There were a total of 160,523 individuals in the EHR system who were registered with a primary care physician (PCP), had at least one PCP visit within 18 months prior to January 1^st, 2021 and had at least one COVID-19 test during the study period (May 1, 2021 to April 28, 2023). We excluded 46,411 individuals who were under 18, had more than 5 doses of vaccines during the study period, or received a vaccine rather than BNT162b2 (Pfizer–BioNTech) or mRNA-1273 (Moderna). A total of 114,112 individuals were included in the final analysis. The study includes baseline variables: age (continuous), gender (Female, Male), race (White or Caucasian, Black or African American, Other), number of hospital visits one year before baseline date (0–4, 5–9, 10–19, 20–50, >50), and Charlson comorbidity (0, 1–2, 3–4, ≥5) [15,16], and time-varying variables, vaccine does (0, 1, 2, 3, 4) and vaccine types (monovalent or bivalent). The detailed dataset information, including data size and key demographic characteristics, are described in Table 3.

Table 3.

Descriptive statistics of study population by infection types from May 1, 2021 to April 28, 2023.

	No infection (N = 78165, 68.5%)	General COVID-19 infection (N = 32018, 28.1%)	COVID-19 Hospitalization (N = 3929, 3.4%)	Total (N = 114112)
Age
Mean (SD)	53.60 (19.70)	50.62 (18.69)	64.67 (17.02)	53.14 (19.50)
Median (Q1, Q3)	55.00 (36.00, 69.00)	50.00 (34.00, 65.00)	67.00 (54.00, 78.00)	54.00 (36.00, 69.00)
Min–Max	18.00–90.00	18.00–90.00	18.00–90.00	18.00–90.00
Gender
Female	50002 (64.0%)	21003 (65.6%)	2175 (55.4%)	73180 (64.1%)
Male	28163 (36.0%)	11015 (34.4%)	1754 (44.6%)	40932 (35.9%)
Race
White	52939 (67.7%)	21183 (66.0%)	2785 (70.9%)	76862 (67.4%)
Black	15033 (22.3%)	7303 (22.8%)	899 (22.9%)	25637 (22.5%)
Other	7791 (10.0%)	3577 (11.2%)	245 (6.2%)	11613 (10.2%)
Number of Visits
0–4	39701 (50.8%)	14943 (46.7%)	1480 (37.7%)	56122 (49.2%)
5–9	19437 (24.9%)	8083 (25.2%)	809 (20.6%)	28329 (24.8%)
10–19	10192 (13.0%)	5028 (15.7%)	404 (10.3%)	15624 (13.7%)
20–50	3124 (4.0%)	1592 (5.0%)	299 (7.6%)	5015 (4.4%)
> 50	5711 (7.3%)	1764 (5.5%)	235 (6.0%)	7710 (6.8%)
Comorbidity Index
0	49632 (63.5%)	20491 (64.0%)	1459 (37.1%)	71582 (62.7%)
1–2	18838 (24.1%)	7935 (24.8%)	1272 (32.4%)	28045 (24.6%)
3–4	6123 (7.8%)	2302 (7.2%)	680 (17.3%)	9105 (8.0%)
≥ 5	3572 (4.6%)	1232 (3.8%)	518 (13.2%)	5322 (4.7%)
Vaccine Statusa
Unvaccinated	25336 (32.4%)	12070 (37.7%)	1670 (42.5%)	39076 (34.2%)
Had at least one dose	52829 (67.6%)	19948 (62.3%)	2259 (57.5%)	75036 (65.8%)

^aVaccine status is time-varying; the vaccine status presented here is as of the end of the study period; subjects might had vaccine before or after infections.

The infection outcome has three categories: no infection (“0”), general COVID-19 infection (“1”), and COVID-19 hospitalization (“2”) based on COVID-19 testing. The general COVID-19 infection is defined as any confirmed positive COVID-19 test that did not result in hospitalization. The COVID-19 hospitalization is defined if a patient’s hospitalization is attributed to COVID-19, based on the definition in [17], the patient had a positive COVID-19 test result during the 14 days before through 72 h after the hospital admission and associated with the COVID-19 illness.

The median age of participants was 54 years (The interquartile range (IQR): 36, 69), with the majority being female (64.1%). Racial composition was predominantly White or Caucasian (67.4%), followed by Black or African American (22.5%) and other races (10.2%). The characteristics of individuals without infection (N = 78,165), general COVID-19 infection (N = 32,018), and COVID-19 hospitalization (N = 3,929) are shown in Table 3. Subjects with COVID-19 hospitalization are older, have a higher comorbidity index and visit the health system more often.

2.2. LSTM with dropouts

In this study, we used the Long Short-Term Memory (LSTM) [18] to learn the associations between vaccination, patient clinical and demographic variables and COVID-19 infections. LSTM is a type of recurrent neural network that captures long-term dependencies for time-series data, which effectively incorporates the time-varying variables, such as number of vaccines received and waning vaccine effects in the learning process.

The key to LSTMs is the cell state, ${\mathbf{\text{c}}}_{\mathbf{\text{t}}}$, the horizontal line running through the top of the diagram in Fig. 2. This cell state enables the LSTM to remember or forget information, such as vaccination history, over long periods. The LSTM’s ability to remove or add information is achieved by components called gates. As illustrated in Eq. (1), these gates—specifically, the input gate ${\mathbf{\text{i}}}_{\mathbf{\text{t}}}$, forget gate ${\mathbf{\text{f}}}_{\mathbf{\text{t}}}$, and output gate ${\mathbf{\text{o}}}_{\mathbf{\text{t}}}$ —decide which information to add, discard and output as the data sequence progresses. For example, the forget gate removes irrelevant past information, and the input gate decides what relevant information should be added as shown in Eq. (2). Additionally, the LSTM updates its memory through a candidate memory cell ${\mathbf{\text{c}}}_{\mathbf{\text{t}}}^{\mathbf{\prime }}$, which integrates new incoming data ${\mathbf{\text{x}}}_{\mathbf{\text{t}}}$ with the existing memory ${\mathbf{\text{h}}}_{\mathbf{\text{t}}-\mathbf{1}}$. This allows the model to evolve its understanding as more data comes in. Finally, the next hidden state, the output of the LSTM cell at the current time step, is determined by the regulated new cell state $tanh\left({\mathbf{\text{c}}}_{\mathbf{\text{t}}}\right)$ and output gate ${\mathbf{\text{o}}}_{\text{t}}$, as shown in Eqs. (2) and (3).

Fig. 2.

An LSTM Cell with Recurrent Dropout (dashed red) and standard feed-forward Dropout (dashed green). This is implemented using the TensorFlow LSTM layer, utilizing the “dropout” argument to drop for the linear transformation of the inputs, and “recurrent_dropout” for dropout within the recurrent connections. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

$$\left(\begin{array}{c}{\mathit{i}}_t\\ {\mathit{f}}_t\\ {\mathit{o}}_t\\ {\mathit{c}}_t^{\prime }\end{array}\right)=\left(\begin{array}{c}\sigma \left({\mathbf{\text{W}}}_i\left[{\mathbf{\text{x}}}_t,{\mathbf{\text{h}}}_{t-1}\right]+{\mathbf{\text{b}}}_i\right)\\ \sigma \left({\mathbf{\text{W}}}_f\left[{\mathbf{\text{x}}}_t,{\mathbf{\text{h}}}_{t-1}\right]+{\mathbf{\text{b}}}_f\right)\\ \sigma \left({\mathbf{\text{W}}}_o\left[{\mathbf{\text{x}}}_t,{\mathbf{\text{h}}}_{t-1}\right]+{\mathbf{\text{b}}}_o\right)\\ \text{tanh}\left({\mathbf{\text{W}}}_{c^{\prime }}\left[{\mathbf{\text{x}}}_t,{\mathbf{\text{h}}}_{t-1}\right]+{\mathbf{\text{b}}}_c\right)\end{array}\right)$$ (1)

$${\mathbf{\text{c}}}_t={\mathbf{\text{f}}}_t\odot {\mathbf{\text{c}}}_{t-1}+{\mathbf{\text{i}}}_t\odot {\mathbf{\text{c}}}_t^{\prime }$$ (2)

$${\mathbf{\text{h}}}_t={\mathbf{\text{o}}}_t\odot \text{tanh}\left({\mathbf{\text{c}}}_t\right)$$ (3)

2.3. Apply LSTM to our data

Fig. 3 shows the architecture of our LSTM, and an illustration of a single LSTM cell is shown in Fig. 2. We transformed the original data into time series data by dividing the study period into 103 weeks and creating indicators for vaccination and infection status in each week. Table 1 shows the hypothetical input data for a particular patient. The input $x_t$ at any $t$ includes time-fixed variables such as age, and gender and time-varying variables such as vaccine dose and vaccine type, and the outcome, $y$, taking 0, 1, or 2. For each row, the outcome refers to the infection status for the following week. By including the time-varying vaccine dose as an input variable, the LSTM takes into account the vaccination history and vaccine waning effect in the modeling. Moreover, the dynamically changing infection outcome data in LSTM incorporates the influence of natural immunity from past infections.

Fig. 3.

Summary of our LSTM architecture. Green arrows represent standard feed-forward dropout, and red arrows indicate recurrent dropout. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Table 1.

Data for a hypothetical patient: a 79-year-old White male who visited Corewell Health East 0–4 times in the past year and had a comorbidity index (Comorb) of 0. The time-varying vaccination status is in “Doses” column. This subject had the first vaccine dose in week 1, the second vaccine dose in week 4, the first booster dose in week 28, and the second bivalent booster dose in week 53. The infection outcome is in the last column: this subject had a general COVID-19 infection in week 2 and a COVID-19 hospitalization in week 28.

Week	Age	Gender	Race	Visits	Comorb	Doses	Bivalent	Outcome
0	79	Male	White	0–4	0	0	NoVax	0
1	79	Male	White	0–4	0	1	No	0
2	79	Male	White	0–4	0	1	No	1
3	79	Male	White	0–4	0	1	No	0
4	79	Male	White	0–4	0	2	No	0
5	79	Male	White	0–4	0	2	No	0
…	…	…	…	…	…	…	…	…
26	79	Male	White	0–4	0	2	No	0
27	79	Male	White	0–4	0	2	No	0
28	79	Male	White	0–4	0	3	No	2
…	…	…	…	…	…	…	…	…
53	79	Male	White	0–4	0	4	Yes	0
54	79	Male	White	0–4	0	4	Yes	0

2.4. Masking to handle varying variable length input

Patients are observed until they die or experience a COVID-19 hospitalization, leading to trajectories of varying lengths. To deal with the varying lengths, we padded short sequences with zeros to match the length of the longest sequence. To calculate the loss, we did not use the padded values such that the model training is only based on the observed data.

2.5. Focal loss to handle rare infection outcome

In this study, it was found that the rate of COVID-19 hospitalizations is rare (3.4%) during the study period; see Table 3. This rarity posed a challenge for the LSTM model in distinguishing the rare COVID-19 hospitalization cases from the more prevalent no-infection cases. To handle this problem, we adopted the focal loss, which assigns a greater weight to the COVID-19 hospitalization cases. As demonstrated in Eq. (4), $p_j$ is the predicted probability of the correct class $j$, and $\gamma$ controls the extent of emphasis on hard cases. When a case is easy to classify ($p_j$ is high), then ${\left(1-p_j\right)}^{\gamma }$ will approach 0, effectively down-weighting its contribution. Besides, it has been shown that it works better by incorporating ${\alpha }_j$ to balance contributions from different cases.

$$FocalLoss=-{\alpha }_j{\left(1-p_j\right)}^{\gamma }\text{log}\left(p_j\right)$$ (4)

2.6. Uncertainties for predictions

Dropout was originally designed to combat overfitting by randomly deactivating neurons at a certain probability [19]. Gal et al. [13] formulated a theoretical framework that interprets the use of dropout as a Bayesian approximation of Gaussian processes. This development has paved the way for the use of Monte Carlo Dropout to estimate the uncertainty of the model prediction. Traditionally, dropout is applied during training and deactivated during testing, where predictions are made through a single deterministic forward pass. In contrast, Monte Carlo Dropout activates dropout during the testing phase, performing multiple forward passes. The variation observed across these passes provides a measure for prediction uncertainty.

There are two kinds of dropout operators in LSTM: standard feed-forward dropout and recurrent dropout to accommodate information transformation across time steps. The standard feed-forward dropout, marked in green in Fig. 3, is utilized when passing input $x_t$ to the first layer LSTM cells, and hidden features to the second layers. It is also illustrated in Eq. (5) where $d$ is a dropout operation defined as $d(\mathbf{\text{x}})=\delta \odot \mathbf{\text{x}}$ where $\delta$ is a vector sampled from the Bernoulli distribution with probability $1-p$ ($p$ is the dropout rate). There are several ways to conduct recurrent dropout [20–22]. Semeniuta et al. [23] proposed to apply dropout to the cell update vector ${\mathbf{\text{c}}}_t^{\prime }$ as demonstrated in Eq. (6) and the red arrow in Fig. 2. This approach has been shown to have a consistent improvement even when combined with conventional feed-forward dropout and was employed in this paper.

$$\left(\begin{array}{c}{\mathit{i}}_t\\ {\mathit{f}}_t\\ {\mathit{o}}_t\\ {\mathit{c}}_t^{\prime }\end{array}\right)=\left(\begin{array}{c}\sigma \left({\mathbf{\text{W}}}_i\left[d\left({\mathbf{\text{x}}}_t\right),{\mathbf{\text{h}}}_{t-1}\right]+{\mathbf{\text{b}}}_i\right)\\ \sigma \left({\mathbf{\text{W}}}_f\left[d\left({\mathbf{\text{x}}}_t\right),{\mathbf{\text{h}}}_{t-1}\right]+{\mathbf{\text{b}}}_f\right)\\ \sigma \left({\mathbf{\text{W}}}_o\left[d\left({\mathbf{\text{x}}}_t\right),{\mathbf{\text{h}}}_{t-1}\right]+{\mathbf{\text{b}}}_o\right)\\ \text{tanh}\left({\mathbf{\text{W}}}_{c^{\prime }}\left[d\left({\mathbf{\text{x}}}_t\right),{\mathbf{\text{h}}}_{t-1}\right]+{\mathbf{\text{b}}}_c\right)\end{array}\right)$$ (5)

$${\mathbf{\text{c}}}_t={\mathbf{\text{f}}}_t\odot {\mathbf{\text{c}}}_{t-1}+{\mathbf{\text{i}}}_t\odot d\left({\mathbf{\text{c}}}_t^{\prime }\right)$$ (6)

2.7. Model training and hyperparameter selection

The model was trained on Tesla M60, utilizing an Adam optimizer with a gradient clipping value set at 0.5. The dataset is partitioned into training, and validation sets with a ratio of 8:2, respectively. The hyperparameters were selected by minimizing the Euclidean distance between the observed and predicted probabilities of COVID-19 hospitalization averaged over the study period, which were obtained from the validation data. See Table 2 for values of hyperparameters.

Table 2.

Model parameters.

Parameter	Value
Epoch	400
Learning Rate	0.001
Hidden Units	[64,64]
Dropout Rate	0.3
Recurrent dropout Rate	0.3
Batch Size	1024
Weight decay	0.01
$\gamma$	2
$\alpha$	[1,20,10]
kernel regularize (penalty)	L2 (0.001)
recurrent regularize (penalty)	L2 (0.001)

2.8. Estimating causal effect on vaccination

The above LTSM creates an environment characterizing the joint distribution of patient covariates, vaccination, and infection outcomes in the real world. Following the idea in g computation [24], we will use LSTM to estimate what would happen to this joint distribution if, hypothetically, we change the vaccination status in this environment. For example, we can generate hypothetical data to answer standard questions, such as the causal booster effect by calculating the marginal risk difference comparing two hypothetical worlds: one in which every subject received the booster dose and the other in which every subject did not receive the booster dose. More generally, we can estimate various “what if” questions in which vaccination data is imagined to be manipulated to values other than their naturally observed values. For example, what if every subject had followed the vaccine policy for the first three doses, and what if only 50% subjects had followed the vaccine policy, providing evidence on the effect of vaccination policy.

3. Analysis results

3.1. Evaluation of LSTM performance

To estimate the probability of infection, we first employed the argmax function to determine the infection outcome (0, 1, or 2) and then calculated the probabilities of each outcome at each week. We evaluated our LSTM performance by comparing the predicted probabilities of infection to the observed probabilities of infection. Fig. 4 shows that weekly predicted probabilities of COVID-19 hospitalization (left) and all infections (general COVID-19 infection and COVID-19 hospitalization) from the LSTM are very close to the observed probabilities from May 2021 to April 2023.

Fig. 4.

Performance of LSTM in predicting COVID-19 hospitalization (left) and all infections (general COVID-19 infection or COVID-19 hospitalization) (right) from May 2021 to April 2023. The blue line represents the weekly infection rates for the observed data. The red line represents the weekly predicted infection rate derived from the LSTM model. The LSTM model first estimated the probability of each outcome (no infection, general COVID-19 infection, or COVID-19 hospitalization) for each individual in each week based on the covariate profiles in the observed data and then the three outcomes were generated from the probabilities and averaged to obtain the predicted risk of COVID-19 infection and COVID-19 hospitalization. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 5 compares the averaged predicted and observed probabilities for different subgroups. The predicted–observed pairs of probabilities centering around the identity line show the accuracy of the LSTM model in predicting infection within these subgroups.

Fig. 5.

LSTM model’s predicted probabilities of COVID-19 hospitalization (left), all infections (general COVID-19 infection and COVID-19 hospitalization) (right) with the observed infection rates, with each colored dot indicating a subgroup based on comorbidity, gender, race, or number of visits. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

3.2. Effectiveness of booster vaccine

Fig. 6 on the left shows the population-averaged cumulative risk of COVID-19 hospitalization under two hypothetical datasets: (1) all of the subjects in the population received a booster dose in November 2021, and (2) none of the subjects in the population received a booster dose. In both datasets, the vaccination history of each subject remained the same up to November 2021 to control for the effect of vaccination history on the booster dose. As shown in Fig. 6 on the left, the group with the booster had dramatically lower cumulative risks of COVID-19 hospitalization than the group without the booster dose, suggesting the effectiveness of the booster vaccine in reducing the risk of COVID-19 hospitalization. Fig. 6 on the right quantifies the vaccine effectiveness (VE) by VE = (1-Relative Risk)×100, where Relative Risk is calculated by the ratio of the cumulative risk of COVID-19 hospitalization in the booster group to that in the no booster group at each week.

Fig. 6.

The left figure shows the cumulative risk of COVID-19 hospitalization over time for the group with booster (blue) and without booster dose (orange), along with 95% confidence bands; the right figure shows the vaccine effectiveness of the booster dose, along with a 95% confidence band. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

This figure shows that the booster vaccine is very effective against COVID-19 hospitalization during the study period, starting with a moderate decline for the first 3 months followed by a steady gradual decrease over time. Even after one year, it still had a VE of 70%.

3.3. Effectiveness of bivalent over monovalent vaccine

To compare the effectiveness of the bivalent booster vaccine against the monovalent booster vaccine, we generated two hypothetical datasets: one with all subjects receiving a bivalent vaccine and the other with all subjects receiving a monovalent vaccine. We simulated the vaccine data of each subject from September 2022 as the bivalent booster was approved on August 31, 2022. Similar to the previous analysis, the vaccination history of each subject remained the same up to September 2022 to control for the confounding effect of vaccination history. To assess the VE, we calculated the relative risk by comparing the risk of COVID-19 hospitalization in each dataset to the risk of a dataset in which all subjects did not receive any booster doses from November 2021 to April 2023 As shown in Fig. 7, the bivalent vaccine had similar, or slightly higher VE than the monovalent vaccine, but it did not reach statistical significance as evidenced by the overlapping of the two 95% confidence bands.

Fig. 7.

Comparison of vaccine effectiveness between monovalent (blue) and bivalent (orange) vaccines from September 2022 to April 2023. The shaded area represents the 95% confidence bands. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

3.4. Risk of COVID-19 hospitalization under full or partial compliance of vaccine policy

To determine the counterfactual outcome of COVID-19 hospitalization if everyone in the population had followed the vaccination policy, we generated a hypothetical dataset where everyone in the population followed the recommended vaccination policy. We completed their vaccine profile of four doses. For subjects with an incomplete vaccine profile, they were supplemented with the missing vaccination doses until they had two primary and two boosters. The initial dose was assigned a random date between January 1 and June 1, 2021. Subsequent doses were scheduled randomly within specified intervals following CDC guidelines 21 to 28 days for the second dose, and 180 to 240 days for each booster dose after the preceding dose; if the second booster dose was added after September 1, 2022, it was considered as bivalent dose; otherwise, it was considered as monovalent dose. Subjects who received four doses (N = 16752 (14.7%)) to begin with retained their original vaccination profiles. Additionally, we simulated a partial policy compliance scenario where 50% of individuals with less than four doses completed their vaccination course. Our trained LSTM model predicted weekly population-averaged probabilities of COVID-19 hospitalization for both scenarios. The full compliance scenario showed significantly lower hospitalization risks compared to observed data, as shown in Fig. 8, while the partial compliance scenario also had a lower risk, but it was slightly higher than what was observed in the full compliance scenario. This model allows us to estimate the varying risks of COVID-19 hospitalization across different levels of vaccine policy compliance.

Fig. 8.

Population-averaged risks of COVID-19 hospitalization: comparing the observed data (blue line) with hypothetical datasets under full vaccine policy compliance (red line) and partial policy compliance (50% of individuals with less than four doses completed their vaccination course, orange line). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

3.5. Comparison with existing method

We compared the Counterfactual Recurrent Network (CRN) model with our LSTM approach. We chose CRN because it was recently developed and has an easy-to-use Python code for implementation. The CRN is a recurrent neuron network with an encoder–decoder architecture to handle time series data and domain adversarial training to remove confounding effects. To estimate the booster vaccine effect, we fit the data before November 2021 into the encoder to get a balancing representation of patients’ histories and then fit the data after November to train the decoder for estimating the counterfactual outcomes. As shown in Fig. 9, the estimated booster’s effectiveness is similar to our result, but our approach provided confidence intervals for the estimation. Compared to our approach, CRN needs to be separately trained to address each different causal question. For example, to compare bivalent to monovalent vaccines, we need to train another CRN with the encoder taking data before September 2022 when the bivalent was approved, and use the adversarial training to balance covariates between the bivalent and monovalent vaccine groups. In contrast, our approach only needs to train the LSTM once to address different causal questions. Furthermore, CRN cannot address the question of compliance with vaccine policy. Lastly, CRN takes much more computation time than our approach for a single task, and it took 10.47 h of training in the study of booster vaccine compared to 1.07 h using our approach on the same Tesla M60 GPU.

Fig. 9.

Vaccine effectiveness of the booster dose estimated from the CRN model.

4. Conclusion and discussion

We devised an LSTM model to learn the relationship between COVID-19 vaccine administrations and COVID-19 infections in the real-world setting, while effectively addressing the difficulty of predicting rare events of COVID-19 hospitalization. Then, we predicted the counterfactual outcomes to address standard causal questions, such as estimating the averaged vaccine effects. We have shown significant effects of booster vaccinations in lowering the risk of COVID-19 hospitalization and showed that the bivalent booster has similar, or slightly better, effectiveness than the second monovalent booster. We also found that full compliance with the vaccine policy significantly decreased the population risk of COVID-19 hospitalization.

Our study has several strengths compared to prior studies. Traditional statistical models, such as the Cox proportional hazards regression and test-negative case-control methods, have limitations in handling dynamically changing vaccination and infection status over time. The LSTM model overcomes these limitations by capturing long-term dependencies for the time series data, providing a robust framework for predicting infection risks under various vaccination scenarios. Moreover, our method is simple and efficient, requiring only a single training phase to address various causal questions. For example, one important causal question we can answer is what the population infection rate would be if all individuals or half of the individuals in the population had followed the recommended vaccination policy.

Our study also has several limitations. First, our study is using EHR data from a single institution and findings may not be generalized to other populations. Second, our model assumes accurate vaccination and infection data in the electronic health records. By linking the EHR data with the immunization data from state of Michigan registry, we can capture vaccination data received outside the health systems. However, we still miss the vaccination data received outside of the state. Third, patients may seek care at other facilities outside Corewell Health or did not report the positive COVID-19 test result done at home or at another health system, leading to missing infection data and biased estimates of vaccine effectiveness. However, the use of the number of visits as covariates controls for the propensity to seek care and testing, mitigating the bias due to differing degrees of interaction with the health system.

Despite these limitations, our study developed a flexible machine learning approach to address various causal questions regarding the vaccine efficacy with uncertainty measures.