Estimating COVID-19 Vaccination and Booster Effectiveness Using Electronic Health Records From an Academic Medical Center in Michigan

HIGHLIGHTS

• •Studying vaccine effectiveness is crucial for informing public health interventions.
• •COVID-19 vaccines lowered infection rates by >60% in 2021.
• •COVID-19 vaccines lowered severe outcomes rates by >74% in 2021.
• •Vaccines’ protection against severe outcomes was persistent throughout 2021.
• •Boosters remarkably improved protection against both severe outcomes and infections.

Abstract

Introduction

Observational studies of COVID-19 vaccines’ effectiveness can provide crucial information regarding the strength and durability of protection against SARS-CoV-2 infection and whether the protective response varies across different patient subpopulations and in the context of different SARS-CoV-2 variants.

Methods

We used a test-negative study design to assess vaccine effectiveness against SARS-CoV-2 infection and severe COVID-19 resulting in hospitalization, intensive care unit admission, or death using electronic health records data of 170,741 adults who had been tested for COVID-19 at the University of Michigan Medical Center between January 1 and December 31, 2021. We estimated vaccine effectiveness by comparing the odds of vaccination between cases and controls during each 2021 calendar quarter and stratified all outcomes by vaccine type, patient demographic and clinical characteristics, and booster status.

Results

Unvaccinated individuals had more than double the rate of infections (12.1% vs 4.7%) and >3 times the rate of severe COVID-19 outcomes (1.4% vs 0.4%) than vaccinated individuals. COVID-19 vaccines were 62.1% (95% CI=60.3, 63.8) effective against a new infection, with protection waning in the last 2 quarters of 2021. The vaccine effectiveness against severe disease overall was 73.7% (95% CI=69.6, 77.3) and remained high throughout 2021. Data from the last quarter of 2021 indicated that adding a booster dose augmented effectiveness against infection up to 87.3% (95% CI=85.0, 89.2) and against severe outcomes up to 94.0% (95% CI=89.5, 96.6). Pfizer-BioNTech and Moderna vaccines showed comparable performance when controlling for vaccination timing. Vaccine effectiveness was greater in more socioeconomically affluent areas and among healthcare workers; otherwise, we did not detect any significant modification of vaccine effectiveness by covariates, including gender, race, and SES.

Conclusions

COVID-19 vaccines were highly protective against infection and severe COVID-19 resulting in hospitalization, intensive care unit admission, or death. Administration of a booster dose significantly increased vaccine effectiveness against both outcomes. Ongoing surveillance is required to assess the durability of these findings.

INTRODUCTION

A total of 3 coronavirus disease 2019 (COVID-19) vaccines were developed, assessed for efficacy against symptomatic COVID-19 disease in placebo-controlled trials, and approved under emergency use authorization in the U.S. by February 2021: mRNA-1273 (Moderna), BNT162b2 (Pfizer-BioNTech), and Ad26.COV2.S (Johnson & Johnson-Janssen).1, 2, 3 As of June 16, 2022, a total of 592 million doses had been administered, and 222 million people had been fully vaccinated in the primary series, meaning that they received at least 1 dose of Janssen or 2 doses of Pfizer-BioNTech or Moderna vaccine in the U.S.⁴ Administration of booster doses has been shown to provide more protection than the primary series,^5,6 but only 47.2% in the U.S. had received a booster as of June 16, 2022.⁷

Observational studies of vaccine effectiveness (VE) can assess real-world effectiveness, estimate duration of protection, identify protection against new variants that arise, and provide guidance on booster requirements.8, 9, 10, 11, 12, 13, 14, 15, 16 Observational studies conducted early in 2021 through COVID-19‒Associated Hospitalization Surveillance Network, a surveillance network in 13 states, including Michigan, found that 2 doses of Pfizer and Moderna provided 96% effectiveness in protecting against hospitalization,¹⁷ and partial vaccination was 64% effective against hospitalization.¹⁸ The Food and Drug Administration approved vaccination boosters,¹⁹ but the protection that the booster doses provide over time and among different vaccines remains unclear. It is critical to assess whether VE and durability of protection vary by patient demographics, comorbidities, history of previous severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, SARS-CoV-2 exposure risk, and other variables that may impact vaccine performance.²⁰ Retrospective analyses using large health systems’ electronic health record (EHR), and other similar databases, provide rich data for this type of research.

We used EHR data from the University of Michigan Health System (i.e., Michigan Medicine [MM]), a large, nationally ranked healthcare center in Michigan, to conduct a time-stratified retrospective cohort study to evaluate VE for the 3 major COVID-19 vaccines that were available in the U.S. Michigan is a state with both large urban population centers and rural communities and with extensive socioeconomic disparities across counties―particularly between Detroit, the most populous Black-majority city in the U.S., and its suburbs. In 32 (39%) of 83 counties in Michigan, vaccination primary series completion rates were below 50%.^21,22 The primary study goals were to determine the following:

• 1.VE against infection and severe disease across 2021;
• 2.VE stratified by the 2 most common vaccines―Pfizer-BioNTech and Moderna―and by sociodemographic and clinical characteristics that are associated with COVID-19 outcomes; and
• 3.VE of booster doses on the basis of the data from the last quarter of 2021.

METHODS

Study Sample

In a test-negative design cohort, eligible individuals included adults (aged ≥18 years) who received primary or other health care at MM and had a reverse transcription-polymerase chain reaction (RT-PCR) test for SARS-CoV-2 infection performed or recorded at MM between January 1 and December 31, 2021. All individuals of this cohort were MM patients; however, some were also healthcare staff.

Measures

RT-PCR Test Data

The RT-PCR test data were collected as part of patient screening on hospital check-in or before hospital admission and healthcare staff screening, regardless of symptom status. Only RT-PCR test results were included in the data set (details are in Appendix Text 1, available online). Antigen test results were excluded because of the large range of test performance characteristics among test types, which limited comparison with RT-PCR tests. This cohort includes individuals who tested positive (i.e., at least 1 RT-PCR‒positive test) and individuals who tested negative (i.e., only negative tests). The test results were considered per quarter of the year 2021, that is, if an individual was tested multiple times within a quarter, we used the first positive test date as the index date for individuals who tested positive at least once in that quarter, whereas we used the last negative test as the index date if an individual never tested positive in that quarter. Because some individuals were tested across multiple quarters of 2021, we carried forward only the latest quarter in which an individual was tested when we report on the overall cohort to ensure maximum follow-up time in the overall analysis.

Vaccination Status

We obtained records of vaccinations for individuals who received a vaccination at MM or who had a recorded vaccination record in the Michigan Care Improvement Registry (MCIR). Michigan's immunization providers are encouraged to report the administration of COVID-19 vaccines to MCIR within 72 hours, and records can be sent from a hospital EHR to MCIR through HL7 messages.²³

Among all individuals, 78,002 had at the date of the corresponding test no documented vaccination and thus were considered unvaccinated. A total of 92,485 individuals had at least 1 documented dose of a COVID-19 vaccine. According to the Centers for Disease Control and Prevention (CDC)’s vaccination guideline,²⁴ we categorized 74,060 individuals as fully vaccinated in the primary series, meaning documentation of 2 doses of Moderna or Pfizer-BioNTech vaccine or a single dose of Janssen vaccine at least 21 days before the corresponding test date.1, 2, 3

The remaining 18,425 vaccinated individuals were excluded because they were partially vaccinated (i.e., had not completed the vaccination primary series [n = 18,321] or received a mixed sequence of vaccines outside of CDC guidelines²⁴ [n = 104]) or received unspecified or other vaccine brands (i.e., Astra Zeneca, Sinopharm, Sinovac, or Novavac) (n = 254) (Appendix Table 1 and Appendix Figure 1B, available online). Individuals were classified as being boosted if they received any additional vaccination at least 21 days after a completed vaccination series. We conducted the booster-related VE analysis only in Quarter (Q) 4 of 2021, covering the period before the CDC guideline for boosters was announced.^24,25

COVID-19 Outcomes

n this study, we focused on 2 COVID-19‒associated outcomes: (1) SARS-CoV-2 infection and (2) severe disease or death. SARS-CoV-2 infection was defined as any documented positive result on an RT-PCR test (symptomatic or asymptomatic, nonsevere or severe disease), whereas severe disease or death was defined as either hospitalization or intensive care unit (ICU) admission between −7 and +30 days relative to the date of a documented positive SARS-CoV-2 RT-PCR test result or death between −7 and +60 days relative to the date of a positive RT-PCR test (Appendix Figure 1A, available online). Data on hospitalizations, ICU admissions, and deaths were obtained from MM's EHR databases as well as from the Michigan Death Registry.

Demographics, Socioeconomic Status, and Other Covariates

We have access to all patients’ structured deidentified EHR data that had been added by an MM healthcare provider. We performed additional analyses to examine whether VE varied by patient demographics (i.e., age, sex, race/ethnicity), weight status (i.e., BMI), clinical characteristics (e.g., comorbidities, history of SARS-CoV-2 infection, or history of cancer), occupational exposure (i.e., documented healthcare worker [HCW] or not), and SES (i.e., neighborhood disadvantage index [NDI]^26,27 and population density). Age, gender, and race/ethnicity were directly extracted from the EHR, whereas history of SARS-CoV-2 infection, history of cancer, immunosuppressed status,^25,28 BMI, the Elixhauser comorbidity score,^29,30 HCW status, NDI, and population density were derived on available variables. Definitions for derived variables are available in Appendix Text 2 (available online). We assumed that covariates included in our adjusted analyses were missing completely at random and performed complete case analyses for each adjustment. The sample sizes of the complete case analyses for various sets of covariates are listed in Appendix Table 4 (available online).

Ethical review and approval were waived for this study because of its qualification for a federal exemption as secondary research for which consent is not required. Determination for exemption was made by the University of Michigan Medical School IRB.

Statistical Analysis

We fit Firth's bias‒corrected logistic regression for each COVID-19‒related outcome $Y_{COVID}$, considering several sets of covariates:

$$\text{logit}\mathrm{P}\left({\mathrm{Y}}_{\text{COVID}}=1|\mathrm{I}\left(\text{Vaccine}\right),\text{Covariates}\right)={\beta }_0+{\beta }_{\text{Vac}}\mathrm{I}\left(\text{Vaccine}\right)+{\beta }_{\text{Cov}}\text{Covariates},$$

where $\text{logit}\left(\mathrm{x}\right)=\text{log}\left(\frac{\mathrm{x}}{1-\mathrm{x}}\right)$, $I\left(Vaccine\right)$ is the indicator of vaccination status, either without or with a booster, that is,

$$\begin{gathered} I\left(\text{Vaccine}\right)=\{\begin{array}{l}0\\ 1\end{array}\begin{array}{c}\mathrm{unvaccinated}\\ \mathrm{vaccinated}\end{array} \\ \text{for the models comparing fully vaccinated individuals, and} \\ I\left(\text{Vaccine}\right)=\{\begin{array}{l}0\\ 1\end{array}\begin{array}{c}\text{unvaccinated}\\ \text{vaccinated and received booster}\end{array} \end{gathered}$$

for the models comparing boosted individuals, and $\text{covariates}$ denotes the vector of covariates adjusted. A total of 5 nested sets of $\text{covariates}$ were evaluated, denoted as Adjustments 1–5 (defined in Appendix Table 4, available online). We reported the final model adjusted for age, gender, race/ethnicity, BMI, Elixhauser score, NDI, population density (persons per square mile), past COVID-19 infection, history of cancer, and HCW status (Adjustment 5).

We estimated the VE using the following formula³¹:

$$\widehat{VE}=1-\widehat{incidencerateratio}=\frac{{\hat{P}}_{Vaccinated}}{{\hat{P}}_{Unvaccinated}},$$

where $incidencerateratio$, the ratio of incidence between the vaccinated group and the unvaccinated group, can be estimated using the Michigan 12-month COVID-19 incidence rate and the estimated OR from the logistic regression mentioned earlier through the following formula³²:

$$Incidencerateratio=\frac{OR}{\left(1-IR\right)+\left(IR*OR\right)}.$$

We estimated an incidence rate of 0.088 on the basis of the cumulative case count through December 29, 2021 (1,710,325 cumulative cases; JHU COVID-19 Dashboard³³ and the resident population of Michigan: census 2020: 10,077,331).²²

We conducted a quarter-stratified analysis to examine the time-varying changes in VE over 2021: Q1, January 1–March 31; Q2, April 1–June 30; Q3, July 1–September 30; and Q4, October 1–December 31. For the estimation of the overall VE in 2021, we used the latest quarter in which a patient was tested (see above).

Finally, we estimated VE separately for individuals who received Pfizer-BioNTech only or Moderna only and provided estimates for the overall cohort as well as by quarter.

To assess the potentially different VE across different strata of risk factors (denoted as X) associated with COVID-19 outcomes, we further conducted interaction analysis by vaccine status using the following model:

$$\text{logit}\mathrm{P}({\mathrm{Y}}_{\text{COVID}}=1|\mathrm{I}\left(\text{vaccine}\right),\mathrm{X},\text{covariates})={\beta }_0+{\beta }_{\text{Vac}}\mathrm{I}\left(\text{vaccine}\right)+{\beta }_{\mathrm{X}}\mathrm{X}+{\beta }_{\text{Int}}\mathrm{X}\times \mathrm{I}\left(\text{vaccine}\right)+{\beta }_{\text{Cov}}\text{covariates}$$

All analyses were performed in R statistical software, Version 4.1.2.³⁴ For each model, we reported VE along with the corresponding 95% Wald CIs and p-values. For the interaction analysis, we obtained the vaccination OR for each stratum of X on the basis of ${\beta }_{Vac}+{\beta }_{Int}$ and reported the strata-specific subgroup VE along with the p-values, $P_{int.}$, by testing the difference of subgroup effects through the null hypothesis $H_0:{\beta }_{int.}=0$. We performed a multiple testing correction on the basis of a Bonferroni-corrected threshold of p<0.00294 (0.05/17) to maintain the overall significance level at 0.05 across the total of 17 performed interaction tests.

RESULTS

Participant Characteristics

In our cohort of 170,487 adults who were tested for SARS-CoV-2 infection between January 1 and December 31, 2021, 74,060 (43.4%) were fully vaccinated, and 78,002 (45.8%) were unvaccinated. In addition, 7,187 (9.7%) individuals from the vaccinated group received at least 1 additional booster vaccine dose after completing the primary series. In the overall cohort, the average time between being fully vaccinated and a recorded test was 136 days. More than 66.5% of individuals were tested within 3 months of being fully vaccinated.

The characteristics of individuals by vaccination status are summarized in Table 1. Vaccinated individuals were slightly older (37.6% of the vaccinated group were aged ≥65 years compared with only 20% of the unvaccinated), were more likely to be established MM primary care patients (i.e., received a primary care visit at MM within the last 24 months; 45% vs 31.9%), and generally resided in less socioeconomically disadvantaged areas.

Table 1.

Characteristics of the Michigan Medicine Cohort of 170,487 Individuals Who Were Tested for COVID-19 Between January and December 2021

Variables	Vaccination status
	Unvaccinated or unknown	Fully vaccinated
		Alla	≥1 booster
n	78,002	74,060	7,187
Patient demographics
Age, median (IQR)	46 (31.8)	58.2 (32.8)	64.5 (26)
Age, years, n (%)
18–49.9	43,430 (55.7)	28,130 (38)	1,981 (27.6)
50–64.9	18,986 (24.3)	18,094 (24.4)	1,670 (23.2)
≥65	15,586 (20)	27,836 (37.6)	3,536 (49.2)
Male gender, n (%)	33,023 (42.3)	30,924 (41.8)	3,122 (43.4)
Race ethnicity, n (%)
Non-Hispanic White	55,111 (70.7)	56,883 (76.8)	5,830 (81.1)
Non-Hispanic Black	9,187 (11.8)	5,287 (7.1)	349 (4.9)
Other or unknown	13,704 (17.6)	11,890 (16.1)	1,008 (14)
Primary care at MM, n (%)	24,847 (31.9)	33,192 (44.8)	3,697 (51.4)
Clinical variables
BMI category, n (%)
<18.5	20,007 (29.7)	20,063 (29.7)	2044 (30)
18.5–24.9	1,497 (2.2)	1,136 (1.7)	94 (1.4)
25–29.9	20,027 (29.8)	21,348 (31.7)	2,210 (32.4)
≥30	25,777 (38.3)	24,894 (36.9)	2,470 (36.2)
Elixhauser score AHRQ, mean (SD)b	3.6 (10.7)	5.9 (12.5)	8.9 (14.2)
Elixhauser score AHRQ category, n (%)b
<0	19,887 (30)	18,374 (27.4)	1,543 (22.8)
0	18,177 (27.4)	15,036 (22.4)	1,135 (16.8)
1–4	7,324 (11.1)	7,036 (10.5)	706 (10.5)
≥5	20,837 (31.5)	26,624 (39.7)	3,370 (49.9)
Past COVID-19 infection, n (%)	1,728 (2.2)	1,401 (1.9)	79 (1.1)
Healthcare worker, n (%)	571 (0.7)	2,786 (3.8)	465 (6.5)
Past immunosuppression, n (%)	5,217 (6.7)	7,249 (9.8)	1,267 (17.6)
Past cancer diagnosis, n (%)	12,326 (15.8)	18,552 (25)	2,584 (36)
Socioeconomic variables
NDI, n (%)c
Quartile 1	22,022 (34.9)	27,826 (45.4)	3,116 (50.9)
Quartile 2	14,860 (23.6)	14,545 (23.7)	1,359 (22.2)
Quartile 3	13,808 (21.9)	11,920 (19.4)	1,128 (18.4)
Quartile 4	12,390 (19.6)	7,038 (11.5)	515 (8.4)
Persons per square mile, n (%)d
Quartile 1	18,615 (29.5)	15,058 (24.6)	1,340 (21.9)
Quartile 2	18,826 (29.8)	20,076 (32.7)	2139 (35)
Quartile 3	19,742 (31.3)	20,778 (33.9)	2,107 (34.4)
Quartile 4	5,897 (9.3)	5,417 (8.8)	532 (8.7)

Note: For individuals who were tested across multiple quarters, only the vaccination status at their last observed quarter is shown.

^aIncludes individuals who received booster.

^bLower score indicates fewer/milder comorbidities.

^cQuartile #1 indicates the least socioeconomically deprived.

^dQuartile #1 indicates the lowest population density.

AHRQ, Agency for Healthcare Research and Quality; MM, Michigan Medicine; NDI, neighborhood disadvantage index.

The vaccinated subgroup had a slightly higher proportion of individuals with multiple comorbidities (Elixhauser Index ≥5). Furthermore, nearly twice as many vaccinated participants had a recent history of immunosuppression and cancer diagnosis as unvaccinated participants (9.8% vs 6.7% and 25% vs 15.8%, respectively) (Table 1 and Appendix Table 1, available online). We report the variables’ missingness by vaccination status in Appendix Table 2 (available online).

Over 62% of the study population lived in the 4 Michigan counties surrounding MM (Washtenaw, Wayne, Livingston, and Oakland), whereas 5.7% lived outside of Michigan. Although Detroit resides in Wayne County, the population of Washtenaw County, where MM is located and most individuals lived, had over 72.6% vaccinated (Figure 1 and Appendix Table 3, available online).

Figure 1.

Geographic representation of the origin of individuals (left) and the rate of the fully vaccinated population (right; aged ≥12 years) across all MI counties at the end of 2021. The location of Michigan Medicine in Washtenaw County is indicated by a white dot.

MI, Michigan.

Rates of Infection and Severe COVID-19 Outcomes Among Unvaccinated and Vaccinated Individuals

Unvaccinated individuals were significantly more likely to test positive for SARS-CoV-2 infection (12.1%) than vaccinated individuals (4.7%). The positivity rate among vaccinated individuals reached a valley with low community transmission in Q2 (Q1: 1.3% and Q2: 1.1%) and surged to 3.2% in Q3 and to 6.7% in Q4 (Table 2). In contrast, the positivity rate among unvaccinated individuals was lowest in Q1 (7.6%), increased to 11.8% in Q2, and decreased to 7.9% in Q3, before reaching a peak of 19.3% in Q4.

Table 2.

COVID-19 Outcome Summary of Included Individuals Who Were Tested for COVID-19 Between January and December 2021

Variables	Overall	Q1/2021	Q2/2021	Q3/2021	Q4/2021
Unvaccinated or unknown
N	78,002	56,953	23,640	13,507	11,101
Outcome, n (%)
Positive	9,464 (12.1)	4,342 (7.6)	2,781 (11.8)	1,073 (7.9)	2,143 (19.3)
Nonseverea	8,359 (10.7)	4,016 (7.1)	2,476 (10.5)	932 (6.9)	1,802 (16.2)
Severe	1,105 (1.4)	326 (0.6)	305 (1.3)	141 (1)	341 (3.1)
Hospitalized	798 (1)	228 (0.4)	222 (0.9)	97 (0.7)	255 (2.3)
ICU admission	166 (0.2)	50 (0.1)	36 (0.2)	31 (0.2)	50 (0.5)
Death	141 (0.2)	48 (0.1)	47 (0.2)	13 (0.1)	36 (0.3)
Fully vaccinated
n	74,060	4,075	25,661	30,237	30,424
Outcome, n (%)
Positive	3,495 (4.7)	51 (1.3)	278 (1.1)	1,056 (3.5)	2,204 (7.2)
Nonseverea	3,207 (4.3)	49 (1.2)	247 (1)	979 (3.2)	2,026 (6.7)
Severe	288 (0.4)	2 (<0.1)	31 (0.1)	77 (0.3)	178 (0.6)
Hospitalized	212 (0.3)	2 (<0.01)	23 (0.1)	50 (0.2)	137 (0.5)
ICU admission	36 (<0.1)	0 (0)	5 (<0.01)	14 (<0.1)	17 (0.1)
Death	40 (0.1)	0 (0)	3 (<0.01)	13 (<0.1)	24 (0.1)
Fully vaccinated + booster
N	7,187	15	111	511	6,755
Outcome n (%)
Positive	245 (3.4)	1 (6.7)	1 (0.9)	11 (2.2)	232 (3.4)
Nonseverea	227 (3.2)	1 (6.7)	1 (0.9)	10 (2)	215 (3.2)
Severe	18 (0.3)	0 (0)	0 (0)	1 (0.2)	17 (0.3)
Hospitalized	15 (0.2)	0 (0)	0 (0)	1 (0.2)	14 (0.2)
ICU admission	1 (<0.1)	0 (0)	0 (0)	0 (0)	1 (<0.1)
Death	2 (<0.1)	0 (0)	0 (0)	0 (0)	2 (<0.1)

Note: Overall numbers are based on the latest quarter in which a patient was tested. For individuals who were tested across multiple quarters, the vaccination status at their last observed quarter is summarized in the overall cohort.

^aIncludes asymptomatic and symptomatic patients who were not hospitalized −7 to +30 days relative to their test date.

ICU, intensive care unit; Q, quarter.

The overall rate of the severe disease, a composite of COVID-19‒related hospitalization, ICU admission, or death, was higher in unvaccinated (1.4%) than in vaccinated individuals (0.4%). Across the 4 quarters of 2021, the rate of severe disease among vaccinated individuals remained relatively low (Q1: <0.01%, Q2: 0.1%, Q3: 0.3%, and Q4: 0.6%) compared with that among the unvaccinated group (Q1: 0.6%, Q2: 1.3%, Q3: 1.0%, Q4: 3.1%). Only 18 severe cases (0.3%) were observed among the 7,187 vaccinated individuals who received a booster vaccination (Table 2).

Vaccine Effectiveness

In Figure 2, we present VE estimates for the overall cohort (using data only from the latest calendar quarter in which each participant was tested) and time-stratified cohort (Q1, Q2, Q3, and Q4). Because the VE estimates were reasonably consistent across adjustments, for example, in the overall cohort, VE against SARS-CoV-2 infection (VE against infection) ranged between 59.4 and 62.1% (Appendix Table 6, available online), we only discuss the estimates from the most conservative setting (Adjustment 5, which included all the covariates mentioned earlier) in this study. Results from all models are listed in the Appendix Material (Appendix Tables 6 and 7, available online). VE against infection in Q1 of 2021 was 82.0% (76.0%, 86.5%), increased in Q2 to 90.9% (89.5%, 92.1%), and was substantially reduced in Q3 (60.1% [55.9%, 64.0%]) and Q4 (68.8% [66.3%, 71.1%]) (Figures 2A and 3^35,36). This pattern was also observed for VE against the composite outcome of severe COVID-19 (VE against severe outcome) estimates, although they were generally higher and did not decrease as dramatically (Q1: 87.4% [58.1%, 96.3%]; Q2: 92.2% [88.3%, 94.8%]; Q3: 74.4% [64.8%, 81.5%], Q4: 83.0% [78.8%, 86.4%]) (Figures 2B and 3^35,36 and Appendix Table 6, available online).

Figure 2.

VE and the rates of (A) RT-PCR‒confirmed SARS-CoV-2 Infection and (B) the composite severe COVID-19 outcome (hospitalization + ICU admission + death) among vaccinated and unvaccinated as well as (C) a comparison of these rates between vaccinated without or with booster in the last quarter of 2021.

Note: Overall vaccine effectiveness is based on the latest quarter in which a patient was tested (Q1: n=39,782; Q2: n=36,058, Q3: n=34,697, Q4: n=41,525; shown in the Methods section). + denotes logistic regression. Adjustment 1: age, gender, race/ethnicity. Adjustment 5: Adjustment 1 + Elixhauser score AHRQ + Persons per square mile + neighborhood disadvantage index + past COVID-19 infection + healthcare worker status.

AHRQ, Agency for Healthcare Research and Quality; ICU, intensive care unit; Q, quarter; RT-PCR, reverse transcription-polymerase chain reaction; VE, vaccine effectiveness.

Figure 3.

VEs in 2021 against infection and severe outcomes and SARS-CoV-2 variant frequencies in the U.S. across the 4 quarters of 2021. The variant frequencies are based on 420 SARS-CoV-2 genomes sampled in the U.S. in 2021. VE estimates and their 95% CIs are shown (VE against SARS-CoV-2 infection: solid line, purple; VE against the composite outcome of severe COVID-19 disease: dashed line, pink). The variants are colored by their WHO variant names, whereas the Nextstrain clade names are shown in the figure.^35,36

VE, vaccine effectiveness.

A stratified analysis by booster status in Q4 showed additional protection of a booster against infection (VE against infection: no booster: 64.0% [61.1%, 66.7%] vs booster: 87.3% [85.0%, 89.2%]) and severe outcomes (VE against severe outcome: no booster: 78.8% [73.5%, 83.0%] vs booster: 94.0% [89.5%, 96.6%]) (Figures 2C and Appendix Figure 2 and Appendix Table 7, available online).

We also performed a sensitivity analysis using only data on individuals who received primary care at MM and thus were more likely to have documented vaccination data. In this analysis, estimates were substantially higher for VE against infection (Q1: 88.7% [81.5%, 93.1%]; Q2: 93.8% [92.3%, 95.0%]; Q3: 68.7% [63.7%, 73.0%]; Q4: 72.3 [68.9%, 75.4%]) (Appendix Figure 2A, available online) and VE against severe outcome (Q1: not available or no severe case among vaccinated; Q2: 95.2% [91.1%, 97.4%]; Q3: 79.4% [68.0%, 86.7%]; Q4: 80.8% [73.8%, 85.9%]) (Appendix Figure 2B, available online).

Comparing Vaccine Effectiveness Between Pfizer-BioNTech and Moderna

Because most of the 74,060 fully vaccinated individuals completed a vaccine series with either Pfizer-BioNTech (n=45,168; 61.0%) or Moderna (n=25,267; 34.1%), we compared the effectiveness of the 2 vaccines. To address that the Pfizer-BioNTech vaccine was available before Moderna, we stratified the analysis on the basis of whether the vaccination occurred <3 months or ≥3 months before the COVID-19 test (Figure 4).

Figure 4.

Comparison of effectiveness between Pfizer-BioNTech and Moderna vaccines. (A) VE against SARS-CoV-2 infection and (B) VE against the composite outcome of severe COVID-19 disease are shown for individuals being fully vaccinated at any time (top), within 3 months (center), or over 3 months (bottom) before being tested for COVID-19.

VE, vaccine effectiveness.

In the overall cohort, we observed lower VE against infection for Pfizer-BioNTech than for Moderna (82.9% [80.7%, 84.9%] vs 88.1% [85.5%, 90.2%]), a trend that was consistent across all the 4 quarters of 2021, although the difference was lowest in Q2 (89.6% [87.6%, 91.3%] vs 91.7% [89.3%, 93.5%]) (Figures 4 and Appendix Figure 4, available online). Contrarily, VE against severe outcome was higher for Pfizer-BioNTech (87.1% [80.3%, 91.6%]) than for Moderna (84.9% [76.2%, 90.5%]) in the overall cohort, consistent in the first half of 2021 but reversed in the last half of 2021. Of note, the VE against severe outcome in Q4 was higher for Moderna (93.4% [5.3%, 99.6%]) than for Pfizer-BioNTech (68.3% [23.6%, 87.2%]), with overlapping CIs (Appendix Figure 5, available online).

Factors Affecting Vaccine Effectiveness

Overall, we found that VE against infection ranged between 57.1 and 65.2% across all strata but was lower for people with a documented past COVID-19 infection (47.6% [24.5%, 63.9%]; p-interaction [p_Int.]=0.065) (Figure 5). Only 182 individuals had a past infection, and only 48 of these were vaccinated (Appendix Table 8, available online). In Figure 5, we found that compared to the trend among those aged <50 years, VE shows a decreasing trend among those aged >65 years against both infection and severe outcomes, but the differences were not statistically significant.

Figure 5.

VE against SARS-CoV-2 infection (infection) and VE against the composite outcome of severe COVID-19 disease (severe outcome) across various covariate strata. Analyses were adjusted for the following set of covariates (after excluding the variable of interest): age, gender, race/ethnicity, Elixhauser score, persons per square mile, neighborhood disadvantage index, past COVID-19 infection, and healthcare worker status.

VE, vaccine effectiveness.

For HCWs, we observed a higher VE against infection than for others (HCW: 70.4% [62.8%, 76.5%] vs non-HCW: 61.7% [59.9%, 63.9%]; p_Int.=0.031).

In addition, we observed decreased VE against infection in individuals living in neighborhoods with increasing NDI quartiles compared with that in those in the lowest NDI quartile (p_Int.≤0.023).

For VE against severe outcome, we observed effective protection against severe COVID-19 across all strata (VE against severe outcome≥66.6%), noting the estimated reduced effectiveness for individuals with a past COVID-19 infection (VE against severe outcome _{Past COVID-19 Infection}=45.3% [0%, 77.1%]; p_Int.=0.133). Owing to the lower counts of individuals with severe COVID-19, CIs were wider (Figure 5).

DISCUSSION

In a study of 170,487 adults tested for COVID-19 within Michigan, we found strong protection provided by Pfizer-BioNTech and Moderna vaccines against infection and severe disease. Booster doses increased effectiveness against both infection and severe disease. In our study, fully vaccinated individuals were less likely to test SARS-CoV-2 positive and were less likely to have the severe disease than unvaccinated individuals. We observed declines in estimated VE over time, indicating potential waning effectiveness, consequences of spreading of the more immune-evasive SARS-CoV-2 B.1.617.2 (Delta) and the B.1.1.159 (Omicron) variants.37, 38, 39 Because uptake in booster shots increased in Q4, both infection and severe disease rates were lowest among boosted individuals. This evidence supports a booster's protection against infections and severe disease in the face of potential waning of VE and emerging virus variants.

Among individuals who received the vaccine within the past 3 months, VE against infection tended to be slightly higher for those who received the Moderna vaccine, and VE against severe outcomes was moderately high for both vaccines. However, the overlapping CIs suggest similar vaccine protection or the need for a larger sample size to tease out the vaccines’ performance.

Our results are largely consistent with previously published findings, although differences among studies make direct comparison challenging. For example, some studies adjusted differently for potential confounders (age only, additional patient characteristics, or time since vaccination),8, 9, 10^,13,40 whereas others characterized the variant associated with each case through whole-genome sequencing.¹¹ Our study spans 2021 and thus likely includes data on multiple variants (including the Omicron variant) and on boosters that have become more common. From other studies conducted from late 2020 to early 2021, VE estimates against hospitalization ranged around 87% (95% CI=55, 100)⁸ and 89% (95% CI=87, 91)⁹ against infection. Studies during mid-2021 reported VE in the ranges of 75.0%, 91.8%,¹³ 79.9%, and 98.4%¹⁰ against infection with different variants, including Delta, and between 89.5% and 95.1%¹³ and 86% (95% CI=82, 88)¹¹ against hospitalization across different variants. A recent study assessing VE from late 2020 through Q3 of 2021 found VE rates for mRNA vaccines to be the highest 2 weeks after a 2-dose regimen (94.5%; 95% CI=94.1, 94.9), lining up with our estimates of VE in Q2 and lowest at 66.6% (95% CI=65.2, 67.8) after 7 months,¹⁴ which corresponds with our VE estimates in Q3. Taken together, these results highlight the need for appropriate adjustment for patient covariates, timing of vaccination, booster status, and replication across different regions and time periods to more fully understand VE and potential modifying factors. Ongoing studies should also be estimating the effectiveness of the vaccine against newer variants and how the previous infection modifies the effectiveness of primary or booster doses of vaccination.

Michigan offers an important location to study COVID-19 VE for several reasons. The southeast corner of the state, where this study is focused, has extensive socioeconomic disparities across its urban, suburban, and rural locations. Notably, Detroit, the most populous Black-majority city in the U.S., was severely affected by COVID-19 outbreaks early in the pandemic. Within the city, there were still substantial differences in COVID-19 vaccination coverage between Black (56%) and White (82%) populations in mid-2021.⁴¹ Because there is substantial evidence that how effective the vaccine is could influence acceptance of the vaccine,⁴² this study, reporting relatively high VE in this area, could be influential. However, we also note that VE, although not significantly different by race/ethnicity, might be lower among individuals living in neighborhoods with higher NDI. This suggests that more effort is needed to study protection from vaccines in more disadvantaged areas. Past studies of other respiratory infections have posited that differences in risk of infection across physical space could include housing overcrowding, limited access to health care, and higher rates of contact with others who are infected.⁴³ Future studies could sample within the community and not just among those with healthcare access.⁴⁴

Strengths of this study are the availability of detailed EHR data for study subjects, COVID-19 test and vaccination data from all the 4 quarters of 2021, and the large number of participants from a single academic medical center in Michigan. This allows us to adjust for potential confounding by patient characteristics, including socioeconomic and demographic characteristics, and relevant medical conditions, including previous COVID-19 testing results and receipt of a booster dose. Furthermore, we considered multiple outcomes of interest and assessed variation in VE by different demographic, clinical, and other factors. The availability of potential confounding patient characteristics allowed us to evaluate the potential modification of VE by patient subgroups. After correcting for multiple testing, none of the stratified analyses were significant. Differences before multiple testing corrections potentially indicate a stricter adherence to the vaccination series or additional protective measures about HCW.^45,46 There are some discrepancies in the literature on the effectiveness, with some literature suggesting that unvaccinated individuals who acquired immunity through a previous COVID-19 infection may benefit less from vaccination than people who were not previously infected.^47,48 However, there is a growing body of literature suggesting that COVID-19 vaccination is the best protection against infection and severe disease⁴⁹ whether or not one has already had COVID-19.^48,50, 51, 52, 53

Limitations

A potential weakness is any missing or incomplete information from the EHR, for example, variant sequencing, undocumented test results, past infections, vaccinations, hospitalizations, or mortality outcomes. Although we cannot test the exact completeness of the EHR vaccination record, for example, it is possible, although unlikely, that they may have been vaccinated elsewhere and that these records were not available, our sensitivity analysis examining MM primary care patients suggests the possibility that VE estimates in the full cohort might be influenced by incomplete vaccination documentation in the EHR. In this scenario, categorizing individuals into an unknown or unvaccinated status tends to produce conservative VE estimates (Appendix Figure 3, available online). Another explanation for these findings regards potential differences in health-seeking behaviors and therefore differential VE between patient groups. Results from the test-negative design are generalizable to individuals tested for COVID-19 and more straightforward with tests having similar sensitivity and specificity properties.^9,44,53 Still, these studies provide insight into the long-term effectiveness and durability of vaccine protection.⁵³ How to combine the RT-PCR test and the rapid antigen tests for VE studies in a statistically valid manner remains a question of interest in the future.

CONCLUSIONS

The study provides robust evidence for the ability of COVID-19 vaccines to protect against infection and severe disease. As new variants arise, continued observational studies will be necessary to assess the effectiveness of currently available vaccine regimens. Overall, these results provide evidence of the effectiveness of the vaccines over time and may encourage those who have not been vaccinated or not received a booster to consider doing so to prevent severe outcomes, including mortality.