Comparative effectiveness of mRNA-1273 and BNT162b2 against symptomatic SARS-CoV-2 infection

Abstract

Background

mRNA coronavirus disease 2019 (COVID-19) vaccines are safe and effective, but increasing reports of breakthrough infections highlight the need to vigilantly monitor and compare the effectiveness of these vaccines.

Methods

We retrospectively compared protection against symptomatic infection conferred by mRNA-1273 and BNT162b2 at Mayo Clinic sites from December 2020 to September 2021. We used a test-negative case-control design to estimate vaccine effectiveness (VE) and to compare the odds of symptomatic infection after full vaccination with mRNA-1273 versus BNT162b2, while adjusting for age, sex, race, ethnicity, geography, comorbidities, and calendar time of vaccination and testing.

Findings

Both vaccines were highly effective over the study duration (VE_mRNA-1273: 84.1%, 95% confidence interval [CI]: 81.6%–86.2%; VE_BNT162b2: 75.6%, 95% CI: 72.2%–78.7%), but their effectiveness was reduced during July–September (VE_mRNA-1273: 75.6%, 95% CI: 70.1%–80%; VE_BNT162b2: 63.5%, 95% CI: 55.8%–69.9%) as compared to December–May (VE_mRNA-1273: 93.7%, 95% CI: 90.4%–95.9%; VE_BNT162b2: 85.7%, 95% CI: 81.4%–88.9%). Adjusted for demographic characteristics, clinical comorbidities, time of vaccination, and time of testing, the odds of experiencing a symptomatic breakthrough infection were lower after full vaccination with mRNA-1273 than with BNT162b2 (odds ratio: 0.60; 95% CI: 0.55–0.67).

Conclusions

Both mRNA-1273 and BNT162b2 strongly protect against symptomatic severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection. It is imperative to continue monitoring and comparing available vaccines over time and with respect to emerging variants to inform public and global health decisions.

Funding

This study was funded by nference.

Graphical abstract

Context and significance

This is a study comparing the effectiveness of the mRNA coronavirus disease 2019 (COVID-19) vaccines developed by Pfizer/BioNTech and Moderna. Although it is well established that both vaccines are safe and effective, it is important to evaluate their effectiveness over time and relative to each other. The results from this study show that both of these vaccines have been effective in reducing the risk of COVID-19, although recipients of the Moderna vaccine were less likely to experience a breakthrough infection than recipients of the Pfizer/BioNTech vaccine. This longitudinal and comparative effectiveness analysis further supports the importance of vaccination in curbing the COVID-19 pandemic and can help to inform recommendations regarding the optimal selection and timing of vaccination series.

Puranik et al. compare the real-world effectiveness of BNT162b2 and mRNA-1273 in a population of over 180,000 vaccinated individuals. Although both vaccines strongly protected against symptomatic infection, their effectiveness declined in recent months. Throughout the study period, mRNA-1273 recipients had lower rates of COVID-19 diagnosis than BNT162b2 recipients.

Introduction

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has infected over 234 million individuals, leading to over 4.8 million deaths attributed to coronavirus disease 2019 (COVID-19).¹ To curb the spread of SARS-CoV-2, mass global vaccination efforts have been initiated, with 6.3 billion vaccine doses administered to date.¹ Controlled clinical trials and real-world studies have clearly demonstrated the effectiveness of the messenger RNA (mRNA) COVID-19 vaccines, which are authorized or approved by the US Food and Drug Administration (FDA) for use in the United States. In a phase 3 clinical trial, BNT162b2, an mRNA vaccine developed by Pfizer/BioNTech, showed 95.0% efficacy (95% confidence interval [CI]: 90.3%–97.6%) in preventing symptomatic COVID-19, with onset 7 or more days after the second dose.² mRNA-1273, an mRNA vaccine developed by Moderna, showed 94.1% efficacy (95% CI: 89.3%–96.8%) in preventing symptomatic infection with onset at least 14 days after the second dose.³ Additional real-world retrospective studies in major health systems in the United States and elsewhere further support the effectiveness and safety of these vaccines.4, 5, 6, 7, 8

However, only about 55% of the United States population is fully vaccinated as of October 2021, with an even lower fraction fully vaccinated across the globe.⁹ Further, there have been reports of reduced vaccine effectiveness over time and against emerging variants, resulting in more breakthrough infections, which have spurred discussions regarding strategies to boost and prolong immunity in previously vaccinated individuals.10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25 With ongoing efforts to distribute initial vaccination series throughout the world and booster doses in the United States and elsewhere, it is important to understand whether one mRNA vaccine confers stronger protection than the other. Here, we begin to address this knowledge gap by retrospectively comparing the rates of symptomatic SARS-CoV-2 infection after full vaccination with mRNA-1273 versus BNT162b2 in the multi-state Mayo Clinic Health System.

Results

Eligible population for test-negative case-control design

The primary analysis was conducted using a test-negative design to compare the odds of testing positive for SARS-CoV-2 given presentation to the clinic with COVID-19 symptoms for individuals vaccinated with mRNA-1273 versus BNT162b2 (Figure 1 A). Based on the longitudinal prevalence of SARS-CoV-2 variants in the states from which the cohorts were derived, we considered three time intervals: the entire study duration (December 1, 2020 through September 22, 2021), an early epoch during which the Alpha variant was the most prevalent (December 1, 2020 through May 31, 2021), and a late epoch during which the Delta variant was the most prevalent (July 1, 2021 through September 22, 2021; Figure S1A).

Figure 1.

Test-negative case-control analysis to compare the odds of symptomatic infection after vaccination with mRNA-1273 versus BNT162b2

(A) The primary analysis was conducted as a test-negative case-control study, with positive symptomatic SARS-CoV-2 PCR tests considered as cases and negative symptomatic tests considered as controls. We compared the odds of symptomatic infection after full vaccination with mRNA-1273 versus after full vaccination with BNT162b2, while adjusting for clinical and demographic covariates.

(B) Odds ratios (ORs) of symptomatic infection obtained in the crude analysis (top) and in the adjusted analysis using conditional logistic regression (CLR) (bottom). In each case, ORs are shown for the entire study duration (December 1, 2020 through September 22, 2021), the early epoch (December 1, 2020 through May 31, 2021), and the late epoch (July 1, 2021 through September 22, 2021). The x axis is log-transformed such that, for example, ORs of 0.5 and 2 are equidistant from the null hypothesis odds ratio of 1. Error bars represent 95% confidence intervals.

(C) Estimated effectiveness of mRNA-1273 and BNT162b2 against symptomatic infection in the early epoch and the late epoch. In each time period, effectiveness estimates were derived by comparing the odds of symptomatic infection after full vaccination versus during the 10 days after the first dose. Error bars represent 95% confidence intervals.

Of 61,743 mRNA-1273 recipients who were at risk for infection at their date of full vaccination, 14,667 subsequently underwent at least one symptomatic test. There were 814 individuals with positive symptomatic tests (cases) and 6,294 negative tests (controls) from 5,879 individuals (Table 1 ). Of 120,334 BNT162b2 recipients who were at risk for infection at their date of full vaccination, 28,990 subsequently underwent at least one symptomatic test, with 2,407 cases and 13,668 controls from 12,763 individuals (Table 1). Most vaccinations occurred during the early epoch (Figure S1B), and over 90% of breakthrough infections were diagnosed during the late epoch (Table 2 ; Figure S1C). Demographic and clinical characteristics of cases, controls, and the underlying populations are shown in Tables 1, S1, and S2.

Table 1.

Demographic and clinical characteristics of cases and controls among BNT162b2 and mRNA-1273 recipients who were considered in the primary test-negative analysis

Characteristic	BNT162b2 case population	BNT162b2 control population	mRNA-1273 case population	mRNA-1273 control population
	1+ positive symptomatic test after full vaccination	1+ negative symptomatic test after full vaccination, sub-sampled	1+ positive symptomatic test after full vaccination	1+ negative symptomatic test after full vaccination, sub-sampled
Number of individuals	2,407	12,763	814	5,879
Number of symptomatic tests	2,407	13,668	814	6,294
Age (years)
Mean (SD)	49.9 (16.1)	51.1 (18.9)	56.7 (17.9)	58.6 (18.2)
18–24	96 (4.0%)	630 (4.9%)	32 (3.9%)	237 (4.0%)
25–34	308 (12.8%)	2,150 (16.8%)	78 (9.6%)	557 (9.5%)
35–44	445 (18.5%)	2,217 (17.4%)	135 (16.6%)	731 (12.4%)
45–54	384 (16.0%)	1,821 (14.3%)	106 (13.0%)	712 (12.1%)
55–64	416 (17.3%)	2,225 (17.4%)	150 (18.4%)	1,077 (18.3%)
65–74	364 (15.1%)	1,865 (14.6%)	191 (23.5%)	1,539 (26.2%)
75–84	292 (12.1%)	1,304 (10.2%)	84 (10.3%)	695 (11.8%)
85+	102 (4.2%)	551 (4.3%)	38 (4.7%)	331 (5.6%)
State of primary residence
Arizona	147 (6.1%)	1,198 (9.4%)	33 (4.1%)	410 (7.0%)
Florida	464 (19.3%)	2,071 (16.2%)	281 (34.5%)	1,741 (29.6%)
Iowa	6 (0.2%)	6 (0.0%)	2 (0.2%)	3 (0.1%)
Minnesota	1,052 (43.7%)	6,020 (47.2%)	342 (42.0%)	2,611 (44.4%)
Wisconsin	737 (30.6%)	3,468 (27.2%)	156 (19.2%)	1,113 (18.9%)
Other	1 (0.0%)			1 (0.0%)
Sex
Female	1,444 (60.0%)	8,071 (63.2%)	431 (52.9%)	3,364 (57.2%)
Male	963 (40.0%)	4,691 (36.8%)	383 (47.1%)	2,514 (42.8%)
Unknown		1 (0.0%)		1 (0.0%)
Race
Asian	56 (2.3%)	475 (3.7%)	10 (1.2%)	176 (3.0%)
Black/African American	54 (2.2%)	322 (2.5%)	17 (2.1%)	169 (2.9%)
Native American	2 (0.1%)	31 (0.2%)	3 (0.4%)	18 (0.3%)
Native Hawaiian/Pacific Islander	2 (0.1%)	9 (0.1%)	2 (0.2%)	1 (0.0%)
White	2,227 (92.5%)	11,501 (90.1%)	744 (91.4%)	5,354 (91.1%)
Other	46 (1.9%)	288 (2.3%)	21 (2.6%)	106 (1.8%)
Unknown	20 (0.8%)	137 (1.1%)	17 (2.1%)	55 (0.9%)
Ethnicity
Hispanic or Latino	85 (3.5%)	459 (3.6%)	40 (4.9%)	236 (4.0%)
Not Hispanic or Latino	2,284 (94.9%)	12,017 (94.2%)	744 (91.4%)	5,535 (94.1%)
Unknown	38 (1.6%)	287 (2.2%)	30 (3.7%)	108 (1.8%)
Elixhauser Comorbidity Index
0	1,725 (71.7%)	9,326 (73.1%)	566 (69.5%)	3,895 (66.3%)
1–4	359 (14.9%)	1,702 (13.3%)	125 (15.4%)	875 (14.9%)
5–9	216 (9.0%)	1,133 (8.9%)	93 (11.4%)	695 (11.8%)
10+	107 (4.4%)	602 (4.7%)	30 (3.7%)	414 (7.0%)
Number of PCR tests prior to first dose
0	1,108 (46.0%)	4,680 (36.7%)	356 (43.7%)	2,277 (38.7%)
1	569 (23.6%)	3,030 (23.7%)	192 (23.6%)	1,407 (23.9%)
2+	730 (30.3%)	5,053 (39.6%)	266 (32.7%)	2,195 (37.3%)
First dose vaccine site
Arizona	58 (2.4%)	450 (3.5%)	18 (2.2%)	218 (3.7%)
Florida	350 (14.5%)	1,661 (13.0%)	226 (27.8%)	1,448 (24.6%)
Mayo Clinic Health System	953 (39.6%)	4,010 (31.4%)	158 (19.4%)	1,249 (21.2%)
Rochester (Minnesota)	526 (21.9%)	3,883 (30.4%)	372 (45.7%)	2,339 (39.8%)
Other/not recorded	520 (21.6%)	2,759 (21.6%)	40 (4.9%)	625 (10.6%)
Dates of full vaccination
Earliest	Jan 20	Jan 18	Feb 8	Feb 1
25th %	Feb 17	Feb 15	Feb 25	Mar 2
Median	Mar 17	Mar 15	Mar 25	Mar 25
75th %	Apr 22	Apr 22	Apr 21	Apr 21
Latest	Sep 2	Aug 30	Aug 1	Aug 30
Days between full vaccination and test
Minimum	0	0	1	0
25th %	113	111	120	114
Median	149	145	147	142
75th %	184	182	167	167
Maximum	238	243	221	221

Cases were defined as the first positive symptomatic test after full vaccination for a given individual, and controls were defined as negative symptomatic tests after full vaccination. Individuals can thus contribute only one case but multiple controls. A negative test is only considered if the individual had no prior positive tests at the time of testing. Controls were sub-sampled to avoid counting multiple negative tests from a single symptomatic illness and to limit the number of negative tests contributed by a single individual over the study duration. Characteristics of the underlying populations from which cases and controls are derived are shown in Tables S1 (BNT162b2) and S2 (mRNA-1273).

Table 2.

Comparison of symptomatic infection odds after full vaccination with mRNA-1273 versus BNT162b2 using a test-negative design

Time period		mRNA-1273	BNT162b2
Study duration Dec 1, 2020– Sept 22, 2021	positive tests (cases)	814	2,407
	negative tests (controls)	6,294	13,668
	crude odds ratio (95% CI)	0.734 (0.675, 0.799)
	adjusted odds ratio (95% CI)	0.604 (0.546, 0.667)
Early epoch Dec 1, 2020– May 31, 2021	positive tests (cases)	24	118
	negative tests (controls)	332	1,085
	crude odds ratio (95% CI)	0.665 (0.421, 1.05)
	adjusted odds ratio (95% CI)	0.414 (0.232, 0.74)
Late epoch Jul 1, 2021–Sept 22, 2021	positive tests (cases)	772	2,242
	negative tests (controls)	5,706	12,171
	crude odds ratio (95% CI)	0.734 (0.673, 0.802)
	adjusted odds ratio (95% CI)	0.611 (0.551, 0.677)

For each time period, the number of cases and controls contributed by the two populations is shown. To compare the odds of symptomatic infection after full vaccination, we calculated the crude odds ratio and an adjusted odds ratio using conditional logistic regression.

Test-negative analysis of odds of symptomatic infection after full vaccination

Throughout the study duration, the odds of symptomatic infection after full vaccination with mRNA-1273 were lower than after full vaccination with BNT162b2 (odds ratio [OR]: 0.73; 95% CI: 0.68–0.80; Table 2; Figure 1B). Similar trends were observed in both the early epoch (OR: 0.67; 95% CI: 0.42–1.05) and the late epoch (OR: 0.73; 95% CI: 0.67–0.80), although the former did not reach statistical significance (Table 2; Figure 1B). Importantly, both groups had similar odds of symptomatic infection in the 10 days following the first dose during each period (OR_{Study Duration}: 1.11, 95% CI: 0.90–1.37; OR_Early: 1.15, 95% CI: 0.92–1.43; OR_Late: 1.47, 95% CI: 0.69–3.11), when the protective effect of vaccination is not yet expected to have set in (Table S3).

After adjusting for age, sex, race, ethnicity, geography, and Elixhauser Comorbidity Index (ECI) score using conditional logistic regression (CLR), the odds of symptomatic infection were still lower after full vaccination with mRNA-1273 across the study duration (OR: 0.60; 95% CI: 0.55–0.67). Similar trends were observed in the early epoch (OR: 0.41; 95% CI: 0.23–0.74) and the late epoch (OR: 0.61; 95% CI: 0.55–0.68; Table 2; Figure 1B). Again, the odds were similar in the 10 days after the first dose during each period (OR_{Study Duration}: 1.14, 95% CI: 0.88–1.48; OR_Early: 1.09, 95% CI: 0.82–1.44; OR_Late: 1.55, 95% CI: 0.79–3.04; Table S3). Results for the other covariates included in the CLR model are provided in Table S4.

Age-stratified test-negative analysis

These trends generally persisted in the age-stratified subanalysis. Among individuals at least 65 years old, the adjusted odds ratios of symptomatic infection after full vaccination with mRNA-1273 versus BNT162b2 were 0.52 (95% CI: 0.44–0.63) across the study duration, 0.70 (95% CI: 0.26–1.87) in the early epoch, and 0.52 (95% CI: 0.43–0.63) in the late epoch (Table S5). Among those less than 65 years old, these adjusted odds were 0.63 (95% CI: 0.55–0.73), 0.37 (95% CI: 0.58; 0.14–0.99), and 0.63 (95% CI: 0.55–0.73; Table S5). Results for the other covariates included in these CLR models are provided in Tables S6 and S7.

Estimated vaccine effectiveness via test-negative analysis

To contextualize these observations, we estimated the effectiveness of mRNA-1273 and BNT162b2 across these three periods by comparing the odds of symptomatic infection after full vaccination versus during the first 10 days after the first dose (see STAR Methods). Across the study duration, mRNA-1273 was 84.1% (95% CI: 81.6%–86.2%; p < 0.001) effective against symptomatic infection, and BNT162b2 was 75.6% (95% CI: 72.2%–78.7%; p < 0.001) effective (Table 3 ). Both vaccines were more effective during the early epoch than in the late epoch, indicated by their non-overlapping 95% CIs (mRNA-1273: 93.7% [90.4%–95.9%] versus 75.6% [70.1%–80%]; BNT162b2: 85.7% [81.4%–88.9%] versus 63.5% [55.8%–69.9%]; Table 3; Figure 1C). Results for the other covariates included in this CLR model are provided in Table S8.

Table 3.

Estimates of vaccine effectiveness against symptomatic infection

Time period		mRNA-1273 or BNT162b2 (days 1–10 after first dose)	mRNA-1273 (fully vaccinated)	BNT162b2 (fully vaccinated)
Study duration Dec 1, 2020–Sept 22, 2021	cases	607	861	2,526
	controls	2,135	11,796	24,199
	vaccine effectiveness (95% CI)		84.1% (81.6%, 86.2%)	75.6% (72.2%, 78.7%)
Early epoch Dec 1, 2020–May 31, 2021	cases	487	28	128
	controls	1,887	2,137	4,948
	vaccine effectiveness (95% CI)		93.7% (90.4%, 95.9%)	85.7% (81.4%, 88.9%)
Late epoch Jul 1, 2021–Sept 22, 2021	cases	119	814	2,349
	controls	234	8,772	17,637
	vaccine effectiveness (95% CI)		75.6% (70.1%, 80%)	63.5% (55.8%, 69.9%)

For each time period, conditional logistic regression was used to calculate odds ratios of symptomatic infection after full vaccination versus during the 10 days after the first dose, adjusted for demographic and clinical covariates. Vaccine effectiveness was calculated as 100% × (1 − adjusted odds ratio).

Unmatched cohort analysis to compare rates of symptomatic infection

There were 55,277 and 108,996 individuals who were eligible for inclusion in the cohort study after vaccination with mRNA-1273 or BNT162b2, respectively (Table 4 ; Figure 2 A). After full vaccination, the incidence rate (IR) of symptomatic infection was significantly lower in the mRNA-1273 cohort (IR_mRNA-1273: 0.067 cases per 1,000 person days; IR_BNT162b2: 0.091; incidence rate ratio [IRR]: 0.74; 95% CI: 0.67–0.83; Table 5 ; Figure 2B). Similar trends were observed during both the early epoch (IR_mRNA-1273: 0.0088; IR_BNT162b2: 0.014; IRR: 0.64; 95% CI: 0.38–1.0) and the late epoch (IR_mRNA-1273: 0.13; IR_BNT162b2: 0.18; IRR: 0.73; 95% CI: 0.65–0.81; Table 5; Figure 2B). Importantly, the IRs of symptomatic infection were similar in the 10 days after the first dose (IR_mRNA-1273: 0.23 cases per 1,000 person days; IR_BNT162b2: 0.22; IRR: 1.1; 95% CI: 0.84–1.3; Table 5; Figure 2B). Further, the likelihood of follow-up after vaccination, as assessed by the incidence rates of negative symptomatic tests, were similar between these groups in each study period (Table S9).

Table 4.

Demographic and clinical characteristics of vaccinated individuals who were eligible for inclusion in the unmatched and matched cohort studies

	mRNA-1273 (unmatched)	BNT162b2 (unmatched)	mRNA-1273 (matched)	BNT162b2 (matched)
Total number of individuals
At least one dose	55,277	108,996	20,890	20,890
Fully vaccinated per protocol AND matched individual also fully vaccinated per protocol	45,534	92,547	15,392	15,392
Age groups in years
18–24	2,987 (5.4%)	7,352 (6.7%)	1,119 (5.4%)	1,119 (5.4%)
25–34	4,675 (8.5%)	14,516 (13.3%)	1,839 (8.8%)	1,839 (8.8%)
35–44	5,777 (10.5%)	15,608 (14.3%)	2,205 (10.6%)	2,205 (10.6%)
45–54	6,789 (12.3%)	15,701 (14.4%)	2,618 (12.5%)	2,618 (12.5%)
55–64	11,077 (20.0%)	21,782 (20.0%)	4,583 (21.9%)	4,583 (21.9%)
65–74	14,939 (27.0%)	17,912 (16.4%)	5,939 (28.4%)	5,939 (28.4%)
75–84	6,650 (12.0%)	12,458 (11.4%)	1,911 (9.1%)	1,911 (9.1%)
85+	2,383 (4.3%)	3,667 (3.4%)	676 (3.2%)	676 (3.2%)
Sex
Female	30,250 (54.7%)	63,864 (58.6%)	12,068 (57.8%)	12,068 (57.8%)
Male	25,019 (45.3%)	45,120 (41.4%)	8,822 (42.2%)	8,822 (42.2%)
Unknown	8 (0.0%)	12 (0.0%)
Race
Asian	1,087 (2.0%)	3,244 (3.0%)	218 (1.0%)	218 (1.0%)
Black/African American	1,505 (2.7%)	3,156 (2.9%)	261 (1.2%)	261 (1.2%)
Native American	204 (0.4%)	304 (0.3%)	2 (0.0%)	2 (0.0%)
Native Hawaiian/Pacific Islander	49 (0.1%)	104 (0.1%)
White	50,527 (91.4%)	98,008 (89.9%)	20,208 (96.7%)	20,208 (96.7%)
Other	1,121 (2.0%)	2,570 (2.4%)	101 (0.5%)	101 (0.5%)
Unknown	784 (1.4%)	1,610 (1.5%)	100 (0.5%)	100 (0.5%)
Ethnicity
Hispanic or Latino	2,143 (3.9%)	4,037 (3.7%)	207 (1.0%)	207 (1.0%)
Not Hispanic or Latino	51,672 (93.5%)	102,021 (93.6%)	20,535 (98.3%)	20,535 (98.3%)
Unknown	1,462 (2.6%)	2,938 (2.7%)	148 (0.7%)	148 (0.7%)
Number of PCR tests taken prior to day of first vaccination
0	0 (0.0%)	0 (0.0%)
1	29,368 (53.1%)	55,413 (50.8%)	11,180 (53.5%)	11,180 (53.5%)
2+	25,909 (46.9%)	53,583 (49.2%)	9,710 (46.5%)	9,710 (46.5%)
Elixhauser Comorbidity Score
0	37,708 (68.2%)	79,024 (72.5%)	15,751 (75.4%)	15,751 (75.4%)
1–4	7,977 (14.4%)	14,863 (13.6%)	2,544 (12.2%)	2,544 (12.2%)
5–9	6,528 (11.8%)	10,511 (9.6%)	1,894 (9.1%)	1,894 (9.1%)
10+	3,064 (5.5%)	4,598 (4.2%)	701 (3.4%)	701 (3.4%)
First dose vaccine site
Arizona	2,351 (4.3%)	4,326 (4.0%)	1,114 (5.3%)	1,114 (5.3%)
Florida	7,538 (13.6%)	8,861 (8.1%)	1,786 (8.5%)	1,786 (8.5%)
Mayo Clinic Health System	11,584 (21.0%)	37,290 (34.2%)	5,644 (27.0%)	5,644 (27.0%)
Rochester (Minnesota)	6,009 (10.9%)	26,614 (24.4%)	3,714 (17.8%)	3,714 (17.8%)
Other/not recorded	27,795 (50.3%)	31,905 (29.3%)	8,632 (41.3%)	8,632 (41.3%)
State of primary residence
Arizona	4,722 (8.5%)	10,746 (9.9%)	2,734 (13.1%)	2,734 (13.1%)
Florida	9,228 (16.7%)	10,818 (9.9%)	2,262 (10.8%)	2,262 (10.8%)
Iowa	1,385 (2.5%)	1,373 (1.3%)	127 (0.6%)	127 (0.6%)
Minnesota	29,110 (52.7%)	59,529 (54.6%)	12,067 (57.8%)	12,067 (57.8%)
Wisconsin	8,288 (15.0%)	22,695 (20.8%)	3,595 (17.2%)	3,595 (17.2%)
Other	2,544 (4.6%)	3,835 (3.5%)	109 (0.5%)	109 (0.5%)
Follow-up days since first dose
Minimum	0	0	1	5
25th percentile	180	169	174	181
Median	207	201	197	203
75th percentile	230	237	217	224
Maximum	293	294	278	288

Characteristics correspond to the set of individuals who received at least one dose in each group. To be considered as fully vaccinated per protocol, an individual had to receive two vaccine doses on schedule (25–35 days apart for mRNA-1273 and 18–28 days apart for BNT162b2) and be at risk for infection as of their date of full vaccination (14 days after the second dose). For the matched analysis, only the matched pairs in which both individuals were fully vaccinated per protocol were considered.

Figure 2.

Cohort analysis to compare the incidence rates of symptomatic infection after vaccination with mRNA-1273 versus BNT162b2

(A) The secondary analysis was conducted as a retrospective cohort study. After matching on the basis of demographic and clinical features, we compared the incidence rates of symptomatic infection after full vaccination with mRNA-1273 versus after full vaccination with BNT162b2.

(B) Incidence rate ratios (IRRs) of symptomatic infection for unmatched (top), matched per protocol (PP) (middle), and matched intention-to-treat (ITT) (bottom) analyses. IRRs are shown for days 1–10 after the first dose (baseline estimate), the entire study duration (December 1, 2020 through September 22, 2021), the early epoch (December 1, 2020 through May 31, 2021), and the late epoch (July 1, 2021 through September 22, 2021). The x axis is log-transformed such that, for example, IRRs of 0.5 and 2 are equidistant from the null hypothesis IRR of 1. Error bars represent 95% confidence intervals.

Table 5.

Incidence rates of symptomatic infection in mRNA-1273 and BNT162b2 cohorts

Study design	Time period	mRNA-1273 incidence rate [cases/1,000 at-risk person days] (number of individuals)	BNT162b2 incidence rate [cases/1,000 at-risk person days] (number of individuals)	Incidence rate ratio (95% CI)
Unmatched cohorts (per protocol)	days 1–10 after first dose	126/550,143.0 [0.23] (n = 55,201)	236/1,085,498.0 [0.22] (n = 108,922)	1.1 (0.84, 1.3)
	study duration	498/7,388,249.0 [0.067] (n = 45,534)	1,389/15,342,928.0 [0.091] (n = 92,547)	0.74 (0.67, 0.83)
	early epoch	22/2,508,776.0 [0.0088] (n = 41,279)	76/5,589,957.0 [0.014] (n = 82,615)	0.64 (0.38, 1)
	late epoch	464/3,550,304.0 [0.13] (n = 45,279)	1,279/7,115,229.0 [0.18] (n = 92,074)	0.73 (0.65, 0.81)
Matched cohorts (per protocol)	days 1–10 after first dose	51/207,988.0 [0.25] (n = 20,862)	35/208,403.0 [0.17] (n = 20,882)	1.5 (0.93, 2.3)
	study duration	139/2,470,204 [0.056] (n = 15,392)	215/2,461,985 [0.087] (n = 15,392)	0.64 (0.52, 0.8)
	early epoch	9/799,534.0 [0.011] (n = 14,351)	12/793,719.0 [0.015] (n = 14,389)	0.74 (0.28, 1.9)
	late epoch	125/1,211,346.0 [0.1] (n = 15,329)	197/1,209,463.0 [0.16] (n = 15,310)	0.63 (0.5, 0.8)
Matched cohorts (intention to treat)	days 1–10 after first dose	51/208,643 [0.24] (n = 20,890)	35/208,574.0 [0.17] (n = 20,889)	1.5 (0.93, 2.3)
	study duration	191/3,265,571.0 [0.058] (n = 20,469)	288/3,260,428.0 [0.088] (n = 20,493)	0.66 (0.55, 0.8)
	early epoch	10/1,027,749.0 [0.0097] (n = 18,565)	16/1,025,958.0 [0.016] (n = 18,596)	0.62 (0.25, 1.5)
	late epoch	176/1,637,131.0 [0.11] (n = 20,379)	265/1,633,836.0 [0.16] (n = 20,388)	0.66 (0.54, 0.81)

The “study duration,” “early epoch,” and “late epoch” rows give the incidence rates of breakthrough symptomatic infections (i.e., positive symptomatic tests occurring after the date of full vaccination). The “days 1–10 after first dose” rows serve as a proxy for the baseline rate of symptomatic infection in these cohorts prior to the onset of vaccine effectiveness. Data are shown for three study designs that were tested: an unmatched per-protocol cohort study, a matched per-protocol cohort study, and a matched intention-to-treat cohort study. In the per-protocol designs, the study duration, early epoch, and late epoch rows consider individuals who received two vaccine doses according to the recommended schedule and were at risk for infection 14 days after their second dose. In the intention-to-treat design, these rows consider any individuals who received at least one dose and were at risk for infection 42 or 35 days after the first dose of mRNA-1273 or BNT162b2, respectively.

Matched cohort analysis to compare rates of symptomatic infection

To account for differences in underlying characteristics of the mRNA-1273 and BNT162b2 populations, we derived cohorts of vaccinated individuals (n = 20,890 each) who were matched on the basis of age, sex, race, ethnicity, residential county, site of vaccination, and history of SARS-CoV-2 PCR testing prior to vaccination (see STAR Methods and Figure 2A). There were 15,392 per-protocol matched pairs, in which both individuals received a second dose according to the recommended schedule and were at risk for infection as of their date of full vaccination.

Among these per-protocol pairs, the IR of symptomatic infection was significantly lower after full vaccination with mRNA-1273 than BNT162b2 throughout the study duration (IR_mRNA-1273: 0.056; IR_BNT162b2: 0.087; IRR: 0.64; 95% CI: 0.52–0.80) and during the late epoch (IR_mRNA-1273: 0.10; IR_BNT162b2: 0.16; IRR: 0.63; 95% CI: 0.50–0.80; Table 5; Figure 2B). A more modest trend was observed during the early epoch (IR_mRNA-1273: 0.011; IR_BNT162b2: 0.015; IRR: 0.74; 95% CI: 0.28–1.9), although this analysis was underpowered due to the low number of breakthrough infections in either cohort during this period (9 for mRNA-1273 and 12 for BNT162b2; Table 5; Figure 2B). The IRs of symptomatic infection between the overall matched cohorts were similar during the 10 days after the first dose (IR_mRNA-1273: 0.25; IR_BNT162b2: 0.17; IRR: 1.5; 95% CI: 0.93–2.3; Table 5; Figure 2B), as were the IRs of negative symptomatic tests during each study period (Table S9).

In an intention to treat (ITT)-like analysis, we considered individuals from the overall matched cohorts who were at risk for infection on their expected dates of full vaccination (42 days after the first dose for mRNA-1273 and 35 days after the first dose for BNT162b2), regardless of whether they actually received the second dose according to the recommended schedule. Across the study duration, the IR of symptomatic infection was significantly lower in the mRNA-1273 ITT cohort (IR_mRNA-1273: 0.058; IR_BNT162b2: 0.088; IRR: 0.66; 95% CI: 0.55–0.80; Table 5; Figure 2B). Similar trends were again observed in the early epoch (IR_mRNA-1273: 0.0097; IR_BNT162b2: 0.016; IRR: 0.62; 95% CI: 0.25–1.5) and the late epoch (IR_mRNA-1273: 0.11; IR_BNT162b2: 0.16; IRR: 0.66; 95% CI: 0.54–0.81; Table 5; Figure 2B), although the former was not statistically significant.

Discussion

The occurrence of breakthrough infections and reports of diminished neutralization of SARS-CoV-2 variants by vaccine-elicited sera mandate the continual monitoring of the comparative effectiveness and durability of COVID-19 vaccines.^{10 , 11} Overall, in our study population, both vaccines were highly effective in reducing symptomatic SARS-CoV-2 infection, although their effectiveness was lower after July as compared to earlier months during the vaccine rollout. Throughout the study duration and during the later Delta variant-dominated months, individuals vaccinated with mRNA-1273 were significantly less likely to experience symptomatic breakthrough infections than individuals vaccinated with BNT162b2. This conclusion was concordant between test-negative and cohort study designs, even after controlling for demographic and clinical covariates in each case.

This study further supports the effectiveness of both vaccines in preventing COVID-19 despite the evolution of more transmissible viral variants. It is important to realize that most widely administered non-COVID-19 vaccines are not 100% effective. For example, the estimated effectiveness of seasonal influenza vaccines has ranged from 19% to 60% over the past decade.²⁶ Although COVID-19 mRNA vaccines were initially shown to be more effective than this, the occurrence of breakthrough infections is still expected. The reduced effectiveness of both mRNA-1273 and BNT162b2 coincides with the surge of the Delta variant in the United States, but this temporal association does not imply causality. Previous studies found that BNT162b2 was modestly less effective against the Delta variant than other variants,^{27 , 28} but there are multiple factors in addition to viral genomic evolution that likely impact vaccine effectiveness over time, including waning immune responses and changing guidelines regarding nonpharmaceutical interventions (e.g., social distancing, masking, and travel). Indeed, we and others have recently found that protection against SARS-CoV-2 infection wanes over time after full vaccination with BNT162b2, and a similar trend has been suggested for mRNA-1273.^{16 , 17 ,} 19, 20, 21 ^{, 28 , 29}

Several factors could contribute to the observed differences in effectiveness of mRNA-1273 and BNT162b2. Although both are nucleoside-modified mRNA vaccines encoding the prefusion stabilized SARS-CoV-2 Spike protein, there are differences in the vaccination regimen and formulation.^{30 , 31} BNT162b2 is administered as 30 μg/0.3 mL (100 μg/mL) doses 21 days apart, and mRNA-1273 is administered as 100 μg/0.5 mL (200 μg/mL) doses 28 days apart.^{32 , 33} Assuming similar-sized constructs, this means that each mRNA-1273 dose provides over three times as many copies of Spike protein mRNA as each BNT162b2 dose, which could result in more effective priming of immune responses. Indeed, mRNA-1273 elicits significantly higher antibody titers than BNT162b2 across age groups, and it has been suggested that neutralizing antibody titers correlate with COVID-19 vaccine effectiveness.^{34 , 35} Further, certain adverse effects (e.g., myalgia and arthralgia) were observed more frequently after vaccination with mRNA-1273 than BNT162b2 in their respective clinical trials, and this increased reactogenicity may be paralleled by increased immunogenicity.^{2 , 3} Finally, there are differences in the lipid composition of the nanoparticles used for packaging the mRNA content of mRNA-1273 and BNT162b2. BNT162b2 has a lipid nanoparticle composed of ALC-0315, ALC-0159, distearoylphosphatidylcholine (DSPC), and cholesterol, whereas the lipid nanoparticle of mRNA-1273 is composed of SM-102, polyethylene glycol-dimyristoyl glycerol (PEG-DMG), DSPC, and cholesterol.³⁶

The FDA and Centers for Disease Control and Prevention (CDC) have recently authorized and recommended booster doses for selected populations starting 6 months after their initial series of either BNT162b2 or mRNA-1273.³⁷ Although eligible individuals were initially recommended to receive a homologous booster dose, CDC recommendations now allow for heterologous boosting. Our data are not sufficient to make a statement on the relative effectiveness of homologous versus heterologous booster strategies, but further systematic evaluations of this topic are warranted.

Our observational study suggests that both mRNA-1273 and BNT162b2 continue to strongly protect against symptomatic infection, but there are differences in their real-world effectiveness relative to each other and relative to prior months of the pandemic. We emphasize that individuals should continue to receive any authorized COVID-19 vaccine that is available to them, for both primary series and booster doses in eligible populations. These data and follow-up studies can help to inform future public and global health recommendations regarding optimal vaccine selection and the necessity for booster doses. Overall, this study emphasizes the continued importance of vaccination to curb the COVID-19 pandemic and the need to vigilantly monitor longitudinal and comparative vaccine effectiveness in the coming months.

Limitations of the study

There are some limitations of this study. First, this is a retrospective observational study that is subject to confounding due to the lack of prospective randomization. Second, the analyzed cohorts are not demographically representative of the American population (Tables 1 and S1), which may limit the generalizability of our findings. Similar studies on larger and more diverse populations from various health systems are warranted. Third, although this study accounts for geographic variability via county-level stratification or matching, there could still be residual confounding by geography due to intra-county variability in SARS-CoV-2 exposure or likelihood of vaccination. Fourth, in the cohort design, requiring at least one negative SARS-CoV-2 PCR test at Mayo Clinic before the first vaccine dose could lead to preferential selection of individuals who are more likely to seek out testing, but we do not expect this bias to differentially affect the mRNA-1273 and BNT162b2 cohorts. Fifth, due to the low number of breakthrough infections during early months of the vaccine rollout, the early epoch comparative analyses suffered from low statistical power. Sixth, because we did not sequence the viral genomes isolated from patients in this study, we cannot specifically determine the contribution of the Delta variant to the reduced vaccine effectiveness observed in the later epoch versus other factors, such as waning immunity. Further, it is important to note that comparison of vaccines can be biased due to underlying differences in the populations receiving them, and this bias is only addressed in the present study to the extent that such differences are captured by the covariates considered.³⁸

Finally, we estimated vaccine effectiveness by comparing the odds of infection after full vaccination versus during the first 10 days after the first dose. This method assumes negligible time-varying changes in behaviors or other underlying risks, and it would inherently underestimate true vaccine effectiveness if vaccination does confer some protection in the 10 days after the first dose. We acknowledge that this approach deviates from the conventional method of estimating vaccine effectiveness in a test-negative design (i.e., comparing the odds of infection between vaccinated and unvaccinated individuals). However, we believe that this conventional strategy is even more limited in this case, because we cannot confidently classify individuals as unvaccinated due to the lack of a high-fidelity national vaccination registry in the United States. Contamination of a putative unvaccinated cohort with fully vaccinated individuals would likely lead to an even more substantial underestimation of true vaccine effectiveness.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Code
Python script to assemble study populations for test-negative and cohort study designs	This study	Data S1
Python script to calculate incidence rates and incidence rate ratios for cohort study	This study	Data S2
R script for Elixhauser comorbidity score determination	This study	Data S4
R script for conditional logistic regression analyses	This study	Data S4
Software and algorithms
Python (version 3.9.5)		https://www.python.org/
Python software package: pandas (version 1.3.4)		https://pandas.pydata.org/
Python software package: numpy (version 1.21.4)		https://numpy.org/
Python software package: scipy (version 1.7.2)		https://scipy.org/
R (version 4.1.0)		https://www.r-project.org/
R software package: survival (version 3.2.11)		https://cran.r-project.org/web/packages/survival/index.html
R software package: comorbidity (version 0.5.3)		https://cran.r-project.org/web/packages/comorbidity/

Resource availability

Lead contact

Further information and requests for information should be directed to and will be fulfilled by the lead contact, Venky Soundararajan (venky@nference.net).

Materials availability

This study did not generate new reagents.

Experimental model and subject details

Human Subjects

This study was reviewed and approved by the Mayo Clinic Institutional Review Board (IRB 20-003278) as a minimal risk study. Subjects were excluded if they did not have a research authorization on file. The approved IRB was titled: Study of COVID-19 patient characteristics with augmented curation of Electronic Health Records (EHR) to inform strategic and operational decisions with the Mayo Clinic. The study was deemed exempt by the Mayo Clinic Institutional Review Board and waived from consent. The following resource provides further information on the Mayo Clinic Institutional Review Board and adherence to basic ethical principles underlying the conduct of research, and ensuring that the rights and well-being of potential research subjects are adequately protected: https://www.mayo.edu/research/institutional-review-board/overview.

Method details

Study design, setting, and population

This is a retrospective analysis of individuals who have a record of receiving an mRNA COVID-19 vaccine and at least one SARS-CoV-2 polymerase chain reaction (PCR) test since February 2020 in the Mayo Clinic electronic health record (EHR). This includes individuals receiving care at a Mayo Clinic facility in Arizona, Florida, Iowa, Minnesota, or Wisconsin. There were 268,818 such individuals in total, from whom subsets were selected for the primary and secondary analyses based on the inclusion criteria described below.

The primary analysis was conducted as a test-negative case-control study, and the secondary analysis was conducted as a retrospective cohort study. In both, the exposure was defined by vaccination status (vaccinated with BNT162b2 versus vaccinated with mRNA-1273), and the outcome was symptomatic SARS-CoV-2 infection. Symptomatic SARS-CoV-2 infection was defined as a positive SARS-CoV-2 PCR test which was not designated as “asymptomatic” by the ordering provider (subsequently referred to as “symptomatic tests”).

Primary analysis: test-negative case-control study

Inclusion and exclusion criteria

To derive the population of individuals eligible to contribute cases and controls, inclusion criteria were defined as follows:

• 1.Age greater than or equal to 18 years as of December 1, 2020.
• 2.Fully vaccinated “per protocol” with an mRNA vaccine with the first dose administered on or after December 1, 2020. Per protocol full vaccination was defined as two vaccine doses separated by 25-35 (mRNA-1273) or 18-28 (BNT162b2) days, followed by 14 days.
• 3.At least one clinical encounter at the Mayo Clinic in the three years preceding the study start date (i.e., between December 1, 2017 and December 1, 2020), per the electronic health record.

Exclusion criteria were defined as follows:

• 1.Any positive SARS-CoV-2 PCR test prior to the date of full vaccination.
• 2.Received one or more doses of Ad26.COV2.S (Janssen adenoviral COVID-19 vaccine) prior to the date of full vaccination.

Definition of cases and controls

Cases were defined as the first positive symptomatic test for a given individual; if an individual contributed multiple positive tests, only their first test was included as a case. Controls were defined as negative symptomatic tests in individuals with no prior positive SARS-CoV-2 PCR tests (asymptomatic or symptomatic). Individuals who met the inclusion and exclusion criteria outlined above were eligible to contribute cases and controls from their date of full vaccination until they (i) had any positive test result (symptomatic or asymptomatic), (ii) received a third dose of any COVID-19 vaccine (mRNA-1273, BNT162b2, or Ad26.COV2.S), (iii) died, or (iv) reached the end of the observation period. If an individual contributed a negative symptomatic test 15 or fewer days before a positive test, that negative test was excluded as a possible false negative. If an individual contributed multiple negative symptomatic tests within 15 days of each other, then one of those tests was randomly selected as a control while the others were dropped; this step was taken to avoid counting multiple controls from a potential single symptomatic illness. Further, if an individual contributed more than three negative symptomatic tests over the study duration, then three tests were randomly selected as controls while the others were dropped, as was recently described in a test-negative case-control study of COVID-19 vaccine effectiveness.²⁷

Comparing odds of symptomatic infection between mRNA-1273 and BNT162b2 recipients

We fit a conditional logistic regression model to estimate the odds of symptomatic infection after full vaccination with mRNA-1273 versus BNT162b2, adjusted for relevant covariates. The primary exposure was the vaccine received (mRNA-1273 or BNT162b2). Additional covariates included age at the study start date (modeled as a linear spline with knots at 25, 35, 45, 55, 65, 75, and 85 years), race (Asian, Black/African American, Native American, Native Hawaiian/Pacific Islander, White, other, or unknown), ethnicity (Hispanic/Latino, not Hispanic/Latino, or unknown), sex, Elixhauser Comorbidity Index (ECI) score (bucketed as 0, 1-4, 5-9, or 10+), number of SARS-CoV-2 PCR tests prior to the first vaccine dose (0, 1, or ≥ 2), and vaccination site (Mayo Clinic Health System site, Arizona, Florida, Rochester, or other/unknown). The regression was stratified on three variables: residential county, date of vaccination (in two week calendar intervals starting on December 1, 2020), and date of PCR testing (also in two week calendar intervals starting on December 1, 2020). The CLR model was thus defined by the equation,

$$\begin{array}{l}log\left(\frac{p_{SymptomaticInfection}}{1-p_{SymptomaticInfection}}\right)={\beta }_0+{\beta }_1\left(\text{Vaccine}\right)+{\mathrm{\beta }}_2\left(\text{Age}\right)+{\mathrm{\beta }}_3\left(\text{Race}\right)+{\mathrm{\beta }}_4\left(\text{Ethnicity}\right)+{\mathrm{\beta }}_5\left(\text{Sex}\right)+\\ {\mathrm{\beta }}_6\left(\text{ECI Score}\right)+{\mathrm{\beta }}_7\left(\text{Number of Prior PCR Tests}\right)+{\mathrm{\beta }}_8\left(\text{Vaccination Site}\right)+\\ \text{Strata}\left[\text{Residential County},\text{ Date of Vaccination},\text{ Date of Test}\right]\end{array}$$

As a control, we also calculated the adjusted odds of symptomatic infection at baseline in the mRNA-1273 versus BNT162b2 populations. To do so, we revised the definitions of cases and controls to only consider symptomatic tests performed during the 10 days after the first vaccine dose, during which the protective effect of vaccination is not yet expected to have set in.^{2 , 3}

Sensitivity analyses

• 1.Crude analysis. We compared the odds of symptomatic infection after full vaccination with mRNA-1273 versus BNT162b2 by calculating the crude OR as follows:

$$O{R}_{mRNA-1273/BNT162b2}=\frac{Case{s}_{mRNA-1273}/Control{s}_{mRNA-1273}}{Case{s}_{BNT162b2}/Control{s}_{BNT162b2}}$$

• 2.Time-stratified analysis. The study duration spanned a period during which the dominant SARS-CoV-2 variant in the United States changed from Alpha to Delta. We thus repeated the test-negative analysis described above for two subintervals: an early epoch during which the Alpha variant was most prevalent (December 1, 2020 to May 31, 2021) and a late epoch during which the Delta variant was most prevalent (July 1, 2021 to September 22, 2021).
• 3.Age-stratified analysis. Individuals meeting the inclusion and exclusion criteria were stratified into older (≥65 years old as of December 1, 2020) and younger (< 65 years old as of December 1, 2020) subgroups. We repeated the analysis described above to compare the odds of symptomatic infection after full vaccination with mRNA-1273 versus BNT162b2 in these subgroups.

Estimation of vaccine effectiveness against symptomatic infection

Vaccine effectiveness (VE) is typically assessed in test-negative studies by comparing the odds of infection in vaccinated versus unvaccinated individuals. Because our study population consisted entirely of vaccinated individuals, we used an alternate strategy to estimate VE against symptomatic infection. We considered cases (positive symptomatic tests) and controls (negative symptomatic tests) that were contributed either within 10 days after the first vaccine dose (“baseline interval”) or after the date of full vaccination. We then modified the covariates and stratifying variables in the conditional logistic regression model described above as follows. The “Vaccine” term (mRNA-1273 versus BNT162b2) was replaced by a “Vaccination Status” term with three levels: (i) received first dose of either mRNA-1273 or BNT162b2 in the past 10 days (considered as the reference level), (ii) fully vaccinated with mRNA-1273, or (iii) fully vaccinated with BNT162b2. The “Date of Vaccination” variable was removed from the set of stratifying variables. Thus, the CLR model to estimate VE was defined by the equation:

$$\begin{array}{l}log\left(\frac{p_{SymptomaticInfection}}{1-p_{SymptomaticInfection}}\right)={\mathrm{\beta }}_0+{\mathrm{\beta }}_1\left(\text{Vaccination Status}\right)+{\mathrm{\beta }}_2\left(\text{Age}\right)+{\mathrm{\beta }}_3\left(\text{Race}\right)+{\mathrm{\beta }}_4\left(\text{Ethnicity}\right)+{\mathrm{\beta }}_5\left(\text{Sex}\right)+{\mathrm{\beta }}_6\left(\text{ECI Score}\right)+{\mathrm{\beta }}_7\left(\text{Number of Prior PCR Tests}\right)+{\mathrm{\beta }}_8\left(\text{Vaccination Site}\right)+\\ \text{Strata}\left[\text{County},\text{ Date of Test}\right]\end{array}$$

Secondary analysis: retrospective cohort study

Inclusion and exclusion criteria

Inclusion criteria were defined as follows:

• 1.Age greater than or equal to 18 years as of December 1, 2020.
• 2.Received at least one dose of an mRNA COVID-19 vaccine (BNT162b2 or mRNA-1273) after December 1, 2020 and on or before September 21, 2021.
• 3.Had at least one negative SARS-CoV-2 PCR test on record prior to the first mRNA vaccine dose.

Exclusion criteria were defined as follows:

• 1.Received one or more doses of Ad26.COV2.S prior to the first mRNA vaccine dose.
• 2.Any positive SARS-CoV-2 PCR test on record prior to the first mRNA vaccine dose.

Matching recipients of mRNA-1273 and BNT162b2

There were 55,277 individuals who met the criteria above for mRNA-1273 and 108,996 individuals who met the criteria for BNT162b2. We derived pairs of individuals who were matched on the following demographic and clinical criteria:

• 1.Age (bucketed match). All individuals were classified into one of the following age buckets: 18-24, 25-34, 35-44, 45-54, 55-64, 65-74, 75-84, or 85+ years.
• 2.Sex (exact match).
• 3.Race (exact match). The categories were Asian, Black/African American, Native American, Native Hawaiian/Pacific Islander, White, other, and unknown.
• 4.Ethnicity (exact match). The categories were Hispanic/Latino, not Hispanic/Latino, and unknown.
• 5.County of residence (exact match). This match helps to control for variability in (i) the vaccine rollout process (i.e., timeline and definition of eligible populations), (ii) community transmission patterns, and (iii) the dynamic landscape of SARS-CoV-2 variant prevalence between and within states.
• 6.Vaccination site (exact match). All individuals were classified as having received their vaccine series at Mayo Clinic Rochester (Minnesota), Mayo Clinic Florida, Mayo Clinic Arizona, a Mayo Clinic Health System site, or another/unknown location.
• 7.SARS-CoV-2 PCR testing history (bucketed match). All individuals were classified as having one or multiple SARS-CoV-2 PCR tests before their first vaccine dose. This is intended to control for access to and/or likelihood of seeking out COVID-19 testing, as well as baseline exposure to SARS-CoV-2.
• 8.ECI score (bucketed match). All individuals were classified as having an ECI score of 0, 1-4, 5-9, or 10+ based on the presence of ICD-9 or ICD-10 codes in the five years leading up to the study period.
• 9.Date of vaccination (window match). For a given individual in the mRNA-1273 cohort, the expected second dose date was taken as 28 days after the first dose. For a given individual in the BNT162b2 cohort, the expected second dose date was taken as 21 days after the first dose. For a given mRNA-1273 recipient, one BNT162b2 recipient was selected who met the first eight matching criteria and had an expected second dose date within one week (i.e., 7 days before to 7 days after) of the expected second dose date for the mRNA-1273 recipient. If no BNT162b2 recipients had an expected second dose date within one week of the given mRNA-1273 recipient, then no match was selected, and that mRNA-1273 recipient was excluded from the analysis.

This matching protocol yielded 20,890 matched pairs. There were 15,392 pairs in which both individuals received a second dose according to the recommended schedule (25-35 days after the first dose for mRNA-1273, 18-28 days after the first dose for BNT162b2) and were at risk for infection as of their date of full vaccination. These 15,392 pairs were considered as the matched per-protocol cohorts among which breakthrough infection rates were assessed for the main analysis.

Comparing rates of symptomatic infection between matched cohorts

Incidence rates (IRs) of symptomatic infection were calculated for each cohort by dividing the number of cases by the total number of at-risk person-days and multiplying by 1000.

To compare baseline symptomatic infection rates, we considered only events occurring during the first 10 days after the first vaccine dose, during which vaccination is not yet expected to confer protection against infection.^{2 , 3} Here, each individual contributed at-risk person days from the date of their first dose until (i) they had a positive SARS-CoV-2 PCR test, (ii) they died, (iii) the end of the observation period, or (iv) ten days after their first dose (whichever came first). To compare breakthrough infection rates, we considered only events occurring in the matched per-protocol cohorts after full vaccination was achieved (i.e., 14 or more days after the second vaccine dose).³⁹ Here, each individual contributed at-risk person days from their date of full vaccination until (i) they had a positive SARS-CoV-2 test, (ii) they died, (iii) they received a third dose of any COVID-19 vaccine, or (iv) the end of the observation period (whichever came first). If one member of a matched pair tested positive or died, the other member of the pair continued contributing at-risk time.⁴⁰ We then calculated the incidence rate ratio along with its 95% confidence interval (CI).

Sensitivity analyses

• 1.Unmatched cohort analysis. The matching of mRNA-1273 and BNT162b2 recipients inherently leads to the exclusion of some vaccine recipients from the primary analysis, which could introduce unintended bias. To assess this, we compared the IRs of symptomatic infection in the unmatched mRNA-1273 and BNT162b2 cohorts as described above.
• 2.Intention to treat (ITT)-like analysis. In the primary analysis, we assessed symptomatic infections only among individuals who had received two doses of a given vaccine according to the recommended schedule and were at risk for infection 14 days after the second dose (breakthrough infections), akin to a per-protocol analysis. To perform an ITT-like analysis, we compared the IRs of infections after expected (rather than actual) dates of full vaccination based on the date of the first dose, including individuals who had only received one dose by this time. The expected date of full vaccination was defined as 42 days after the first dose for mRNA-1273 recipients and 35 days after the first dose for BNT162b2 recipients.
• 3.Time-stratified analyses. We repeated the analyses described above for the two subintervals defined previously: the early epoch (December 1, 2020 to May 31, 2021) and the late epoch (July 1, 2021 to September 22, 2021).

Comparing rates of negative symptomatic testing between cohorts

A retrospective cohort study can be confounded by differences in the rates of follow-up between the cohorts. Specifically in this study, the comparison of positive symptomatic SARS-CoV-2 test rates could be confounded by differences in the likelihood of seeking out testing between mRNA-1273 recipients and BNT162b2 recipients. To assess whether such confounding is likely to impact the previous cohort-based analyses, we compared the incidence rates of negative symptomatic tests for each cohort during the study duration, the early epoch, and the late epoch by calculating IRRs and 95% CIs as described above. Individuals were eligible to contribute multiple negative tests, and negative tests which occurred after an individual had tested positive for SARS-CoV-2 were excluded.

Determination of Elixhauser Comorbidity Index (ECI) score

We used the comorbidity package (version 0.5.3) in R (version 4.1.0, http://www.r-project.org, Vienna, Austria) to identify ICD-9 and ICD-10 codes that correspond to each Elixhauser comorbidity. For each individual, we extracted all such diagnosis codes in the Mayo Clinic EHR from the five years preceding this study (i.e., between December 1, 2015 and December 1, 2020). The ECI score was defined as the total number of Elixhauser comorbidities present in at least one record during this five-year period. For subsequent analyses, these values were bucketed as described above (0, 1-4, 5-9, ≥ 10).

Assessing longitudinal prevalence of SARS-CoV-2 variants

Genomic sequence data from the GISAID initiative was used to estimate the longitudinal prevalence of SARS-CoV-2 variants in the states from which study participants were selected (Minnesota, Arizona, Florida, Iowa, and Wisconsin).⁴¹ Specifically, we quantified the weekly prevalence of the Pango lineages corresponding to CDC-labeled variants being monitored (VBMs), variants of interest (VOIs), variants of concern (VOCs), or variants of high consequence (VOHC) in each state. Pango lineages were mapped to these variants based on definitions provided by the Centers for Disease Control and Prevention (CDC) and the World Health Organization (WHO).^{42 , 43} For a given variant, prevalence was calculated as the number of sequences corresponding to that variant deposited in that state during the given week divided by the total number of sequences deposited in that state during the same interval, multiplied by 100. A total of 147,957 SARS-CoV-2 genome sequences collected between December 2020 and September 2021 were included in this analysis. The total number of deposited sequences split by state was as follows: Florida - 68,282; Minnesota - 39,898; Wisconsin −14,989; Arizona - 20,243; Iowa - 4,545.

Quantification and statistical analysis

Primary analysis: test-negative design

Adjusted odds ratios were calculated using conditional logistic regression (CLR). Conditional estimation was applied due to sparsity within strata (i.e., if the strata indicators were instead treated as covariates for unconditional estimation, the degrees of freedom would be high relative to the number of cases). All CLR models were fit using the clogit function from the survival package (version 3.2.11) in R (Version, 4.1.0, http://www.r-project.org, Vienna, Austria). Confidence intervals and tests were based upon the Wald method. Odds were considered significantly different if the confidence interval of the odds ratio (OR) did not include 1.

To estimate vaccine effectiveness (VE) for each vaccine, the CLR model defined by the equation given previously was fit using clogit as described above. The exponentiated coefficients of the “Vaccination Status” term represent the adjusted odds of symptomatic infection after full vaccination with mRNA-1273 or BNT162b2 versus the unprotected (baseline) state. VE for each vaccine was then calculated as 100% × (1 - Adjusted Odds Ratio).

Secondary analysis: retrospective cohort design

The incidence rate ratio (IRR) was calculated as the IR of the matched mRNA-1273 cohort divided by the IR of the matched BNT162b2 cohort. The 95% confidence interval (CI) of the IRR was calculated using the Poisson distribution and test-based methods as described previously.⁴⁴ Specifically, the IRR and 95% CI were defined as follows:

$$IRR=\left(\frac{m}{P{T}_M}\right)/\left(\frac{b}{P{T}_B}\right)\text{, and}$$

$$IRR95\%CI=\left(\frac{P{T}_B}{P{T}_M}\right)\left(\frac{m}{b+1}\right)\left(\frac{1}{F_{0.025,2(b+1),2m}}\right)to\left(\frac{P{T}_B}{P{T}_M}\right)\left(\frac{m+1}{b}\right)\left(\frac{1}{F_{0.025,2(m+1),2b}}\right)\text{, where}$$

m = the number of cases in the matched mRNA-1273 cohort,

b = number of cases in the matched BNT162b2 cohort,

PT_M = at-risk person time contributed by the matched mRNA-1273 cohort,

PT_B = at-risk person time contributed by the matched BNT162b2 cohort.

IRs and IRRs were calculated in Python (version 3.8.8) using the pandas (version 1.3.4), numpy (version 1.21.4), and scipy (version 1.7.2) packages. The IRs of symptomatic infection were considered significantly different between the matched mRNA-1273 and BNT162b2 cohorts if the 95% CI of the IRR did not include 1.

Published: December 11, 2021

Data and code availability

• •Data: The datasets supporting the current study have not been deposited because they contain personally identifiable information from human subjects. This data may be made available from the corresponding author on request. A proposal with a detailed description of the study objectives and statistical analysis plan will be needed to evaluate the reasonability of requests. Deidentified data will be provided after approval from the lead contact and the Mayo Clinic’s standard IRB process for such requests.
• •Code: All original code is available in this paper’s supplemental information as Data S1, S2, S3, and S4. The contents of the individual files are as follows:
• •Data S1: Python script to assemble study populations for test-negative and cohort study designs. Related to Tables 1 and 4.
• •Data S2: Python script to calculate incidence rates and incidence rate ratios for cohort study. Related to Table 5.
• •Data S3: R script for Elixhauser comorbidity score determination. Related to STAR Methods.
• •Data S4: R script for conditional logistic regression analyses. Related to Tables 2 and 3.
• •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.