Effectiveness of Covid-19 Vaccines in Ambulatory and Inpatient Care Settings

Mark G Thompson; Edward Stenehjem; Shaun Grannis; Sarah W Ball; Allison L Naleway; Toan C Ong; Malini B DeSilva; Karthik Natarajan; Catherine H Bozio; Ned Lewis; Kristin Dascomb; Brian E Dixon; Rebecca J Birch; Stephanie A Irving; Suchitra Rao; Elyse Kharbanda; Jungmi Han; Sue Reynolds; Kristin Goddard; Nancy Grisel; William F Fadel; Matthew E Levy; Jill Ferdinands; Bruce Fireman; Julie Arndorfer; Nimish R Valvi; Elizabeth A Rowley; Palak Patel; Ousseny Zerbo; Eric P Griggs; Rachael M Porter; Maria Demarco; Lenee Blanton; Andrea Steffens; Yan Zhuang; Natalie Olson; Michelle Barron; Patricia Shifflett; Stephanie J Schrag; Jennifer R Verani; Alicia Fry; Manjusha Gaglani; Eduardo Azziz-Baumgartner; Nicola P Klein

doi:10.1056/NEJMoa2110362

. 2021 Sep 8:NEJMoa2110362. doi: 10.1056/NEJMoa2110362

Effectiveness of Covid-19 Vaccines in Ambulatory and Inpatient Care Settings

Mark G Thompson ^1,^✉, Edward Stenehjem ¹, Shaun Grannis ¹, Sarah W Ball ¹, Allison L Naleway ¹, Toan C Ong ¹, Malini B DeSilva ¹, Karthik Natarajan ¹, Catherine H Bozio ¹, Ned Lewis ¹, Kristin Dascomb ¹, Brian E Dixon ¹, Rebecca J Birch ¹, Stephanie A Irving ¹, Suchitra Rao ¹, Elyse Kharbanda ¹, Jungmi Han ¹, Sue Reynolds ¹, Kristin Goddard ¹, Nancy Grisel ¹, William F Fadel ¹, Matthew E Levy ¹, Jill Ferdinands ¹, Bruce Fireman ¹, Julie Arndorfer ¹, Nimish R Valvi ¹, Elizabeth A Rowley ¹, Palak Patel ¹, Ousseny Zerbo ¹, Eric P Griggs ¹, Rachael M Porter ¹, Maria Demarco ¹, Lenee Blanton ¹, Andrea Steffens ¹, Yan Zhuang ¹, Natalie Olson ¹, Michelle Barron ¹, Patricia Shifflett ¹, Stephanie J Schrag ¹, Jennifer R Verani ¹, Alicia Fry ¹, Manjusha Gaglani ¹, Eduardo Azziz-Baumgartner ¹, Nicola P Klein ¹

PMCID: PMC8451184 PMID: 34496194

Abstract

Background

There are limited data on the effectiveness of the vaccines against symptomatic coronavirus disease 2019 (Covid-19) currently authorized in the United States with respect to hospitalization, admission to an intensive care unit (ICU), or ambulatory care in an emergency department or urgent care clinic.

Methods

We conducted a study involving adults (≥50 years of age) with Covid-19–like illness who underwent molecular testing for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). We assessed 41,552 admissions to 187 hospitals and 21,522 visits to 221 emergency departments or urgent care clinics during the period from January 1 through June 22, 2021, in multiple states. The patients’ vaccination status was documented in electronic health records and immunization registries. We used a test-negative design to estimate vaccine effectiveness by comparing the odds of a positive test for SARS-CoV-2 infection among vaccinated patients with those among unvaccinated patients. Vaccine effectiveness was adjusted with weights based on propensity-for-vaccination scores and according to age, geographic region, calendar time (days from January 1, 2021, to the index date for each medical visit), and local virus circulation.

Results

The effectiveness of full messenger RNA (mRNA) vaccination (≥14 days after the second dose) was 89% (95% confidence interval [CI], 87 to 91) against laboratory-confirmed SARS-CoV-2 infection leading to hospitalization, 90% (95% CI, 86 to 93) against infection leading to an ICU admission, and 91% (95% CI, 89 to 93) against infection leading to an emergency department or urgent care clinic visit. The effectiveness of full vaccination with respect to a Covid-19–associated hospitalization or emergency department or urgent care clinic visit was similar with the BNT162b2 and mRNA-1273 vaccines and ranged from 81% to 95% among adults 85 years of age or older, persons with chronic medical conditions, and Black or Hispanic adults. The effectiveness of the Ad26.COV2.S vaccine was 68% (95% CI, 50 to 79) against laboratory-confirmed SARS-CoV-2 infection leading to hospitalization and 73% (95% CI, 59 to 82) against infection leading to an emergency department or urgent care clinic visit.

Conclusions

Covid-19 vaccines in the United States were highly effective against SARS-CoV-2 infection requiring hospitalization, ICU admission, or an emergency department or urgent care clinic visit. This vaccine effectiveness extended to populations that are disproportionately affected by SARS-CoV-2 infection. (Funded by the Centers for Disease Control and Prevention.)

Three currently authorized coronavirus disease 2019 (Covid-19) vaccines in the United States were shown to be highly effective in preventing symptomatic Covid-19 in randomized, placebo-controlled phase 3 trials^1-3 and in subsequent observational vaccine-effectiveness studies of messenger RNA (mRNA) Covid-19 vaccines.^4-7 However, less is known about how well these vaccines protect against more severe illness due to severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) resulting in hospitalization, admission to an intensive care unit (ICU), or ambulatory care in an emergency department or urgent care clinic. In addition, estimates of vaccine effectiveness have been limited in populations that have been disproportionately affected by Covid-19, including older adults, persons with chronic medical conditions, and Black or Hispanic populations.^8-14 Data are lacking from real-world estimates of effectiveness of all three Covid-19 vaccines authorized in the United States: BNT162b2 (Pfizer–BioNTech), mRNA-1273 (Moderna), and Ad26.COV2.S (Johnson & Johnson–Janssen).

The Centers for Disease Control and Prevention (CDC), in collaboration with seven U.S. health care systems and research centers with integrated medical, laboratory, and vaccination records, established the VISION Network to assess the effectiveness of Covid-19 vaccines with respect to laboratory-confirmed SARS-CoV-2 infection–associated hospitalizations, ICU admissions, or visits to emergency departments or urgent care clinics from January 1 through June 22, 2021. Here, we report the findings of this assessment according to age group, race or ethnic group, underlying medical condition, and type of Covid-19 vaccine.

Methods

Study Design

The VISION network includes Columbia University Irving Medical Center (CUIMC; New York), HealthPartners Institute (Minnesota and Wisconsin), Intermountain Healthcare (Utah), Kaiser Permanente Northern California (KPNC), Kaiser Permanente Northwest Center for Health Research (KPNW; Oregon and Washington), Regenstrief Institute (Indiana), and the University of Colorado. All seven network partners contributed data on hospitalizations and ICU visits from a total of 187 hospitals; three partners also contributed data on visits to a total of 167 emergency departments and a total of 54 urgent care clinics. The partners categorized their medical facilities into a total of 36 geographic subregions (Table 1, and Table S1 in the Supplementary Appendix, available with the full text of this article at NEJM.org).

Table 1. Health Care Settings, Vaccination Data, and Covid-19 Vaccines Administered, According to Type of Medical Visit.^*.

Network Partner (State, Geographic Subregions)	No. of Hospitals (N=187)	No. of EDs (N=167)	No. of Urgent Care Clinics (N=54)	Vaccine Availability in 2021, According to Age^†	Source of Vaccination Records	Interval before Record Updated	Covid-19 Vaccines Received
CUIMC (New York, 1 subregion)	3	NA	NA	≥75 yr: Jan. 8 ≥65 yr: Jan. 12 ≥60 yr: March 10 ≥50 yr: March 23	New York City registry and EHRs	1 wk	Among hospitalized patients: 71% BNT162b2, 25% mRNA-1273, 4% Ad26.COV2.S
HealthPartners (Minnesota and Wisconsin, 2 subregions)	9	NA	NA	≥85 yr: Jan. 3 ≥75 yr: Jan. 22 ≥65 yr: Feb. 16 ≥50 yr: Feb. 26	Claims data, pharmacy data, EHRs, and Minnesota Immunization Information Connection	6 wk	Among hospitalized patients: 52% BNT162b2, 47% mRNA-1273, 1% Ad26.COV2.S
Intermountain Healthcare (Utah, 8 subregions)	22	21	34	≥75 yr: Jan. 18 ≥65 yr: Feb. 18 ≥50 yr: March 8	Utah Statewide Immunization Information System and EHRs	1 wk	Among hospitalized patients: 54% BNT162b2, 42% mRNA-1273, 4% Ad26.COV2.S;
							Among patients presenting to EDs and urgent care clinics: 53% BNT162b2, 43% mRNA-1273, 5% Ad26.COV2.S
KPNC (California, 8 subregions)	29	NA	NA	≥65 yr: Feb. 28 ≥50 yr: April 2	California Immunization Registry, Care Everywhere (Epic interhospital system for vaccination record sharing), pharmacy data, claims data, and EHRs	2 wk	Among hospitalized patients: 55% BNT162b2, 41% mRNA-1273, 4% Ad26.COV2.S
KPNW (Oregon and Washington, 3 subregions)	11	43	20	Oregon: ≥80 yr: Feb. 8 ≥75 yr: Feb. 15 ≥70 yr: Feb. 22 ≥65 yr: March 1 ≥50 yr: April 19 Washington: ≥65 yr: Jan. 18 ≥60 yr: March 31 ≥50 yr: April 15	Oregon Immunization Information System (ALERT Immunization Information System), Washington State Immunization Information System, claims data, and EHRs	2 wk	Among hospitalized patients: 67% BNT162b2, 28% mRNA-1273, 5% Ad26.COV2.S;
							Among patients presenting to EDs and urgent care clinics: 67% BNT162b2, 28% mRNA-1273, 4% Ad26.COV2.S
Regenstrief Institute (Indiana, 11 subregions)	101	103	NA	≥80 yr: Jan. 8 ≥70 yr: Jan. 13 ≥65 yr: Feb. 1 ≥60 yr: Feb. 23 ≥55 yr: March 2 ≥50 yr: March 3	Children and Hoosier Immunization Registry Program and EHRs	1 wk	Among hospitalized patients: 42% BNT162b2, 55% mRNA-1273, 3% Ad26.COV2.S;
							Among patients presenting to EDs: 43% BNT162b2, 53% mRNA-1273, 5% Ad26.COV2.S
University of Colorado (3 subregions)	12	NA	NA	≥70 yr: Jan. 8 ≥65 yr: Feb. 8 ≥60 yr: March 5 ≥50 yr: March 19	Colorado Immunization Information System and EHRs	2 wk	Among hospitalized patients: 56% BNT162b2, 44% mRNA-1273, 0% Ad26.COV2.S

Open in a new tab

Covid-19 denotes coronavirus disease 2019, CUIMC Columbia University Irving Medical Center, ED emergency department, EHR electronic health record, KPNC Kaiser Permanente Northern California, KPNW Kaiser Permanente Northwest Center for Health Research, and NA not applicable.

^†

Medical visits were reported 14 days after the date of general vaccine availability to community-dwelling adults in each age group, regardless of the patients’ occupation or underlying medical conditions. Dates of general vaccine availability were provided by each network partner according to state guidelines and local vaccine distribution.

A full description of the study methods is available in the Supplementary Appendix. In brief, in a study involving adults (≥50 years of age), we used a test-negative design to assess the effectiveness of Covid-19 vaccines with respect to hospitalization lasting more than 24 hours, ICU admission (as a subset of hospitalization), or an emergency department or urgent care clinic visit associated with laboratory-confirmed SARS-CoV-2 infection and a diagnosis consistent with Covid-19–like illness.^15-17

Covid-19–like illness was defined as a clinical diagnosis of acute respiratory illness (e.g., Covid-19, respiratory failure, or pneumonia) or signs or symptoms (e.g., cough, fever, dyspnea, vomiting, or diarrhea) that have been associated with Covid-19 in previous studies.^12,13,18 We identified Covid-19–like illness using discharge codes (Table S2) from the International Classification of Diseases, 9th and 10th Revisions (ICD) that are based on previous studies of other viral respiratory diseases.^19,20 Data on hospital readmissions within 30 days after discharge, repeat emergency department visits within 24 hours, or repeat visits to urgent care clinics within 24 hours were combined and analyzed as single medical visits within each clinical setting.

The demographic characteristics of the patients and their underlying medical conditions (defined according to the ICD codes that were assigned at the visit) were extracted from medical records. The study protocol, available at NEJM.org, was reviewed and approved by the institutional review boards of the study partners. The study sponsor did not place limitations on publication or require confidentiality in the reporting of results.

SARS-CoV-2 Infection, Covid-19 Vaccination, and Viral Circulation

SARS-CoV-2 infection was confirmed by molecular assays (e.g., real-time reverse-transcriptase–polymerase-chain-reaction [RT-PCR] assays) that had been ordered by clinicians to detect infection that occurred within 14 days before to less than 72 hours after a hospital admission or an emergency department or urgent care clinic visit. Covid-19 vaccination status was documented by state immunization registries (or, in the case of CUIMC, by a city registry), electronic health records, and claims data. The dates when Covid-19 vaccines generally became available locally according to the age group of the recipients (regardless of their occupation or medical conditions) and the time required for vaccination records to be updated were provided by each partner.

As a measure of local SARS-CoV-2 circulation on the day of each medical visit, the 7-day moving average of the percentage of RT-PCR tests that were SARS-CoV-2–positive within each of the 36 geographic subregions was extracted from public health records. Details are provided in the Supplementary Appendix.

Statistical Analysis

The effectiveness of Covid-19 vaccines was assessed with the use of a test-negative design to compare the odds of testing positive for SARS-CoV-2 among vaccinated patients with the odds among unvaccinated patients. The test-negative design is widely used in postlicensure evaluations of vaccine effectiveness and is thought to minimize biases associated with access to vaccination or health care utilization.^15-17 A patient with Covid-19 was defined as a person with at least one ICD code that was consistent with Covid-19–like illness and at least one positive SARS-CoV-2 test result; control patients were defined as persons with at least one ICD code for Covid-19–like illness and only negative SARS-CoV-2 test results.

The index date for each medical visit was defined as either the date of collection of a respiratory specimen associated with the most recent positive or negative SARS-CoV-2 test result before the medical visit or the date of the medical visit (if testing occurred only after the admission or visit date). We included medical visits that had occurred at least 14 days after the local eligibility date for age-specific Covid-19 vaccination (Table 1), so all the patients were assumed to have had an opportunity to be at least partially vaccinated before their index date.

In univariate comparisons of SARS-CoV-2–positive patients with negative controls and of unvaccinated patients with vaccinated patients (≥14 days after the first dose), a standardized mean or proportion difference of more than 0.2 was considered to be noteworthy.²¹ Vaccine effectiveness was estimated with the use of multivariable logistic-regression models adjusted for potential confounding in the associations between vaccination status and laboratory confirmation of SARS-CoV-2 infection through inclusion of a priori covariates and through weights based on propensity-for-vaccination scores. Using established methods for estimating propensity scores within case–control studies,²² we first estimated propensity-for-vaccination scores among test-negative controls and then used the fitted model to calculate propensity-for-vaccination scores for test-positive observations.

Because the test-negative design has also been interpreted as an indirect cohort design¹⁵ and because Covid-19 is not currently a rare outcome, we also used all study observations in a sensitivity analysis to estimate propensity-for-vaccination scores. Each observation was weighted by the inverse of the propensity to be vaccinated (if the patient was vaccinated) or unvaccinated (if patient was not vaccinated). Generalized boosted regression trees were used to estimate the propensity to be vaccinated with a set of explanatory variables including sociodemographic characteristics, underlying medical conditions, and characteristics of the hospital or emergency department or urgent care clinic.²³ Separate weights were calculated for each vaccine-effectiveness model and were truncated at the 99.9 percentile of the distribution of weights within each model.²⁴ Age, geographic subregion, calendar time (days from January 1, 2021, to the index date), and local virus circulation were included both in weight calculations and as covariates in the vaccine-effectiveness regression models.²⁵

In a secondary analysis, vaccine effectiveness was estimated for patients at 14-day intervals after each dose of vaccine.^²⁶ Vaccine effectiveness among patients who received the first dose less than 14 days before the index date should approach zero in an unbiased model, since protective immunity is unlikely immediately after vaccination.⁷

Four sensitivity analyses that were conducted for the pooled estimates of mRNA-based vaccine effectiveness are described in detail in the Supplementary Appendix. First, to examine possible heterogeneity in vaccine effectiveness among health care systems, vaccine effectiveness was stratified according to network partner. Second, to explore possible heterogeneity in vaccine effectiveness according to outcome, vaccine effectiveness models were narrowed to medical visits in which an ICD code for Covid-19–like illness was listed as the first or primary diagnosis code, and according to medical visits that included an ICD code for pneumonia. Third, vaccine effectiveness was estimated after excluding patients with any SARS-CoV-2–positive result on molecular or antigen assays more than 14 days before the index date. Fourth, to examine potential bias introduced by testing ordered by clinicians, we calculated inverse weights that accounted for both the propensity to be vaccinated and the propensity to be tested.

To aid in the interpretation of estimates of vaccine effectiveness, a simulation model was used to assess the influence of possible misclassification of vaccination exposure or outcomes²⁷ (Section S4). All the analyses were conducted with the use of SAS software, version 9.4 (SAS Institute) or R software, version 4.0.2 (R Foundation for Statistical Computing).

Results

Study Sample

A total of 103,199 hospitalizations of patients with Covid-19–like illness who were 50 years of age or older were identified by the seven VISION partners; of these hospitalizations, 64,400 (62%) occurred after the dates of age-specific Covid-19 vaccine eligibility and the time required for vaccination records to be updated (Table S3). The hospitalizations occurred during the period from January 1 through June 22, 2021. Among unvaccinated patients who were hospitalized, the median duration from vaccine eligibility to the index date was 39 days (interquartile range, 16 to 70) (Table S4). SARS-CoV-2 testing with a molecular assay ordered by clinicians was conducted for 74% of the patients who were hospitalized (range across network partners, 55 to 99). During the period from January 1 through June 22, a total of 121,709 visits to emergency departments or urgent care clinics for Covid-19–like illness were identified by three partners; 76,220 visits (63%) occurred after vaccine age eligibility and updates to vaccination records (Table S5). Among the patients who visited an emergency department or urgent care clinic, the median duration from vaccine eligibility to the index date was 39 days (interquartile range, 15 to 70); 30% (range, 25 to 41) of these patients were tested by means of molecular assay. Across the partners, 1872 hospitalizations and 1350 emergency department or urgent care clinic visits were excluded because the index dates occurred 1 to 13 days after the patient received the first dose of Covid-19 vaccine and immunity was considered indeterminant.

Our analytic sample included 41,552 hospitalizations and 21,522 emergency department or urgent care clinic visits; 3% of the hospitalizations and 14% of the emergency department or urgent care clinic visits were repeat medical visits by the same patient (Table 2). Characteristics of the patients are listed in Table 2, and characteristics of the patients according to network partner are provided in Tables S6 through S11. The median age was 74 years (interquartile range, 66 to 82) among hospitalized patients and 70 years (interquartile range, 61 to 78) among those who visited an emergency department or urgent care clinic. Black patients and Hispanic patients accounted for a larger percentage of medical visits in the hospitalization sample (9% and 11%, respectively) than in the emergency department or urgent care sample (4% and 5%); these findings reflect in part the differing demographic characteristics of the network partners that contributed data on emergency department or urgent care clinic visits. The percentage of patients with underlying medical conditions was higher among hospitalized patients than among those who visited an emergency department or urgent care clinic.

Table 2. Characteristics of the Patients According to SARS-CoV-2 Test Results and Vaccination Status.^*.

Characteristic	All Visits	Assay for SARS-CoV-2		Standardized Mean Difference^†	Vaccination Status^‡					Standardized Mean Difference^§
		Negative	Positive		Unvaccinated	Partial, 1 Dose of mRNA Vaccine	Partial, 2 Doses of mRNA Vaccine	Full, 2 Doses of mRNA Vaccine	Full, Ad26.COV2.S Vaccine
	number (percent of all visits)				number (percent of all visits)
Hospitalization
All hospitalizations	41,552 (100)	37,231 (90)	4321 (10)	−0.13	20,406 (49)	3083 (7)	2482 (6)	14,874 (36)	707 (2)	0.05
First	40,367 (97)	36,096 (89)	4271 (11)		19,903 (49)	3003 (7)	2456 (6)	14,361 (36)	644 (2)
Repeat	1,185 (3)	1,135 (96)	50 (4)		503 (42)	80 (7)	26 (2)	513 (43)	63 (5)
Partner				0.56						0.78
CUIMC	3,126 (8)	2,458 (79)	668 (21)		2,181 (70)	216 (7)	103 (3)	588 (19)	38 (1)
HealthPartners	989 (2)	928 (94)	61 (6)		389 (39)	98 (10)	104 (11)	390 (39)	8 (1)
Intermountain Healthcare	3,910 (9)	3,353 (86)	557 (14)		1,917 (49)	392 (10)	278 (7)	1,236 (32)	87 (2)
KPNC	13,824 (33)	13,159 (95)	665 (5)		3,559 (26)	1021 (7)	980 (7)	7,874 (57)	390 (3)
KPNW	1,952 (5)	1,834 (94)	118 (6)		906 (46)	210 (11)	132 (7)	651 (33)	53 (3)
Regenstrief Institute	12,437 (30)	10,776 (87)	1661 (13)		7,593 (61)	925 (7)	674 (5)	3,114 (25)	131 (1)
University of Colorado	5,314 (13)	4,723 (89)	591 (11)		3,861 (73)	221 (4)	211 (4)	1,021 (19)	0
Age group				0.26						0.34
50–64 yr	8,764 (21)	7,436 (85)	1328 (15)		5,532 (63)	585 (7)	467 (5)	1,898 (22)	282 (3)
65–74 yr	13,158 (32)	11,839 (90)	1319 (10)		6,681 (51)	1014 (8)	795 (6)	4,481 (34)	187 (1)
75–84 yr	12,282 (30)	11,193 (91)	1089 (9)		5,233 (43)	935 (8)	772 (6)	5,189 (42)	153 (1)
≥85 yr	7,348 (18)	6,763 (92)	585 (8)		2,960 (40)	549 (7)	448 (6)	3,306 (45)	85 (1)
ICU admission				0.01						−0.10
Yes	7,394 (18)	6,606 (89)	788 (11)		4,024 (54)	512 (7)	388 (5)	2,359 (32)	111 (2)
No	34,158 (82)	30,625 (90)	3533 (10)		16,382 (48)	2571 (8)	2094 (6)	12,515 (37)	596 (2)
Covid-19 vaccine				0.95						NA
Unvaccinated	20,406 (49)	16,711 (82)	3695 (18)		20,406 (100)
BNT162b2	11,292 (27)	10,932 (97)	360 (3)			1444 (13)	1348 (12)	8,500 (75)
mRNA-1273	9,147 (22)	8,911 (97)	236 (3)			1639 (18)	1134 (12)	6,374 (70)
Ad26.COV2.S	707 (2)	677 (96)	30 (4)						707 (100)
ED or urgent care visit
All ED or urgent care visits	21,522 (100)	18,271 (85)	3251 (15)	0.09	11,812 (55)	1920 (9)	1269 (6)	6,065 (28)	456 (2)	0.00
First	18,537 (86)	15,822 (85)	2715 (15)		10,190 (55)	1688 (9)	1088 (6)	5,181 (28)	390 (2)
Repeat	2,985 (14)	2,449 (82)	536 (18)		1,622 (54)	232 (8)	181 (6)	884 (30)	66 (2)
Medical setting				0.00						0.14
ED	18,375 (85)	15,598 (85)	2777 (15)		10,351 (56)	1582 (9)	1069 (6)	5,000 (27)	373 (2)
Urgent care clinic	3,147 (15)	2,673 (85)	474 (15)		1,461 (46)	338 (11)	200 (6)	1,065 (34)	83 (3)
Partner				0.31						0.33
Intermountain Healthcare	8,993 (42)	7,600 (85)	1393 (15)		4,612 (51)	908 (10)	557 (6)	2,700 (30)	216 (2)
KPNW	4,682 (22)	4,277 (91)	405 (9)		2,100 (45)	440 (9)	327 (7)	1,702 (36)	113 (2)
Regenstrief Institute	7,847 (36)	6,394 (81)	1453 (19)		5,100 (65)	572 (7)	385 (5)	1,663 (21)	127 (2)
Age group				0.35						0.41
50–64 yr	7,352 (34)	5,819 (79)	1533 (21)		4,908 (67)	547 (7)	325 (4)	1,336 (18)	236 (3)
65–74 yr	6,569 (31)	5,616 (85)	953 (15)		3,518 (54)	611 (9)	403 (6)	1,915 (29)	122 (2)
75–84 yr	5,235 (24)	4,656 (89)	579 (11)		2,359 (45)	539 (10)	369 (7)	1,893 (36)	75 (1)
≥85 yr	2,366 (11)	2,180 (92)	186 (8)		1,027 (43)	223 (9)	172 (7)	921 (39)	23 (1)
Covid-19 vaccine				0.93						NA
Unvaccinated	11,812 (55)	8,965 (76)	2847 (24)		11,812 (100)	0	0	0	0
BNT162b2	5,212 (24)	4,988 (96)	224 (4)		0	912 (17)	711 (14)	3,589 (69)	0
mRNA-1273	4,042 (19)	3,891 (96)	151 (4)		0	1008 (25)	558 (14)	2,476 (61)	0
Ad26.COV2.S	456 (2)	427 (94)	29 (6)		0	0	0	0	456 (100)

Open in a new tab

Percentages may not total 100 because of rounding. All patients with indeterminate immunization status (receipt of the first dose of mRNA-based vaccine 1 to 13 days before the index date for the medical visit) were excluded. ED denotes emergency department, ICU intensive care unit, and mRNA messenger RNA.

^†

The standardized mean difference is the difference between the number of uninfected patients and the number of infected patients

^‡

^§

The standardized mean difference is the difference between the number of unvaccinated patients and the number of patients who received partial vaccination (first or second dose) or full vaccination.

Covid-19–Associated Medical Care

We identified 4321 patients with Covid-19 who had laboratory-confirmed SARS-CoV-2 infection among 41,552 patients who were hospitalized (10%; range across network partners, 5 to 21); the remaining 37,231 hospitalized patients (90%) had discharge codes for Covid-19–like illness but were SARS-CoV-2–negative. Laboratory-confirmed SARS-CoV-2 infection was identified in 3251 of 21,522 patients who visited an emergency department or urgent care clinic (15%; range across network partners, 9 to 19); the remaining 18,271 patients who visited an emergency department or urgent care clinic (85%) were SARS-CoV-2–negative (Table 2). The percentage of SARS-CoV-2–positive patients also varied among network partners (Tables S12 and S13).

The percentage of patients with laboratory-confirmed SARS-CoV-2 infection decreased with age among hospitalized patients and among those with emergency department or urgent care clinic visits. In both care settings, the percentage of infected patients was higher among unvaccinated patients and lower among White patients, non-Hispanic patients, and those with chronic nonrespiratory conditions. The numbers of both SARS-CoV-2–positive patients and SARS-CoV-2–negative patients with medical visits on each day are provided in Figures S1 through S10.

Covid-19 Vaccination Status

On the index date, unvaccinated patients composed approximately half the patients who were hospitalized (49%; range across network partners, 26 to 73) or visited an emergency department or urgent care clinic (55%; range, 45 to 65) (Table 2). In both samples, the largest differences between vaccinated and unvaccinated patients were age, network partner, calendar time, and local SARS-CoV-2 circulation on the index date. These same differences were noted when the sample was limited to SARS-CoV-2–positive patients only (Tables S14 and S15). As described in the Supplementary Appendix, the application of inverse propensity-to-be-vaccinated weighting reduced the differences between vaccinated and unvaccinated patients with respect to these factors and other patient characteristics to a standard mean difference of less than 0.2.

Among vaccinated patients, 53.4% of those who were hospitalized and 53.7% of those who visited an emergency department or urgent care clinic had received the BNT162b2 vaccine, 43.3% and 41.6%, respectively, had received the mRNA-1273 vaccine, and 3.3% and 4.7%, respectively, had received the Ad26.COV2.S vaccine. The median days from full vaccination to the index date were similar with the three types of Covid-19 vaccines and with both samples (hospitalization and emergency department or urgent care clinic) (range, 42 to 53). Among the patients who received the BNT162b2 vaccine, the median duration from partial vaccination (one dose) to the index date of hospitalization was 21 days and the median duration from partial vaccination to the index date of an emergency department or urgent care visit was 20 days; among patients who received the mRNA-1273 vaccine, these durations were 26 days and 24 days, respectively. These findings reflected the different dosing schedules of these vaccines.

mRNA-Based Vaccine and Hospitalization

Among adults who were 50 years of age or older, the effectiveness of full two-dose mRNA-based vaccination (≥14 days after the second dose) was 89% (95% confidence interval [CI], 87 to 91) against laboratory-confirmed SARS-CoV-2 infection leading to hospitalization; the vaccine-effectiveness point estimates were similar (differences, ≤5 percentage points) with the BNT162b2 and mRNA-1273 vaccines (Figure 1 and Figure 2). The effectiveness of full mRNA-based vaccination was 83% (95% CI, 77 to 87) among patients who were at least 85 years of age, 86% (95% CI, 75 to 92) among Black patients, 90% (95% CI, 85 to 93) among Hispanic patients, 90% (95% CI, 88 to 92) among patients with chronic respiratory conditions, and 88% (95% CI, 86 to 90) among patients with chronic nonrespiratory conditions (Figure 2). When the hospital sample was limited to 7283 admissions to an ICU, the effectiveness of full mRNA-based vaccination against laboratory-confirmed SARS-CoV-2 infection leading to ICU admission was 90% (95% CI, 86 to 93) (Table S16).

Patients who were partially vaccinated with one dose of a messenger RNA (mRNA)–based vaccine received the first dose at least 14 days before the index date for the medical visit and had not received the second dose by the index date. Patients who were partially vaccinated with two doses of an mRNA-based vaccine received the second dose 1 to 13 days before the index date. Fully vaccinated patients received a single dose of the Ad26.COV2.S vaccine or the second dose of an mRNA-based vaccine at least 14 days before the index date. CI denotes confidence interval, and SARS-CoV-2 severe acute respiratory syndrome coronavirus 2.

Patients who were partially vaccinated with one dose of mRNA-based vaccine received the first dose at least 14 days before the index date and had not received the second dose by the index date. Patients who were partially vaccinated with two doses of mRNA-based vaccine received the second dose 1 to 13 days before the index date. Among patients who received an mRNA-based vaccine, the effectiveness of partial one-dose vaccination (≥14 days after the first dose, but without the second dose) was 54% (95% CI, 47 to 61) against SARS-CoV-2 infection leading to hospitalization, and the effectiveness of partial two-dose vaccination (1 to 13 days after the second dose) was 73% (95% CI, 66% to 79). With both the BNT162b2 and mRNA-1273 vaccines, the effectiveness of full vaccination with respect to Covid-19–associated hospitalization was higher than that of partial vaccination (first dose) (with 95% confidence intervals that did not overlap) (Figure 1). A similar pattern of higher vaccine-effectiveness point estimates for full mRNA-based vaccination than for partial mRNA-based vaccination was noted in all stratified analyses (Table S17). The effectiveness after partial vaccination (first dose) was lower with BNT162b2 than with mRNA-1273 (Figure 1).

The estimates of the effectiveness of full mRNA-based vaccination were similar when stratified according to the six network partners that contributed the most data on hospitalizations (range, 82 to 97%); however, heterogeneity was observed among the partners in the estimates of effectiveness of partial vaccination (first dose). Vaccine effectiveness also remained consistent in the other sensitivity analyses (Section S5). Our simulation model suggested that if both misclassification of outcome and of exposure occur, vaccine effectiveness could be underestimated by as much as 10 percentage points, given the rates of clinical testing, percent positivity, and vaccination coverage observed in our hospitalization sample.

In secondary analyses, we stratified mRNA-based vaccine exposure according to 14-day intervals after administration (Figure 3) and according to type of vaccine (Table S18). Vaccine effectiveness with respect to Covid-19–associated hospitalization was null 0 to 13 days after the first dose, and vaccine-effectiveness point estimates increased through 55 days after the first dose. Vaccine-effectiveness point estimates for full mRNA-based vaccination remained consistently high (>80%) through at least 112 days after the second dose.

The total number of hospitalizations shown is higher than the total number in the main analysis because this secondary analysis was conducted weeks after the main analysis and incorporated updated information from vaccination records and registries; specifically, an additional 212 hospitalizations among unvaccinated patients and 831 hospitalizations among vaccinated patients with confirmed vaccination status were included.

mRNA-Based Vaccine and Emergency Department and Urgent Care Visits

The effectiveness of full two-dose mRNA-based vaccination was 91% (95% CI, 89 to 93) against laboratory-confirmed SARS-CoV-2 infection leading to emergency department or urgent care clinic visits (Figure 4); the vaccine-effectiveness point estimates were similar (3 percentage points) with the BNT162b2 and mRNA-1273 vaccines (Figure 1). The effectiveness of full mRNA-based vaccination was 84% (95% CI, 73 to 91) among adults who were 85 years of age or older, 95% (95% CI, 84 to 98) among Black patients, 81% (95% CI, 70 to 88) among Hispanic patients, and 90% (95% CI, 86 to 93) and 90% (95% CI, 87 to 92) among patients with chronic respiratory conditions and those with chronic nonrespiratory conditions, respectively (Figure 4). The effectiveness of partial (one-dose) mRNA-based vaccination (both types) against SARS-CoV-2 infection leading to emergency department or urgent care clinic visits was 68% (95% CI, 61 to 74), and the effectiveness of partial (two-dose) vaccination was 80% (95% CI, 73 to 85) (Table S19). With both the BNT162b2 and mRNA-1273 vaccines, the effectiveness of full vaccination against SARS-CoV-2 infection leading to emergency department or urgent care clinic visits was higher than the effectiveness with partial vaccination (one dose) (Figure 1).

In sensitivity analyses, vaccine-effectiveness point estimates for full mRNA-based vaccination against SARS-CoV-2 infection leading to emergency department or urgent care clinic visits ranged from 89 to 97% across the three network partners. Estimates of vaccine effectiveness also remained consistent in other sensitivity analyses (Section S5).

In secondary analyses, vaccine effectiveness against SARS-CoV-2 infection leading to emergency department or urgent care clinic visits was null 0 to 13 days after the first dose, and then vaccine-effectiveness point estimates increased through 55 days after the first dose. Vaccine-effectiveness point estimates for full mRNA-based vaccination remained consistently high (≥86%) through at least 112 days after the second dose (Figure 3). Estimates of effectiveness according to the type of Covid-19 vaccine are provided in Table S20.

Effectiveness of Ad26.COV2.S Vaccine

Estimates of the effectiveness of Ad26.COV2.S vaccine were limited to five network partners with Ad26.COV2.S vaccine recipients (CUIMC, Intermountain Healthcare, KPNC, KPNW, and Regenstrief Institute). These analyses included 11,468 hospitalizations and 8917 emergency department or urgent care clinic visits that occurred after the index date for the first patient who was fully vaccinated with Ad26.COV2.S for each network partner (Figure 1). The effectiveness of the full one-dose Ad26.COV2.S vaccine was 68% (95% CI, 50 to 79) with respect to Covid-19–associated hospitalization; the effectiveness of full vaccination against SARS-CoV-2 infection leading to emergency department or urgent care clinic visits was 73% (95% CI, 59 to 82) (Figure 1).

Discussion

In this multistate analysis of more than 63,000 medical visits (from January 1 through June 22, 2021) in the VISION network, among patients who were at least 50 years of age, who presented with Covid-19–like illness, and who were clinically tested for SARS-CoV-2 infection with a molecular assay, estimates of vaccine effectiveness against symptomatic Covid-19 were consistently high in a wide range of medical care settings, from emergency department or urgent care clinics to ICUs. Specifically, after adjustment for age, geographic region, calendar time, local virus circulation, and the propensity to be vaccinated, the effectiveness of mRNA-based vaccines was 88% against a SARS-CoV-2 infection leading to hospitalization, 90% against infection leading to ICU admission, and 91% against infection leading to an emergency department or urgent care clinic visit. Our findings complement those in other recent studies of the effectiveness of mRNA-based vaccines with respect to Covid-19–associated hospitalization^6,7 and expand our knowledge of the clinical protection offered by Covid-19 vaccines in at least four ways.

First, mRNA-based vaccines were highly effective among adults who were 85 years of age or older and persons with chronic medical conditions. This finding is reassuring, given that these groups can have impaired immune responses to other vaccines^28-30 and are most at risk for severe and prolonged manifestations of Covid-19 and death.^8,9,13,14 Second, in Black adults and Hispanic adults, who have been disproportionately affected by Covid-19, mRNA-based vaccines were similarly effective with respect to Covid-19–associated hospitalization and an emergency department or urgent care clinic visit.^9-11,14,31 Third, the effectiveness of full mRNA-based vaccination remained consistently high at least until 112 days after the second dose, which was the longest interval since vaccination during our study period. Fourth, we examined the effectiveness of all three authorized Covid-19 vaccines in the United States.

This study complements other published estimates of the effectiveness of the BNT162b2 vaccine,^{5,6,^32,33} and it confirms that the effectiveness of full vaccination with mRNA-1273 is very similar to that of BNT162b2 with respect to both hospital and emergency department or urgent care outcomes. Among adults 50 years of age or older, the effectiveness of the Ad26.COV2.S vaccine was 73% with respect to a Covid-19–associated emergency department or urgent care visit and 68% with respect to Covid-19–associated hospitalization; these estimates were lower than those for full vaccination with the mRNA-based vaccines, but, pending further research, this finding should be interpreted with caution because of the relatively small number of observations.

The effectiveness of partial (one-dose) mRNA-based vaccination (≥14 days after the first dose but before the second dose) was 54% against SARS-CoV-2 infection leading to hospitalization and 68% against SARS-CoV-2 infection leading to emergency department or urgent care clinic visits; these vaccine-effectiveness point estimates are similar to recent estimates^6,33 of vaccine effectiveness against medically attended Covid-19 and at the lower end of the wide range of reported vaccine effectiveness against symptomatic Covid-19 (50% to 80%).^1,2,5,32,34 These vaccine-effectiveness point estimates, and in particular the partial (one-dose) effectiveness of the BNT162b2 vaccine, should be interpreted with caution, given the heterogeneity in estimates among partners and because vaccine-effectiveness estimates are probably dependent on the observation period; these point estimates for both hospitalization and emergency department or urgent care clinic visits were lowest 14 to 27 days after the first dose and increased through 55 days after the first dose. Nonetheless, during our study period, the effectiveness of full two-dose mRNA-based vaccination was consistently higher than the effectiveness of partial (one-dose) vaccination; this finding reinforces the importance of completing vaccination with both doses of mRNA-based vaccines.

Our study relied on a test-negative design in which the study patients were those who sought care for the clinical syndrome of Covid-19–like illness. This design minimizes bias arising from differences in health care–seeking behavior between vaccinated and unvaccinated patients and is analogous to an indirect cohort design.¹⁵ If certain assumptions are met, the estimate of vaccine effectiveness against symptomatic Covid-19 can be generalized to the source population of adults in the communities served by the participating emergency departments or urgent care clinics and hospital facilities in the study sample. The incidence of Covid-19–like illness that was unrelated to Covid-19 (i.e., in SARS-CoV-2–negative patients) should not vary according to Covid-19 vaccination status. This assumption could be violated if the effect of Covid-19 vaccines on the risk of infection with other pathogens or if differences in disease among SARS-CoV-2–negative patients according to vaccination status are not sufficiently adjusted in the statistical model.

In addition, vaccine effectiveness should be the same among patients who would seek and receive medical attention and those who would not, given similar Covid-19 severity. Under these assumptions, our estimates can be interpreted as the effectiveness of the vaccine for prevention of SARS-CoV-2 infection that is severe enough for medical attention among patients who would seek and have access to care if Covid-19–like illness developed. These assumptions, although plausible, are not directly testable with the study data but are consistent with the assumptions in studies showing the similarity between estimates derived from the test-negative designs and those derived from randomized clinical trials³⁵ and from simulations of cohort studies.¹⁷

Among the strengths of the study are the large number of medical visits examined in a network of 187 hospitals, 167 emergency departments, and 54 urgent care clinics across multiple health care systems and geographic regions of the United States. In addition, a common protocol was used to identify diagnoses consistent with Covid-19–like illness and to detect SARS-CoV-2 infection with highly sensitive and specific molecular assays. The type of Covid-19 vaccines and vaccination dates were documented in electronic health records for vaccines administered within the health care system and by state and city immunization registries for vaccines administered in pharmacies, public health facilities, and other health care settings. Our estimation of vaccine effectiveness stratified according to age, race or ethnic group, and chronic medical conditions allowed us to examine vaccine effectiveness in populations with a high Covid-19 burden and historically limited participation in vaccine trials,^36,37 while at the same time reducing bias that may be introduced because health care–seeking behavior and vaccination uptake may vary according to sociodemographic groups.

Our study is subject to at least five limitations. First, although inverse weights balanced vaccinated and unvaccinated medical visits on sociodemographic and health characteristics, and we further adjusted for age, geographic region, calendar time, and local virus circulation, unmeasured and residual confounding may have biased our estimates. For example, we did not know the occupations of the patients, which is associated with exposure to virus and access to and use of vaccination and personal protective equipment. However, we attempted to minimize this bias by limiting the observation period to weeks after Covid-19 vaccines became available to the general public, regardless of occupation or chronic medical conditions. Second, the percentage of patients who were clinically tested for SARS-CoV-2 by molecular assay differed across network partners and clinical settings, and vaccine-effectiveness estimates can be biased if clinicians make testing decisions based on vaccination status.^38,39

Third, our simulation models suggest that misclassification of vaccine exposures or outcomes could bias our vaccine-effectiveness estimates downward.²⁷ Fourth, the circulation of SARS-CoV-2 variants of concern increased during the study period, and we were unable to evaluate whether vaccine effectiveness differed according to SARS-CoV-2 lineage. Nonetheless, most of our observations occurred during months when SARS-CoV-2 variants of concern — and, in particular, the B.1.1.7 variant (alpha) — were predominant viruses in the geographic regions covered by the network. To the extent that potential biases (by testing practices, vaccine coverage, and virus circulation) may have varied across study partners, it was reassuring to find in stratified models across partners similar effectiveness of full mRNA-based vaccination with respect to Covid-19 hospitalization (87 to 91%) and emergency department or urgent care clinic visits (89 to 92%) (Figure 1). Finally, although the consistency of vaccine effectiveness across different levels of medical care that we observed is not indicative of vaccine attenuation of disease severity, to the extent that Covid-19 vaccines may attenuate viral shedding and reduce virus detection among breakthrough infections,⁴⁰ this would result in overestimation of vaccine effectiveness against SARS-CoV-2 infection.

Our analysis of medical visits during a 6-month period showed that mRNA-based vaccines, when fully administered, are highly effective against laboratory-confirmed SARS-CoV-2 infection leading to hospitalization, ICU admission, and emergency department or urgent care clinic visits among adults 50 years of age or older, including those most at risk for severe Covid-19 because of advanced age, underlying medical conditions, or race or ethnic group. Early findings suggest that the Ad26.COV2.S vaccine also has high effectiveness, although additional data are needed. These findings reinforce the value of widespread Covid-19 vaccination, underscore the importance of completing vaccination for both mRNA-based vaccines, and may help to motivate persons who remain hesitant to be vaccinated, including Black adults and Hispanic adults.

Protocol

Click here for additional data file.^{(1.3MB, pdf)}

Supplementary Appendix

Click here for additional data file.^{(6.6MB, pdf)}

Disclosure Forms

Click here for additional data file.^{(816.3KB, pdf)}

Data Sharing Statement

Click here for additional data file.^{(69.5KB, pdf)}

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

This article was published on September 8, 2021, at NEJM.org.

A data sharing statement provided by the authors is available with the full text of this article at NEJM.org.

Footnotes

Supported by the Centers for Disease Control and Prevention (contracts 75D30120C07986 to Westat and 75D30120C07765 to Kaiser Foundation Hospitals).

Disclosure forms provided by the authors are available with the full text of this article at NEJM.org.

References

1.Baden LR, El Sahly HM, Essink B, et al. Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine. N Engl J Med 2021;384:403-416. [DOI] [PMC free article] [PubMed] [Google Scholar]
2.Polack FP, Thomas SJ, Kitchin N, et al. Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine. N Engl J Med 2020;383:2603-2615. [DOI] [PMC free article] [PubMed] [Google Scholar]
3.Sadoff J, Gray G, Vandebosch A, et al. Safety and efficacy of single-dose Ad26.COV2.S vaccine against Covid-19. N Engl J Med 2021;384:2187-2201. [DOI] [PMC free article] [PubMed] [Google Scholar]
4.Thompson MG, Burgess JL, Naleway AL, et al. Interim estimates of vaccine effectiveness of BNT162b2 and mRNA-1273 COVID-19 vaccines in preventing SARS-CoV-2 infection among health care personnel, first responders, and other essential and frontline workers — eight U.S. locations, December 2020–March 2021. MMWR Morb Mortal Wkly Rep 2021;70:495-500. [DOI] [PMC free article] [PubMed] [Google Scholar]
5.Hall VJ, Foulkes S, Saei A, et al. COVID-19 vaccine coverage in health-care workers in England and effectiveness of BNT162b2 mRNA vaccine against infection (SIREN): a prospective, multicentre, cohort study. Lancet 2021;397:1725-1735. [DOI] [PMC free article] [PubMed] [Google Scholar]
6.Dagan N, Barda N, Kepten E, et al. BNT162b2 mRNA Covid-19 vaccine in a nationwide mass vaccination setting. N Engl J Med 2021;384:1412-1423. [DOI] [PMC free article] [PubMed] [Google Scholar]
7.Tenforde MW, Olson SM, Self WH, et al. Effectiveness of Pfizer-BioNTech and Moderna vaccines against COVID-19 among hospitalized adults aged ≥65 years — United States, January–March 2021. MMWR Morb Mortal Wkly Rep 2021;70:674-679. [DOI] [PMC free article] [PubMed] [Google Scholar]
8.Stokes EK, Zambrano LD, Anderson KN, et al. Coronavirus Disease 2019 Case Surveillance — United States, January 22–May 30, 2020. MMWR Morb Mortal Wkly Rep 2020;69:759-765. [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Killerby ME, Link-Gelles R, Haight SC, et al. Characteristics associated with hospitalization among patients with COVID-19 — metropolitan Atlanta, Georgia, March–April 2020. MMWR Morb Mortal Wkly Rep 2020;69:790-794. [DOI] [PMC free article] [PubMed] [Google Scholar]
10.Gu T, Mack JA, Salvatore M, et al. Characteristics asociated with racial/ethnic disparities in COVID-19 outcomes in an academic health care system. JAMA Netw Open 2020;3(10):e2025197-e2025197. [DOI] [PMC free article] [PubMed] [Google Scholar]
11.Price-Haywood EG, Burton J, Fort D, Seoane L. Hospitalization and mortality among Black patients and White patients with Covid-19. N Engl J Med 2020;382:2534-2543.. [DOI] [PMC free article] [PubMed] [Google Scholar]
12.Garg S, Kim L, Whitaker M, et al. Hospitalization rates and characteristics of patients hospitalized with laboratory-confirmed Coronavirus disease 2019 — COVID-NET, 14 states, March 1-30, 2020. MMWR Morb Mortal Wkly Rep 2020;69:458-464. [DOI] [PMC free article] [PubMed] [Google Scholar]
13.Kim L, Garg S, O’Halloran A, et al. Risk factors for intensive care unit admission and in-hospital mortality among hospitalized adults identified through the US Coronavirus Disease 2019 (COVID-19)–Associated Hospitalization Surveillance Network (COVID-NET). Clin Infect Dis 2021;72(9):e206-e214. [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Wingert A, Pillay J, Gates M, et al. Risk factors for severity of COVID-19: a rapid review to inform vaccine prioritisation in Canada. BMJ Open 2021;11(5):e044684-e044684. [DOI] [PMC free article] [PubMed] [Google Scholar]
15.Jackson ML, Nelson JC. The test-negative design for estimating influenza vaccine effectiveness. Vaccine 2013;31:2165-2168. [DOI] [PubMed] [Google Scholar]
16.Chua H, Feng S, Lewnard JA, et al. The use of test-negative controls to monitor vaccine effectiveness: a systematic review of methodology. Epidemiology 2020;31:43-64. [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Foppa IM, Ferdinands JM, Chaves SS, et al. The case test-negative design for studies of the effectiveness of influenza vaccine in inpatient settings. Int J Epidemiol 2016;45:2052-2059. [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Cates J, Lucero-Obusan C, Dahl RM, et al. Risk for in-hospital complications associated with COVID-19 and influenza — Veterans Health Administration, United States, October 1, 2018–May 31, 2020. MMWR Morb Mortal Wkly Rep 2020;69:1528-1534. [DOI] [PMC free article] [PubMed] [Google Scholar]
19.Thompson MG, Kwong JC, Regan AK, et al. Influenza vaccine effectiveness in preventing influenza-associated hospitalizations during pregnancy: a multi-country retrospective test negative design study, 2010–2016. Clin Infect Dis 2019;68:1444-1453. [DOI] [PubMed] [Google Scholar]
20.Ferdinands JM, Gaglani M, Martin ET, et al. Prevention of influenza hospitalization among adults in the United States, 2015–2016: results from the US Hospitalized Adult Influenza Vaccine Effectiveness Network (HAIVEN). J Infect Dis 2019;220:1265-1275. [DOI] [PMC free article] [PubMed] [Google Scholar]
21.Cohen J. Statistical power analysis in the behavioral sciences. 2nd ed. New York: Routledge, 1988. [Google Scholar]
22.Månsson R, Joffe MM, Sun W, Hennessy S. On the estimation and use of propensity scores in case-control and case-cohort studies. Am J Epidemiol 2007;166:332-339. [DOI] [PubMed] [Google Scholar]
23.McCaffrey DF, Ridgeway G, Morral AR. Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychol Methods 2004;9:403-425. [DOI] [PubMed] [Google Scholar]
24.Austin PC, Stuart EA. Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Stat Med 2015;34:3661-3679. [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Nguyen TL, Collins GS, Spence J, et al. Double-adjustment in propensity score matching analysis: choosing a threshold for considering residual imbalance. BMC Med Res Methodol 2017;17:78-78. [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Chung H, He S, Nascreen S, et al. Effectiveness of BNT162b2 and mRNA-1273 COVID-19 vaccines against symptomatic SARS-CoV-2 infection and severe COVID-19 outcomes in Ontario, Canada. May 28, 2021. (https://www.medrxiv.org/content/10.1101/2021.05.24.21257744v1). preprint. [DOI] [PMC free article] [PubMed]
27.Ferdinands JM, Shay DK. Magnitude of potential biases in a simulated case-control study of the effectiveness of influenza vaccination. Clin Infect Dis 2012;54:25-32. [DOI] [PubMed] [Google Scholar]
28.Andrew MK, McElhaney JE. Age and frailty in COVID-19 vaccine development. Lancet 2021;396:1942-1944. [DOI] [PMC free article] [PubMed] [Google Scholar]
29.Andrew MK, Schmader KE, Rockwood K, Clarke B, McElhaney JE. Considering frailty in SARS-CoV-2 vaccine development: how geriatricians can assist. Clin Interv Aging 2021;16:731-738. [DOI] [PMC free article] [PubMed] [Google Scholar]
30.Zimmermann P, Curtis N. Factors that influence the immune response to vaccination. Clin Microbiol Rev 2019;32(2):e00084-e00118. [DOI] [PMC free article] [PubMed] [Google Scholar]
31.Azar KMJ, Shen Z, Romanelli RJ, et al. Disparities in outcomes among COVID-19 patients in a large health care system in California. Health Aff (Millwood) 2020;39:1253-1262. [DOI] [PubMed] [Google Scholar]
32.Amit S, Regev-Yochay G, Afek A, Kreiss Y, Leshem E. Early rate reductions of SARS-CoV-2 infection and COVID-19 in BNT162b2 vaccine recipients. Lancet 2021;397:875-877. [DOI] [PMC free article] [PubMed] [Google Scholar]
33.Lopez Bernal J, Andrews N, Gower C, et al. Effectiveness of the Pfizer-BioNTech and Oxford-AstraZeneca vaccines on covid-19 related symptoms, hospital admissions, and mortality in older adults in England: test negative case-control study. BMJ 2021;373:n1088-n1088. [DOI] [PMC free article] [PubMed] [Google Scholar]
34.Skowronski DM, De Serres G. Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine. N Engl J Med 2021;384:1576-1577. [DOI] [PubMed] [Google Scholar]
35.De Serres G, Skowronski DM, Wu XW, Ambrose CS. The test-negative design: validity, accuracy and precision of vaccine efficacy estimates compared to the gold standard of randomised placebo-controlled clinical trials. Euro Surveill 2013;18:20585-20585. [DOI] [PubMed] [Google Scholar]
36.Cassidy EL, Baird E, Sheikh JI. Recruitment and retention of elderly patients in clinical trials: issues and strategies. Am J Geriatr Psychiatry 2001;9:136-140. [PubMed] [Google Scholar]
37.George S, Duran N, Norris K. A systematic review of barriers and facilitators to minority research participation among African Americans, Latinos, Asian Americans, and Pacific Islanders. Am J Public Health 2014;104(2):e16-e31. [DOI] [PMC free article] [PubMed] [Google Scholar]
38.Ferdinands JM, Belongia EA, Nwasike C, Shay DK. Influenza vaccination status is not associated with influenza testing among children: implications for observational studies of vaccine effectiveness. Vaccine 2011;29:1935-1940. [DOI] [PubMed] [Google Scholar]
39.Kwong JC, Campitelli MA, Gubbay JB, et al. Vaccine effectiveness against laboratory-confirmed influenza hospitalizations among elderly adults during the 2010-2011 season. Clin Infect Dis 2013;57:820-827. [DOI] [PMC free article] [PubMed] [Google Scholar]
40.Thompson MG, Burgess JL, Naleway AL, et al. Prevention and attenuation of Covid-19 with the BNT162b2 and mRNA-1273 vaccines. N Engl J Med 2021;385:320-329. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Click here for additional data file.^{(1.3MB, pdf)}

Click here for additional data file.^{(6.6MB, pdf)}

Click here for additional data file.^{(816.3KB, pdf)}

Click here for additional data file.^{(69.5KB, pdf)}

[r1] 1.Baden LR, El Sahly HM, Essink B, et al. Efficacy and safety of the mRNA-1273 SARS-CoV-2 vaccine. N Engl J Med 2021;384:403-416. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r2] 2.Polack FP, Thomas SJ, Kitchin N, et al. Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine. N Engl J Med 2020;383:2603-2615. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r3] 3.Sadoff J, Gray G, Vandebosch A, et al. Safety and efficacy of single-dose Ad26.COV2.S vaccine against Covid-19. N Engl J Med 2021;384:2187-2201. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r4] 4.Thompson MG, Burgess JL, Naleway AL, et al. Interim estimates of vaccine effectiveness of BNT162b2 and mRNA-1273 COVID-19 vaccines in preventing SARS-CoV-2 infection among health care personnel, first responders, and other essential and frontline workers — eight U.S. locations, December 2020–March 2021. MMWR Morb Mortal Wkly Rep 2021;70:495-500. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r5] 5.Hall VJ, Foulkes S, Saei A, et al. COVID-19 vaccine coverage in health-care workers in England and effectiveness of BNT162b2 mRNA vaccine against infection (SIREN): a prospective, multicentre, cohort study. Lancet 2021;397:1725-1735. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r6] 6.Dagan N, Barda N, Kepten E, et al. BNT162b2 mRNA Covid-19 vaccine in a nationwide mass vaccination setting. N Engl J Med 2021;384:1412-1423. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r7] 7.Tenforde MW, Olson SM, Self WH, et al. Effectiveness of Pfizer-BioNTech and Moderna vaccines against COVID-19 among hospitalized adults aged ≥65 years — United States, January–March 2021. MMWR Morb Mortal Wkly Rep 2021;70:674-679. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r8] 8.Stokes EK, Zambrano LD, Anderson KN, et al. Coronavirus Disease 2019 Case Surveillance — United States, January 22–May 30, 2020. MMWR Morb Mortal Wkly Rep 2020;69:759-765. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r9] 9.Killerby ME, Link-Gelles R, Haight SC, et al. Characteristics associated with hospitalization among patients with COVID-19 — metropolitan Atlanta, Georgia, March–April 2020. MMWR Morb Mortal Wkly Rep 2020;69:790-794. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r10] 10.Gu T, Mack JA, Salvatore M, et al. Characteristics asociated with racial/ethnic disparities in COVID-19 outcomes in an academic health care system. JAMA Netw Open 2020;3(10):e2025197-e2025197. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r11] 11.Price-Haywood EG, Burton J, Fort D, Seoane L. Hospitalization and mortality among Black patients and White patients with Covid-19. N Engl J Med 2020;382:2534-2543.. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r12] 12.Garg S, Kim L, Whitaker M, et al. Hospitalization rates and characteristics of patients hospitalized with laboratory-confirmed Coronavirus disease 2019 — COVID-NET, 14 states, March 1-30, 2020. MMWR Morb Mortal Wkly Rep 2020;69:458-464. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r13] 13.Kim L, Garg S, O’Halloran A, et al. Risk factors for intensive care unit admission and in-hospital mortality among hospitalized adults identified through the US Coronavirus Disease 2019 (COVID-19)–Associated Hospitalization Surveillance Network (COVID-NET). Clin Infect Dis 2021;72(9):e206-e214. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r14] 14.Wingert A, Pillay J, Gates M, et al. Risk factors for severity of COVID-19: a rapid review to inform vaccine prioritisation in Canada. BMJ Open 2021;11(5):e044684-e044684. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r15] 15.Jackson ML, Nelson JC. The test-negative design for estimating influenza vaccine effectiveness. Vaccine 2013;31:2165-2168. [DOI] [PubMed] [Google Scholar]

[r16] 16.Chua H, Feng S, Lewnard JA, et al. The use of test-negative controls to monitor vaccine effectiveness: a systematic review of methodology. Epidemiology 2020;31:43-64. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r17] 17.Foppa IM, Ferdinands JM, Chaves SS, et al. The case test-negative design for studies of the effectiveness of influenza vaccine in inpatient settings. Int J Epidemiol 2016;45:2052-2059. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r18] 18.Cates J, Lucero-Obusan C, Dahl RM, et al. Risk for in-hospital complications associated with COVID-19 and influenza — Veterans Health Administration, United States, October 1, 2018–May 31, 2020. MMWR Morb Mortal Wkly Rep 2020;69:1528-1534. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r19] 19.Thompson MG, Kwong JC, Regan AK, et al. Influenza vaccine effectiveness in preventing influenza-associated hospitalizations during pregnancy: a multi-country retrospective test negative design study, 2010–2016. Clin Infect Dis 2019;68:1444-1453. [DOI] [PubMed] [Google Scholar]

[r20] 20.Ferdinands JM, Gaglani M, Martin ET, et al. Prevention of influenza hospitalization among adults in the United States, 2015–2016: results from the US Hospitalized Adult Influenza Vaccine Effectiveness Network (HAIVEN). J Infect Dis 2019;220:1265-1275. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r21] 21.Cohen J. Statistical power analysis in the behavioral sciences. 2nd ed. New York: Routledge, 1988. [Google Scholar]

[r22] 22.Månsson R, Joffe MM, Sun W, Hennessy S. On the estimation and use of propensity scores in case-control and case-cohort studies. Am J Epidemiol 2007;166:332-339. [DOI] [PubMed] [Google Scholar]

[r23] 23.McCaffrey DF, Ridgeway G, Morral AR. Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychol Methods 2004;9:403-425. [DOI] [PubMed] [Google Scholar]

[r24] 24.Austin PC, Stuart EA. Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Stat Med 2015;34:3661-3679. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r25] 25.Nguyen TL, Collins GS, Spence J, et al. Double-adjustment in propensity score matching analysis: choosing a threshold for considering residual imbalance. BMC Med Res Methodol 2017;17:78-78. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r26] 26.Chung H, He S, Nascreen S, et al. Effectiveness of BNT162b2 and mRNA-1273 COVID-19 vaccines against symptomatic SARS-CoV-2 infection and severe COVID-19 outcomes in Ontario, Canada. May 28, 2021. (https://www.medrxiv.org/content/10.1101/2021.05.24.21257744v1). preprint. [DOI] [PMC free article] [PubMed]

[r27] 27.Ferdinands JM, Shay DK. Magnitude of potential biases in a simulated case-control study of the effectiveness of influenza vaccination. Clin Infect Dis 2012;54:25-32. [DOI] [PubMed] [Google Scholar]

[r28] 28.Andrew MK, McElhaney JE. Age and frailty in COVID-19 vaccine development. Lancet 2021;396:1942-1944. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r29] 29.Andrew MK, Schmader KE, Rockwood K, Clarke B, McElhaney JE. Considering frailty in SARS-CoV-2 vaccine development: how geriatricians can assist. Clin Interv Aging 2021;16:731-738. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r30] 30.Zimmermann P, Curtis N. Factors that influence the immune response to vaccination. Clin Microbiol Rev 2019;32(2):e00084-e00118. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r31] 31.Azar KMJ, Shen Z, Romanelli RJ, et al. Disparities in outcomes among COVID-19 patients in a large health care system in California. Health Aff (Millwood) 2020;39:1253-1262. [DOI] [PubMed] [Google Scholar]

[r32] 32.Amit S, Regev-Yochay G, Afek A, Kreiss Y, Leshem E. Early rate reductions of SARS-CoV-2 infection and COVID-19 in BNT162b2 vaccine recipients. Lancet 2021;397:875-877. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r33] 33.Lopez Bernal J, Andrews N, Gower C, et al. Effectiveness of the Pfizer-BioNTech and Oxford-AstraZeneca vaccines on covid-19 related symptoms, hospital admissions, and mortality in older adults in England: test negative case-control study. BMJ 2021;373:n1088-n1088. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r34] 34.Skowronski DM, De Serres G. Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine. N Engl J Med 2021;384:1576-1577. [DOI] [PubMed] [Google Scholar]

[r35] 35.De Serres G, Skowronski DM, Wu XW, Ambrose CS. The test-negative design: validity, accuracy and precision of vaccine efficacy estimates compared to the gold standard of randomised placebo-controlled clinical trials. Euro Surveill 2013;18:20585-20585. [DOI] [PubMed] [Google Scholar]

[r36] 36.Cassidy EL, Baird E, Sheikh JI. Recruitment and retention of elderly patients in clinical trials: issues and strategies. Am J Geriatr Psychiatry 2001;9:136-140. [PubMed] [Google Scholar]

[r37] 37.George S, Duran N, Norris K. A systematic review of barriers and facilitators to minority research participation among African Americans, Latinos, Asian Americans, and Pacific Islanders. Am J Public Health 2014;104(2):e16-e31. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r38] 38.Ferdinands JM, Belongia EA, Nwasike C, Shay DK. Influenza vaccination status is not associated with influenza testing among children: implications for observational studies of vaccine effectiveness. Vaccine 2011;29:1935-1940. [DOI] [PubMed] [Google Scholar]

[r39] 39.Kwong JC, Campitelli MA, Gubbay JB, et al. Vaccine effectiveness against laboratory-confirmed influenza hospitalizations among elderly adults during the 2010-2011 season. Clin Infect Dis 2013;57:820-827. [DOI] [PMC free article] [PubMed] [Google Scholar]

[r40] 40.Thompson MG, Burgess JL, Naleway AL, et al. Prevention and attenuation of Covid-19 with the BNT162b2 and mRNA-1273 vaccines. N Engl J Med 2021;385:320-329. [DOI] [PMC free article] [PubMed] [Google Scholar]

PERMALINK

Effectiveness of Covid-19 Vaccines in Ambulatory and Inpatient Care Settings

Mark G Thompson, Ph.D.

Edward Stenehjem, M.D.

Shaun Grannis, M.D.

Sarah W Ball, Sc.D.

Allison L Naleway, Ph.D.

Toan C Ong, Ph.D.

Malini B DeSilva, M.D., M.P.H.

Karthik Natarajan, Ph.D.

Catherine H Bozio, Ph.D., M.P.H.

Ned Lewis, M.P.H.

Kristin Dascomb, M.D., Ph.D.

Brian E Dixon, M.P.A., Ph.D.

Rebecca J Birch, M.P.H.

Stephanie A Irving, M.H.S.

Suchitra Rao, M.D.

Elyse Kharbanda, M.D., M.P.H.

Jungmi Han, B.S.

Sue Reynolds, Ph.D., M.P.H.

Kristin Goddard, M.P.H.

Nancy Grisel, M.P.P.

William F Fadel, Ph.D.

Matthew E Levy, Ph.D.

Jill Ferdinands, Ph.D.

Bruce Fireman, M.A.

Julie Arndorfer, M.P.H.

Nimish R Valvi, Dr.P.H.

Elizabeth A Rowley, Dr.P.H.

Palak Patel, M.B., B.S., M.P.H.

Ousseny Zerbo, Ph.D.

Eric P Griggs, M.P.H.

Rachael M Porter, M.P.H.

Maria Demarco, Ph.D.

Lenee Blanton, M.P.H.

Andrea Steffens, M.P.H.

Yan Zhuang, Ph.D.

Natalie Olson, M.P.H.

Michelle Barron, M.D.

Patricia Shifflett, R.N.

Stephanie J Schrag, D.Phil.

Jennifer R Verani, M.D., M.P.H.

Alicia Fry, M.D., M.P.H.

Manjusha Gaglani, M.B., B.S.

Eduardo Azziz-Baumgartner, M.D., M.P.H.

Nicola P Klein, M.D., Ph.D.

Abstract

Background

Methods

Results

Conclusions

Methods

Study Design

Table 1. Health Care Settings, Vaccination Data, and Covid-19 Vaccines Administered, According to Type of Medical Visit.*.

SARS-CoV-2 Infection, Covid-19 Vaccination, and Viral Circulation

Statistical Analysis

Results

Study Sample

Table 2. Characteristics of the Patients According to SARS-CoV-2 Test Results and Vaccination Status.*.

Covid-19–Associated Medical Care

Covid-19 Vaccination Status

mRNA-Based Vaccine and Hospitalization

Figure 1. Estimated Vaccine Effectiveness against SARS-CoV-2 Infection Leading to Hospitalization or an Emergency Department or Urgent Care Clinic Visit, According to the Type of Vaccine.

Figure 2. Estimated Effectiveness of Full Two-Dose mRNA Vaccination against SARS-CoV-2 Infection Leading to Hospitalization, According to Age, Race or Ethnic Group, and Underlying Medical Conditions.

Figure 3. Estimated Effectiveness of mRNA-Based Vaccination against SARS-CoV-2 Infection Leading to Hospitalization or an Emergency Department or Urgent Care Visit, According to the Days since the Most Recent Dose Was Administered.

mRNA-Based Vaccine and Emergency Department and Urgent Care Visits

Figure 4. Estimated Effectiveness of Full Two-Dose mRNA-Based Vaccination against SARS-CoV-2 Infection Leading to an Emergency Department or Urgent Care Clinic Visit, According to Age, Race or Ethnic Group, and Underlying Medical Conditions.

Effectiveness of Ad26.COV2.S Vaccine

Discussion

Protocol

Supplementary Appendix

Disclosure Forms

Data Sharing Statement

Footnotes

References

Associated Data

Supplementary Materials

ACTIONS

PERMALINK

RESOURCES

Table 1. Health Care Settings, Vaccination Data, and Covid-19 Vaccines Administered, According to Type of Medical Visit.^*.

Table 2. Characteristics of the Patients According to SARS-CoV-2 Test Results and Vaccination Status.^*.