Reduced COVID-19 hospitalizations among New York City residents following age-based SARS-CoV-2 vaccine eligibility: Evidence from a regression discontinuity design

Highlights

• •For 8 weeks in NYC, age ≥ 65 years was a SARS-CoV-2 vaccine eligibility requirement.
• •We assessed local vaccine effectiveness using a regression discontinuity design.
• •Vaccination reduced COVID-19 hospitalizations among 65–84-year-olds by 15%.
• •Equitable vaccine access is needed to reduce inequities in COVID-19 outcomes.

Abstract

Background

In clinical trials, several SARS-CoV-2 vaccines were shown to reduce risk of severe COVID-19 illness. Local, population-level, real-world evidence of vaccine effectiveness is accumulating. We assessed vaccine effectiveness for community-dwelling New York City (NYC) residents using a quasi-experimental, regression discontinuity design, leveraging a period (January 12–March 9, 2021) when ≥ 65-year-olds were vaccine-eligible but younger persons, excluding essential workers, were not.

Methods

We constructed segmented, negative binomial regression models of age-specific COVID-19 hospitalization rates among 45–84-year-old NYC residents during a post-vaccination program implementation period (February 21–April 17, 2021), with a discontinuity at age 65 years. The relationship between age and hospitalization rates in an unvaccinated population was incorporated using a pre-implementation period (December 20, 2020–February 13, 2021). We calculated the rate ratio (RR) and 95% confidence interval (CI) for the interaction between implementation period (pre or post) and age-based eligibility (45–64 or 65–84 years). Analyses were stratified by race/ethnicity and borough of residence. Similar analyses were conducted for COVID-19 deaths.

Results

Hospitalization rates among 65–84-year-olds decreased from pre- to post-implementation periods (RR 0.85, 95% CI: 0.74–0.97), controlling for trends among 45–64-year-olds. Accordingly, an estimated 721 (95% CI: 126–1,241) hospitalizations were averted. Residents just above the eligibility threshold (65–66-year-olds) had lower hospitalization rates than those below (63–64-year-olds). Racial/ethnic groups and boroughs with higher vaccine coverage generally experienced greater reductions in RR point estimates. Uncertainty was greater for the decrease in COVID-19 death rates (RR 0.85, 95% CI: 0.66–1.10).

Conclusion

The vaccination program in NYC reduced COVID-19 hospitalizations among the initially age-eligible ≥ 65-year-old population by approximately 15% in the first eight weeks. The real-world evidence of vaccine effectiveness makes it more imperative to improve vaccine access and uptake to reduce inequities in COVID-19 outcomes.

Introduction

The SARS-CoV-2 vaccines authorized and recommended for emergency use in the United States were demonstrated in randomized clinical trials to reduce risk of severe COVID-19 illness [1], [2], [3]. Post-authorization, several studies have demonstrated real-world evidence of SARS-CoV-2 vaccine effectiveness in various settings, such as among skilled nursing facility residents and essential workers [4], [5], or at the national level [6], [7]. Local, population-level evidence of effectiveness can support public messaging to promote the importance of vaccination [8], [9].

In New York City (NYC), guidelines for vaccine eligibility were established by the Office of the Governor of New York State. When vaccinations began on December 14, 2020 [10], eligibility was initially restricted to health care workers and residents and staff of long-term care facilities. Eligibility expanded to ≥ 75-year-olds in the general population and essential workers (workers in education, public safety, and public transit, and first responders) on January 11, 2021 [11], to ≥ 65-year-olds on January 12 [12], to ≥ 60-year-olds and additional categories of public-facing essential workers on March 10 [13], to ≥ 50-year-olds and individuals with comorbidities and underlying conditions on March 23 [14], to ≥ 30-year-olds on March 30 [15], to ≥ 16-year-olds on April 6 [15], and to ≥ 12-year-olds on May 12 [16]. Screeners at vaccination sites verified age-based eligibility by requiring proof of age, such as a driver’s license, IDNYC (a free municipal identification card for NYC residents), birth certificate, passport, permanent resident card, certificate of naturalization or citizenship, or life insurance policy or marriage certificate with birthdate [17]. Vaccinations were provided at no cost to recipients. Notably, vaccine eligibility for ≥ 65-year-olds as of mid-January coincided with the second peak of COVID-19 hospitalizations in NYC [18], such that the vaccination program was established concurrently with a waning epidemic period.

Vaccinees and non-vaccinees are likely systematically different in ways that are difficult to observe (e.g., adherence to social distancing recommendations, presence of underlying conditions, and history of prior infection) yet influence their probability of SARS-CoV-2 infection and testing and COVID-19 hospitalization and death [19]. Cross-sectional studies comparing outcome rates among vaccinated and unvaccinated populations are susceptible to confounding due to population differences arising from volunteer selection bias, healthy vaccinee effects, and frailty bias [20].

The NYC Department of Health and Mental Hygiene (DOHMH) sought to assess evidence of population-level vaccine effectiveness in NYC, citywide and stratified by subpopulations with different vaccination coverage. We used a quasi-experimental observational study design to leverage an 8-week period (January 12–March 9, 2021) when community residents at an age threshold of ≥ 65-years were vaccine-eligible but younger persons (excluding essential workers) were not.

Methods

COVID-19 hospitalization and death data

Confirmed and probable cases of COVID-19 [21] among NYC residents are reported to NYC DOHMH through electronic laboratory reporting, and hospitalizations and deaths for these patients are ascertained by routinely importing and matching data from supplemental systems, as previously described [22]. The supplemental systems for hospitalization data included emergency department syndromic surveillance, regional health information organizations, NYC public hospitals, DOHMH’s electronic death registry system, and remote access to hospitals’ electronic health record systems, limited only to COVID-19 patients [22]. COVID-19 hospitalizations were defined as NYC residents admitted within +/- 14 days of the first date of specimen collection that tested positive for SARS-CoV-2 by a molecular or antigen test; or if not laboratory-positive but a symptomatic contact of a confirmed or probable case, then admitted within +/- 14 days of illness onset. Hospitalizations with missing admission date (n = 940, 3.3%) were omitted from analysis. COVID-19 deaths were defined as NYC residents who had a positive molecular test and (a) the cause-of-death on the death certificate was COVID-19 or similar, or b) COVID-19 was not a cause-of-death on the death certificate but the patient died within 60 days of a positive molecular test, and the death was not due to external causes such as injury (“confirmed deaths”); or the cause-of-death on the death certificate was COVID-19 or similar, but a positive molecular test was not reported (“probable deaths”) [22]. Patient age was calculated as of January 12, 2021, not as of hospitalization or death date.

Two categories of patients were excluded from analysis. First, given low hospitalization and death rates [23], patients < 45 years-old as of January 12, 2021 were excluded from the comparator for trends among vaccine-eligible ≥ 65-year-olds. Patients ≥ 85 years-old were also excluded for sparsity, such that the study population was restricted to 45–84-year-olds, i.e., +/- 20 years around the vaccine eligibility threshold of age 65 years. Second, patients residing in congregate settings (e.g., long-term care or correctional facilities) were excluded because their vaccine eligibility timing was different from community residents and less dependent on age. Such patients were identified by geocoding the residential address at time of report and matching to facility lists.

Vaccination and population denominator data

The cumulative percentage of NYC residents having received at least the first dose of a SARS-CoV-2 vaccine, by vaccination date and age at first dose, was obtained from the NYC DOHMH Citywide Immunization Registry [24], as reported by immunizing facilities. For patients admitted after the vaccination program was implemented, vaccination status as of the 14 days prior to COVID-19 diagnosis was obtained by matching with the Citywide Immunization Registry. Single-year of age population estimates for 2019 for the five NYC boroughs (equivalent to counties) were downloaded from the National Center for Health Statistics [25].

Program implementation timing

We assumed that population-level vaccine effects on hospitalizations would not be apparent until 4 weeks after age-based eligibility was established. Within that period, we accounted for 1 week to begin substantial vaccination uptake in the newly eligible population, an additional 2 weeks after receipt of the first dose for a partially protective effect from vaccination, and an additional 1 week for hospitalizations to occur among those infected. Following the same logic, we assumed that vaccine effects on deaths would not be apparent until 6 weeks after age-based eligibility was established, additionally accounting for an average lag of approximately 2 weeks between COVID-19 hospitalization and death.

We defined pre- and post-vaccination program implementation periods of 8 weeks each, defining weeks as Sundays–Saturdays. We chose 8 weeks to correspond with the duration of the period (January 12–March 9, 2021) when only ≥ 65-year-olds had age-based eligibility. For the primary analysis for hospitalizations, we defined the pre-implementation period as December 20, 2020–February 13, 2021, i.e., an 8-week period ending 4 weeks after ≥ 65-year-olds became eligible on Jan. 12. Imposing a 1-week washout period, we defined the post-implementation period as February 21–April 17, 2021. For the primary analysis for deaths, we defined the pre-implementation period as January 3–February 27, 2021, i.e., an 8-week period ending 6 weeks after ≥ 65-year-olds became eligible on Jan. 12. Imposing a 2-week washout period to account for additional ambiguity in the timing of vaccine effects, we defined the post-implementation period as March 14–May 8, 2021. Data were frozen as of June 28, 2021, capturing hospitalizations ascertained within 72 days and deaths ascertained within 51 days after the ends of the post-implementation periods for the primary analysis. In sensitivity analyses, we shifted implementation period definitions and imposed washout periods of different lengths.

Regression discontinuity design

We constructed segmented, negative binomial regression models of the age-specific hospitalization (and death) rates during the post-vaccination program implementation period, with a discontinuity at age 65 years. We used pre-implementation period data to incorporate the observed relationship between age and hospitalization rates in an unvaccinated population. We expected the overall hospitalization rate in the post-implementation period to be lower due to the waning stage of the epidemic, but for trends across age to persist. We specified the model as a standard regression discontinuity design with a control group and indexed and centered the values for age and their corresponding interaction terms for appropriate interpretations of regression coefficients of interest (Appendix A) [26]. The analytic dataset for the hospitalizations primary analysis is provided for reproducibility (Appendix B). Analyses were conducted using PROC GENMOD in SAS Version 9.4 (SAS Institute, Cary, NC). This work was deemed public health surveillance that is non-research by the NYC DOHMH Institutional Review Board.

The key parameter of interest was ${\beta }_6$, the interaction term between vaccine program implementation period (pre or post) and age-based eligibility (45–64 or 65–84 years), representing the adjusted difference in log rates (intercept change) for 65-year-olds following age-based eligibility [27], [28], [29], [30] (Appendix A). We exponentiated this parameter estimate and 95% confidence interval (CI) to obtain the rate ratio (RR) of interest and 95% CI.

Estimating hospitalizations and deaths averted

We estimated hospitalizations and deaths among 65–84-year-olds during the post-implementation period under the counterfactual scenario in which there were no effects of vaccination. That is, we estimated hospitalizations and deaths in this group had they experienced the same intercept and slope change from the pre- to post-implementation periods as the 45–64-year-olds but experienced no discontinuity due to the implementation of the age-based vaccination policy. As above, let ${\beta }_6$ equal the parameter of a fitted negative binomial model describing the log difference of hospitalization rates between pre- and post-implementation periods among 65–84-year-olds, controlling for the decrease in hospitalizations in the post-implementation period due to the waning epidemic. Therefore, $e^{\beta 6}$ is the rate ratio of this parameter.

Let $Y_{65-84}$ equal the observed hospitalization rate among 65–84-year-olds in the post-implementation period and $N_{65-84}$ equal the total population of 65–84-year-olds in NYC. Therefore, the counterfactual scenario in which there was no change in the hospitalization rate between 65 and 84-year-olds in the pre- and post-implementation periods, controlling for the decrease in hospitalizations in the post-implementation period is:

$${\dot{Y}}_{65-84}=Y_{65-84}\ast \left(1+\left(1-e^{\beta 6}\right)\right)$$

and the expected number of hospitalizations in the counterfactual is:

$${\dot{H}}_{65-84}=N_{65-84}\ast {\dot{Y}}_{65-84}$$

while the observed number of hospitalizations is:

$$H_{65-84}=N_{65-84}\ast Y_{65-84}$$

Finally, assuming ${\beta }_6$ is negative, the number of averted hospitalizations is defined as:

$$A_{65-84}={\dot{H}}_{65-84}-H_{65-84}$$

Therefore, the final equation of averted hospitalizations is:

$$A_{65-84}=\left[N_{65-84}\ast Y_{65-84}\ast \left(1+\left(1-e^{\beta 6}\right)\right)\right]-\left[N_{65-84}\ast Y_{65-84}\right]$$

Similarly, for the 95% CI, we replaced ${\beta }_6$ with the upper and lower limits of the parameter's CI. The same approach was used to calculate averted deaths.

Stratified analyses and negative controls

To assess heterogeneity of findings for hospitalizations across subpopulations with different vaccination rates, stratified analyses were conducted by race/ethnicity and borough of residence, where non-missing. Stratified analyses were not conducted for deaths given sparsity, nor for sub-borough areas because borough was the smallest geographic resolution available with population denominators by single year of age [25].

We used negative controls to assess whether findings for citywide COVID-19 hospitalizations might be attributable to unknown sources of error [31]. If the vaccination program were effective, then these methods should demonstrate reduced COVID-19 hospitalization rates when applied at the ≥ 65 years age threshold during the period around vaccine program implementation, but null effects when applied to a different age threshold or to earlier periods when no SARS-CoV-2 vaccines were available.

First, we modified our primary analysis by redefining the age groups from 45–64 and 65–84 (corresponding to the true age threshold of ≥ 65 years) to 30–49 and 50–79 (corresponding to a false age threshold of ≥ 50 years). Second, we modified the primary analysis by shifting the 16-week study period with 1-week washout period to three negative control points earlier in the epidemic based on trends in citywide hospitalizations [18], defining the start of the post-implementation period as May 10, 2020 (when hospitalizations were at a similar magnitude and waning during the first epidemic wave), August 16, 2020 (when hospitalizations were low between epidemic waves), and December 20, 2020 (when hospitalizations were waxing during the second wave).

Results

Trends in vaccination coverage and COVID-19 hospitalizations

As of January 16, 2021 (the end of the week age-based eligibility began for ≥ 65-year-olds), the cumulative percentage of 65–84-year-old NYC residents having received at least the first dose of a SARS-CoV-2 vaccine was 6.5% (Fig. 1). By March 9 (the last day prior to ≥ 60-year-olds also becoming vaccine-eligible), 47.6% of 65–84-year-olds were vaccinated, compared with only 22.2% of 45–64-year-olds. Vaccination coverage of 65–84-year-olds by March 9 varied widely by race/ethnicity, ranging from 26.2% of Black/African-American NYC residents to 45.9% of Asian/Pacific Islander NYC residents. Vaccination coverage of 65–84-year-olds as of March 20 (4 weeks before the end of the post-implementation period) was 55.8%. During the second COVID-19 wave in NYC, the timing of peak hospitalizations was similar for younger (45–64) and older (65–84) age groups (Fig. 1).

Fig. 1.

Weekly COVID-19 hospitalizations and cumulative vaccine coverage among 45–64 and 65–84-year-olds in relation to timing of age-based SARS-CoV-2 vaccine eligibility, New York City, December 13, 2020–April 17, 2021.

Regression discontinuity design, primary analysis

Among 2,027,014 45–64-year-old NYC residents, 5,563 COVID-19 hospitalizations occurred during the pre-implementation period and 4,977 occurred during the post-implementation period (Appendix B). Among 1,101,467 65–84-year-old NYC residents, the number of hospitalizations during pre- and post-implementation periods were 7,557 and 4,780, respectively. During the post-implementation period, the percentage of persons with COVID-19 hospitalization who had not received at least the first dose of a SARS-CoV-2 vaccine ≥ 14 days prior to diagnosis was higher for 45–64-year-olds (96.5% unvaccinated, 4,804/4,977) than for 65–84-year-olds (90.0% unvaccinated, 4,303/4,780). During the pre-implementation period, hospitalization rates increased with increasing age in years (Fig. 2).

Fig. 2.

COVID-19 hospitalization rates among New York City residents by year of age during 8-week pre- (December 20, 2020–Feb 13, 2021) and post- (February 21–April 17, 2021) implementation periods for the SARS-CoV-2 vaccine program.

The hospitalization rate in the post-implementation period was lower across all ages when compared with the pre-implementation period, likely due to the waning epidemic. Even so, there was a significant negative intercept shift in the hospitalization rate of 65–84-year-olds in the post-implementation period, and 65- and 66-year-olds just above the age threshold for eligibility had lower hospitalization rates than 63- and 64-year-olds just below the threshold (Fig. 2). Hospitalization rates among 65–84-year-olds during the post-implementation period decreased compared with the pre-implementation period (RR 0.85, 95% CI: 0.74–0.97, P = 0.02), controlling for the epidemic trend among 45–64-year-olds, a group without concurrent age-based vaccine eligibility (Table 1). As expected, the hospitalization rate increased with age in both pre- and post-implementation periods. The 3-way interaction between vaccine program implementation, age-based eligibility, and age in years was null, indicating that the trajectory of the hospitalization rate with increasing age did not differ before and after program implementation among 65–84-year-olds.

Table 1.

Reductions in COVID-19 hospitalizations and deaths among 65–84-year-old New York City community residents following age-based SARS-CoV-2 vaccine eligibility.

			Coverage for ≥ 1 dose of a SARS-CoV-2 vaccine administered by Mar 9, 20211		Rate ratio for outcome among 65–84-year-olds post-vaccine program implementation			Outcomes averted among 65–84-year-olds post-vaccine program implementation
Outcome	Stratum	Level	% of 65–84-year-olds	% of 45–64-year-olds	Rate ratio $\left(e^{{\beta }_{6}Z{X}_y}\right)$	95% CI	P-value	N	95% CI
Hospitalizations	Primary analysis2	Citywide	47.6	22.2	0.85	0.74–0.97	0.02	721	126–1,241
	Race/ethnicity3	Asian/Pacific Islander	45.9	20.6	0.80	0.59–1.09	0.16	125	−55–258
		White	44.9	20.6	0.79	0.61–1.01	0.06	255	−11–463
		Hispanic/Latino4	30.5	13.2	0.83	0.67–1.03	0.09	210	−34–408
		Black/African-American	26.2	12.8	0.95	0.79–1.16	0.63	57	−196–266
	Borough	Manhattan	60.9	27.2	0.76	0.57–1.02	0.07	151	−15–275
		Staten Island	55.7	25.9	0.65	0.44–0.98	0.04	88	5–143
		Queens	46.1	22.1	0.79	0.63–1.01	0.06	289	−8–523
		Bronx	45.5	22.1	1.02	0.82–1.27	0.86	−17	−238–160
		Brooklyn	38.7	18.2	0.88	0.73–1.06	0.17	194	−89–430
	Sensitivity analyses: study period shift from primary analysis of post-implementation period, washout period length	2 weeks earlier, 1 week washout	N/A		0.90	0.78–1.05	0.17	556	−253–1,253
		1 week earlier, 1 week washout	N/A		0.88	0.76–1.02	0.09	631	−97–1,259
		1 week earlier, 2 weeks washout	N/A		0.87	0.75–1.02	0.08	660	−79–1,296
		0 shift, 2 weeks washout	N/A		0.85	0.73–0.98	0.03	725	89–1,275
		1 week later, 2 weeks washout	N/A		0.84	0.74–0.96	0.01	684	179–1,127
		1 week earlier, 0 washout	N/A		0.88	0.76–1.01	0.07	627	−57–1,222
	Negative control: age 50 as threshold	N/A	N/A		1.04	0.90–1.21	0.59	N/A
	Negative controls: start of post-implementation period	May 10, 2020	N/A		1.12	0.91–1.37	0.29	N/A
		Aug 16, 2020	N/A		1.06	0.75–1.48	0.75	N/A
		Dec 20, 2020	N/A		0.99	0.84–1.16	0.89	N/A
Deaths	Primary analysis5	Citywide	47.6	22.2	0.85	0.66–1.10	0.22	155	−106–358
	Sensitivity analyses: study period shift from primary analysis of post-implementation period, washout period length	3 weeks earlier, 1 week washout	N/A		1.04	0.81–1.33	0.76	−56	−468–266
		2 weeks earlier, 1 week washout	N/A		0.97	0.76–1.24	0.79	42	−306–314
		1 week earlier, 1 week washout	N/A		0.94	0.73–1.20	0.61	75	−236–318
		2 weeks earlier, 2 weeks washout	N/A		0.99	0.77–1.27	0.93	15	−346–297
		1 week earlier, 2 weeks washout	N/A		0.93	0.73–1.19	0.57	82	−231–326
		1 week earlier, 0 washout	N/A		0.97	0.76–1.24	0.83	34	−312–305
		1 week later, 1 week washout	N/A		0.86	0.67–1.11	0.25	145	−117–349
		1 week later, 2 weeks washout	N/A		0.84	0.65–1.09	0.18	155	−84–340
		2 weeks later, 1 week washout	N/A		0.78	0.60–1.00	0.05	220	1–390
		2 weeks later, 2 weeks washout	N/A		0.78	0.60–1.02	0.07	184	−14–336

¹March 9, 2021 was the last date in New York State when only ≥ 65-year-olds had age-based vaccine eligibility.

²For the primary analysis for hospitalizations, the pre-vaccine program implementation period was December 20, 2020–February 13, 2021, the 1-week washout period was February 14–20, 2021, and the post-vaccine program implementation period was February 21–April 17, 2021.

³Of 22,877 COVID-19 hospitalizations of NYC residents 45–84 years-old during the period in the primary analysis, 1,519 (6.6%) were missing race/ethnicity and 706 (3.1%) identified as other racial/ethnic categories (e.g., Native American/Alaska Native or multi-racial). Of 974,113 NYC residents 45–84 years-old with ≥ 1 dose of a SARS-CoV-2 vaccine administered by Mar 9, 2021, 119,387 (12.3%) were missing race/ethnicity and 106,292 (10.9%) identified as other racial/ethnic categories.

⁴The Hispanic/Latino category included people of any race.

⁵For the primary analysis for deaths, the pre-vaccine program implementation period was January 3–February 27, 2021, the 2-week washout period was February 28–March 13, 2021, and the post-vaccine program implementation period was March 14–May 8, 2021.

Thus, there was a 15.1% (95% CI: 2.6%–26.0%) reduction in the hospitalization rate among 65–84-year-olds during the post-implementation period compared with the pre-implementation period and accounting for the overall decrease in hospitalizations in the post-implementation period. This translates to an estimated 721 (95% CI: 126–1,241) hospitalizations averted during the 8-week post-implementation period, which are causally attributable to the vaccination program as the only intervention to reduce COVID-19 illness implemented at the ≥ 65-year-old threshold during this period.

In stratified analyses (Table 1), boroughs where residents had higher vaccination coverage generally had greater reductions in the hospitalization rate ratio point estimate, although only the citywide and Staten Island estimates were statistically significant at α = 0.05. The Bronx was an outlier, with no change in the hospitalization rate ratio (RR 1.02, 95% CI: 0.82–1.27). Black/African American individuals had the lowest vaccination coverage as of March 9 and experienced the least reduction in hospitalization rates, although differences across race/ethnic groups were not statistically significant.

Citywide during the post-implementation period, the decrease in death rates among 65–84-year-olds was of a similar magnitude as the decrease in hospitalization rates, but deaths were sparser so uncertainty around this estimate was wide (RR 0.85, 95% CI: 0.66–1.10, P = 0.22) (Table 1).

Sensitivity analyses and negative controls

Findings were generally robust to program implementation period definitions. For hospitalizations, shifting the period and imposing washout periods of different lengths had minor effects on the rate ratio point estimates but influenced whether results were statistically significant. The effect for death rates was closer to the null in sensitivity analyses using earlier period definitions and further from the null using later definitions. The strongest reduction in death rates was observed when shifting the implementation period 2 weeks later and with a 1-week washout period (RR 0.78, 95% CI: 0.60–0.999, P = 0.05). As expected, negative controls using a false age threshold or periods prior to vaccine availability yielded no reductions in COVID-19 hospitalization rates among the older age group (Table 1).

Discussion

The SARS-CoV-2 vaccination program in NYC reduced the COVID-19 hospitalization rate among the initially age-eligible population by approximately 15% and was responsible for approximately 721 averted hospitalizations during the 8 weeks following program implementation. These are remarkable reductions, especially with low vaccination coverage among 65–84-year-olds during the first 8 weeks of eligibility, when demand generally exceeded supply and the cumulative percentage having received at least the first dose increased from 6.5% to only 47.6%. Our findings were robust to sensitivity analyses and negative controls.

With a regression discontinuity design, the main threat to validity would be other interventions or events to reduce COVID-19 hospitalizations that were implemented according to the same ≥ 65-year-old threshold and at the same time. We are unaware of any other such interventions or events, so confounding is unlikely, supporting a causal interpretation of the results [32]. The vaccination program in NYC began when the epidemic was already waning. As a point of comparison, in nearby Massachusetts, age-based eligibility began for ≥ 75-year-olds on February 1, 2021 [33], 1 month after the second wave of COVID-19 hospitalizations peaked in early January [34]. A strength of the regression discontinuity design (as opposed to, for example, an interrupted time series design) is avoiding the misattribution of reduced hospitalizations to vaccination as opposed to epidemic trends. Had we used time as the continuous running variable, the epidemic peak in the pre-implementation period would have complicated efforts to disentangle the vaccination program from other secular trends. While there was no discontinuity in hospitalizations over time (Fig. 1), there was a sharp discontinuity in hospitalizations by age at the vaccine eligibility threshold of 65-years-old (Fig. 2).

In stratified analyses, we observed suggestive but not statistically significant associations between increasing vaccination coverage and stronger reductions in hospitalization rate ratios. Residents of the Bronx and Brooklyn, as well as Black/African-American NYC residents, had lower vaccination coverage and appeared to experience the least reduction in COVID-19 hospitalization rates. Equitable vaccine access is urgently needed to reduce pronounced inequities in COVID-19 outcomes [35].

Limitations

The primary limitation is that our estimates of reduced COVID-19 hospitalizations in NYC attributable to the SARS-CoV-2 vaccination program are likely underestimates. The regression discontinuity design in this setting was “fuzzy” and akin to a randomized trial with imperfect adherence [36]. Vaccination coverage did not increase instantaneously following the January 12, 2021 eligibility date. Many individuals above the ≥ 65 years age eligibility threshold were unvaccinated, while many essential workers < 65 years-old became eligible concurrently [11] and were vaccinated. A small proportion of persons who were 64 years-old as of January 12, 2021 and vaccine-ineligible would have turned 65 years-old and vaccine-eligible during the post-implementation period. Any vaccination coverage among < 65-year-olds (although beneficial for vaccine recipients through the direct effects of vaccination and those around them through indirect effects [20]) would have diluted differences between older and younger groups and biased estimates of vaccine effects for the older group toward the null. Slightly faster convergence in vaccination rates between older and younger groups might partially explain why the Bronx was an outlier in stratified analyses, with no change in the hospitalization rate ratio; a mass vaccination site at Yankee Stadium, restricted to eligible Bronx residents, opened on February 5, 2021 [37], contributing to relatively quicker vaccine uptake in the Bronx among the younger age group. Similar mass vaccination sites were not opened in other boroughs until almost 3 weeks later on February 24 [38]. In addition, of the five boroughs, the age-adjusted case incidence rate was highest in the Bronx for much of the early epidemic period prior to vaccine availability [22], and mobility patterns consistent with commuting by essential workers declined least in the Bronx [39]. Disproportionately higher infection-induced immunity among 45–64 year-old Bronx residents compared with residents of other boroughs during the post-implementation period might have further contributed to the lack of change in the hospitalization rate ratio observed in the Bronx.

This study leveraged a brief, 8-week period during which age-based eligibility was restricted to ≥ 65-year-olds, which was only a limited period for observing reductions in hospitalization and death rates overall and any heterogeneity across subpopulations with different vaccination coverage. These estimates could not be updated using the regression discontinuity design as vaccination coverage further increased or as age eligibility expanded to younger persons, given lack of an appropriate comparator. Additionally, vaccine effects on asymptomatic or mild infections could not be assessed because vaccinated patients could have been less likely to seek testing, so estimates based on reported cases could have been biased.

Findings from the regression discontinuity design would have been further strengthened with a negative control outcome [31], [40], i.e., demonstrating no reduction among ≥ 65-year-olds post-vaccine program implementation in hospitalizations for a different cause also associated with age, such as myocardial infarctions. However, hospitalization data for COVID-19, but not for other causes, were ascertained through emergency response efforts to import and match data from supplemental systems [22]. Such an analysis could be explored in the future once comprehensive, all-payer hospitalization data for NYC residents become available from the Statewide Planning and Research Cooperative System (SPARCS) [41]. SPARCS data could also be used to assess effects on all-cause hospitalizations.

Finally, data included in this analysis were preliminary and subject to missing observations, missing values, and misclassification. Hospitalizations were incompletely ascertained via matching with external sources [22]. Additionally, a proportion of hospitalizations classified as COVID-19-attributable might have been due to other causes (e.g., injuries) or were misclassified because DOHMH quality assurance processes might not have eliminated all patients who only had an encounter at an emergency department but were not admitted to a hospital. However, missing or misclassified hospitalizations would bias findings only if differential both by age and time, which is unlikely. Immunizations were incompletely ascertained for NYC residents who were vaccinated outside of New York State or by federal programs. Demographic data, notably for race/ethnicity, were incomplete for both hospitalizations and immunizations. Population denominators by single year of age were unavailable for small geographic areas or for disaggregated race/ethnic groups, limiting the ability to further examine inequities [42]. Furthermore, the denominators were vintage 2019 and did not account for any population changes between pre- and post-implementation periods or pandemic-related deaths and outmigration from NYC [43]; however, large-scale population changes during the brief period examined are unlikely, and the denominators yielded hospitalization rates with the expected positive association with increasing age during the pre-implementation period (Fig. 2).

Conclusion

We demonstrated real-world evidence of SARS-CoV-2 vaccination effectiveness in protecting NYC residents in the community setting from severe COVID-19 illness. The regression discontinuity design is valid for causal inference and is low-cost to implement as an ecological, observational study with no requirement to ascertain individual-level vaccination status. This design could be used by other local and state health departments to demonstrate vaccination effects in their own jurisdictions to support public and provider messaging about the importance of vaccination. Such an approach would complement other methods for evaluating real-world vaccine effectiveness, including mathematical model-based approaches for estimating outcomes averted [44], [45], [46] and test-negative designs [47].

Funding

SKG was supported by the Public Health Emergency Preparedness Cooperative Agreement (grant No. NU90TP922035-02), funded by the US Centers for Disease Control and Prevention (CDC). ALR was supported by ELC CARES (grant No. NU50CK000517-01–09), funded by CDC. RK was supported by the U.S. National Cancer Institute Seronet cooperative agreement U01CA261277. This article’s contents are solely the responsibility of the authors and do not necessarily represent the official views of CDC or the Department of Health and Human Services. The funders had no role in study design; data collection, analysis, or interpretation; writing of the article; or in the decision to submit the article for publication.

CRediT authorship contribution statement

Sharon K. Greene: Conceptualization, Methodology, Writing – original draft, Writing – review & editing, Supervision. Alison Levin-Rector: Methodology, Data curation, Software, Formal analysis, Visualization, Writing – review & editing. Emily McGibbon: Data curation, Validation, Writing – review & editing. Jennifer Baumgartner: Data curation, Writing – review & editing. Katelynn Devinney: Data curation, Writing – review & editing. Alexandra Ternier: Data curation, Writing – review & editing. Jessica Sell: Data curation, Writing – review & editing. Rebecca Kahn: Methodology, Writing – review & editing. Nishant Kishore: Methodology, Validation, Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A and Appendix B. Supplementary data

The following are the Supplementary data to this article: