Benefit-risk assessment of COVID-19 vaccine, mRNA (Comirnaty) for age 16–29 years

Abstract

Since authorization of the Pfizer-BioNTech COVID-19 Vaccine, mRNA (Comirnaty), real-world evidence has indicated the vaccines are effective in preventing COVID-19 cases and related hospitalizations and deaths. However, increased cases of myocarditis/pericarditis have been reported in the United States associated with vaccination, particularly in adolescents and young adults. FDA conducted a benefit-risk assessment to determine whether the benefits of vaccination outweigh the risks among various age (16–17, 18–24, 25–29) and sex (M/F) subgroups being considered for approved use of the vaccine. We conducted a simulation study with sensitivity analysis of the benefits and risks of the vaccine across possible pandemic scenarios. The model results show benefits outweigh the risks for all scenarios including the high-risk subgroup, males 16–17 years old. Our worst-case scenario used sex and age subgroup-specific incidences for COVID-19 cases (47–98 per million per day) and hospitalizations (1–4 per million per day) which are the US COVID-19 incidences as of July 10, 2021, vaccine efficacy of 70% against COVID-19 cases and 80% against hospitalization, and unlikely, pessimistic, non-zero vaccine-attributable myocarditis death rate. For males 16–17 years old, the model predicts prevented COVID cases, hospitalizations, ICUs, and deaths of 13577, 127, 41, and 1, respectively; while the predicted ranges for excess myocarditis/pericarditis cases, hospitalizations, and deaths attributable to the vaccine are [98–196], [98–196], and 0, respectively, for the worst-case scenario. Considering the different clinical implications of hospitalization due to COVID-19 infection versus vaccine-attributable myocarditis/pericarditis cases, we determine the benefits still outweigh the risks even for this high-risk subgroup. Our results demonstrate that the benefits of the vaccine outweigh its risks for all age and sex subgroups we analyze in this study. Uncertainties exist in this assessment as both benefits and risks of vaccination may change with the continuing evolution of the pandemic.

1. Introduction

The Pfizer-BioNTech COVID-19 Vaccine, CVmRNA (Comirnaty), had been recommended for persons 12 years of age and older in the United States under Food and Drug Administration’s (FDA’s) Emergency Use Authorization (EUA). Since authorization of mRNA COVID-19 vaccines (Pfizer-BioNTech and Moderna), real-world evidence has indicated the vaccines are effective in preventing COVID-19 cases and related hospitalizations and deaths. However, increased cases of myocarditis and pericarditis (inflammation of the heart or lining of the heart) have been reported in the United States associated with mRNA COVID-19 vaccination, particularly in male adolescents and young adults [1], [2], [3]. The US FDA and CDC use the Biologics Effectiveness and Safety (BEST) database and the Vaccine Safety Datalink (VSD) to monitor the incidence of myocarditis and pericarditis cases post-vaccination since EUA approval of the COVID-19 vaccines. The BEST initiative is part of FDA’s Center for Biologics Evaluation and Research (CBER) active surveillance program for the safety and effectiveness of biologic products such as vaccines, blood and blood products, human tissues and cellular products, and gene therapies. The BEST system consists of large-scale administrative claims, Electronic Health Records (EHRs), and linked claims-EHR data sources. The VSD is a collaborative project between the US CDC’s Immunization Safety Office and US health care organizations. It monitors safety of vaccines and studies rare and serious adverse events following immunization. The VSD database includes comprehensive medical records from participating US health care organizations.

In 2021, FDA conducted a benefit-risk assessment to inform regulatory decisions related to the Biologics License Application (BLA) for use of CVmRNA among ages 16 years and older. The assessment followed the structured benefit-risk framework (BRF) consisting of four dimensions - Analysis of Condition, Current Treatment Options, Benefits, and Risks and Risk Management [14]. In FDA risk assessment we considered the impact of the COVID-19 pandemic on the public health, unavailability of treatment options, urgent need for a vaccine to prevent the disease and control the pandemic, availability of evidence, and uncertainty associated with vaccine effectiveness and adverse reactions. The benefit endpoints considered in this benefit-risk assessment include preventable COVID-19 cases, hospitalizations, ICU admissions, and deaths. These endpoints were selected since they are the most explicit, trackable, and quantifiable outcomes that have the greatest public health significance. The increase of myocarditis/pericarditis post-vaccination was considered as a key risk post COVID-19 vaccination due to its potentially severe consequences. Vaccine attributable myocarditis/pericarditis cases, hospitalizations, intensive care unit (ICU) admissions, and deaths were selected to pair with benefit endpoints for benefit-risk comparison. The regulatory question to be answered is whether the benefits of vaccination outweigh the risks among various age and sex subgroups being considered for approved use of the vaccine, in particular among the young males, given the potentially elevated myocarditis/pericarditis risk after vaccination suggested by post-authorization safety surveillance among this population.

2. Methods

2.1. Model overview

We assessed the benefits and risks per million individuals who are vaccinated with two complete doses of CVmRNA. We conducted the analysis for the subgroups stratified by combinations of sex and age (16–17, 18–24, and 25–29 years). The post-market safety surveillance indicated elevated myocarditis/pericarditis incidence post vaccination among young males with the highest incidence rate among males aged 16–17 years. In this analysis we focused on young adults (age 16–29 years) and further stratified the population by smaller age/sex subgroups to assess the benefit-risk of vaccination for these subgroups with differing risk of myocarditis/pericarditis. The model assesses the benefits of vaccine-preventable COVID-19 cases, hospitalizations, ICU visits, and deaths, and the risks of vaccine-attributable excess myocarditis/pericarditis cases, hospitalizations, and deaths (Fig. 1 ). The key model inputs include duration of vaccine protection, vaccine efficacy against COVID-19 cases and hospitalizations, age/sex specific COVID-19 case and hospitalization incidence rates, age/sex specific vaccine-attributable myocarditis case rate, and myocarditis death rate (Table 1 ). To evaluate the impact of uncertainty of the key model inputs on the benefits and risks, low and high values from different sources of data are used in a sensitivity analysis (Table 1 and Supplemental Table S1) to formulate model scenarios reflecting our assumptions about the model input parameters.

Fig. 1.

Benefits-risks value tree.

Table 1.

The three main model scenarios with different combinations of model input values.

Scenario	Protection period	Efficacy against cases	Efficacy against hospitalization	COVID-19 case incidence	COVID-19 hospitalization incidence	Vaccine attributable myocarditis/pericarditis death rate
Scenario 1	6 months	90%	90%	July 10, 2021 rate	July 10, 2021 rate	0%
Scenario 2 (Most Likely)	6 months	70%	80%	10× July 10, 2021 rate	4× July 10,2021 rate	0%
Scenario 3 (Worst-Case)	6 months	70%	80%	July 10, 2021 rate	July 10, 2021 rate	0.002%

Our model generated benefit-risk outcomes for seven scenarios with different combinations of the input values. The model results for male 16–19 ages under the first three scenarios (Table 1), which we considered most important, are presented in the main body of this paper.

Scenario 1, a base scenario, uses the US COVID-19 incidence and estimated vaccine-attributable myocarditis/pericarditis rate as of July 10, 2021 (the latest date available at the time of analysis) as model inputs. In this scenario we assumed vaccine efficacy of 90% for both case and hospitalization and zero vaccine attributable myocarditis/pericarditis death rate. Scenario 1 served as a starting point for sensitivity analysis and other model scenarios.

Scenario 2, the most likely scenario, applies the multipliers of 10x and 4x respectively to the US COVID-19 case and hospitalization incidences used in scenario 1. When the COVID-19 incidence is high, the model predicted benefits of the vaccine will be significantly larger, and vice versa. By the time of our analysis (the 2nd week of August 2021 with the highest incidence since the vaccine became available) the US COVID-19 case and hospitalization incidences had increased by approximately 6x and 3x respectively from the July 10, 2021 (incidence used in scenario 1) and the pandemic was in an upward trend due to the surge of Delta variant. We projected the pandemic to continue rising during the subsequent 6 months. For scenario 2, we assumed a 10x and 4x increase of incidence from July 10, 2021, respectively for COVID-19 case and hospitalization. This scenario was considered to represent the most-likely scenario based on the observed trend of the pandemic at the time of the analysis (August 2021). Of note, later the pandemic reached higher peaks than we projected for both COVID-19 cases and hospitalizations during the Omicron wave. In this scenario we assume 70% vaccine efficacy for case and 80% for hospitalization and zero myocarditis/pericarditis death rate (as scenario 1).

For scenario 3, the worst-case scenario (i.e., most pessimistic in terms of vaccine benefit), we used US COVID-19 case and hospitalization incidences as of July 10, 2021 (scenario 1), where COVID incidence was close to pandemic low; vaccine efficacy of 70% for cases and 80% for hospitalization (as scenario 2) and a pessimistic non-zero vaccine-attributable myocarditis death rate as model inputs.

The results of these three scenarios are presented for the male subgroup (age 16–29 years) in Table 4 and Fig. 2, Fig. 3, Fig. 4 . The results for other age/sex subgroups and scenarios are summarized in the supplementary materials of the paper (Supplemental Table S1 and Supplementary Figures).

Table 4.

Model predicted benefit-risk outcomes of Scenarios 1–3 for the 16–17-year-old subgroups.

Scenario	Benefits				Risks
	Prevented COVID-19 Cases	Prevented COVID-19 Hospitalizations	Prevented COVID-19 ICUs	Prevented COVID-19 Deaths	Excess Myocarditis Cases	Excess Myocarditis Hospitalizations	Excess Myocarditis Deaths
Males & Females
Scenario 1	19,425	241	59	2	116	116	0
Scenario 2	151,080	855	210	6	116	116	0
Scenario 3	15,108	214	52	1	116	116	0
Males only
Scenario 1	17,456	142	47	1	196	196	0
Scenario 2	135,771	506	166	4	196	196	0
Scenario 3	13,577	127	41	1	196	196	0
Females only
Scenario 1	21,657	350	68	2	36	36	0
Scenario 2	168,443	1245	243	9	36	36	0
Scenario 3	16,844	311	61	2	36	36	0

Fig. 2.

Results of Scenario 1 for the male population.

Fig. 3.

Results of Scenario 2 for the male population.

Fig. 4.

Results of Scenario 3 for the male population.

2.2. Benefits

2.2.1. Calculation of benefits

Our benefit-risk model has four benefit endpoints (Fig. 1): preventable COVID-19 cases, hospitalizations, ICU admissions, and deaths. We selected these endpoints since they are some of the most explicit, trackable, and quantifiable outcomes that have the greatest public health significance. To calculate the potential COVID-19 cases preventable by vaccine ($C_P$), we used Eq. (1).

$$C_P=\frac{I_C}{P_U}LD{E}_C$$ (1)

where $I_C$ is the COVID-19 case incidence rate, $P_U$ is the proportion of the population that is at risk (i.e., unvaccinated), L is the duration of vaccine protection, D is the number of individuals fully vaccinated with two doses of vaccine (fixed at 1 million to provide a consistent metric that allows benefit-risk comparison among different age/sex subgroups regardless of potential variation in vaccine uptake among those subgroups), and E_C is the vaccine efficacy against COVID-19 cases. In this analysis we did not model individuals who obtain partial protection after exposure/recovery from infection, which may lead to somewhat over-estimating the benefit of the vaccine. For preventable COVID-19 hospitalizations ($H_P$), we use a similar equation (Eq. (2)) in which we consider the COVID-19 hospitalization incidence rate ($I_H$) and vaccine efficacy against hospitalization ($E_H$).

$$H_P=\frac{I_H}{P_U}LD{E}_H$$ (2)

The preventable COVID-19 ICUs ($I_P$) and preventable COVID-19 deaths ($D_P$) are fractions of $H_P$, such that $I_P=f_{\mathit{IH}}{H}_P$ and $D_P=f_{\mathit{DH}}{H}_P$. $f_{\mathit{IH}}$ denotes the fraction of hospitalizations going to ICU and $f_{\mathit{DH}}$ denotes the fraction of hospitalized patients who die. We perform these calculations over the individual age and sex subgroups and combined male and female subgroups.

2.2.2. Data and assumptions

2.2.2.1. Duration of vaccine protection

We assumed the vaccine has at least 6 months of protection since this is the period examined by Pfizer in their ongoing study at the time of this analysis [4]. The model assesses the benefits for a period of 6 months post 2nd dose of vaccination. For the sensitivity analysis in the supplement, we use a protection period of 12 months in Scenario 5. For simplicity, the model does not account for the benefits of partial vaccination (partial protection between the first and second dose).

2.2.2.2. Incidences of COVID-19 case, hospitalization, ICU, and death

We assumed the incidence rates of COVID-19 cases and hospitalizations remain constant over the assessment period (6 or 12 months post 2nd dose). We obtained the incidence rates of COVID-19 cases for the week of July 10, 2021 from CDC COVID Data Tracker for all sex/age subgroups. Four-week averages of hospitalizations (June 26–July 10, 2021) are used due to the variability in rates given the small numbers of hospitalizations per age/sex subgroups. We estimated $f_{\mathit{IH}}$ (the fraction of hospitalizations going to ICU) and $f_{\mathit{DH}}$ (the fraction of hospitalized patients who die) based on cumulative rates of hospitalizations, ICUs, and deaths for each sex/age subgroups reported on COVID NET since March 2020. All the incidence data for these factors are summarized in Table 2 . Comparing the incidence from the 2nd week of August 2021 (the time of our analysis) with those reported in July 2021 (around the pandemic low), which is used in scenario 1 and considering the continuing upward trend at the time of analysis, we assumed a 10× incidence and 4× hospitalization increase over a 6-week period.

Table 2.

Vaccine coverage and COVID incidences by sex and age subgroups.

Sex	Age subgroup	Population1	Vaccinated population2	COVID-19 cases/100 k persons2	Hospitalizati ons/100 k persons3	Percent of hospitalized going to ICU3	Percent of hospitalized who die3
Female
	16–17	4,119,686	1,985,672	47.9	1.593	19.5	0.7
	18–24	14,923,948	8,033,040	64.6	2.025	8.1	1
	25–29	11,428,122	5,918,524	68.6	2.45	5.9	0.3
Male
	16–17	4,300,731	1,826,299	42.9	0.35	32.7	0.7
	18–24	15,633,953	7,217,945	53.3	0.8	22.2	0.6
	25–29	12,036,982	5,592,473	57.8	0.875	22.7	1.5

Source: 1-CDC Wonder, 2-COVID Data Tracker, 3-COVID NET.

2.2.2.3. Unvaccinated population

We estimated the unvaccinated population among each age/sex subgroup using US census data and “Age groups of people with at least one dose” from COVID data tracker. Data for Texas were not contained in COVID data tracker at the time of analysis, so we imputed proportional vaccination counts based on population averages from the census data. The incidence of COVID-19 cases and hospitalization, described in section 2.2.2.2 “Incidences of COVID-19 case, hospitalization, ICU and death,” are converted into the incidence of COVID-19 cases and hospitalizations among unvaccinated individuals of each age/sex subgroup.

2.2.2.4. Vaccine efficacy

We used vaccine efficacy rates of protection against COVID-19 cases of 70% and 90% and vaccine efficacy rates of protection against COVID-19 hospitalizations of 80% and 90% in the different scenarios. The high efficacy of 90% represents the lower bound of the confidence interval from the clinical trial data as discussed in [5]. The low efficacy of 70% for cases and 80% for hospitalization represents a conservative efficacy rate given the uncertainty of the vaccine’s protection against the new Delta variant. Early studies on the vaccine’s efficacy against cases from the Delta variant suggested 79% in Scotland [6], 87% in Canada [7], and 88% in India [8]. The low and high values for vaccine efficacy were determined based on critical review of available information and deliberation among subject matter experts within the FDA.

2.3. Risks

2.3.1. Calculation of risks

The vaccine-attributable myocarditis/pericarditis was considered as a key risk post vaccination of COVID-19 vaccine due to their potential severe consequence. Our benefit-risk model has three risk endpoints (Fig. 1): vaccine-attributable myocarditis/pericarditis cases (for the purpose of this analysis we use the sum of reported myocarditis and pericarditis cases), hospitalizations, and deaths, which mirror the benefit endpoints for comparison. Similar to the benefits, these endpoints are the most explicit, trackable, and quantifiable outcomes that have the greatest public health significance. We calculated estimates of excess cases of myocarditis/pericarditis by subtracting the background rate of myocarditis in Optum health claims database sample population from 2019 from the rate of myocarditis in the study window from December 10, 2020 – July 10, 2021 (described in detail in Section 2.3.2.1). We then used Equation (3) to calculate excess cases of myocarditis/pericarditis (M_Exc) per one million fully vaccinated individuals.

$${\mathrm{M}}_{\mathit{Exc}}=\left(M_{Obs1}-M_{Exp1}+M_{Obs2}-M_{Exp2}\right)\ast F$$ (3)

M_obs1 and M_Exp1 are observed and expected myocarditis/pericarditis case rates post dose 1, M_obs2 and M_Exp2 are corresponding case rates post dose 2, and F is a multiplier for unit conversion (described in detail below). Expected myocarditis/pericarditis case rates are the predicted background case rates unassociated with vaccine.

The number of myocarditis hospitalization (M_H) and deaths (M_D) are fractions of excess myocarditis/pericarditis cases (M_Exc), such that M_H = M_Exc * f_HM and M_D = M_Exc * f_DM. $f_{\mathit{HM}}$ denotes the fraction of myocarditis cases that needed hospitalization and $f_{\mathit{DM}}$ denotes the fraction of deaths within myocarditis cases.

2.3.2. Data and assumptions

2.3.2.1. Myocarditis/pericarditis attributable to vaccine

We used myocarditis/pericarditis reports data from the Optum health claims database which is part of the FDA’s BEST system (Table 3 ). The Optum data (through OptumServe) includes enrollment and pre-adjudicated hospital, physician, and prescription drug health insurance claims. The pre-adjudicated claims database includes de-identified claims data for privately insured and Medicare Advantage (MA) enrollees. Hospital and physician claims undergo initial processing daily from many providers across the US who accept patients with health insurance. OptumServe has established an ongoing weekly update schedule to incorporate newly processed claims into the pre-adjudicated claims database. This data source was utilized for this study to reduce the delay between the occurrence of healthcare services and their presence in the database. The pre-adjudicated claims have an approximately two-month delay for 90% completeness for inpatient claims and over 70% completeness at one-month for outpatient claims. Acumen reports cases of myocarditis in 7-, 21-, and 42-day risk windows from each vaccine dose. Our analysis focuses on the 7-day risk window post vaccination where most cases are found for all subgroups.

Table 3.

Estimated excess number of myocarditis/pericarditis cases per 1 million people fully vaccinated with Comirnaty. 95% confidence intervals are shown in brackets.

Sex	Age (years)	Rate of excess myocarditis/pericarditis per 1 million fully vaccinated
Male
	16–17	196 [36, 424]
	18–25	131 [27, 224]
	26–35	49 [0, 123]
Female
	16–17	36 [0, 298]
	18–25	57 [9, 147]
	26–35	2 [0, 80]

Source: Optum Database pre-adjudicated claims 12/11/2020–07/10/2021.

The database contains rates of expected (M_Exp1 and M_Exp2) and observed (M_Obs1 and M_Obs2) myocarditis/pericarditis in 100 k person-years for the 1st and 2nd dose of the vaccine. We first divided the rates given in units of per 100 k person-years by 365 days and then multiplied by 7 days (risk window) to derive the total number of myocarditis/pericarditis cases among the 100 k fully vaccinated. Then, we multiplied that number by 10 to derive the total cases among one million full vaccinations. As a result, converting from 100 k person-years to one million vaccinated individuals’ risk in a 7-day risk window post vaccination, we multiplied the rates by a factor F = (7 * 10)/365 as seen in Eq. (3). Confidence intervals for the myocarditis cases are calculated using the chi-square method for Poisson distribution of rare events [9].

Vaccine attributable myocarditis/pericarditis is a rare outcome, and its statistical analysis has limited power. The only data available for our initial analysis (3.1, 3.4) was Optum’s data, which is a part of FDA’s BEST system. Subsequently, CDC presented an analysis using the Vaccine Safety Datalink (VSD) [10] data up to October 9, 2021. The estimated excess rate of myocarditis from their analysis for 12–17 year old’s was 54 cases per million fully vaccinated individuals of both sexes combined, which is approximately half of the Optum rate (116 cases per million) for the same age subgroup of combined sexes [10]. The case rate for male age 16–17 years is likely higher than the rate for overall age 12–17 year old’s based on both US [11] and Israeli [12] data. Since we have no access to VSD data for smaller age/sex subgroups, we conducted a sensitivity analysis and present risk outcome results as a range with lower bound based on half of the Optum myocarditis/pericarditis rate (to simulate the VSD rate) and an upper bound based on the Optum rate. Another reporting service, the Vaccine Adverse Event Reporting System (VAERS) is a passive US national early warning system for potential safety problems associated with vaccines licensed for use in the US. We decided not to use data from VAERS as it has the significant drawback of the potential for underreporting or misreporting of cases due to the voluntary reporting nature of passive surveillance.

2.3.2.2. Myocarditis/pericarditis hospitalization and death rate

Almost all adolescent and young adult patients with suspected myocarditis/pericarditis cases are hospitalized and monitored for the condition. In this model, we assume all myocarditis/pericarditis cases are hospitalized. The Vaccine Safety Database (VSD) data show one day being the median length of stay, primarily for clinical observation [10].

A total of 1061 myocarditis/pericarditis cases and two deaths were reported through VAERS among US individuals <30 year of age post vaccination. After complete review of medical records, CDC concluded these two deaths were unlikely to be caused by the vaccine. In our model, we assume the death rate related to the vaccine is most likely to be zero in the base-case and most likely scenarios (Scenario 1 and 2). However, we use 2/1,061 as the death rate for the worst-case scenario (Scenario 3) to account for the very unlikely outcome of vaccine-attributable myocarditis/pericarditis death in those two cases.

3. Results

We developed a Microsoft Excel model (available upon request) for our analysis as described in the Methods section. This section summarizes the results for the three main model scenarios and the alternative scenarios. The upper bound of risk outcomes represents Optum’s data and the lower bound represents the approximation of VSD data (half the rate from Optum data as described above in Section 2.3.2.1 “Myocarditis/pericarditis attributable to vaccine”).

3.1. Scenario 1: Base case

Our model scenarios starting with the base case used the latest available incidence data of July 10, 2021 and assumed a 6-month vaccine protection period, 90% vaccine efficacy against both COVID-19 cases and hospitalizations, and a myocarditis/pericarditis death rate of zero.

Figs. 2, S13, and S14 summarize the results for analyses of male only, male/female combined, and female only categories, respectively. The results indicate a clearly favorable benefit-risk for males/females combined, females only, and males >18 years old. For males 16–17 years old, the model also predicts far more prevented COVID-19 cases compared to excess myocarditis/pericarditis cases. Depending on the data sources for myocarditis/pericarditis, the model estimates of prevented COVID-19 hospitalizations are either higher (VSD) or lower (Optum) than the vaccine attributable myocarditis/pericarditis hospitalizations. The model predicted 142 prevented COVID-19 hospitalizations vs. 196 vaccine-attributable myocarditis/pericarditis hospitalizations (based on Optum data) and 98 (based on half of Optum rate for approximation of VSD data) among males 16–17 years. Hospitalizations associated with COVID-19 have more severe clinical outcomes than hospitalization due to myocarditis/pericarditis. Hospitalization due to COVID-19 typically requires longer stays (median three to six days [15], [16]) with more potential care including intensive care and mechanical ventilation. In contrast, for vaccine-attributable myocarditis/pericarditis the median length of stay in the hospital is one day [10] and is primarily for monitoring the condition without intensive treatment. The majority of cases resolved in a few days. For this reason, we still consider the benefits of the vaccine to outweigh the risks in this scenario for males aged 16–17 years old. See Table 4 for details of the benefit-risk results for 16–17-year-olds.

3.2. Scenario 2: Most likely scenario

We constructed a scenario that we believed most likely represented the short-term moving direction of the pandemic at the time of analysis. We assumed 6-month vaccine protection, 10x higher COVID-19 case incidence, and 4x higher COVID-19 hospitalizations incidence compared to the incidence on July 10, 2021. We also assumed lower vaccine efficacy (70% against COVID-19 case, 80% against hospitalization) against the newly emerging virus variants such as the Delta or Omicron variant. We assumed a myocarditis/pericarditis death rate of zero based on CDC medical review of vaccine-attributable myocarditis.

Figs. 3, S15, and S16 summarize the results for analyses of male only, male/female combined, and female only, respectively. For all age/sex subgroups and across all attributes, the benefits clearly outweigh the risks in this scenario. See Table 4 for details and benefit-risk results for the 16–17-year-old age subgroup.

3.3. Scenario 3: Worst case scenario

We also constructed the worst-case scenario using the most conservative assumptions for all the model inputs. We assumed 6-month vaccine protection, the COVID-19 incidence as of July 10, 2021, 70% vaccine efficacy against COVID-19 case, 80% vaccine efficacy against COVID-19 hospitalization, and 0.002% (2/1,061) myocarditis/pericarditis death rate based on two initially reported deaths. CDC later determined the causes of these two death cases were not associated with vaccine after medical chart review.

Figs. 4, S17, and S18 summarize the results for analyses of male only, male/female combined, and female only, respectively. Even with the conservative assumption on myocarditis/pericarditis death rate, the model predicts no deaths associated with myocarditis/pericarditis compared to one prevented COVID-19 death among males aged 16–17 years. The model predicts 127 prevented COVID-19 hospitalizations vs. 196 myocarditis/pericarditis hospitalizations (Optum data) and 98 (VSD data approximation) for males 16–17 years old. Considering the differential clinical outcomes of the hospitalization from the two different causes (i.e., COVID-19 outcomes are more severe than myocarditis/pericarditis), we considered the benefits of the vaccine to still outweigh the risks in this “worst-case scenario”. See Table 4 for details of the benefit-risk results for the 16–17-year-old age subgroup.

3.4. The alternative scenarios

We altered key model inputs including vaccine protection duration, vaccine efficacy, COVID-19 case and hospitalization incidence rates, vaccine-attributable myocarditis/pericarditis cases (Tables 1 and S1) for alternative scenarios (i.e., Scenario 4–7). Results of the simulated alternative scenarios are in Figs. S1–S12 and we observed that overall benefits of CVmRNA outweighed its risks.

4. Conclusions and discussion

Our results demonstrate that the benefits of CVmRNA clearly outweigh its risks for all age and sex subgroups we analyzed. Under the base case scenario and the worst-case scenario (Scenario 1 and 3), we predicted a higher number of myocarditis/pericarditis hospitalizations than the COVID-19 hospitalizations among males 16–17 years old; however, considering the differential clinical implications of COVID-19 and myocarditis/pericarditis hospitalization, we still consider the benefits of the vaccine to outweigh its risks. Moreover, under all other scenarios including the most likely (Scenario 2), our model predicts that preventable COVID-19 cases, hospitalizations, and deaths exceed the myocarditis/pericarditis cases and related hospitalizations and deaths for all age and sex subgroups.

We note that COVID-19 incidence highly influences the predicted benefits of the vaccine. If the disease incidence is higher, the benefits of the vaccine will be greater, and vice versa. Therefore, the benefit-risk conclusion may change if the COVID-19 incidence rate becomes very low in the future. Also, “the worst-case scenario” (in term of vaccine benefit) presented here is the worst only among the modeled scenarios. Scenarios worse than Scenario 3 could occur if the data fall outside the ranges of model inputs we used, such as lower COVID-19 incidence than those reported on July 10, 2021, lower vaccine effectiveness against COVID-19 cases (<70%) and against hospitalizations (<80%), and shorter vaccine protection duration (<6 months).

Our approach has several limitations. First, the constant COVID-19 incidence rate assumption in our model generates high uncertainty on the estimate of benefits considering the uncertain dynamics of the pandemic. Our analysis was conducted before the emergence of the Omicron variant. The estimated benefits of the vaccine would decrease if the vaccine became less effective against novel variants of COVID-19, but benefit would increase if COVID-19 incidence increased. The durability of vaccine protection is another source of uncertainty for the model. Any significant waning of vaccine-induced immunity before 6 or 12 months would reduce the benefit of the vaccine.

Additionally, there is uncertainty in the myocarditis/pericarditis case rates attributable to the vaccine. Three sources of data for vaccine attributable myocarditis/pericarditis case rates are available: Optum, VSD, and VAERS. Optum is a health claims database in FDA’s CBER BEST System which has inherent limitations, such as small sample sizes and imperfect sensitivity of ICD-10 codes to identify these rare outcomes. We also acknowledge that the cases were not validated by a complete review of patient’s medical records. The lack of chart review validation may lead to overestimation of myocarditis/pericarditis case rate. However, for myocarditis in the young population, arguably these symptomatic cases will be medically seen and hospitalized and captured in health insurance. One key advantage of VSD data is that it is chart confirmed by CDC. However, VSD has a smaller database than the BEST system and its generalizability is limited to patients who have health maintenance organization (HMO) insurance with a high emphasis on wellness/preventative care. In addition, age/sex specific myocarditis/pericarditis case rates are not available from VSD data. VAERS includes all reported cases regardless of causality associated with vaccine. Underreporting and lack of a denominator to determine a rate are critical issues associated with VAERS voluntary reporting. Although VAERS is efficient for rapid detection of potential safety signals, it is not fit for the purpose of determining the event rates. To estimate the myocarditis/pericarditis case rate attributable to the vaccine, we consider Optum and VSD data to be more appropriate. Therefore, the results based on these two sets of data are presented as the range of risk outcomes in this paper.

A third limitation is that some benefit-risk endpoints in our assessment are difficult to compare directly, for example, hospitalizations from COVID-19 and myocarditis hospitalizations. In addition, we did not model partial protection after exposure/recovery from infection. There was no high quality data on the seroprevalence among the population at the time of analysis, and it is unclear what level of antibody is sufficient to render protection against COVID-19 and how long that protection could last. This may lead to an over-estimate of the benefit of the vaccine to some degree. This benefit-risk assessment does not consider the potential long-term health impacts of COVID-19 or myocarditis. Also, it does not include secondary benefits and risks, such as the benefit of the vaccine on controlling the pandemic and reducing the viral transmission in the population. In this analysis, we did not investigate the benefits and risks of subpopulations with specific comorbidities due to limited information. The benefit-risk profile could be different depending on an individual’s health condition.

This benefit-risk assessment helped to inform FDA’s regulatory decision on CVmRNA licensure. FDA considered the impact of COVID-19 pandemic on the public health, unavailability of treatment options, urgent need for a vaccine to prevent the disease and control the pandemic, together with available evidence and uncertainty associated with vaccine effectiveness and vaccine-attributable myocarditis/pericarditis risk. “The FDA Review Committee is in agreement that the risk/benefit balance for COMIRNATY is favorable and supports approval for use in individuals 16 years of age and older” [13]. The purpose for publication of this benefit-risk assessment is to increase the transparency of regulatory action, communicate to the public that the benefits of the CVmRNA vaccine clearly outweigh the risks even among the male adolescent population that is at higher risk of myocarditis/pericarditis, increase the public confidence of the vaccine, and promote vaccination to fight against the COVID-19 pandemic. However, the benefit-risk assessment is an iterative process. We may need to reassess the benefits and risks of the vaccine in the future if the pandemic slows down or if the emergence of a new variant leads to significant reduction of vaccine efficacy.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary material

The following are the Supplementary data to this article: