The critical role of background rates of possible adverse events in the assessment of COVID-19 vaccine safety

Abstract

Beginning in December of 2019, a novel coronavirus, SARS-CoV-2, emerged in China and is now a global pandemic with extensive morbidity and mortality. With the emergence of this threat, an unprecedented effort to develop vaccines against this virus began. As vaccines are now being introduced globally, we face the prospect of millions of people being vaccinated with multiple types of vaccines many of which use new vaccine platforms. Since medical events happen without vaccines, it will be important to know at what rate events occur in the background so that when adverse events are identified one has a frame of reference with which to compare the rates of these events so as to make an initial assessment as to whether there is a potential safety concern or not. Background rates vary over time, by geography, by sex, socioeconomic status and by age group. Here we describe two key steps for post-introduction safety evaluation of COVID-19 vaccines: Defining a dynamic list of Adverse Events of Special Interest (AESI) and establishing background rates for these AESI. We use multiple examples to illustrate use of rates and caveats for their use. In addition we discuss tools available from the Brighton Collaboration that facilitate case evaluation and understanding of AESI.

1. Introduction

Beginning in December of 2019, a novel coronavirus, SARS-CoV-2, emerged in China and is now a global pandemic with extensive morbidity and mortality. With the emergence of this threat, an unprecedented effort to develop vaccines against this virus began. Several manufacturers have now developed and evaluated vaccines in less than one year; some are now being used extensively under emergency or conditional use authorizations in many countries. While unprecedented and based upon thorough evaluation of high quality data, this rapid development and authorization process has meant that vaccines have been introduced for use in the population based on studies with limited population heterogeneity (e.g., no children, pregnant, immunocompromised persons), and limited follow up (~2 months) on recipients in clinical trials. Furthermore, despite the fact that large phase III trials were conducted and completed, the “warp speed” with which the vaccine development process proceeded has augmented vaccine hesitancy concerns expressed by many anti-vaccination groups, the general public, and some health care workers that the process was “rushed” or incomplete. These concerns have arisen despite the fact that large phase III clinical trials were appropriately conducted and completed and that while several administrative and regulatory factors helped to speed up the process, none of the routine safety and efficacy evaluation steps have been bypassed. Indeed, this speed was possible for several reasons: First, because of the emergency, abundant funding was available; second, clinical studies that were usually done consecutively were done in parallel and internationally; third, enrollment into the trials was very rapid as many volunteers wanted to participate; and finally, regulators conducted rolling reviews of studies minimizing regulatory time delays.

Evidence regarding vaccine safety plays a key role in public acceptance of a new vaccine. The most convincing evidence of a causal relationship between a new vaccine and an adverse event derives from randomized controlled trials (RCT) examining whether a statistically significant higher rate of an adverse event (e.g., fever) occurs in the vaccinated group compared to the control group, plus other factors such as biologic plausibility and clustering of onset interval (time elapsed between vaccination and onset of the adverse event). We must recognize that even though these phase III COVID-19 vaccine trials were large (N ~ 30,000–60,000), the follow-up information on individuals in the trials was relatively short prior to authorization (8–12 + weeks) and more importantly, any large phase III trial has limited statistical power to detect rare events that might be be recognized when millions and even billions of people globally are vaccinated to control a pandemic. For example, the risk of anaphylaxis following the first two m-RNA COVID-19 vaccines appears to be ~2.8–5/million doses from post-introduction pharmacovigilance - a rate that was too low to be detected in the phase III trials which as currently conducted have an ability to detect events that occur at a frequency of 1:10,000 [1]. Using a combination of passive [e.g., UK Yellow Card, US Vaccine Adverse Event Reporting System (VAERS)] and active [e.g., EU Access pe, US Vaccine Safety Datalink (VSD and V-Safe] and Global Vaccine Data Network surveillance systems are essential to provide clear and transparent information on the safety of COVID-19 vaccines when they are in widespread use.

Once a new vaccine is approved by a National Regulatory Authority [NRA) and recommended for use by the National Immunization Technical Advisory Group, many experts state that it is no longer ethical to withhold a vaccine from an approved target population [2]. Therefore, the ability to infer causality for adverse events from an RCT by comparison with a randomized unvaccinated arm may no longer be feasible. Since humans fall ill due to many causes in the absence of COVID-19 vaccine (including COVID-19 disease), how do we then distinguish if an adverse event is causally related to the COVID-19 vaccine or not?

The first way is to see if there is unique mechanistic laboratory-based evidence. For example, a vaccinee is found to have developed a disease caused by the virus strain such as isolation of mumps vaccine strain virus from the CSF of a patient with aseptic meningitis [3]. The second way is if vaccine recipients develop a unique or relatively unique clinical syndrome not otherwise found in this population [4]. The third way is if the adverse event recurs after a second dose of the same vaccination, i.e. a “re-challenge” phenomenon [5]. The fourth way is to collect data in as unbiased a manner as possible from active surveillance observational epidemiologic studies to detect whether the rate of a given event or syndrome in vaccinated individuals exceeds that expected among unvaccinated individuals.

In the modern era of rapid dissemination of rumors on social media, a rapid response to a vaccine safety signal is needed to maintain public confidence. Thus, now that large vaccination programs have started, it is critical to provide ongoing information to the public on the safety of the vaccines rapidly and transparently. Already, a concern regarding the risk of anaphylaxis following either of the m-RNA vaccines as well as death in the frail elderly has raised public concern [6], [7]. Because the first three ways of establishing causal relationship between a vaccine and an adverse event are quite rare, the fourth way, observational epidemiologic studies, is the main method available. In such studies one investigates whether the occurrence of the event changes after vaccination as any factor that is causal should have an impact on disease occurrence. One then compares the risk of an event between vaccinated and unvaccinated individuals or, in some analytic frameworks, between exposed and unexposed follow up time in the same individual [8].

A rapid response to a vaccine safety signal is key to maintaining public confidence in mass vaccination programs. Traditionally when a signal is identified, the first step is to calculate whether the occurrence of the event is more frequent than one would expect. In order to do this, one needs background rates of the occurrence of the event to assess what the incidence is in the absence of a vaccine. Such background rates are preferably established before vaccine introduction and therefore a list of adverse events of interest should be established.

Accordingly, here we describe two key steps for post-introduction safety evaluation of COVID-19 vaccines:

• •Defining a dynamic list of Adverse Events of Special Interest (AESI)
• •Establishing background rates for these AESI

2. Defining adverse events of Special Interest (AESI) for COVID-19 vaccines

Given that COVID-19 is a new disease and SARS-CoV-2 is a newly emerged and still emerging virus, attempts have been made to systematically try and understand what safety events might occur following a variety of SARS-CoV-2 vaccine platforms. As part of its work to support the safety assessment of vaccine development funded by the Coalition for Epidemic Preparedness (CEPI), the Brighton Collaboration supported the Safety Platform for Emergency vACcines.

(SPEAC) project which has developed a list of Adverse Events of Special Interest (AESI) for each pathogen targeted by CEPI. AESI are events that have not been associated with COVID-19 vaccines, but are events for which regulators and public health authorities need to be prepared to address should a signal occur. The May 2020 version of this list has subsequently been endorsed by the WHO Global Advisory Committee on Vaccine Safety and adaptations have been made by the EMA funded ACCESS project [9], the Global Vaccine Data Network [10], and public health agencies in many countries. The AESI list is reviewed and updated quarterly <https://brightoncollaboration.us/wp-content/uploads/2021/01/COVID-19-updated-AESI-list.pdf> and contains four categories of outcomes: events that have been associated with vaccines in general such as anaphylaxis, events associated with a specific vaccine platform such as thrombocytopenia following live attenuated measles vaccines, events that have been associated with COVID-19 disease itself such as myocarditis, and a fourth emerging category of adverse events that have been observed in COVID-19 vaccine clinical trials such as neuroinflammatory disorders (e.g. transverse myelitis). In addition to the list developed by the SPEAC project, other similar lists have been developed by the US CDC, the European EMA funded ACCESS project, the US FDA, the Global Vaccine Data Network and by agencies in other countries such as the MHRA and HPA in the UK and agencies in Australia and New Zealand. Table One summarizes these outcomes and compares those from different sources. Importantly, one should recognize that lists of AESIs are dynamic and will be updated as knowledge regarding possible events of concern are identified.

3. What are background rates and how can they be used?

Most simply stated, the background rate of any adverse event (e.g., thrombosis, heart attack, seizure, death) is the incidence rate of the event one would observe in a given population in the absence of receipt of the vaccine being tested or any other intervention. This background rate can be assessed in the general population, in a subpopulation such as pregnant women or children, or in populations with pre-specified co-morbidities. The background rate is used to calculate the number of expected cases of an event in a given population and time period in the absence of vaccine or other intervention that can be compared with the number observed following vaccination. Preferably such data are reasonably contemporaneous given that event incidence can change over time even without a vaccine being introduced.

In order to understand the rationale for and use of background rates, it is important to understand the difference between temporal association and causality. It is often said that “bad things happen all the time”. That is, in the absence of the use of a vaccine, adverse events or “bad things” will occur in a population because most diseases have multiple causes. Under routine conditions without receipt of vaccine, people will develop transverse myelitis, anaphylaxis, or die unexpectedly. In addition, pregnant women will experience spontaneous abortions. The fact that such events occur within a plausible time window following vaccination does not necessarily mean that the vaccine caused the event, i.e. temporality is not causality. However, an observation of temporal association can provide a hypothesis for further evaluation. A given health event might have occurred without the individual having received a vaccine dose, as is frequently observed in the placebo arms of clinical trials. To epidemiologically assess whether the vaccine is associated with an event, one must show that the risk of the event is higher than in a comparable non-vaccinated group.

As explained above, many background rates for AESI can be assessed in advance of the introduction of a vaccine and thus can be used to rapidly evaluate and respond to potential safety concerns after vaccine rollout. Background rates can be obtained from the literature or generated de novo from retrospective analysis of electronic healthcare databases. Availability of background rates is important for rapid evaluation of a safety signal that may be identified through passive reporting or through social media. In the face of a dramatic media report of a serious event that followed a new vaccine at the start of a public vaccination campaign, knowing and communicating the expected background rate can provide reassurance to the public. If a cluster of adverse events that is potentially vaccine-associated is observed, one can rapidly calculate the rate of this event in vaccinees and compare this to the “expected number of events” or background rate. If the rate of the event in vaccinees is equal to or less than the background rate, then one can be reassured that the events may have occurred following vaccination by chance. However, one should not use or apply rates without due caution as rates can vary over time, by age, sex, geography and other factors as discussed below. It should also be noted that background rates in a comparison group or in unexposed follow up time are a critical component of active surveillance studies using a cohort or self control case series approach respectively.

For events in Table 1 , case definitions and evaluation tools are being prepared by the SPEAC project. These are available on the Brighton Website <https://brightoncollaboration.us/covid-19/> and discussed further below. For convenience we have categorized illustrative examples of background rates as relatively common, somewhat rare, and extremely rare in Table 2 . Some outcomes are remarkably common; for example, up to 21.2% of all pregnancies end in a spontaneous abortion in women 18–24 years of age in Finland. Other outcomes are very rare such as Kawasaki disease in Europe with only 1.55 cases occurring in a population of 10 million people being observed over a one week time period.

Table 1.

Adverse Events of Special Interest (AESI) proposed by SPEAC, USA Center for Disease Control and Prevention (CDC), USA Food and Drug Agency (FDA) and/or European Medicines Agency (EMA).

*FDA included AESIs are part of draft plans for determining background rates.

¹Specify encephalopathy and ataxia as separate entities. For SPEAC these are included as part of encephalitis/ADEM.

²Specify chronic inflammatory demyelinating polyneuritis(CIDP), multiple sclerosis, optic neuritis

³Specify other acute demyelinating diseases.

⁴Includes single organ cutaneous vasculitis.

⁵Includes Kawasaki disease, autoimmune diseases (CDCP specifies for VAERS reporting but not for VSD surveillance).

⁶Specify and distinguish hemorrhagic and non-hemorrhagic stroke.

⁷EMA specifies adverse foetal outcomes as well as pregnancy outcomes.

⁸Brighton case definition exists for Sudden unexpected death in infancy (up to age 2 years). None for older individuals.

Table 2.

Selected background rates by person time, and number expected in an immunized population of 10 million within various time periods (one day, six weeks,) showing differences in different populations.

Outcome	Population	Age Group (yrs)	Rate/100,000 person-years (except where noted)	Number of events/10 million vaccinees without vaccine in various time windows			Rate used to estimation of events/10 million vaccinees
				One day	One week	Six weeks
Common
Myocardial Infarction	Japan [15]	All
	1979		7.4
	2008		27.0	7.40	51.8	310.1	Japan Rate for 2008
	USA [16]	>30
	2000		287
	2008		208	57.0	398.9	2393	USA Rate for 2008
Preterm Labor or Delivery (>37 wk) [15]			% of Pregnancies
	Finland	All	5.7% pregnancies
	France		6.27% pregnancies
	Sweden		5.8% pregnancies
	USA		10.4–11.5% pregnancies
Seizures [15]	Finland	0–17	106.6	27.40	192.3	1,154	USA Rate for > 18 years old
		18–44	23.44
		45–64	39.72
		≥65	54.64
	Switzerland	0–4	460
	USA	>18	100
Spontaneous Abortion [17]	Australia		% of Pregnancies
		18–23	3.5%
		25–30	9.8%
	Finland	28–33	14.5%
		18–24	21.2%
		25–29	12.1%
	UK	30–34	11.9%	197	2,780	16,684	UK Rate for all ages
	USA	All ages	12.0%
		<24	10.4%
		25–29	13.6%
		30–34	22.3%
Rare
Anaphylaxis	USA National [18]	0–18
	2006		10.1
	2008		24.9	6.82	47.7	286.5	USA 2008 all age rate
	USA New York [19]	0–18
	2008		1.0
	2014		4.7
	Korea [20]
	2008	All Ages	16.02
	2010	All Ages	19.42
	2014	All Ages	32.19
		0–19	21.26
		20–39	24.23
		40–69	28.47
		≥70	29.49	8.07	56.6	339.3	Korea ≥ 70 year old rate
Bell’s Palsy	Israel [21]	1–4	18.9
		5–14	30.9
		15–24	47.7
		25–34	72.3
		35–44	91.1
		45–54	118.4
		55–64	154.4
		65–74	190.9
		75–84	190.74	44.39	310.69	1,864	Background rate in Males > 85 yo
		>85	162 M; 102 F, 125 overall
	USA [22]	0–9	4.2
		10–19	15.3
		20–29	24.0
		30–39	30.9
		40–49	35.2
		50–59	35.8
		60–69	37.1
		≥70	52.7
		All ages	25.2	6.9	48.3	289.0	Background rate all ages USA
	Italy	All ages	53.3
Death	USA [23]	15–24	74.0				US All Cause Mortality rates
	2017	25–34	132.8
		35–44	195.2				USA Age 35–55
		45–54	401.5
		55–64	885.8
		65–74	1791
		75–84	4473
		≥85	13,574				USA Age >85
	Guinea-Bissau [24]	All	3135 M; 2565 F				Crude death Rate INDEPTH DHSS
Very Rare
Transverse Myelitis	USA	10–17	0.7 M; 0.4 F
	N. California [25]	18–25	0.4 M; 1.1 F
		26–62	2.4 M; 4.9 F
	USA
	New Mexico [26]	All	0.46	0.126	0.882	5.29	Background rate for all ages in New Mexico
	Finland [17]	0–17	7.27
		18–44	4.06
		45–64	5.39
		≥64	9.04
Guillain Barre	USA [27]	≤17	0.81
		18–39	1.34
		40–59	2.84
		≥60	3.25				USA Background rate for all ages
		All ages	1.68	0.46	3.22	19.3319.33
	Taiwan [28]
		0–9	0.76
		10–19	0.56
		20–29	0.92
		30–39	1.04
		40–49	1.36
		50–59	2.12
		60–69	4.10
		70–79	6.35
		≥80	6.34
		All ages	1.65
Kawasaki Disease	Europe [29]	All	0.81 M; 0.52 F	0.222	1.55	9.32	European Advance Project rate in males.
Multiple Sclerosis  [17]	UK [30]	25–29	5.57 M; 10.75 F
		30–39	5.78 M; 16.05 F
		40–49	7.22 M; 18.04 F	4.94	34.6	207.6	UK Background incidence rate in Females age 4--49
		50–59	5.64 M; 8.77 F
		60–69	2.69 M; 3,23 F
		70–74	0.32 M; 0.21 F
Narcolepsy	Europe [29]	All	1.04 M ; 1.12 F	0.307	2.15	12.9	European Advance Project rate in females.

M = males F = female.

Other rates are available from the Brighton Collaboration Website https://brightoncollaboration.us.

4. Potential issues and caveats regarding the use of background rates

While it is easy to search the literature and identify potential sources of background rates for most outcomes, care must be taken to use appropriate comparisons and to be aware of the limitations of the data. In order to reliably evaluate the rate of an AESI or other event and compare it to a background rate, a standard case definition should be used to validate cases. However, it is important to recognize that many published background rates are “crude” rates in that the cases have not been validated. For appropriate comparisons, “like should be compared with like” – that is, validated rates with validated rates and unvalidated rates with unvalidated rates. In addition, due to changes over time and by season for some events, rates should be compared in a similar time frame when possible. This is especially critical for rare events.

For a given event, several factors may impact on a background rate including: the year(s) when measured; the age and gender distribution of the population, geographic location, co-morbidities, socioeconomic status, medication use, and study methodology. Some specific examples are provided in Table two and discussed below.

As can be seen, rates vary by the time period of observation. For example, there was a three-fold increase in the rate of acute myocardial infarction (AMI) in Japan between 1979 and 2008. Variation is also seen in the rates of anaphylaxis in both the US and Korea with rates more than doubling in all three studies sited, and primarily among women. Such trends may be due to changes in reporting, but the presence of the same trend in the US and Korean studies for anaphylaxis over time means that the true incidence may also be increasing. Thus, caution must be used in making comparisons for some outcomes using rates from the distant past.

Background rates can also vary by age and sex for many outcomes. This is quite dramatic for a neuroinflammatory disease such as multiple sclerosis where the incidence rate is 18.04 cases/100,000p-y in women 40–49 years of age but only 0.21 cases/100,000p-y in women 70–74 years of age – a >85-fold difference. For the same outcome, incidence also dramatically varies by sex with the incidence in 40–49 year olds being 2.5 time higher in females. Similarly, the rate of Bell’s Palsy increases with age and in individuals over 85 years of age is 1.6 times more common in males than females. Background rates can also vary by country. This is unfortunate since rates for many outcomes are only available from selected geographic locations. This is dramatically seen for acute myocardial infarction with the rate in Japan in 2008 being 27 cases/100,000 p-y as compared to 208 cases/100,000 p-y in the US in the same year – a 7.7 fold-difference.

Another issue with comparing rates is that the age strata used to report rates in the literature are often not standardized making direct comparisons of different studies problematic. This can be seen, for example, in Table two where different age strata were used in the reporting of Bell’s Palsy rates from the US and Israel.

Another factor that must be considered in making comparisons of observed post-vaccination data to background rates is the “healthy vaccinee effect”. Since sick people are less likely to be vaccinated, the risk of adverse events may be lower in vaccinees immediately following vaccination [11].

Changes in health care utilization over time may also impact rates. This is highly relevant in the COVID-19 era, where the rates of medical utilization (on which many rates are based) have changed dramatically for many outcomes. Elective procedures have significantly diminished in frequency given surge demands on hospitals but so also have rates of premature births and outcomes related to other infectious diseases [12]. Therefore, for background rates to be most relevant in the evaluation of COVID-19 vaccines, they either need to have been assessed in the post-February 2020 era or possible differences that might have occurred in the most recent time period need to be considered. Thus, historical background rates are very important for rapid assessment of a signal but may have inherent limitations. For a full evaluation of risk, contemporaneous reference cohorts (non-vaccinated or a comparison vaccine) need to be utilized. Several template protocols have been created by the ACCESS project for such evaluations based on electronic health care data or hospital-based data [13].

5. Case evaluation tools

In order to reliably evaluate individual cases or clusters of cases and compare the rates of outcomes with a background rate, it is critical to classify the cases accurately. The Brighton Collaboration has developed standardized case definitions for several outcomes that have generally been considered the gold standard for vaccine safety review. More recently the SPEAC project has developed additional case definitions for AESI that might be associated with COVID-19 vaccines. These definitions are available on the Brighton website and as individual manuscripts that have been published in Vaccine <https://brightoncollaboration.us/category/pubs-tools/case-definitions/>. In addition, the SPEAC project has developed “companion guides” to assist investigators in case evaluation. In addition to the key components of the case definition, along with algorithms to simplify determination of level of certainty, the companion guide summarizes what is known about the outcome, risk factors, information on reported vaccine association, background rates and ICD-9/10 and MedDRA codes. Once potential cases of an adverse event have been evaluated and classified as to their level of diagnostic certainty, a rate of confirmed events can be calculated and a possible association fully evaluated.

An example of this process is the “companion guide” instrument developed for anaphylaxis. Evaluation <Submitted separately during manuscript submission>. The “companion guide” includes a description of risk factors for anaphylaxis, a summary of background rates of anaphylaxis from various countries and age groups from the literature, the Brighton Collaboration Anaphylaxis Case Definition, anaphylaxis diagnostic codes in ICD9/10-CM as well as MedDRA, a prototype data abstraction for medical chart review as well as both tabular and pictorial presentations of key case definition criteria for case classification. The goal of this effort is to attempt to assist with and standardize evaluations of SARS-2-CoV-2 AESI.

6. Summary

Background rates of events can serve as a useful evaluation tool for a rapid initial response to a vaccine safety concern. Many such rates are available on the Brighton website and from the published literature. These may be complemented by more recent rates that are being generated for the post-February 2020 era. To realize the full potential of such rates, tools are available for standardized case classification and assessment. While comparison of background rates is an important tool, it nonetheless must be realized that background rates are only a single tool and that for possible safety signals, the use of other tools such as rapid cycle analyses [14] and epidemiologic observational studies that employ case validation and classification methods need to be considered.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary material

The following are the Supplementary data to this article: