Ascertainment Bias in Anaphylaxis Safety Data of COVID-19 Vaccines

Vaccination against coronavirus disease 2019 (COVID-19) began in the United Kingdom and the United States in December 2020. In the United Kingdom, 3 initial reports of anaphylaxis resulted in a temporary embargo on the vaccination of those with a history of anaphylaxis.¹ Initial US surveillance data reported an anaphylaxis rate of 11.1 cases per million doses for the Pfizer-BioNTech COVID-19 vaccine² and 2.5 cases per million doses for the Moderna vaccine,³ highlighting an apparently much higher occurrence of anaphylaxis reported into Centers for Disease Control and Prevention (CDC) passive vaccine surveillance systems, compared with historical data from existing vaccines, typically around 1.3 per million doses.⁴

The phenomenon of an initial “spike” in adverse event reporting that subsequently “settles” with time is common with safety reporting systems designed to be very sensitive to event detection. As the denominator (number of vaccines given) increases and additional information becomes available, case definitions are revised and reporting fatigue occurs.⁵ As COVID-19 vaccine roll out accelerates, regulatory agencies must constantly review safety decisions on the basis of information generated from relatively small case numbers from the earliest stages, potentially biased by the reporting spike phenomenon. It is imperative to hold the analysis and interpretation of adverse event reporting to the highest standards.

It is not uncommon for any adverse reaction occurring in association with a vaccination to be considered “allergic,” for example, nausea or subjective oropharyngeal symptoms that are often psychogenic and difficult to refute on physical examination, in particular by nonallergy specialists who may lack robust experience in making such determinations. Hoarseness can represent laryngeal edema from an allergic reaction, but also vocal cord dysfunction, which can be anxiety-related.⁶ Evidence from placebo-controlled allergen challenges shows that even urticaria can be stress-related.

Anaphylaxis treatment guidelines advocate the early use of intramuscular epinephrine to treat possible or likely anaphylaxis, but such epinephrine doses may be also administered “early” for only skin symptoms that might have resolved without progression to anaphylaxis. Usage of epinephrine also varies between physicians. Thus, epinephrine use (and resolution of symptoms) cannot be considered a surrogate in a post hoc diagnosis of anaphylaxis,⁷ and we are reassured that none of the currently used scoring schemes does so, though we recognize that, outside the allergy community, use of epinephrine still may be interpreted mistakenly as such.

In the context of new vaccines developed and licensed at an unprecedented pace in response to the COVID-19 pandemic, there is additional pressure to properly distinguish symptoms that are a direct result of mast cell degranulation from other mechanisms of action, to discern true immune-mediated allergic reactions within the larger umbrella of adverse reactions.

The Brighton Collaboration case definition criteria were developed for the remote, post hoc evaluation of vaccine-related adverse events.⁸ However, the Brighton criteria are not used in clinical allergy or in non–vaccine-related research settings. In contrast, allergists in both clinical and research settings apply other, more widely used, clinical criteria for anaphylaxis—National Institute of Allergy and Infectious Disease (NIAID) 2005⁹ and World Allergy Organization (WAO) 2020¹⁰—which differ fundamentally from the Brighton criteria in that they do not contain levels of “diagnostic certainty” (Table I ). We have reviewed the recently published data from the CDC COVID-19 Vaccine Task Force,^{2 , 3} and conclude that, when using the Brighton case definition, a large proportion of the apparent anaphylaxis reactions reported to date do not actually meet the Brighton criteria, despite classification as such (Table II ). Furthermore, on application of the more widely used clinical criteria for anaphylaxis (NIAID/WAO), many (up to 71%) of the Brighton-defined anaphylactic reports for both the Pfizer-BioNTech and Moderna vaccines would not be classified as anaphylaxis (Table II).

Table I.

Comparison of criteria for the grading of anaphylaxis

Brighton Collaboration Criteria8	NIAID Criteria (2005)9	WAO Criteria (2020)10∗
For all levels of diagnostic certainty Anaphylaxis is a clinical syndrome characterized by • sudden onset AND • rapid progression of signs and symptoms AND • involving multiple (≥2) organ systems, as follows Level 1 of diagnostic certainty •≥1 major dermatological AND • ≥1 major cardiovascular AND/OR ≥1 major respiratory criterion Level 2 of diagnostic certainty •≥1 major cardiovascular AND ≥1 major respiratory criterion OR • ≥1 major cardiovascular OR respiratory criterion AND • ≥1 minor criterion involving ≥1 different system (other than cardiovascular or respiratory systems) OR • (≥1 major dermatologic) AND (≥1 minor cardiovascular AND/OR minor respiratory criterion) Level 3 of diagnostic certainty •≥1 minor cardiovascular OR respiratory criterion AND • ≥1 minor criterion from each of ≥2 different systems/categories Major criteria Dermatologic or mucosal •generalized urticaria (hives) or generalized erythema •angioedema,∗ localized or generalized •generalized pruritus with skin rash Cardiovascular •measured hypotension •clinical diagnosis of uncompensated shock, indicated by the combination of at least 3 of the following: •tachycardia •capillary refill time >3 s •reduced central pulse volume •decreased level of consciousness or loss of consciousness Respiratory •bilateral wheeze (bronchospasm) •stridor •upper airway swelling (lip, tongue, throat, uvula, or larynx) •respiratory distress—2 or more of the following: •tachypnea •increased use of accessory respiratory muscles (sternocleidomastoid, intercostals, etc) •recession •cyanosis •grunting Minor criteria Dermatologic or mucosal •generalized pruritus without skin rash •generalized prickle sensation •localized injection-site urticaria •red and itchy eyes Cardiovascular •reduced peripheral circulation as indicated by the combination of at least 2 of • tachycardia and • a capillary refill time of >3 s without hypotension •a decreased level of consciousness Respiratory •persistent dry cough •hoarse voice •difficulty breathing without wheeze or stridor •sensation of throat closure •sneezing, rhinorrhea Gastrointestinal •diarrhea •abdominal pain •nausea •vomiting Laboratory •Mast cell tryptase elevation > upper normal limit	Anaphylaxis is highly likely when any 1 of the following 3 criteria is fulfilled: 1.Acute onset of an illness (minutes to several hours) with involvement of the skin, mucosal tissue, or both (eg, generalized hives, pruritus or flushing, and swollen lips-tongue-uvula) AND AT LEAST 1 OF THE FOLLOWING a. Respiratory compromise (eg, dyspnea, wheeze-bronchospasm, stridor, reduced PEF, and hypoxemia) b. Reduced BP or associated symptoms of end-organ dysfunction (eg, hypotonia (collapse], syncope, and incontinence) 2. Two or more of the following that occur rapidly after exposure to a likely allergen for that patient (minutes to several hours): a. Involvement of the skin-mucosal tissue (eg, generalized hives, itch-flush, and swollen lips-tongue-uvula) b. Respiratory compromise (eg, dyspnea, wheeze-bronchospasm, stridor, reduced PEF, and hypoxemia) c. Reduced BP or associated symptoms (eg, hypotonia [collapse], syncope, and incontinence) d. Persistent gastrointestinal symptoms (eg, crampy abdominal pain and vomiting) 3. Reduced BP after exposure to known allergen for that patient (minutes to several hours): a. Infants and children: low systolic BP (age specific) or >30% decrease in systolic BP b. Adults: systolic BP of <90 mm Hg or >30% decrease from that person's baseline	Anaphylaxis is highly likely when any 1 of the following 2 criteria is fulfilled: 1.Acute onset of an illness (minutes to several hours) with involvement of the skin, mucosal tissue, or both (eg, generalized hives, pruritus or flushing, and swollen lips-tongue-uvula) AND AT LEAST 1 OF THE FOLLOWING: a. Respiratory compromise (eg, dyspnea, wheeze-bronchospasm, stridor, reduced PEF, and hypoxemia) b. Reduced BP or associated symptoms of end-organ dysfunction (eg, hypotonia [collapse], syncope, and incontinence) c. Severe gastrointestinal symptoms (eg, severe crampy abdominal pain and repetitive vomiting), especially after exposure to nonfood allergens 2.Acute onset of hypotension† or bronchospasm or laryngeal involvement after exposure to a known or highly probable allergen for that patient (minutes to several hours), even in the absence of typical skin involvement

AAAAI, American Academy of Allergy, Asthma & Immunology; BP, blood pressure; EAACI, European Academy of Allergy and Clinical Immunology; PEF, peak expiratory flow.

^∗Not hereditary angioedema.

^†WAO Criteria are coendorsed by 50 global allergy societies, but not AAAAI or EAACI.

Table II.

Reported cases of anaphylaxis per million doses of COVID-19 vaccines in the literature, evaluated independently by the authors and final consensus achieved.

Anaphylaxis definition	Certainty	CDC COVID-19 Vaccine Task Force		Non–COVID vaccines 2009-2011, USA4
		Pfizer-BioNTech December 14-23, 20202	Moderna December 21 2020-January 10, 20213
Brighton criteria (reported)	Levels 1-3	21 cases in 1,893,360 doses 11.1/million (95% CI, 6.9-17.0)	10 cases in 4,041,396 doses 2.5/million (95% CI, 1.2-4.6)	33 cases in 25,173,965 doses 1.3/million (95% CI, 0.9-1.8)
Brighton criteria (reassessed)	Levels 1-3	15 cases 7.9/million (95% CI, 4.4-13.1)	5 cases 1.2/million (95% CI, 0.4-2.9)	31 cases 1.2/million (95% CI, 0.8-1.8)
National Institute of Allergy and Infectious Disease		4 cases 2.1/million (95% CI, 0.6-5.4)	3 cases 1.5/million (95% CI, 0.2-2.2)	18 cases 0.7/million (95% CI, 0.4-1.1)
World Allergy Organization		10 cases 5.3/million (95% CI, 2.5-9.7)	4 cases 1.0/million (95% CI, 0.3-2.5)	25 cases 1.0/million (95% CI, 0.6-1.5)

If these more stringent criteria are also applied to the report by McNeil et al⁴ summarizing reactions to non-COVID vaccinations reported to the US Vaccine Adverse Event Reporting System between 2009 and 2011, far fewer cases (defined according to the Brighton criteria) have to be excluded. This is evidence that the Brighton criteria are open to overinterpretation and thus to ascertainment bias. Recent publication of a case series from a large medical center in the United States supports this conclusion, with a similar pattern of reclassification when the Brigton and NIAID criteria were both used.¹¹

Overestimation of the anaphylaxis rate after COVID-19 vaccination may delay or deter vaccination through increased hesitancy. Public health bodies and health care professionals must apply the most accurate criteria on the basis of sufficient clinical information to assign a diagnosis of anaphylaxis in the context of vaccine adverse event surveillance. This may imply that perhaps the Brighton Collaboration Criteria no longer are the best choice for vaccine-related anaphylaxis. The Pfizer-BioNTech vaccine currently appears to be associated with a 3- to 5-fold higher rate of anaphylaxis compared with the rate of anaphylaxis after non-COVID vaccines, but this rate is still very low and in contrast to other allergens used in immunotherapy, no anaphylaxis fatalities have been reported to date. This rate may even fall further with more data points.

The medical community has a duty to ensure transparency and availability of reliable data to inform the public and policy, and to not risk contributing to vaccine hesitancy. Covid-19 vaccine hesitancy is a worldwide threat to timely achievement of COVID-19 herd immunity. A US survey¹² recently showed that more than a fifth of its participants reported vaccine hesitancy, most commonly among 35- to 45-year-old women, a similar cohort to that represented in the CDC COVID-19 vaccine anaphylaxis reports.^{2 , 3} Vaccine safety concern is an important determinant of both vaccine hesitancy and resistance. Thus, early reporting of overestimated anaphylaxis rates to COVID-19 vaccines is a likely contributor to reduced vaccine uptake, to undue referral for evaluation by immunologists or allergists, and to diversion of both human and other resources to hospital-supervised vaccination.

The variable national responses to COVID-19 showed that rapid activation of public health measures was the most effective public health strategy before vaccines became available. Now that vaccination is underway using new vaccines, the allergy community must reassure governments, public health bodies, and vaccination programs to proceed with vaccination at the fastest possible pace.

We propose that the criteria for defining anaphylaxis in the context of vaccine adverse event surveillance should be those used with current clinical and research-based consensus. Anaphylaxis, defined using the most rigorous and clinically relevant criteria, is not much more common with the COVID-19 vaccines than with other vaccines that have been used safely for more than 70 years and are widely accepted by the public.