Diagnostic accuracy of vaccine and vaccine excipient testing in the setting of allergic reactions to COVID‐19 vaccines: A systematic review and meta‐analysis

Abstract

For persons with immediate allergic reactions to mRNA COVID‐19 vaccines, skin testing (ST) to the vaccine/excipients (polyethylene glycol[PEG] and polysorbate 80 [PS]) has been recommended, but has unknown accuracy. To assess vaccine/excipient ST accuracy in predicting all‐severity immediate allergic reactions upon re‐vaccination, systematic review was performed searching Medline, EMBASE, Web of Science, and the WHO global coronavirus database (inception‐Oct 4, 2021) for studies addressing immediate (≤4 h post‐vaccination) all‐severity allergic reactions to 2nd mRNA COVID‐19 vaccination in persons with 1st dose immediate allergic reactions. Cases evaluating delayed reactions, change of vaccine platform, or revaccination without vaccine/excipient ST were excluded. Meta‐analysis of diagnostic testing accuracy was performed using Bayesian methods. The GRADE approach evaluated certainty of the evidence, and QUADAS‐2 assessed risk of bias. Among 20 studies of mRNA COVID‐19 first dose vaccine reactions, 317 individuals underwent 578 ST to any one or combination of vaccine, PEG, or PS, and were re‐vaccinated with the same vaccine. Test sensitivity for either mRNA vaccine was 0.2 (95%CrI 0.01–0.52) and specificity 0.97 (95%CrI 0.9–1). PEG test sensitivity was 0.02 (95%CrI 0.00–0.07) and specificity 0.99 (95%CrI 0.96–1). PS test sensitivity was 0.03 (95%CrI 0.00–0.0.11) and specificity 0.97 (95%CrI 0.91–1). Combined for use of any of the 3 testing agents, sensitivity was 0.03 (95%CrI 0.00–0.08) and specificity was 0.98 (95%CrI 0.95–1.00). Certainty of evidence was moderate. ST has low sensitivity but high specificity in predicting all‐severity repeat immediate allergic reactions to the same agent, among persons with 1st dose immediate allergic reactions to mRNA COVID‐19 vaccines. mRNA COVID‐19 vaccine or excipient ST has limited risk assessment utility.

Abbreviations

COVID‐19

Coronavirus disease 2019

CrI

credible interval

EUA

emergency use authorization

GRADE

Grading of Recommendation, Assessment, Development, and Evaluation

ST

skin test/skin testing

1. INTRODUCTION

Over 12.2 billion COVID‐19 vaccine doses have been administered worldwide for a SARS‐CoV‐2 pandemic which has infected over 609 million persons and resulted in more than 6.5 million fatalities. ¹ Concerns regarding immediate allergic reactions primarily to mRNA COVID‐19 vaccines have emerged and been attributed to a potential culprit excipient, polyethylene glycol (PEG). Theoretical cross‐reactivity of PEG with a related compound, polysorbate 80 (PS), the excipient in adenoviral‐vector COVID‐19 vaccines, has been discussed. However, increasing evidence suggests that immediate allergic reactions to mRNA COVID‐19 vaccines are very rare, and may not necessarily be the result of an excipient‐driven, IgE‐mediated process. Two recent meta‐analyses demonstrated that globally, immediate severe allergic reactions following the COVID‐19 vaccines are rare (7.9 per 1 million vaccinations), ² and repeat severe immediate allergic reactions among individuals re‐vaccinated to the same mRNA COVID‐19 vaccine are also rare (occurring with frequency of 0.16%). ³ Despite mounting evidence to the contrary, the vaccine remains largely contraindicated internationally in persons with an immediate allergic reaction to their 1st vaccine dose or history of allergy (of any severity) to the vaccine excipients. ³ , ⁴ , ⁵ , ⁶

In the face of uncertainty regarding new vaccine excipients, an algorithm that included skin tests (ST) to both PEG and PS prior to a 1st dose mRNA COVID‐19 vaccine in individuals with suspected immediate reaction to PEG/PS, or before receiving a 2nd dose in persons with a suspected 1st dose immediate reaction was recommended. Testing to the vaccine directly was not recommended at that time due to issues of vaccine availability, equity/ethics of using the vaccine as a ST reagent given limited worldwide availability, and that using vaccine as a ST reagent was not explicitly discussed under the vaccine emergency use authorization (EUA). ⁷ , ⁸ The initial testing guidance was made prior to data regarding the test accuracy for using the excipients to assess COVID‐19 vaccination outcomes, ⁷ but generally followed prior Joint Task Force guidance for the approach to adverse reactions to vaccines. ⁹ This guidance to withhold vaccination in excipient sensitized persons contradicts current non‐COVID vaccine allergy specialist practice parameter recommendations, ⁹ but follows the COVID‐19 vaccine FDA and CDC guidance. ⁵ Since that time, several small studies have reported on the diagnostic utility of COVID‐19 vaccine/vaccine excipient testing, but there remains no systematic evidence synthesis regarding this issue which could help influence vaccine allergy guidelines. This is an unmet need with substantial implications on an individual and broader public health level during the current pandemic.

To better inform the evidence base regarding such practices, we systematically reviewed and performed meta‐analysis of the testing accuracy of ST to mRNA COVID‐19 vaccines, PEG, and PS to diagnose mRNA COVID‐19 vaccine allergy.

2. METHODS

To assess the diagnostic testing accuracy of mRNA COVID‐19 vaccine and vaccine excipient ST, we performed a systematic review using a nested search strategy and protocol from a distinct question for a previously planned and published meta‐analysis regarding the incidence of immediate severe allergic reactions following a 2nd dose of a mRNA COVID‐19 vaccine (either BNT162b2 vaccine from Pfizer‐BioNTech[Pfizer Inc, New York, NY; and BioNTech Manufacturing, Mainz, Germany], or mRNA‐1273 vaccine from Moderna [Cambridge, MA]). The base systematic review protocol was modified to focus on an additional question regarding vaccine and vaccine excipient ST, ³ that could be answered from the articles already identified within the original search. Additionally, the outcome measures, risk of bias assessment, and data synthesis methods were updated as appropriate for meta‐analysis of diagnostic testing (eMethods).

We searched Medline, Embase, and The WHO Global Coronavirus database (a database aggregating COVID‐19 published and pre‐print reports daily from 116 other literature databases), from inception through Oct 4, 2021 for studies of any design addressing (a) the risk of repeat immediate allergic reactions following mRNA COVID‐19 vaccines of any severity among individuals who had a prior mRNA COVID‐19 vaccine allergic reaction, and (b) accuracy of ST to the vaccine attributed to causing the allergic reaction with the 1st dose, as well as to either or both PEG and PS (eMethods). We additionally searched Web of Science (all databases) using forward and backward citation analysis to identify any additional relevant records. Studies that detailed delayed (>4 h post vaccine) reactions, involved mRNA COVID‐19 re‐vaccination but did not address individuals with prior allergic reactions and where ST was not performed were excluded. Three reviewers (MG, DC, MS), using Covidence (Veritas Health Innovation, Melbourne, Australia), independently and in duplicate screened records, and four reviewers (MG, EA, DG, MS) independently and in duplicate extracted data. Consensus among the reviewers was used to resolve conflicts. The PRISMA diagram for the literature search is detailed in Figure 1. We extracted the number of true/false positive and true/false negative ST (inclusive of both skin prick test and any dilution intradermal testing) for the vaccine (both BNT162b2 and MRNA‐1273), PEG, and PS (index tests) in assessing an outcome of 2nd dose immediate allergic reactions (reference test), among persons undergoing both ST and re‐vaccination with the same agent to which they had an immediate allergic reaction to with their 1st dose. ⁷ , ⁸ , ⁹ Because patients in the included studies were usually tested simultaneously to multiple potential agents, the vaccination outcomes were analyzed by individual test for maximal sensitivity, which implies the case would be separately counted as a true positive for each agent in any patient with an immediate reaction to the 2nd dose who was sensitized to multiple items. Reaction severity was defined as indicated by the investigator in the included study, with non‐severe allergic reactions defined as mild or self‐limiting subjective or objective symptoms that either spontaneously resolved or resolved with anti‐histamine treatment, and severe allergic reaction as either anaphylaxis or a reaction requiring injectable epinephrine administration. ³

FIGURE 1.

PRISMA diagram

Study authors were individually contacted by email to verify final data extraction, to clarify if any cases were duplicated if the author group had multiple included publications and clarify any study design questions. Pooled data were analyzed with the updated MIDAS program for meta‐analysis for diagnostic tests which uses Bayesian methods for data with multiple zero count cells (e.g., “sparse” data), ¹⁰ using Stata version 15. Summary sensitivity and specificity and 95% credible interval (CrI)—which is the Bayesian analog to confidence interval—were calculated using bivariate generalized linear mixed modeling. ¹¹ , ¹² We used Bayesian Hamiltonian Monte Carlo simulation with non‐informative hyperpriors and implemented with the Statastan program. ¹³ We ran 4 chains of 10,000 iterations each after a warmup of 1000 iterations. We applied the Gelman and Rubin's convergence diagnostic that computes the potential scale reduction factor (PSRF). A PSRF value close to 1 indicates model convergence and in practice, the value of 1.1 has been recommended as the threshold to gauge whether the model has converged. ¹⁴ , ¹⁵

The primary outcome was the ST test sensitivity and specificity, with 95% credible intervals for immediate allergic reactions of any severity. Forrest plots were generated for visual display of the data, with heterogeneity assessed using the I ² statistic. Prespecified subgroup analyses were performed to analyze test sensitivity/specificity for severe 2nd dose reactions, for studies that included patients with 1st dose anaphylaxis, and if graded dosing and/or premedication were allowed in the individual study. The Grading of Recommendation, Assessment, Development, and Evaluation (GRADE) approach provided assessment of quality of the body of the evidence, ¹⁶ the QUADAS‐2 Risk of Bias tool was used to rate the risk of bias. ¹⁷

3. RESULTS

We identified 20 studies (all single‐arm cohorts, case series, and case reports, but no randomized clinical trials) detailing ST to the BNT162b2 and mRNA‐1273 mRNA vaccines, PEG, and PS in individuals with a history of immediate allergic reaction to their 1st mRNA COVID‐19 vaccine dose, who underwent such evaluation prior to re‐vaccination with the same mRNA COVID‐19 vaccine eliciting the index reaction. This included a total of 317 individuals who underwent a total of 578 ST to one or more of the vaccines or vaccine excipients (not including dilutions). ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁷ Table 1 details the study characteristics, and Table E1 details the QUADAS‐2 risk of bias ratings for the included studies. All studies were designated as having unclear risk of bias due to performing only selected tests and/or the lack of blinding of test results though no study precluded vaccination based on a positive test; it was not possible in the current context to achieve a lower risk of bias. All re‐vaccinations occurred in adults, under the guidance of an allergy specialist, and used mRNA vaccines. Overall, for any testing reagent in predicting an immediate allergic reaction of any severity, sensitivity was 0.03 (95% CrI 0.01–0.08) and specificity was 0.98 (95%CrI 0.95–1.00) (Figure 2). Receiver operating characteristic (ROC) curves for this analysis are presented in Figure E1.

TABLE 1.

Characteristics of the included studies

Author (all 2021)	Country	Design	Female %	Age	Number of 1st dose reactions evaluated a	Number who received 2nd dose	Number of 1st dose immediate reactions skin tested	Number not skin tested	Reason not skin tested	Skin tested and received 2nd dose	2nd dose deferred	Reason 2nd dose deferred	2nd dose anaphylaxis	Included patients with anaphylaxis to 1st dose (n tested)	Graded dosing	Premedication allowed	Tested to vaccine g	Tested to excipient g
Tuong et al. 18	US	Case series	100	42.8 y (21–64 y)	15	15	11	4	Patients declined testing	11	0	NA	2 b	Yes (n = 1)	Yes	No	Yes (both)	Yes (both)
Krantz et al. 19	US, Denmark	Case series	100	44.8 y (29–54 y)	8	8	8	0	NA	8	0	NA	0	Yes (n = 4)	No	Yes	No	Yes (both)
Rassmussen et al. 20	Denmark	Case series	88.5	46 y (18–88 y)	16	16	16	0	NA	16	0	NA	0	Yes (n = 4)	No	No	Yes (Pfizer)	Yes (PEG)
Wolfson et al. 21	US	Case series	89	40.9 y (sd 13.6)	65	58	65	0	NA	58	7	2 PEG+ and 5 PEG‐ skin test patients declined	3 c	Yes (n = 3)	No	Yes	No	Yes (both)
Kessel et al. 22	Israel	Case series	77.8	54.3 y (23–75 y)	18	18	16	2	Not specified	16	0		0	Yes (n = 7)	No	Yes	Yes (Pfizer)	Yes (PEG)
Kelso et al. 23	US	Case report	100	48.6 y (43–56 y)	4	3	4	0	NA	3	1	Skin test ‐ patient declined because of fear of needing epinephrine	0	Yes (n = 3)	No	No	Yes (Pfizer)	No
Mustafa et al. 24	US	Case report	100	64 y, 39 y	2	2	2	0	NA	2	0	NA	0	No	Yes	No	Yes (Moderna)	Yes (both)
Vanijcharoenkarn et al. 25	US	Case series	92	Not specified	88	73	16	72	Investigators stopped testing after the first 16 patients due to utility.	16	15	9 deferred without explanation, 6 lost to follow up. None were tested	0	Yes (n = 4)	Yes	No	No	Yes (both)
Park et al. 26	US	Case report	100	34 y	1	1	1	0	NA	1	0	NA	0	Yes (n = 1)	No	No	Yes (Pfizer)	Yes (PEG)
Loli‐Ausejo et al. 27	Spain	Case series	81.8	39 y (29.5–56.5)	6	6	6	0	NA	6	0	NA	0	No	Yes	Yes	Yes (Pfizer)	Yes (PEG)
Pitlick et al. 28	US	Case series	80	48 y (20–90 y)	41	41	41	0	NA	41 d	0	NA	0	Yes (n = 4)	Yes	No	Yes (both)	Yes (both)
Kohli‐Pamnani et al.  29	US	Case series	87	56 y (sd 16)	18	16	18	0	NA	16	2	2 skin test negative patients preferred to receive Janssen vaccine.	0	Yes (n = 1)	No	yes	Yes (both)	Yes (both)
Warren et al. 30	US	Case series	91	40.9 y (sd 10.3)	22	11	11	11	Patients declined testing	11 d	11	Not stated	1 e	Yes (n = 11)	No	No	Yes (both)	Yes (both)
Carpenter et al. 31	US	Case report	100	60 y	2	1	2	0	Received Janssen vaccine	1	1	A skin test negative patient preferred to receive the Janssen vaccine	0	No	No	No	No	Yes (both)
Kaplan et al. 32	US	Case series	86.7	48 y (19–89 y)	34	27	31	3	The investigators stopped testing	27	4	3 patients with +PS testing were lost to follow up; 1 patient with ‐PS testing deferred	0	Yes (N = 1)	Yes	Yes	No	Yes (both)
AlMuhizi et al. 33	Canada	Case series	86.9	55 y (43.25–65 y)	40	40	29	0	NA	40	0	NA	0	Yes (n = 15)	Yes	No	Yes f (Pfizer)	Yes (both)
Van Meerbeke et al. 34	US	Case Series	80	50.2 y (31–59)	8	8	8	0	NA	8	0	NA	0	Yes (n = 4)	Yes	Yes	Yes (both)	Yes (both)
Otani et al. 35	US	Case Series	89	45 y (24–78 y)	42	42	14	28	The investigators stopped testing	14	0	NA	2	Yes (n = 6)	No	No	No	Yes (both)
Csuth et al. 36	Sweden	Case Series	80.9	45 y (16–90 y)	21	20	21	0	NA	20	1	1 skin test negative patient declined. Had received epinephrine with first dose but did not meet Brighton Level 1–3 criteria	0	Yes (n = 7)	No	No	No	Yes (PEG)
Cahil and Kan 37	Canada	Case Report	100	44 y (35, 52 y)	2	2	2	0	NA	2	0	NA	0	Yes (n = 2)	Yes	Yes	Yes (Pfizer)	Yes (PEG)

^a Column represents the total number of 1st dose reactions that were evaluated in the study. Not all of these patients underwent skin testing, re‐vaccination, or both. Thus, some patients met our selection criteria, and some were vaccinated without testing.

^b Both cases of anaphylaxis occurred on the last step of a multi‐dose desensitization, and one of these patients was sensitized to the Moderna vaccine but negative to the excipients. Neither had 1st dose anaphylaxis.

^c All 3 anaphylaxis cases had both anaphylaxis to their first dose and negative skin testing to the excipients prior to the 2nd dose.

^d Totals supplemented by author personal communication, which account for additional vaccination of persons who initially deferred vaccination after testing at the time of initial publication.

^e Patient had anaphylaxis to both the first and second dose, with positive skin testing to vaccine (Pfizer) but negative skin testing to the excipients prior to 2nd dose.

^f Only a single patient was tested to vaccine (Pfizer) in this study.

^g Indicates which vaccine or excipient was used for testing in persons who met inclusion criteria and were re‐vaccinated.

FIGURE 2.

Sensitivity and specificity of any skin test to evaluate the risk of a second dose reaction. Forrest plot of the sensitivity and specificity for the combined analysis of skin testing to polyethylene glycol, polysorbate, or either mRNA COVID‐19 vaccine in predicting the risk of a 2nd dose immediate allergic reaction to a mRNA COVID‐19 vaccine. M, Moderna; PEG, polyethylene glycol; PB, Pfizer‐BioNTech; PS, polysorbate.

3.1. Accuracy using BNT162b2 or mRNA‐1273 vaccine as a ST reagent

The pooled test sensitivity for any patient undergoing ST to either mRNA vaccine agent (14 studies, 117 patients tested) was 0.2 (95%CrI 0.01–0.52) and specificity was 0.97 (95%CrI 0.9–1) (Figure 3). Test sensitivity for mRNA‐1273 vaccine (6 studies, 36 patients tested) was 0.59 (95%CrI 0.01–1), and test specificity 0.76 (95%CrI 0.08–0.98). ¹⁸ , ²⁴ , ²⁸ , ²⁹ , ³⁰ , ³⁴ For BNT162b2 vaccine (12 studies, 81 patients tested), test sensitivity was 0.1 (95%CrI 0.00–0.38) and test specificity was 0.99 (95%CrI 0.94–1). ¹⁸ , ²⁰ , ²² , ²³ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³³ , ³⁴ , ³⁷ There were only 3 true positive tests out of 117 patients tested to either vaccine across all studies (2 for mRNA‐1273 vaccine, 1 for BNT162b2), including 2 true positives in patients who had a severe reaction (one for each vaccine brand). Certainty of evidence was low (Table E2). Receiver operating characteristic (ROC) curves for this analysis are presented in Figure E1.

FIGURE 3.

Sensitivity and specificity of either mRNA vaccine skin test to evaluate the risk of a second dose reaction. Forrest plot of the sensitivity and specificity for skin testing to either mRNA COVID‐19 vaccine in predicting the risk of a 2nd dose immediate allergic reaction to a mRNA COVID‐19 vaccine. M, Moderna; PB, Pfizer‐BioNTech.

3.2. Accuracy using polyethylene glycol or polysorbate as a ST reagent

For PEG (19 studies, 297 patients tested), test sensitivity was 0.02 (95%CrI 0.00–0.07) and test specificity was 0.99 (95%CrI 0.96–1) (Figure 4A). ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² , ³³ , ³⁴ , ³⁵ , ³⁶ , ³⁷ For PS (13 studies, 164 patients tested), testing sensitivity was 0.03 (95%CrI 0.00–0.11) and testing specificity 0.97 (95%CrI 0.91–1) (Figure 4B). ¹⁸ , ¹⁹ , ²¹ , ²⁴ , ²⁵ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² , ³³ , ³⁴ , ³⁵ Certainty of evidence was moderate (Table E2).One study did note several positive PS tests using Refresh Tears™ as a reagent, but this was an irritant and as such, positive PS tests to that reagent from that study were considered false positive. ²¹ There was a single PEG‐sensitized patient who reacted to their 2nd dose, but did not have a severe reaction. Among 6 patients with severe 2nd dose reactions, 5 of these 6 patients had false negative testing to PEG, and 4 of 5 had false negative testing to PS. Receiver operating characteristic (ROC) curves for this analysis are presented in Figure E1.

FIGURE 4.

Sensitivity and specificity of excipient skin testing to evaluate the risk of a second dose reaction. Forrest plot of the sensitivity and specificity for skin testing to polyethylene glycol (panel A) and polysorbate (panel B) in predicting the risk of a 2nd dose immediate allergic reaction to a mRNA COVID‐19 vaccine.

3.3. Sensitivity analysis

Sensitivity and subgroup analyses, including accounting for studies that permitted use of graded dosing (n = 9 studies), premedication (n = 8 studies), or patients with 1st dose anaphylaxis (n = 17 studies) did not alter the main findings (Table 2). However, when specifically analyzing test accuracy relative to predicting only severe reactions, there was an increase in sensitivity and decrease in specificity for ST to the mRNA‐1273 vaccine (sensitivity 0.75[CrI 0.1–1], specificity 0.82[CrI 0.07–1]), and increased sensitivity for ST to the BNT162b2 vaccine (sensitivity 0.82[CrI 0.13–1], specificity 0.99 [CrI 0.96–1]) (Figure E2). Among the 6 total severe immediate 2nd dose allergic reactions noted, 2 occurred among vaccine sensitized individuals who were not sensitized to either excipient, and 4 occurred among individuals not sensitized to either vaccine or both excipients tested.

TABLE 2.

Summary of test reagent sensitivity and specificity

Testing reagent	TP	FP	FN	TN	Sensitivity	95% CrI	Specificity	95% CrI	PLR	NLR
All agents, combined	5	27	88	458	0.03	0.00–0.08	0.98	0.95–1	2.75	0.99
Either mRNA vaccine agent	3	8	11	95	0.2	0.01–0.52	0.97	0.9–1	4.75	0.84
mRNA‐1273 vaccine (Moderna)	2	6	1	27	0.59	0.08–0.98	0.76	0.01–1	2.3	0.52
BNT162b2 vaccine (Pfizer‐BioNTech)	1	2	10	68	0.1	0.00–0.38	0.99	0.94–1	1.3	0.92
Polyethylene Glycol a	1	11	45	240	0.02	0.00–0.07	0.99	0.96–1	2	0.99
Polysorbate b	1	8	32	123	0.03	0.00–0.11	0.97	0.91–1	1.5	0.99

Sensitivity analysis (all agents, combined)	TP	FP	FN	TN	Sensitivity	95% CI	Specificity	95% CrI
1st dose anaphylaxis patients included in the study	5	26	82	435	0.03	0.00–0.09	0.98	0.95–1	6.5	0.89
No 1st dose anaphylaxis patients included in the study	0	0	10	10	0.07	0.00–0.45	0.98	0.76–1	2.5	0.97
Graded challenge allowed	4	15	45	197	0.08	0.01–0.23	0.97	0.9–1	1.4	0.98
No graded challenge allowed	1	11	47	248	0.01	0.00–0.06	0.99	0.96–1	2	0.99
Premedication allowed	0	11	54	183	0.00	0.00–0.03	0.98	0.93–1	0.5	1
No premedication allowed	5	15	38	280	0.09	0.01–0.25	0.98	0.94–1	3.67	0.92
Predictive only of 2nd dose anaphylaxis	2	26	12	433	0.11	0.01–0.32	0.98	0.95–1	6.5	0.89

Abbreviations: CrI, credible interval; FN, false negative; FP, false positive; NLR, negative likelihood ratio; PLR, positive likelihood ratio; TN, true negative; TP, true positive.

^a Inclusive of any molecular weight polyethylene glycol tested.

^b Inclusive of any polyoxyethylene group number tested.

4. DISCUSSION

This systematic review and meta‐analysis found moderate‐certainty evidence of very low pooled sensitivity (3%) with high pooled specificity (98%) for ST to either the mRNA vaccine or its excipients as a means of risk assessment regarding repeat immediate allergic reactions of any severity among those with 1st vaccine dose immediate allergic reaction. The majority of second dose reactions occurred in non‐sensitized persons, with few patients demonstrating true positive ST to the vaccine (3 persons), PEG (1 person), or PS (0 persons). Of the 6 severe reactions to second dose mRNA vaccine, 4 occurred in individuals who were not sensitized to either the vaccine or excipient. ¹⁸ , ²¹ , ³⁰ Test sensitivity was poor, and, altogether, would only identify approximately 3 out of 100 persons with immediate revaccination reactions by skin testing to the vaccine, PEG, or PS. The very low sensitivity is concerning when all cases came exclusively from a pool of persons believed to have had an IgE‐mediated allergy to their 1st vaccine dose, previously assumed to be attributable to the excipient content in the vaccine (or the vaccine itself). As such, in this meta‐analysis, these tests are unable to reliably discriminate if someone will react to, or tolerate, a 2nd dose after a reaction to a 1st dose any more than testing to an unrelated inert reagent would. While our findings might be explained by the poor performance of these tests in detecting IgE to PEG, PS, or mRNA vaccines, findings could also represent a non‐IgE‐mediated mechanism being the dominant immediate reaction phenotype that renders IgE tests futile, or only useful to a much smaller subset of patients. This analysis was not designed to evaluate whether skin tests can predict allergic reactions to PEG, only whether they predict repeat reaction to mRNA COVID‐19 vaccines.

Few international allergy societies (or their COVID‐19 taskforces) have formally recommended for or against testing for assessing COVID‐19 vaccine reactions. However, Canadian Society of Allergy and Clinical Immunology has recently specifically recommended against such ST. ⁷ , ⁸ , ³⁸ , ³⁹ , ⁴⁰ The CDC and multiple other health authorities have considered a history of allergy to PEG, PS, or an immediate reaction to a prior dose of COVID‐19 vaccine a contraindication to receiving an mRNA COVID‐19 vaccine, although this approach is not uniform internationally and is evolving as new data emerge. ⁵

Meta‐analyses noting both low prevalence of PEG allergy and sensitivity of PEG ST for risk‐assessment of non‐COVID vaccination/medication reactions informed earlier GRADE‐based guidance conditionally recommending against PEG/PS ST before vaccination in persons with a history of PEG/PS allergy or COVID‐19 vaccine reaction. ² This meta‐analysis now supplements and bolsters those data within the context of the utility of PEG/PS and vaccine ST after an initial reaction to an mRNA COVID‐19 vaccine, for risk stratification. There is scant evidence to suggest that PEG may be a culprit causing IgE‐mediated mRNA COVID‐19 vaccine immediate allergic reactions. While there are case reports that describe mRNA COVID reactions occurring with initial doses in known PEG allergic individuals (who were not re‐vaccinated), there are case series in which patients with PEG allergy tolerated an mRNA COVID vaccine. This suggests that it may be the very rare PEG allergic patient that has an IgE mediated reaction to the mRNA vaccine. Case series have been published which also suggested that PEG allergic patients may tolerate PS containing vaccines (or vice‐versa). ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁴⁹ , ⁵⁰ , ⁵¹ , ⁵² Currently, it is hypothesized that these reactions occur primarily through a non‐IgE mediated mechanism, which is supported by recent meta‐analytic data demonstrating a low risk of repeat reactions with 2nd doses, and could explain the poor test accuracy in this present analysis. ³

While the pooled data in this analysis show testing has poor sensitivity, the high specificity does not indicate that these tests accurately identify persons who are not allergic to the vaccine. The testing performs quite poorly in identifying vaccine‐allergic persons from a pool of subjects with a 100% pre‐test probability to have had immediate reactions to their 1st dose triggered by the vaccine or excipient. In this context, this could be explained by these reagents not containing relevant allergens, or that mRNA vaccine reactions are only rarely caused by IgE mediated PEG allergy. Regardless of the reasons why, this analysis demonstrates that these tests do not predict allergic reactions to a 2nd dose of mRNA vaccine. In this case, the test accuracy is comparable to historically reported rates of positive saline control testing. ⁵³ , ⁵⁴ Under a shared decision‐making paradigm, some patients and clinicians may still ultimately decide ST is necessary to be comfortable to proceed with being re‐vaccinated. An additional history consistent with PEG or polysorbate allergy may be an independent factor that should guide these decisions.

This study has a number of important limitations. First, given the limited number of included studies, and small numbers of participants in these studies who underwent both ST and re‐vaccination, there is a risk for imprecision. This risk may be highest for estimates for mRNA vaccine ST, particularly mRNA‐1273 vaccine, where just 30 patients were included and no standardized ST protocol or non‐irritant concentration has yet to be described. However, even the limited numbers of reactors captured among the studies denote the vaccine is being successfully administered to at‐risk individuals, and multiple sensitivity analyses did not alter the estimates. Further, this is planned as a living systematic review and meta‐analysis, and the data will be updated as more studies meeting selection criteria become available. Second, there is risk of selection bias among the included studies, in terms of who was offered testing, who underwent testing, who ultimately was administered a 2nd dose, and that few studies of this nature have been performed. Analysis of the included studies did not detect critical issues related to flow and timing of testing, use of an index test or reference standard, applicability and being administered re‐vaccination that would be of concern and confer high risk of bias. However, all studies were designated as having unclear risk of bias due to performing only selected tests and/or the lack of blinding of test results. It was not possible in the current context to achieve a lower risk of bias. While it could be argued that any non‐blinded study where the team and patients are aware of the index test before performing the reference test necessarily has a high risk of bias, no study in our meta‐analysis precluded re‐vaccination based on the test result. Thus, we felt the actual effect of these influences most accurately reflected an “unclear” risk of bias. Many studies had patients who refused to undergo partial or full evaluation, had an investigator‐driven decision stop testing patients in the middle of the study, or where patients were lost to follow up, which could also introduce selection bias. Deferral of the 2nd dose in a number of patients introduces a potential “reverse work‐up bias”. In order to accurately calculate diagnostic test characteristics, inclusion criteria required individuals to be revaccinated with the same vaccine. While many patients met these criteria for both the excipient ST and vaccine ST analyses, it is possible that this inclusion criteria excluded those individuals with the most concerning reactions thought most likely to be IgE as they may have declined revaccination or chose an alternative vaccine. However, to account for this, sensitivity analysis that presumed 25% and 50% of the total deferrals across all studies underwent full evaluation and were considered as true positive cases. Even in this “best‐case scenario” (using frequentist and unweighted analysis of the raw totals), the ST sensitivity improved to 0.22 (any test), 0.32 (PEG), and 0.48 (any vaccine) (Table E3). Third, as was noted in our earlier meta‐analysis using these search parameters, severe reactions were, in part, defined as requiring injectable epinephrine, though other potential definitions could apply. ³ , ⁵⁵ For consistency between the analyses we used the same designation, and this severity definition is accepted as standard within the allergy field. ⁵⁴ Fourth, all included studies were conducted with allergy specialist guidance, which could limit generalizability, though it would not be expected that patients would undergo allergy diagnostic testing in a primary care or general vaccine setting. Fifth, chance (random error) could explain why ST with the mRNA‐1273 vs. the BNT162b2 vaccine would differ so greatly in sensitivity. The excipient quantity and composition of the vaccines are similar, and it would not be predicted that one agent should be a more accurate testing reagent. Sixth, non‐irritating concentrations of mRNA vaccines have not been widely studied. ⁵⁶ Although studies largely followed published protocols for excipient ST, no protocols were available for mRNA COVID‐19 vaccines ST beyond the current Allergy Joint Task Force on Practice Parameter guidelines, which pre‐date COVID‐19 vaccine development. ⁹ As such, in this analysis we considered ST positive or negative as indicated by the authors, without consideration of ST protocol. Seventh, we caution that there are small numbers comprising the mRNA‐1273 estimates, and that the pooled estimate combining both vaccines is a more realistic measure of testing accuracy in predicting reactions. Eighth, these data only address 2nd dose mRNA vaccinations. While it is plausible that the test accuracy would not change if this were the case of pre‐emptive testing prior to the 1st dose (i.e., screening tests before any vaccination), or assessment for a reaction after a 2nd dose in evaluation for a 3rd dose, neither were the context studied in this analysis. Ninth, this analysis was not intended to evaluate approaches to vaccinating patients with confirmed PEG allergy, or the performance of ST to the vaccine or vaccine excipients in such persons prior to receiving an initial mRNA COVID‐19 vaccine dose. ² Rather, this focused on patients with a 1st dose vaccine reaction. Importantly, PEG allergy may not predict mRNA COVID vaccine reactions, mRNA COVID vaccine tolerance may not exclude PEG allergy, and it is possible to be both PEG allergic and tolerate the volume/concentration of PEG in mRNA COVID‐19 vaccines. ⁴³ Finally, this analysis does not address additional considerations for performing PEG ST in persons with suspected PEG allergy presenting for allergy care after a first dose reaction, where the COVID vaccination reaction is incidental to a history of other reactions to PEG‐containing items. Allergy assessment would still be highly important in such individuals.

This meta‐analysis demonstrates poor accuracy for ST with mRNA COVID‐19 vaccine or vaccine excipients to predict vaccine tolerance with a 2nd vaccination after an immediate reaction to a 1st dose, as a means of sole risk‐stratification. ST has very poor sensitivity despite high specificity and had no impact on 2nd dose vaccination outcomes. It remains unclear if allergic reactions to mRNA COVID‐19 vaccines are IgE‐mediated or related to PEG contained in mRNA vaccines; however, these testing data question the utility of ST to the vaccine or vaccine excipient in evaluating mRNA COVID‐19 vaccine allergic reactions. Recent meta‐analyses noted the incidence of a severe immediate allergic reaction to mRNA COVID‐19 vaccines is rare, and among those with a 1st dose immediate reaction, the rate of severe immediate allergic reactions to a 2nd dose is also rare. ² , ³ While further research of this topic is ongoing, our results demonstrate that ST to the vaccine or vaccine excipient has very limited utility in determining outcomes of persons being re‐vaccinated to a 2nd dose after having experienced an immediate reaction of any severity to their 1st dose, and if testing is considered, it may have the best utility in patients with an additional compelling history of prior PEG allergy.

ACKNOWLEDGMENTS

We would like to thank and acknowledge Shazahd Mustafa, MD; Allison Ramsey, MD; Miguel Park, MD; Mitchell Pitlick, MD; Carsten Bindslev‐Jensen, MD, PhD, DMSci; Trine Rasmussen, MD; Arnon Elizur, MD, PhD; John Kelso, MD; Kari Nadeau, MD, PhD; Christopehr Warren, PhD; Anita Kohli‐Pamnani MD; Pamela L.Kwittken, MD; Lacey Robinson, MD, MPH; Aleena Banerji, MD; Lene Garvey, MD, PhD; Elizabeth Phillips, MD, PhD; MD; Blanca Kaplan, MD, Merin Kuruvilla, MD; Moshe Ben Shoshan, MD, Msc; Faisal Al Muhizi, MD; Ghislaine Isabwe, MD; Julia Cahil, MD, Manstein Kan, MD; and Iris Otani, MD for their assistance with verifying data and study design questions pertaining to their included works.

CONFLICT OF INTEREST

Matthew Greenhawt: is a consultant for Aquestive; is a member of physician/medical advisory boards for DBV Technologies, Sanofi/Regeneron, Nutricia, Novartis, Acquestive, Allergy Therapeutics, AstraZeneca, ALK‐Abello, and Prota, with all activity unrelated to vacines/vaccine development or COVID‐19 treatment; is an unpaid member of the scientific advisory council for the National Peanut Board and medical advisory board of the International Food Protein Induced Enterocolitis Syndrome Association; is a member of the Brighton Collaboration Criteria Vaccine Anaphylaxis 2.0 working group; is the senior associate editor for the Annals of Allergy, Asthma, and Immunology, and is member of the Joint Taskforce on Allergy Practice Parameters. He has received honorarium for lectures from ImSci, MedLearningGroup, RMEI Medical Education and multiple state/local allergy societies. He received past research support ending in 2020 from the Agency for Healthcare Quality and Research (K08‐HS024599). Elissa Abrams: is a collaborator with the Institute for Health Metrics and Evaluation. She is na employee of Public Health Agency of Canada (PHAC) but the views expressed are her own and not that of PHAC. Marcus Shaker: member of the Joint Taskforce on Allergy Practice Parameters; has a family member who is CEO of Altrix Medical; is an associate editor for the Annals of Allergy, Asthma, and Immunology, and serves on the editorial boards of the Journal of Allergy and Clinical Immunology In Practice and the Journal of Food Allergy. Kimberly Blumenthal: receives grant support from the NIH/NIAID (R01AI150295, 2UM1AI109565‐08), Phadia Ab (Thermo Fisher Scientific), and the Massachusetts General Hospital; personal fees for legal case review from Weekley Shulte Valdes Murman Tonelli, Piedmont Liability Trust, Vasios Kelly and Strollo PA, and Publix Supermarkets; and royalties from UpToDate, outside the submitted work. Anna Wolfson: no conflicts to declare. Cosby Stone: receives research support from the AAAAI Foundation Faculty Development Award. Matthew Krantz: no conflicts to declare. David Golden: Speakers bureau honoraria from Genentech, Kaleo; Clinical trial support from Genentech, Thermo Fisher, Novartis, Pfizer, GSK, and Regeneron, all unrelated to vaccine/vaccine development or COVID‐19 treatment; Consulting fees from Aquestive, Novartis, ALK; Royalties from UpToDate (section editor). Derek Chu: no conflicts to declare. Ben Dwamena: no conflicts to report.

Supporting information

Supporting information S1

Figure S1

Figure S2

Table S1

Table S2

Table S3

Greenhawt M, Shaker M, Golden DBK, et al. Diagnostic accuracy of vaccine and vaccine excipient testing in the setting of allergic reactions to COVID‐19 vaccines: A systematic review and meta‐analysis. Allergy. 2023;78:71‐83. doi: 10.1111/all.15571