Accuracy of penicillin allergy diagnostic tests: A systematic review and meta-analysis

Abstract

Background:

Having a penicillin allergy label associates with a higher risk for antibiotic resistance and increased health care use.

Objective:

We sought to assess the accuracy of skin tests and specific IgE quantification in the diagnostic evaluation of patients reporting a penicillin/β-lactam allergy.

Methods:

We performed a systematic review and diagnostic accuracy meta-analysis, searching on MEDLINE, Scopus, and Web of Science. We included studies conducted in patients reporting a penicillin allergy and in whom skin tests and/or specific IgE quantification were performed and compared with drug challenge results. We quantitatively assessed the accuracy of diagnostic tests with bivariate random-effects meta-analyses. Meta-regression and subgroup analyses were performed to explore causes of heterogeneity. Studies’ quality was evaluated using QUADAS-2 criteria.

Results:

We included 105 primary studies, assessing 31,761 participants. Twenty-seven studies were assessed by bivariate meta-analysis. Skin tests had a summary sensitivity of 30.7% (95% CI, 18.9%−45.9%) and a specificity of 96.8% (95% CI, 94.2%−98.3%), with a partial area under the summary receiver-operating characteristic curve of 0.686 (I² = 38.2%). Similar results were observed for subanalyses restricted to patients reporting nonimmediate maculopapular exanthema or urticaria/angioedema. Specific IgE had a summary sensitivity of 19.3% (95% CI, 12.0%−29.4%) and a specificity of 97.4% (95% CI, 95.2%−98.6%), with a partial area under the summary receiver-operating characteristic curve of 0.420 (I² = 8.5%). Projected predictive values mainly reflect the low frequency of true penicillin allergy.

Conclusions:

Skin tests and specific IgE quantification appear to have low sensitivity and high specificity. Because current evidence is insufficient for assessing the role of these tests in stratifying patients for delabeling, we identified key requirements needed for future studies.

Graphical Abstract

Beta-lactams are a class of broad-spectrum antibiotics that include penicillins, cephalosporins, carbapenems, and monobactams. Because of their high effectiveness, safety, and low costs, β-lactams are the most frequently prescribed antibiotics in the world, and those for which hypersensitivity reactions are most often reported.^1,2 Penicillin allergy reporting is particularly frequent—in Europe and North America, allergy to penicillins is reported by up to 10% of the general population and 15% of hospitalized patients.^1,3–5 However, several studies have shown that there is an overdiagnosis of penicillin allergy, because only 2% to 10% of individuals who report penicillin allergy are truly allergic.^3,6

Without a proper diagnostic approach, many patients with a penicillin allergy label may unnecessarily receive second-line, broad-spectrum antibiotics, leading to an increased risk of drug failure, hospital-acquired infections, and higher length of stay.^7–10 These consequences, along with the use of more expensive antibiotics, explain the higher costs of treating patients with a penicillin allergy label,^7,8 and why penicillin allergy testing programs have been identified as a vital tool to improve antimicrobial stewardship and curb antibiotic resistance.^3,10

Patients reporting a penicillin allergy should have their diagnosis confirmed. The International Consensus on Drug Allergy recommends an approach based on skin tests and, in case of negative results, a confirmatory drug provocation test (DPT; “drug challenge”).¹¹ Two types of skin tests are usually done—(1) skin prick tests, which are the first to be performed, consisting in pricking the skin with small amounts of penicillin reagents, and (2) intradermal tests, which are performed when skin prick test results are negative, involving placing penicillin reagents just underneath the skin.¹² In patients reporting immediate reactions, a blood test for quantification of penicillins specific IgE (sIgE) is indicated by European recommendations.¹³ If negative results are obtained, the patient undergoes a DPT, receiving a penicillin drug under clinical observation.

The DPT is the reference standard for testing patients reporting a penicillin allergy. However, because it is typically performed after negative results with skin tests and/or sIgE quantification, there is still scarce information on the accuracy of these recommended diagnostic tests. In fact, although skin tests appear to have a high negative predictive value (NPV), there are conflicting reports on their sensitivity and positive predictive value (PPV), with wide ranges of values reported. Therefore, this systematic review aimed to evaluate the accuracy of the diagnostic tests—skin tests and sIgE quantification—used in the evaluation of patients reporting a penicillin/β-lactam allergy, and to identify factors that may explain differences in diagnostic parameter estimates.

METHODS

Search strategy and inclusion criteria

This study is a systematic review with meta-analysis of diagnostic test accuracy, complying with the Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies (PRISMA-DTA) Statement.¹⁴

We included original studies conducted in patients in whom penicillin/β-lactam allergy was reported, and in whom the results of DPT had been compared with those of skin tests (skin prick and/or intradermal tests) and/or sIgE quantification (“index tests”). We excluded studies conducted only in patients with nonpenicillin β-lactam allergy, and studies evaluating only patients with specific diseases (eg, patients with rheumatic fever) or occupations (eg, nurses).

Three electronic bibliographic databases (MEDLINE/PubMed, Web of Science, and Scopus) were searched in July 2018. The queries (presented in Table E1 in this article’s Online Repository at www.jacionline.org) included a database adaptation of the strategy presented by Astin et al¹⁵ to search for diagnostic performance studies. No restrictions based on primary studies’ language or publication date were applied.

After eliminating duplicates, the titles and abstracts of the obtained studies were independently evaluated by 2 researchers. Subsequently, the full texts of selected studies were independently read by 2 investigators, who extracted information regarding the year of publication, country, sampling method, study setting (outpatients, inpatients, or emergency department), tested drugs/determinants, participants’ number, sex, and age distributions, and inclusion criteria regarding initial reactions—initial reactions were assessed on the basis of involved drug classes (whether studies included participants reporting an allergy to any β-lactam or specifically to penicillins), clinical manifestations, and timing. Immediate reactions were defined as initial reactions occurring within the first hour after contact with the suspected drug, whereas non-immediate reactions were defined as those occurring after the first hour of drug contact. In addition, we collected information related to each diagnostic index test, including the performed tests, number of participants with positive and negative results, and number of participants submitted to DPT. For primary studies assessing patients with any reported β-lactam allergy but presenting discriminated results for those who specifically reported a penicillin allergy, we retrieved data concerning only the latter.

Data were extracted by means of a purposely-built form—we developed a pilot version, which was modified after assessment of the first 10 studies. For publications whose full texts were not available, we contacted corresponding authors. Authors were also contacted to provide relevant missing information. Disagreements in study selection or data extraction were solved by consensus.

Study quality was independently assessed by 2 researchers according to QUADAS-2 judgments on primary studies’ risk of bias and applicability.¹⁶

Quantitative synthesis

For each study, we retrieved the frequency of positive results to each index test. On the basis of that and on DPT results, we were able to calculate, for each index test, the number of true positives, true negatives, false positives, and/or false negatives.

For tests for which we had more than 4 primary studies simultaneously presenting results for true positives, true negatives, false positives, and false negatives, we performed meta-analyses of diagnostic test accuracy using a bivariate random-effects approach.¹⁷ Bivariate random-effects meta-analyses allow for the calculation of summary properties of diagnostic tests (namely, sensitivity, specificity, and area under the receiver-operating characteristic curve), by pooling the results of different diagnostic accuracy studies, accommodating between-study heterogeneity. This method (which is equivalent to the hierarchical summary receiver operating characteristic [sROC] model) has the advantage of jointly estimating sensitivity and specificity, accounting for correlation between them.¹⁸ Performing bivariate meta-analysis, we calculated areas under the sROC curve (AUC-sROC), and summary sensitivities, specificities, likelihood ratios, and diagnostic odds ratios. A correlation coefficient (r) of more than 0.6 between sensitivity and false-positive rate was considered to represent considerable threshold effect (ie, negative correlation between sensitivity and specificity possibility accounted by variability in positivity criteria).¹⁹ Heterogeneity (ie, between-studies variability beyond expected because of chance or random sampling errors alone) was assessed using an I² statistic estimate for bivariate meta-analysis.²⁰ An I² value of more than 50% was considered to represent substantial heterogeneity.

On the basis of obtained estimates of pooled sensitivity and specificity, and considering a prevalence of true penicillin allergy ranging from 0% to 10%, we estimated the PPV and NPV of index tests. We performed univariate sensitivity analyses, to obtain PPV and NPV estimates for different prevalence values. Probabilistic sensitivity analyses based on Monte-Carlo simulations were additionally performed to account for uncertainty in pooled sensitivity and specificity estimates.

Whenever there were sufficient primary studies, sensitivity analyses were performed to address the possibility of selection bias and/or to account for possible clinically relevant sources of heterogeneity. Such sensitivity analyses were restricted to (1) studies in which at least 50% of both patients with positive and negative results on index tests received a DPT, (2) studies assessing a mix of participants with immediate and nonimmediate reactions (excluding only those with very severe index reactions), and testing all with index tests and DPT, (3) studies assessing consecutive samples of patients specifically reporting a history of penicillin allergy and—in the case of skin tests—testing for the determinants recommended by the European Network of Drug Allergy (penicillin polylysine, minor determinant mixture, benzylpenicillin, and an aminopenicillin), and (4) studies assessing particular clinical presentations or age groups. In addition, we performed bivariate meta-regression and applied the likelihood-ratio test to identify moderator variables (ie, variables explaining heterogeneity²¹). Meta-regression tested the following covariates: year of publication, proportion of female subjects, participants’ age group (children, adults, or adults and children), study region (Europe, North America, Middle East, or Australia and Far East), setting, sampling method (consecutive sample including all participants with reported allergy, consecutive sample including all tested participants, or convenience sample), number of tested drugs/determinants (and whether the index test assessed at least an aminopenicillin, a cephalosporin, and/or the suspected drug), reported allergy (to overall β-lactams, penicillins, or aminopenicillins only), and timing of the initial reaction.

Not all studies allowed for obtaining, for each index test, the number of true positives, true negatives, false positives, and/or false negatives. Therefore, in addition to bivariate random-effects meta-analysis, we performed random-effects meta-analysis of individual diagnostic parameter estimates (methods fully described in this article’s Online Repository at www.jacionline.org). The latter corresponds to a less robust approach in the context of meta-analysis of diagnostic studies, because it does not account for sensitivity-specificity correlation.¹⁷

Meta-analyses were performed using software R (version 3.5.0), using mada, meta for, and glmulti packages. Simulations of predictive value estimates were obtained using TreeAge Pro software (TreeAge Software, Inc, Williamstown, Mass). Significance level was defined at P <.05.

RESULTS

Our search yielded 3952 publications, of which 1387 were duplicates; 2419 studies were excluded on the basis of application of eligibility criteria (Fig 1). We could not obtain the full text of 24 publications despite contacting corresponding authors—12 of these studies had been written in languages other than English and 16 had been published before 2000. Of the 122 articles meeting the eligibility criteria, 17 were excluded because some participants had also been assessed (receiving the same tests) by studies assessing larger samples. Therefore, a total of 105 studies were included in this systematic review.^13,22–125 For studies in which some degree of participant overlap with other study(ies) occurred (see Table E2 in this article’s Online Repository at www.jacionline.org),^{36,37,40–42,59,62,69,72,83,106,109} we presented only unique information from each study (eg, results from diagnostic tests performed in 1 study that had not been performed in the other study assessing the same participants), so that no participant has been included more than once.

FIG 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram illustrating study selection process.

Of the 105 included primary studies, 27 were included in bivariate random-effects meta-analyses^22–48 (summary of information in Table I and Table E3 in this article’s Online Repository at www.jacionline.org). The remainder were only quantitatively assessed by means of individual diagnostic parameter estimates meta-analysis, whose description and results can be consulted in this article’s Online Repository (see Tables E2–E4 in this article’s Online Repository at www.jacionline.org). We developed an interactive app (https://penaccuracy.med.up.pt; see this article’s Online Repository at www.jacionline.org), which also presents results from all analyses, as well as results for other tests whose studies were retrieved by our query.

TABLE I.

Description of studies included in the systematic review and assessed by means of bivariate random-effects meta-analyses

Reference	Country	No. of participants (% females)	Participants’ age group	Setting	Sampling method	Performed tests*	Drugs participants reported to be allergic	Timing of reactions	Index reactions of included participants
Levine and Zolov,22 1969	USA	218†	Children and adults	Patients (not specified)	Consecutive sample ‡	Skin tests (SPT; IDT§)	Penicillins	Immediate and nonimmediate	MPE; U/A; others unspecified
Warrington et al,23 1978	Canada	253†	Children and adults	Inpatients and outpatients	Convenience sample	Skin tests (SPT; IDT§)	Penicillins	Immediate and nonimmediate	MPE; U/A; others (including anaphylaxis and arthralgias/ arthritis)
Blanca et al,24 1990	Spain	288†	Children and adults	Outpatients	Convenience sample	Skin tests (IDT); sIgE	Penicillins	Immediate and nonimmediate	||
Sogn et al,251992	USA	825†	Adults	Inpatients	Consecutive sample ‡	Skin tests (SPT; IDT§)	Penicillins	Immediate and nonimmediate	||
Romano et al,26 1995	Italy	60†	Children and adults	Patients (not specified)	Consecutive sample ¶	Skin tests (SPT; IDT); sIgE	Aminopenicillins	Nonimmediate	MPE
Sastre et al,27 1996	Spain	576†	Children and adults	Outpatients	Consecutive sample ¶	Skin tests (SPT; IDT§)	Penicillins	Immediate	||
Cederbrant et al,28 1998	Sweden	17 (82)	Adults	Outpatients	Convenience sample	Skin tests (SPT; IDT); sIgE	Penicillins	Immediate and nonimmediate	MPE; U/A; others (including GI reactions and pruritus)
Loza Cortina,29 1998	Spain	76†	Children	Outpatients	Convenience sample	Skin tests (SPT; IDT§); sIgE	β-Lactams	#	Exclusion only of anaphylactic shock (most cases with MPE or U/A)
Blanca et al,30 2001	Spain	81 (67)	Children and adults	Patients (not specified)	Convenience sample	sIgE	Penicillins	Immediate and nonimmediate	Anaphylaxis; U/A; MPE
Forrest et al,31 2001	Canada	72†	Adults	Inpatients	Convenience sample	Skin tests	Penicillins	Immediate	||
Romano et al,32 2002	Italy	259 (72.6)	Children and adults	Patients (not specified)	Consecutive sample ¶	Skin tests (IDT)	Penicillins	Nonimmediate	MPE; U/A
Brusca et al,33 2007	Italy	122 (57.4)	Children and adults	Outpatients	Consecutive sample ¶	sIgE	Penicillins**	Immediate	U/A
Goldberg and Confino-Cohen,34 2008	Israel	169 (62.7)	Children and adults	Outpatients	Consecutive sample ¶	Skin tests (IDT)	Penicillins	Immediate and nonimmediate	Exclusion only of life-threatening reactions
Tantikul et al,35 2008	Thailand	4†	Children	Inpatients	Consecutive sample ¶	Skin tests (SPT)	Amoxicillin**	Immediate and nonimmediate	MPE; U/A
Caubet et al,36 2010	Switzerland	88 (50)	Children	ED	Consecutive sample ¶	sIgE	β-Lactams	Nonimmediate	MPE; U/A
Macy et al,37 2010	USA	150 (69.3)	Adults	Outpatients	Convenience sample	Skin tests (SPT; IDT§); sIgE	Penicillins	Immediate and nonimmediate	Exclusion only of severe mucocutaneous reactions, hemolytic anemia, or nephritis
Richter et al,38 2011	UK	47†	Adults	Outpatients	Consecutive sample††	Skin tests (SPT; IDT); sIgE	Penicillins**	Immediate and nonimmediate	U/A; rash; pruritus; other isolated reactions; anaphylaxis
Kopac et al,39 2012	Slovenia	606 (75.9)	Children and adults	Outpatients	Consecutive sample ¶	Skin tests (SPT; IDT); sIgE	Penicillins	Immediate and nonimmediate	||
Caubet et al,40 2014	Switzerland	96†	Children	Outpatients and ED	Convenience sample	Skin tests (IDT)	β-Lactams	Nonimmediate	MPE; U/A
Barni et al,41 2015	Italy	352 (50.3)	Children	Outpatients	Convenience sample	Skin tests (SPT; IDT§)	Amoxicillin	Nonimmediate	MPE; U/A
Mori et al,42 2015	Italy	190 (47.9)	Children	Outpatients	Consecutive sample ¶	Skin tests (SPT; IDT); sIgE	Amoxicillin**	Immediate and nonimmediate	MPE; U/A; others (including anaphylaxis and undefined reactions)
Rosenfield et al,43 2015	Canada	240 (72.9)	Children and adults	Outpatients	Convenience sample	Skin tests (SPT; IDT§)	β-Lactams	Immediate and nonimmediate	Rash; pruritus; U/A; cardiovascular reactions; others (including anaphylaxis and unknown reactions)
Mawhirt et al,44 2016	USA	49†	Adults	Inpatients and outpatients	Convenience sample	Skin tests (SPT; IDT§)	Penicillins**	Immediate	Cutaneous or other isolated reactions; mucocutaneous reactions; anaphylaxis
Meng et al,45 2016	UK	84 (66)	#	Outpatients	Consecutive sample ¶	Skin tests (SPT; IDT); sIgE	Penicillins	Immediate and nonimmediate	Rash; U/A; anaphylaxis; other/unknown
Confino-Cohen et al,46 2017	Israel	642 (51.4)	Children and adults	Outpatients	Consecutive sample ¶	Skin tests (IDT)	Penicillins	Nonimmediate	MPE (90%); U/A; other nonsevere delayed reactions
Tannert et al,47 2017	Denmark	21 (71)	Adults	Outpatients	Convenience sample	Skin tests (IDT); sIgE	Penicillins	Immediate and nonimmediate	Anaphylaxis; U/A; rash
Ibáñez et al,48 2018	Spain	732 (48.8)	Children	Outpatients and ED	Consecutive sample ¶	Skin tests (SPT; IDT); sIgE	Penicillin; amoxicillin-clavulanate; cloxacillin	Immediate and nonimmediate	Exclusion only of severe cutaneous reactions or severe anaphylaxis

ED, Emergency department; GI, gastrointestinal; IDT, intradermal test; MPE, maculopapular exanthema; SPT, skin prick test; U/A, urticaria/angioedema.

^*For each primary study, only those diagnostic tests whose accuracy could be calculated are listed.

^†Sex discrimination not available.

^‡Consecutive sample consisting of all patients with a history of penicillin allergy and in whom penicillin is indicated.

^§Skin prick tests and intradermal test results were not presented separately.

^||Not clearly specified for all patients.

^¶Consecutive sample consisting of all participants with history/events compatible with penicillin allergy.

^#Information not available.

^**Participants with reported cephalosporin allergy are not being counted.

^††Consecutive sample consisting of all participants tested for penicillin allergy.

The meta-analysis assessing the diagnostic accuracy of skin tests comprised 20 studies^{22–29,31,32,34,35,38,40,41,43–46,48} (Fig 2, A; Table II). These studies had been mostly performed in Europe^{24,26–29,32,38,40,41,45,48} (n = 11) and North America^{22,23,25,31,43,44} (n = 6), and mostly assessed outpatients^{23,24,27–29,34,38,40,41,43–46,48} (n = 14). Adults and children were exclusively assessed by 5 studies each,^{25,28,29,31,35,38,40,41,44,48} whereas 9 studies assessed patients of all age groups_.^{22–24,26,27,32,34,43,46} In 9 studies,^{26,27,32,34,35,38,45,46,48} aconsecutive sample was assessed, whereas other 9 reported a convenience sample.^{23,24,28,29,31,40,41,43,44} We obtained a partial AUC-sROC of 0.686, with a summary sensitivity of 30.7% (95% CI, 18.9%−45.8%) and a specificity of 96.8% (95% CI, 94.2%−98.3%). No considerable threshold effect was observed (r = 0.363). Similar results were obtained in sensitivity analyses addressing selection bias or possible sources of confounding, albeit with less precise estimates (Table II). Heterogeneity was found to be moderate (I² = 38.2%), with the year of publication and performance in Middle East found to be moderators of heterogeneity (Table III). Predictive values estimates are depicted in Fig 3.

FIG 2.

sROC curves for the diagnostic performance of penicillin skin tests (all reactions—A; nonimmediate reactions—B), intradermal tests (all reactions—C; nonimmediate reactions—D), and sIgE quantification (all reactions—E; immediate reactions—F): For each test, the sROC curve plots the respective predicted sensitivity against the false-positive rate. Within it, the summary point indicates the summary (“pooled”) sensitivity and specificity of the respective test (its precision is indicated by a 95% confidence region). Both the summary point and the sROC curve were calculated on the basis of diagnostic measures presented in the included primary studies. Between-studies variability is plotted through 95% prediction regions, which indicate the region where we are most confident the test diagnostic performance lies. For immediate reactions, we were not able to plot sROC curves, as we did not have a sufficient number of studies presenting values for true positives, true negatives, false positives, and false negatives.

TABLE II.

Results of the bivariate diagnostic random-effects meta-analyses for penicillin allergy diagnostic tests

Diagnostic test	No. of studies	AUC-sROC	Partial AUC-sROC	Summary measures (95% CI)					Correlation coefficient*	I2
				Sensitivity	Specificity	LR+	LR−	DOR
Skin tests	20	0.841	0.686	30.7% (18.9%–45.8%)	96.8% (94.2%–98.3%)	10.1 (4.9–18.6)	0.7 (0.6–0.8)	14.7 (6.2–29.7)	0.363	38.2%
More than 50% participants†	8	0.842	0.667	39.2% (14.6%–70.0%)	93.5% (83.0%–97.8%)	6.6 (2.1–16.0)	0.6 (0.3–0.9)	11.6 (2.5–34.5)	0.545	45.5%
Generalized provocation tests‡	4	0.694	0.480	41.2% (14.2%–75.0%)	87.4% (54.0%–97.6%)	3.6 (1.4–8.5)	0.7 (0.4–0.9)	5.5 (1.9–12.6)	1.000	36.4%
Consecutive sample§	6	0.655	0.449	23.3% (9.3%–47.3%)	95.6% (83.8%–98.9%)	6.5 (1.4–19.8)	0.8 (0.6–1.0)	8.7 (1.5–28.7)	0.416	65.5%
Adult participants	5	0.677	0.228	24.2% (11.7%–43.5%)	97.6% (94.1%–99.0%)	12.2 (2.7–34.6)	0.8 (0.6–0.9)	17.2 (3.0–55.9)	−1.000	0%
Pediatric participants	5	0.857	0.338	31.5% (10.6%–64.1%)	96.0% (87.8%–98.7%)	7.9 (3.9–14.5)	0.7 (0.4–0.9)	11.8 (5.0–23.8)	1.000	20.2%
Skin tests (nonimmediate reactions)||,¶	7	0.926	0.396	44.2% (14.2%–79.1%)	95.1% (88.3%–98.0%)	9.7 (2.6–23.5)	0.6 (0.2–0.9)	21.4 (3.0–76.7)	0.335	22.2%
Intradermal tests||	10	0.778	0.582	29.9% (15.0%–50.8%)	95.1% (88.1%–98.0%)	6.5 (2.6–13.6)	0.7 (0.5–0.9)	9.2 (3.2–21.1)	0.607	54.8%
Intradermal tests (nonimmediate reactions)||,¶	6	0.939	0.404	43.1% (12.8%–79.5%)	95.1% (90.9%–97.5%)	9.7 (2.1–23.5)	0.6 (0.2–0.9)	23.0 (2.3–94.1)	−0.194	35.3%
Specific IgE	11	0.760	0.420	19.3% (12.0%–29.4%)	97.4% (95.2%–98.6%)	7.7 (3.8–13.8)	0.8 (0.7–0.9)	8.8 (4.3–17.9)	1.000	8.5%
More than 50% participants†	6	0.910	0.132	17.4% (8.7%–31.0%)	97.6% (96.0%–98.7%)	7.9 (3.4–15.6)	0.8 (0.7–0.9)	9.7 (3.7–21.0)	1.000	0%#
Specific IgE (immediate reactions)||	4	0.279	0.262	26.2% (18.7%–35.3%)	92.1% (64.7%–98.7%)	5.2 (0.7–21.0)	0.8 (0.7–1.2)	6.8 (0.6–29.2)	−1.000	0%#

DOR, Diagnostic odds ratio; LR, likelihood ratio; z, z score.

^*Correlation coefficient between sensitivity and false-positive rate.

^†Sensitivity analysis restricted to studies in which at least 50% of both patients with positive and negative results on index tests received a DPT.

^‡Studies assessing a mix of participants with immediate and nonimmediate reactions (excluding only those with very severe index reactions), and testing all with index tests and DPT.

^§Sensitivity analysis restricted to consecutive samples of patients specifically reporting a history of penicillin allergy and testing for penicillin polylysine, minor determinant mixture, benzylpenicillin, and an aminopenicillin.

^||Insufficient number of studies to perform sensitivity analyses as prespecified.

^¶All participants had maculopapular exanthema or urticaria/angioedema as their index reaction.

^#Obtained with the Higgins estimator.

TABLE III.

Variables identified as associated with heterogeneity in the bivariate diagnostic random-effects meta-analyses for penicillin allergy diagnostic tests

Diagnostic test	Variables associated with heterogeneity*	Variables associated with heterogeneity in summary sensitivity estimation†	Variables associated with heterogeneity in summary specificity estimation†
Skin tests	Year (P = .002); Middle East (P = .002)	Year (z = −2.8; P = .005)	Middle East (z = −3.4; P = .001)
Skin tests (nonimmediate reactions)	Year (P = .02); Europe (P = .05); Middle East (P = .02); suspected drug testing (P = .02)	Year (z = −2.9; P = .004)	—
Intradermal tests	Europe (P = .001); Middle East (P = .001); suspected drug testing (P = .02)	—	Europe (z = 2.4; P = .02); Middle East (z = −2.4; P = .02)
Intradermal tests (nonimmediate reactions)	Year (P = .010); Europe (P = .05); Middle East (P = .05); suspected drug testing (P = .05)	Year (z = −2.7; P = .006)	Reported allergy to aminopenicillins (z = −2.2; P = .03)
Specific IgE‡	Adults (P = .04)	Emergency department (z = −2.0; P = .05)	Adults (z = −2.6; P = .009)

Adults, Assessment of adult patients; Year, publication year; z, z score.

^*As assessed by the likelihood-ratio test.

^†As assessed by meta-regression.

^‡Insufficient number of studies assessing only participants reporting immediate reactions to perform meta-regression.

FIG 3.

Estimates of the positive and negative predictive values based on the pooled sensitivity and specificity obtained by bivariate meta-analysis. A, Estimates of the PPVs across a range of penicillin allergy prevalences (univariate sensitivity analysis). B, Probability distributions (“Probability (%)”) of the PPVs obtained by means of probabilistic sensitivity analyses. C, Estimates of the negative predictive value across a range of penicillin allergy prevalences (univariate sensitivity analysis). D, Probability distributions (“Probability (%)”) of the negative predictive values obtained by means of probabilistic sensitivity analyses.

For nonimmediate reactions (and contrary to immediate reactions), we were able to perform a bivariate random-effects meta-analysis on the diagnostic accuracy of skin tests. This analysis was based on 7 studies^{26,28,32,35,40,41,46} (Fig 2, B; Table II), with almost all participants reporting either maculopapular exanthema or urticaria/angioedema. The partial AUC-sROC was of 0.396 (r = 0.335), the summary sensitivity was of 44.2% (95% CI, 14.2%−79.1%), the specificity was of 95.1% (95% CI, 88.3%−98.0%), and an I² of 22.2% was obtained.

For skin prick tests, we were able to estimate only pooled single diagnostic measures (Table E4). Regarding intradermal tests, we performed a bivariate diagnostic random-effects meta-analysis (Fig 2, C; Table II) based on 10 _studies^{24,26,28,32,34,38,40,42,46,48}—we observed a partial AUC-sROC of 0.582, a summary sensitivity of 29.9% (95% CI, 15.0%−50.8%), and a specificity of 95.1% (95% CI, 88.1%−98.0%). A borderline threshold effect was observed (r = 0.607) along with severe heterogeneity (I² = 54.8%). Performance in Europe and testing the suspected culprit drug were identified as moderators of heterogeneity (Table III). Among patients reporting nonimmediate reactions (largely maculopapular exanthema or urticaria/angioedema),^{26,28,32,40,42,46} a partial AUC-sROC of 0.404 was obtained (Fig 2, D; Table II), with a summary sensitivity of 43.1% (95% CI, 12.8%−79.5%) and a specificity of 95.1% (95% CI, 90.9%−97.5%). No relevant threshold effect was observed (r = −0.194), and moderate heterogeneity was observed (I² = 35.3%).

Eleven studies were included in the meta-analysis assessing the accuracy of specific IgE.^{24,28,30,33,36–39,42,47,48} The studies had been mostly performed in Europe (n = 10),^{24,28,30,33,36,38,39,42,47,48} and among outpatients (n = 9).^{24,28,33,36–39,42,47,48} Four studies assessed adults,^28,37,38,47 whereas 3 assessed children^36,42,48 and other 4 studies patients of all age groups.^24,30,33,39 The sampling method was consecutive for 6 studies^{33,36,38,39,42,48} and of convenience for 5 studies.^{24,28,30,37,47} We obtained a partial AUC-sROC of 0.420, but considerable threshold effect was observed (r = 1.000) (Fig 2, E; Table II). This model resulted in a summary sensitivity of 19.3% (95% CI, 12.0%−29.4%) and a specificity of 97.4% (95% CI, 95.2%−98.6%). Similar results were obtained in a sensitivity analysis (Table II). Heterogeneity was found to be low (I² = 8.5%), with participants’ age identified as a moderator of heterogeneity (Table III). Predictive value estimates are depicted in Fig 3. Among patients reporting immediate reactions,^28,33,42,47 we obtained a partial AUC-sROC of 0.262, with a summary sensitivity of 26.2% (95% CI, 18.7%−35.3%) and a summary specificity of 92.1% (95% CI, 64.7%−98.7%).

Studies’ quality was assessed using the QUADAS-2 tool¹⁶—Fig E1 depicts the risk-of-bias graph, whereas Table E5 presents the risk-of-bias summary. The main quality concerns of the assessed studies are related to patients’ flow (in most studies, not all tested patients received a DPT) and to “patient selection domain” (reflecting concerns on sample representativity). Of all included primary studies, only 2 (1.9%) were judged as “low risk” on all bias-related domains,^42,46 while 12 (11.4%) were judged as “low concern” on all applicability domains.^{24,27,32,34,39,69,77,102,111,114,120,123}

DISCUSSION

In this systematic review, we assessed the diagnostic accuracy of skin tests and sIgE quantification for patients reporting a penicillin allergy. In all performed analyses, both skin tests and sIgE assessment were found to have a high specificity but a sensitivity to predict hypersensitivity reactions of less than 50%. Based on these obtained values, and on a plausible range of penicillin allergy prevalences, these tests were projected to have a high NPV, but a poor PPV.

Importantly, predictive values are influenced by the prevalence of the studied condition, in this case corresponding to the pretest probability of having a true penicillin allergy. For most primary studies we do not have individual participant-level information on the presentation/severity of the index reaction (nor were defined homogeneous inclusion criteria regarding the latter). However, we have information that most participants reported a mild index reaction, reflecting the fact that most included studies consist of a description of a mostly low-risk cohort of patients undergoing a diagnostic workup for penicillin allergy. This may explain why the PPVs projected by our models were lower than those estimated by a recent review, which identified a PPV of 81%.¹²⁶ However, that latter review included only studies with detailed characterizations of each participant’s index reaction (and tested with the culprit drug), who, as such, had a higher pretest probability of having a true/severe allergic reaction. Although we were not able to estimate the diagnostic accuracy of skin tests and specific IgE for each clinical presentation or risk group,¹²⁷ we were able to estimate the sensitivity, specificity, and predictive values for penicillin skin tests in patients reporting a delayed maculopapular exanthema or urticaria/angioedema. Therefore, our results apply largely to the most common patients encountered, particularly in North America, namely low-risk patients with rashes and/or swelling.

This study has some limitations, particularly associated with the included primary studies (Table IV lists the limitations of current evidence and key requirements needed for future studies). First, only a minority of them performed DPT in participants with positive index results. As a result, our bivariate diagnostic meta-analysis included fewer primary studies than the corresponding individual diagnostic measures meta-analyses. Nevertheless, we obtained comparable results for the analyses in which the 2 meta-analytic methods were used. Another limitation concerns the possibility of selection bias—in several studies, not all participants who were subject to an index test underwent a DPT. This is particularly true for patients with positive index test results, with possible implications for sensitivity estimation. In fact, there were studies that only challenged patients with mild or borderline positive skin test results.^23,43 To account for this possible bias, we performed sensitivity analyses restricted to (1) studies with at least 50% patients with positive results on the index test performing a DPT and (2) studies assessing a mix of participants with immediate and nonimmediate reactions (excluding only those with very severe index reactions), and testing all with index tests and DPTs. Those sensitivity analyses yielded similar results, although with less precision due to the lower number of primary studies. Nevertheless, even in those studies, most participants reported a clinical history compatible with a mild index reaction.

TABLE IV.

Limitations of current evidence and key aspects for future studies

Limitations of current evidence	Key requirements needed for future studies
- Inconsistent eligibility criteria often creating heterogeneous patient mixes;	- Definition of more homogeneous inclusion criteria to have less patient variation;
- Inconsistent reporting of demographic and clinical details (eg, timing or clinical manifestations of index reactions);	- Standardization of testing protocols;
- Limited ability to assess accuracy of skin tests and specific IgE for different clinical phenotypes or risk groups (except nonimmediate maculopapular exanthema or urticaria/angioedema);	- Limitation of selection bias through reporting of consecutive samples;
- Potential selection bias related to using DPTs;	- Inclusion of comprehensive details of index reactions (eg, clinical manifestations and timing);
- Limited data from Latin America, Asia, and Africa.	- Presentation of results stratified by different reaction phenotypes and risk groups OR publication of anonymized patient-level data.

In addition, in several studies, more than 1 index test was performed, with DPT being done only in those patients who had negative results in all index tests—one particularly common example is the sequential execution of skin prick and intradermal tests before DPT. Therefore, our pooled estimates do not strictly measure the “pure” accuracy of each index test. Nevertheless, for individual diagnostic measures meta-analyses, similar results are obtained when analyzing only studies performing DPT in all patients with negative and/or positive results in the corresponding index tests. We were not able to assess the accuracy of skin prick tests by bivariate random-effects meta-analysis. Nevertheless, results of individual diagnostic measures meta-analyses point to a lower sensitivity of skin prick tests (14.6%) when compared with intradermal tests (29.9%) or with the performance of both skin prick and intradermal tests (30.7%)—the performance of skin prick tests without intradermal tests can result in nonidentifying a larger proportion of patients with allergy.

Other important limitations concern the low quality of most included primary studies and—for some studies— lack of discriminated information regarding the timing of initial reactions (on the other hand, patient-reported initial reaction timing may be subject to recall bias). Finally, although DPT was defined as the reference standard, its accuracy is not 100%, and protocol variability might affect the primary studies results.

This study also has several strengths. In particular, we combined 2 different methods—bivariate diagnostic and individual measure meta-analyses—to estimate summary measures, obtaining comparable results. To minimize the impact of publication bias, we searched 3 different electronic bibliographic databases without applying exclusion criteria based on the date or language of publication. In addition, we performed meta-regression and sensitivity and subgroup analyses to explore sources of heterogeneity and hint at variables possibly associated with differences in penicillin allergy tests’ diagnostic accuracy. Heterogeneity was low/moderate in most bivariate random-effects meta-analyses, with substantial heterogeneity observed in only 2 analyses (in the remaining 10, heterogeneity was found to be low or moderate). The year of publication, the region, and whether the suspected culprit drug was tested were identified as the main variables explaining heterogeneity. A more recent year was associated with a decreased summary sensitivity of skin tests—this may indicate an improvement in DPT procedures, with more recent studies following/contacting patients for a longer period of time after performing a DPT, and not simply a worsening accuracy of skin tests.¹²⁸ Regarding the region, recent surveys found important regional differences in the diagnostic workup of penicillin allergy.^129,130 The included primary studies presented several other differences in their diagnostic approaches (eg, regarding testing for minor determinant mixture, whose individual role we were not able to adequately assess). However, these differences were frequently not identified as moderators of heterogeneity, and similar results were obtained with the sensitivity analysis restricted to studies with a “more homogeneous approach” (which included testing for penicillin polylysine, minor determinant mixture, benzylpenicillin, and an aminopenicillin).

The topic of this systematic review is particularly relevant and timely because penicillin allergy delabeling is now encouraged as a tool of antibiotic stewardship and quality improvement by several organizations—including the American Academy of Allergy, Asthma & Immunology, the European Academy of Allergy and Clinical Immunology, the Centers for Disease Control and Prevention, and the Infectious Diseases Society of America. Overall, our results suggest that, at least in patients reporting a delayed mild index reaction, skin tests may have a high specificity and NPV, but a low PPV and sensitivity in identifying patients who will develop a hypersensitivity reaction when exposed to penicillins. If that trend is confirmed by subsequent studies, direct performance of DPT may be a feasible and safe option at least in some of those patients—in fact, recent studies have shown that patients whose clinical history is nonsevere and imprecise or not compatible with a true reaction may possibly safely undergo DPT without prior skin tests, particularly if the reaction did not occur within 1 hour of penicillin ingestion.^131,132 In addition, our systematic review points to the need for further studies minimizing the risk of selection bias and providing, if not individual-level participant data, at least results by type of index reaction clinical presentation. Such evidence would help in patient stratification, so that penicillin allergy delabeling could be safely performed in the most accurate way for each patient.

Clinical implications:

In patients reporting a penicillin allergy, skin tests (at least in nonimmediate mild reactions) and sIgE quantification appear to have high specificity and negative predictive value, but low sensitivity.

Abbreviations used

AUC-sROC

Area under the summary receiver-operating characteristic curve

DPT

Drug provocation test

NPV

Negative predictive value

PPV

Positive predictive value

sIgE

Specific IgE