A Statistical Approach for Determination of Disk Diffusion-Based Cutoff Values for Systematic Characterization of Wild-Type and Non-Wild-Type Bacterial Populations in Antimicrobial Susceptibility Testing

Abstract

In this study, we introduce a new approach for determination of epidemiologic cutoffs (ECOFFs) and resistant-population cutoffs (RCOFFs) based on receiver operating characteristic (ROC) curves. As an example, the method was applied for determination of ECOFFs for seven different beta-lactam antibiotics and wild-type populations of Escherichia coli, Klebsiella pneumoniae, and Enterobacter cloacae. In addition, RCOFFs were determined for bacterial populations with defined resistance mechanisms (“resistotypes”), i.e., extended-spectrum beta-lactamase (ESBL)-positive E. coli, ESBL-positive K. pneumoniae, and ESBL-positive E. cloacae; AmpC cephalosporinase-positive E. coli and AmpC-positive K. pneumoniae; and broad-spectrum beta-lactamase (BSBL)-positive E. coli. RCOFFs and ECOFFs are instrumental for a systematic characterization of associations between resistotypes and wild-type populations.

INTRODUCTION

In antimicrobial susceptibility testing (AST), clinical isolates are classified as resistant or susceptible on the basis of clinical breakpoints (CBPs), i.e., a threshold inhibition zone diameter size to which the treatment outcome is correlated. Organizations such as the European Committee on Antimicrobial Susceptibility Testing (EUCAST) and the Clinical and Laboratory Standards Institute (CLSI) recommend CBPs for a wide range of antibiotic-pathogen combinations (1, 2). CBPs are derived from a combination of pharmacokinetic and pharmacodynamic (PK/PD) data, clinical outcome data, and in vitro MIC data (3, 4). EUCAST has revived the concept of the microbiological breakpoint proposed by Williams (5) and introduced the epidemiological cutoff (ECOFF), which is used as microbiological evidence in the process of CBP setting. EUCAST defined the ECOFF as “the upper MIC value of the wild-type distribution” and as the cutoff which “separates microorganisms without (wild-type) and with acquired resistance mechanisms (non-wild-type) to the agent in question” (1, 3). In an accompanying paper, Valsesia et al. (6) argue that in order to improve the description of antimicrobial susceptibility patterns in bacterial populations, the relationship between wild-type and non-wild-type populations should be considered. A complementary cutoff term is proposed for this purpose: the resistant-population cutoff (RCOFF). The RCOFF is defined as the largest inhibition zone diameter (or the lowest MIC) that delineates a non-wild-type population and complements the ECOFF for improved discrimination of populations carrying a resistance mechanism (the “resistotypes”) from wild-type bacterial populations.

To date, several methods have been proposed to determine inhibition zone diameter cutoffs, such as the error rate-bounded method (7), a modified error rate-bounded method (8), the normalized resistance interpretation method (9, 10), and few statistical inference approaches (11, 12). However, visual inspection of distributions (eyeball method) has usually been applied to determine cutoffs (12–16). While this method can be applied for cutoff estimation in bimodal distributions, it is flawed in overlapping distributions and also lacks reproducibility (12). EUCAST publishes ECOFF values for a large number of disk diameter and MIC distributions (1). However, a detailed disclosure on the methodology applied for cutoff determination is not available (12). A recent publication reveals the use of the eyeball method for determination of most EUCAST ECOFFs, while suggesting that a statistical characterization of the wild-type distributions by nonlinear regression fitting of cumulative log-normal distributions (12) might have been applied for some antibiotic-species combinations (16).

The aim of this study was to establish a robust, reproducible, and practical statistical method to determine inhibition zone diameter cutoffs. The proposed approach is based on a receiver operating characteristic (ROC) curve methodology, which enables concomitant determination of ECOFFs and RCOFFs. The ROC curve methodology was first developed in the 1950s for signal analysis and has been frequently used since than also for the assessment of the accuracy of medical imaging devices (17, 18). Over the years, ROC curve analysis has also been increasingly applied for evaluation of microbiological diagnostic tests (19–22). In this study, we used the ROC curve methodology for the selection of antibiotic-specific cutoff values. The method was exemplified by generating cutoff values for the distributions of clinical isolates of Escherichia coli, Klebsiella pneumoniae, and Enterobacter cloacae for seven different beta-lactam antibiotics. ROC curve-based analysis was performed for wild-type populations and populations carrying different resistance mechanisms against beta-lactam antibiotics, i.e., extended-spectrum beta-lactamases (ESBLs), broad-spectrum beta-lactamases (BSBL), and AmpC-type beta-lactamases.

MATERIALS AND METHODS

Clinical isolates.

A total of 9,513 nonduplicate E. coli strains, 2,137 nonduplicate K. pneumoniae strains, and 516 nonduplicate E. cloacae strains were included in this study. Isolates of the same species were considered duplicates if they (i) originated from the same patient and (ii) showed one major and two minor differences in antibiotic susceptibility testing (AST) interpretation at maximum (23). All duplicates were excluded from the analysis. The bacterial strains were isolated from January 2010 until March 2014 in the clinical microbiology laboratory of the Institute for Medical Microbiology (IMM), University of Zurich. Specific numbers for each species/drug combination are indicated in below (see Table 2). Different isolate numbers reflect missing values for certain antibiotic-isolate combinations.

TABLE 2.

Beta-lactam cutoff values determined at the Institute of Medical Microbiology, University of Zurich, on the basis of ROC curves for E. coli beta-lactamase resistotypes (AmpC, ESBL, and BSBL) and E. coli wild type, K. pneumoniae beta-lactamase resistotypes (AmpC and ESBL) and K. pneumoniae wild type, and E. cloacae ESBL and E. cloacae wild type^a

Species	Antibiotic	Resistotype/phenotype	No. of isolates analyzed at the IMMb	IMM ECOFF (mm)	IMM RCOFF (mm)	AUC	95% CI(AUC)
E. coli	Piperacillin-tazobactam	AmpC	172	20	30	0.86	0.83–0.90
		ESBL	899	20	31	0.79	0.77–0.81
		BSBL	3,649	20	32	0.64	0.63–0.66
		Wild type	4,779	20
	Cefuroxime	AmpC	172	19	26	0.97	0.95–0.98
		ESBL	900	19	19	1.00	0.99–1.00
		BSBL	3,649	19	31	0.57	0.56–0.58
		Wild type	4,778	19
	Cefotaxime	AmpC	172	23	31	0.96	0.95–0.98
		ESBL	901	23	23	1.00	0.99–1.00
		BSBL	3,652	23	39	0.53	0.51–0.54
		Wild type	4,787	23
	Ceftazidime	AmpC	172	21	29	0.96	0.94–0.98
		ESBL	900	21	29	0.96	0.95–0.97
		BSBL	3,648	21	35	0.53	0.51–0.54
		Wild type	4,774	21
	Cefepime	AmpC	172	25	38	0.75	0.71–0.79
		ESBL	901	25	29	0.99	0.99–1.00
		BSBL	3,651	25	39	0.56	0.55–0.57
		Wild type	4,787	25
	Ertapenem	AmpC	172	25	38	0.81	0.78–0.84
		ESBL	901	25	39	0.79	0.77–0.80
		BSBL	3,649	25	41	0.52	0.50–0.53
		Wild type	4,784	25
	Meropenem	AmpC	172	25	39	0.60	0.56–0.64
		ESBL	901	25	40	0.60	0.58–0.62
		BSBL	3,653	25	41	0.50	0.49–0.51
		Wild type	4,786	25
K. pneumoniae	Piperacillin-tazobactam	AmpC	23	18	27	0.91	0.83–0.99
		ESBL	254	18	27	0.81	0.78–0.84
		Wild type	1,858	18
	Cefuroxime	AmpC	23	18	25	0.97	0.93–1.00
		ESBL	255	18	24	0.99	0.98–1.00
		Wild type	1,855	18
	Cefotaxime	AmpC	22	22	33	0.96	0.89–1.00
		ESBL	255	22	25	0.99	0.99–1.00
		Wild type	1,857	22
	Ceftazidime	AmpC	23	21	32	0.95	0.87–1.00
		ESBL	255	21	27	0.98	0.97–0.99
		Wild type	1,858	21
	Cefepime	AmpC	23	22	37	0.53	0.41–0.64
		ESBL	255	23	28	0.99	0.99–1.00
		Wild type	1,856	23
	Ertapenem	AmpC	23	24	33	0.95	0.89–1.00
		ESBL	253	24	38	0.76	0.72–0.79
		Wild type	1,856	24
	Meropenem	AmpC	22	23	35	0.59	0.48–0.69
		ESBL	253	23	37	0.56	0.52–0.59
		Wild type	1,859	23
E. cloacae	Piperacillin-tazobactam	ESBL	61	19	30	0.82	0.76–0.89
		Wild type	455	19
	Cefuroxime	ESBL	61	17	20	1.00	0.99–1.00
		Wild type	455	17
	Cefotaxime	ESBL	61	20	21	1.00	0.99–1.00
		Wild type	454	20
	Ceftazidime	ESBL	61	21	30	0.97	0.94–1.00
		Wild type	455	21
	Cefepime	ESBL	61	25	30	0.98	0.97–1.00
		Wild type	455	25
	Ertapenem	ESBL	61	21	34	0.79	0.73–0.85
		Wild type	454	21
	Meropenem	ESBL	61	25	38	0.62	0.54–0.70
		Wild type	455	25

^aCutoff values for 7 beta-lactam antibiotics were determined on the basis of the ROC curves for E. coli beta-lactamase resistotypes (AmpC, ESBL, and BSBL) and E. coli wild type, K. pneumoniae beta-lactamase resistotypes (AmpC and ESBL) and K. pneumoniae wild type, and E. cloacae ESBL and E. cloacae wild type. The area under the curve (AUC) together with the confidence interval (95% CI[AUC]) reflects the magnitude of discrimination between resistotypes and wild-type populations with respect to the inhibition zone diameter. AUC values close to 1 indicate a perfect discrimination between both populations and low in vitro activity of the antibiotic for the resistotype. AUC values close to 0.5 indicate low ability to discriminate between wild-type and non-wild-type populations and high in vitro antibiotic activity.

^bIMM, Institute of Medical Microbiology, University of Zurich.

Antibiotic susceptibility testing.

The AST of clinical strains was performed following standard procedures as previously described (24–26). Müller-Hinton agar and antibiotic discs were obtained from Becton-Dickinson (Franklin Lakes, NJ) or i2a (Montpellier, France). The inhibition zone diameters were recorded with the Sirweb/Sirscan system (i2a). The phenotypic screening for AmpC-type cephalosporinases was done by cefoxitin screening and confirmed by combined disk testing with cefoxitin and cefoxitin-cloxacillin (27). The phenotypic screening for ESBLs was done using the cefpodoxime inhibition diameter size (<21 mm) and/or the presence of a synergy phenomenon between third-generation cephalosporins and amoxicillin-clavulanic acid (28–30). The phenotypic confirmation of ESBL was done by combined disk testing using cephalosporin disks with or without clavulanic acid, as described previously (29). The phenotypic carbapenemase screening was done by determination of the inhibition zone diameters. Isolates were considered suspicious for carbapenemase production if they were nonsusceptible to ertapenem and/or meropenem and/or imipenem (intermediate or resistant zone diameters according to EUCAST CBPs) (31). The presence of carbapenemases was confirmed by combined disk testing using carbapenem disks with and without EDTA and boronic acid as specific inhibitors for class A and B carbapenemases, respectively (31). The isolates that had inconsistent results in the AmpC, ESBL, or carbapenemase confirmation tests were subjected to molecular analysis (27, 29, 31).

Isolates showing inhibition zone diameters corresponding to the upper or lower limits of a distribution were checked by visual reevaluation of the recorded inhibition zone diameters and by cross-checking with clinical laboratory reports.

Phenotype/resistotype definitions.

AST data for all clinical isolates were derived from our database of inhibition zone diameters. The bacterial populations were classified in subpopulations according to their phenotype/resistotype profile. Following a previous study (32), the wild-type E. coli population lacking any resistance mechanism against beta-lactam antibiotics was defined as (i) susceptible to ampicillin to exclude the presence of BSBLs, (ii) susceptible to cephalothin to exclude the presence of cephalosporinases, (iii) susceptible to cefoxitin to exclude permeability changes due to porin deficiency or increased efflux, and (iv) devoid of ESBLs, AmpC cephalosporinases, and carbapenemases (Table 1). Wild-type K. pneumoniae is intrinsically resistant to penicillins, due to the presence of the bla_SHV gene, and was defined as being (i) resistant to ampicillin, (ii) susceptible to cephalothin to exclude cephalosporinases, (iii) susceptible to cefoxitin to exclude porin deficiency or presence of efflux systems, (iv) susceptible to amoxicillin-clavulanic acid to exclude inhibitor-resistant (IRT) TEM or SHV (33), and (v) devoid of ESBLs, AmpC cephalosporinases, and carbapenemases (Table 1). E. cloacae is naturally resistant to penicillins, early-generation cephalosporins, and cephamycins, due to intrinsic production of AmpC (34). The E. cloacae wild-type population was defined as (i) resistant to ampicillin, (ii) resistant to amoxicillin-clavulanic acid, (iii) resistant to cephalothin, (iv) resistant to cefoxitin, (v) susceptible to cefpodoxime to exclude hyperproduction of AmpC, and (vi) devoid of ESBLs and carbapenemases (Table 1). The BSBL E. coli population was defined as (i) resistant to ampicillin (substrate of BSBL), (ii) susceptible to cephalothin (not a substrate of BSBL) to exclude the presence of cephalosporinases, (iii susceptible to cefoxitin to exclude permeability changes due to porin deficiency or increased efflux, and (iv) absence of ESBLs, AmpC cephalosporinases, and carbapenemases (Table 1). The AmpC resistotype was defined as strains that were positive in the AmpC confirmation assay but were negative for ESBLs and carbapenemases (Table 1). The ESBL resistotype was defined as strains that were positive in the ESBL confirmation assay but negative for AmpC and carbapenemase assays (Table 1). Isolates carrying multiple resistance mechanisms were excluded from the study population.

TABLE 1.

E. coli, K. pneumoniae, and E. cloacae strains isolated in the diagnostic laboratory of the Institute of Medical Microbiology, University of Zurich, between January 2010 and March 2014

Bacterial species	Subpopulation (phenotype/resistotype)	Defining antibiotic/testsa	No. of isolates	Prevalence (%)
E. coli	Wild type	Ampicillin S, cefoxitin S, cephalothin S, AmpC negative, ESBL negative, carbapenemase negative	4,787	49
	BSBL	Ampicillin R, cefoxitin S, cephalothin S, AmpC negative, ESBL negative, carbapenemase negative	3,653	37
	ESBL	ESBL positive, AmpC negative, carbapenemase negative	901	9
	AmpC	AmpC positive, ESBL negative, carbapenemase negative	172	2
	Other resistotypesb		333	3
	Total		9,846	100
K. pneumoniae	Wild type	Ampicillin R, cefoxitin S, cephalothin S, amoxicillin-clavulanic acid S, AmpC negative, ESBL negative, carbapenemase negative	1,859	65
	ESBL	ESBL positive, AmpC negative, carbapenemase negative	255	9
	AmpC	AmpC positive, ESBL negative, carbapenemase negative	23	1
	Other resistotypesc		737	26
	Total		2,874	100
E. cloacae	Wild type	Ampicillin R, cefoxitin R, cephalothin R, amoxicillin-clavulanic acid R, ESBL negative, carbapenemase negative	455	28
	ESBL	ESBL positive, carbapenemase negative	61	4
	Other resistotypesd		1,092	68
	Total		1,608	100

^aS, susceptible; R, resistant.

^bFor example, combinations of BSBL and permeability modifications (PM), beta-lactamase groups 2c, 2e, and 2d ± PM, inhibitor-resistant (IRT) beta-lactamase groups 2er and 2dr ± PM, carbepenemases ± PM, AmpC and ESBL ± PM, ESBL and carbepenemases ± PM, and AmpC and carbapenemases ± PM.

^cFor example, combinations of IRT BSBL, BSBL hyperproducers (Hy) ± PM; beta-lactamase groups 2c, 2e, and 2d ± PM, IRT beta-lactamase groups 2er and 2dr ± PM, carbapenemases ± PM, AmpC and ESBL, ESBL and carbapenemases ± PM, and AmpC and carbapenemases ± PM. Isolates susceptible to ampicillin were also included in this group.

^dFor example, combinations of AmpC Hy ± PM, carbapenemases ± PM, and ESBL and carbapenemases ± PM. Isolates susceptible to ampicillin and/or amoxicillin-clavulanic acid and/or cefoxitin and/or cephalothin were also included in this group.

Statistical analysis and determination of cutoffs.

According to EUCAST, the ECOFF is defined as the smallest inhibition zone size of a wild-type distribution (3). As proposed in an accompanying paper by Valsesia et al. (6), the RCOFF is defined as the largest inhibition zone diameter of defined non-wild-type populations (the resistotypes).

Cutoffs that separated distributions of wild-type and non-wild-type populations of E. coli, K. pneumoniae, and E. cloacae were determined for seven different beta-lactam antibiotics (piperacillin-tazobactam, cefuroxime, cefotaxime, ceftazidime, cefepime, ertapenem, and meropenem). Three E. coli resistotype-wild-type population combinations were investigated, i.e., AmpC-wild-type, ESBL-wild-type, and BSBL-wild type. For K. pneumoniae, the AmpC-wild-type and ESBL-wild-type population combinations were analyzed, and for E. cloacae the ESBL-wild-type combination was studied. Cutoffs were determined by means of the receiver operating characteristic (ROC) curve methodology, which hinges on the assignment of isolates to non-wild-type and wild-type populations by independent external tests (Table 1). To ensure independent determination of cutoff values, antibiotics used to define subpopulations (ampicillin, amoxicillin-clavulanic acid, cephalothin, cefoxitin, and cefpodoxime) were excluded from the analysis.

ROC curves were generated for inhibition zone diameter values with respect to the isolates' affiliation either to the wild-type or to the non-wild-type population. ROC curves provide sensitivity and specificity levels for thresholds imposed on inhibition zone diameters by applying the following procedure. A threshold is set at each observed inhibition zone diameter. Isolates with inhibition zone diameters below the threshold are classified as non-wild type by the test, whereas isolates with inhibition zone diameters above the threshold are regarded as wild type. Sensitivity and specificity are computed on the basis of the test classification of the isolates in non-wild-type and wild type populations. Sensitivity estimates the conditional probability of testing an isolate as non-wild type when it is truly non-wild type. Specificity estimates the conditional probability of testing an isolate as wild type when it is truly wild type. According to this approach, cutoff values with a defined sensitivity or susceptibility level which correspond to a given inhibition zone size can be selected. The cutoff values were derived from the coordinates of the ROC curves, and the ECOFF was set as the nearest full diameter to the specificity level of 99%, so that ≥98.5% of the disk diameter values lie within the cutoff range. The RCOFF was chosen as the nearest full diameter to the 99% sensitivity threshold in order to include ≥98.5% of the disk diameter values in the cutoff range (Fig. 1).

FIG 1.

ROC curve generated for cefuroxime inhibition zone diameters of E. coli AmpC/wild-type populations. Graphic representation and coordinates of the ROC curve generated for cefuroxime inhibition zone diameters of E. coli AmpC/wild-type populations. The non-wild-type subpopulation (AmpC) is classified as “positive” actual state (value of 1.00) in the test evaluation, while the wild-type population is classified as “negative.” ECOFFs and RCOFFs with a specificity/sensitivity level of 100% or 99% are indicated.

The area under the curve (AUC) together with the corresponding 95% confidence interval (95% CI[AUC]) was computed as described elsewhere (35, 36). In AST, the AUC is a measure of the ability of a specific antibiotic to discriminate between the wild-type and non-wild-type populations based on the inhibition zone diameter sizes. AUC values close to 1 indicate a perfect ability of a specific antibiotic to discriminate between the non-wild-type and the wild-type populations and low in vitro activity of the antibiotic against the resistotype. In contrast, AUC values close to 0.5 for a given antibiotic are indicators of low discriminative power and high in vitro activity of the antibiotic. Following Greiner et al. (17) and Swets (37), we suggest that AUC values of 1 indicate a perfect discrimination between the wild-type and non-wild-type condition by a specific antibiotic, AUC values of ≥0.90 indicate a highly accurate discrimination, AUC values between 0.71 and 0.89 indicate moderate accuracy, and AUC values between 0.51 and 0.70 indicate low discrimination power of the antibiotic. AUC values of 0.50 are noninformative, as the same level of discrimination can be achieved by mere chance.

Software.

Data were coded in Microsoft Excel 2010 software (Microsoft Corporation, Redmond, VA) and analyzed using SPSS Statistics software version 22 (IBM Corp., Armonk, NY).

RESULTS

The inhibition zone diameters for 9,513 E. coli, 2,137 K. pneumoniae, and 516 E. cloacae clinical isolates were analyzed in this study. Isolates were classified as either wild type or non-wild type, depending on the presence or absence of resistance mechanisms against beta-lactam antibiotics. For E. coli, three resistotypes were analyzed (ESBL, AmpC, and BSBL), for K. pneumoniae, two resistotypes were investigated (ESBL and AmpC), and for E. cloacae, the ESBL resistotype was analyzed (Table 1).

The ECOFF is defined by EUCAST as the smallest inhibition zone size of a wild-type distribution (3). ECOFF values (IMM ECOFFs) for each antibiotic were derived from the ROC curves (Table 2 and Fig. 2). For E. coli, the IMM ECOFFs were mostly similar to the EUCAST ECOFFs, with only minor differences (Δ = 1 mm), e.g., 19 mm versus 18 mm for cefuroxime. However, substantial discrepancies (Δ = 3 to 4 mm) were observed for cefepime and ertapenem (25 mm versus 28 to 29 mm) (Table 3). For K. pneumoniae, only minor differences (Δ = 1 to 2 mm) between the IMM and EUCAST values were observed, e.g., 18 mm versus 17 mm for piperacillin-tazobactam and 21 mm versus 19 mm for ceftazidime. For E. cloacae, the IMM ECOFF values were identical to the EUCAST values for meropenem and piperacillin-tazobactam. A comparison of the IMM and EUCAST ECOFFs for other beta-lactam antibiotics was not possible for E. cloacae, as the EUCAST ECOFFs are not available.

FIG 2.

Graphic representation of ECOFFs and RCOFFs for resistotype-beta-lactam antibiotic combinations of E. coli (A), K. pneumoniae (B), and E. cloacae (C). Also, inhibition zone diameter distributions for the each resistotype and the wild-type populations are depicted, as well as the median values of both distributions.

TABLE 3.

ECOFF values determined at the Institute of Medical Microbiology, University of Zurich, on the basis of the ROC curves for wild-type populations of E. coli, K. pneumoniae, E. cloacae, and beta-lactam antibiotics, compared with ECOFF values published by EUCAST^a

Species	Antibiotic class	Antibiotic	IMM ECOFF (mm)	No. of isolates analyzed at the IMM	EUCAST ECOFF (mm)	Isolates analyzed EUCAST (n)	Data sources (no. of laboratories)	EUCAST S CBP (≥, mm)	EUCAST R CBP (<, mm)
E. coli	Beta-lactams	Piperacillin-tazobactam	20	4,779	20	6,033	9	20	17
		Cefuroxime	19	4,778	18	334	4	18	18
		Cefotaxime	23	4,787	23	14,654	6	20	17
		Ceftazidime	21	4,774	22	14,658	6	22	19
		Cefepime	25	4,787	28	280	3	24	21
		Ertapenem	25	4,784	29	738	3	25	22
		Meropenem	25	4,786	25	7,027	7	22	16
K. pneumoniae	Beta-lactams	Piperacillin-tazobactam	18	1,858	17	770	7	20	17
		Cefuroxime	18	1,855	18	217	4	18	18
		Cefotaxime	22	1,857	21	1,525	6	20	17
		Ceftazidime	21	1,858	19	1,523	6	22	19
		Cefepime	23	1,856	24	216	4	24	21
		Ertapenem	24	1,856	25	220	4	25	22
		Meropenem	23	1,859	25	936	6	22	16
E. cloacae	Beta-lactams	Piperacillin-tazobactam	19	455	19	228	4	20	17
		Cefuroxime	17	455	NAb	96	2	18	18
		Cefotaxime	20	454	NA	366	3	20	17
		Ceftazidime	21	455	NA	366	3	22	19
		Cefepime	25	455	NA	94	2	24	21
		Ertapenem	21	454	NA	113	4	25	22
		Meropenem	25	455	25	259	3	22	16

^aECOFF values were determined at the Institute of Medical Microbiology, University of Zurich (IMM ECOFFs), on the basis of the ROC curves for wild-type populations of E. coli, K. pneumoniae, E. cloacae, and beta-lactam antibiotics and compared with ECOFF values (EUCAST ECOFFs) published by EUCAST (1). The number of isolates analyzed and the numbers of data sources for the EUCAST ECOFFs are indicated.

^bNA, not available.

The RCOFF is defined in an accompanying paper by Valsesia et al. (6) as the largest inhibition zone diameter of a resistotype. Resistotype-specific RCOFF values (IMM RCOFFs) were derived from the coordinates of the ROC curves. The RCOFF values for the majority of E. coli, K. pneumoniae, and E. cloacae resistotype-drug combinations were larger than the respective wild-type ECOFFs (Table 2). For example, the ceftazidime, cefepime, ertapenem, and meropenem RCOFFs for the E. coli ESBL resistotype were 29 mm, 29 mm, 39 mm, and 40 mm, while the respective ECOFFs were 21 mm, 25 mm, 25 mm, and 25 mm (Table 2).

To quantify the relative drug activity for specific resistotypes, AUC values were generated for each ROC curve, illustrating the ability of a specific antibiotic to discriminate between the wild-type and non-wild-type populations on the basis of inhibition zone diameter values (Table 2). AUC values are indicative of the in vitro activity of an antibiotic against a certain resistotype. AUC values of ≥0.90 were observed with the cephalosporins for E. coli ESBL, K. pneumoniae ESBL, and E. cloacae ESBL, indicating accurate discrimination between the resistotype and wild-type populations (i.e., clear separation of the populations) and low in vitro activity of those antibiotics against ESBL-positive isolates (Table 2 and Fig. 2). Lower AUC values were found for piperacillin-tazobactam (0.79, 0.81, and 0.82 for E. coli ESBL, K. pneumoniae ESBL, and E. cloacae ESBL, respectively) and ertapenem (0.79, 0.76, and 0.79, respectively), suggesting moderate discriminative power and moderate in vitro activity (Table 2 and Fig. 2). Little discrimination between the wild type and the resistotype was observed with meropenem (AUC values of 0.60, 0.59, and 0.62, respectively), indicating high in vitro activity of this compound; the ESBL resistotype and the wild type showed similar (coincident) inhibition zone diameter distributions (Table 2 and Fig. 2).

High AUC values (≥0.90) were observed for AmpC E. coli and AmpC K. pneumoniae with beta-lactam-inhibitor combinations and cephalosporins, with the exception of cefepime (0.75 and 0.53, respectively). Meropenem showed low AUC values for AmpC E. coli and AmpC K. pneumoniae (0.60 and 0.59, respectively), indicating high in vitro activity. In contrast, ertapenem had AUC values of 0.81 and 0.95 for AmpC E. coli and AmpC K. pneumoniae, respectively, indicating reduced in vitro activity (Table 2 and Fig. 2).

Amoxicillin-clavulanic acid was the only discriminative antibiotic for E. coli BSBL (AUC value of 0.86, data not shown), followed by piperacillin-tazobactam (AUC value of 0.64), cephalosporins and carbapenems (AUC values ranging from 0.52 to 0.57), which were poorly discriminative (Table 2). The latter observation suggests good activity levels for cephalosporins and carbapenems against E. coli BSBL and relatively low activity for the inhibitor combinations (Table 2; Fig. 2A).

DISCUSSION

Epidemiological cutoffs are microbiological parameters used by EUCAST in the process of CBP setting (3, 14) and correspond to the upper MIC/lower inhibition zone size of a wild-type distribution (1). Several methods for determination of inhibition zone diameter cutoffs are available (7–11, 13, 14). However, up to this time no consensus has been reached on the best approach to determine cutoff values (4).

An ROC curve methodology can be applied for determination of wild-type cutoffs (ECOFFs) and resistotype cutoffs (RCOFFs) in a binary classification test such as AST, if subpopulations have been previously and independently classified as displaying a “positive condition,” i.e., carrying a resistance mechanism, or a “negative condition,” i.e., being devoid of any resistance mechanism against the same antibiotic class (35, 36, 38). In this study, the ROC curve-based approach for concomitant determination of ECOFFs and RCOFFs was validated by analyzing the AST data for beta-lactam antibiotics and various populations of E. coli, K. pneumoniae, and E. cloacae, the species most frequently isolated from clinical specimens in our diagnostic laboratory. Identical ECOFF values for each antibiotic were generated from the ROC curves for different resistotype-wild-type combinations, indicating the robustness of the methodology. The two available EUCAST ECOFFs for E. cloacae were identical to the IMM ECOFFs. For E. coli, three IMM ECOFFs were identical to their equivalent EUCAST ECOFFs with minor discrepancies in two cases; major discrepancies were observed in two cases (Table 3), namely, for cefepime and ertapenem. Similarly, for K. pneumoniae, one IMM ECOFF value was identical to the EUCAST ECOFF, and minor discrepancies were observed for six antibiotics. ECOFF values do not depend on local epidemiology, as distribution of wild-type populations are independent of geographical origin and do not change over time (2, 39). A factor possibly influencing ECOFFs is sample size. While our data set was constant, with approximately 4,787 E. coli and 1,859 K. pneumoniae wild-type isolates analyzed for each antibiotic, population sizes analyzed by EUCAST fluctuate between 237 and 14,658 isolates for E. coli and between 142 and 1,525 for K. pneumoniae (Table 2). Although no data on the impact of sample size on cutoff determination are available, we suggest that cutoff values determined either by statistical methods or by visual estimation are likely to be sample size dependent. Cutoff values should stabilize to a constant value once a minimum population size is reached. Our analysis indicates that sample size might play a relevant role for accurate cutoff determination, but further analysis is needed to confirm this hypothesis. Even though EUCAST uses data from multiple laboratories to determine ECOFFs, our results suggest that data from a single laboratory may be sufficient to calculate ECOFFs, since for those antibiotics with comparable sample sizes (cefotaxime, piperacillin-tazobactam, and meropenem), we could exactly reproduce the EUCAST ECOFF values (Table 2). Surprisingly, identical ECOFFs were generated for K. pneumoniae wild-type distributions with cefuroxime and ceftriaxone, despite remarkable sample size differences. In order to understand this inconsistency, detailed information on the specific methodology applied by EUCAST for ECOFF determination for each species-drug combination is desirable.

The RCOFF is introduced in an accompanying paper by Valsesia et al. (6) as a cutoff complementing the ECOFF to systematically characterize non-wild-type and wild-type populations. ECOFFs are proposed to discriminate between wild-type and non-wild-type bacteria (40). However, our results show that resistotype-specific RCOFF values frequently exceed the respective wild-type-specific ECOFFs. This indicates that ECOFFs do not allow a reliable separation of the wild-type population and resistotype populations, e.g., in the case of ceftazidime for E. coli ESBL and cefepime for E. coli AmpC (see Fig. S1 in the supplemental material). As a consequence, classification of non-wild-type and wild-type isolates based solely on ECOFFs is frequently inadequate and may lead to misclassification of isolates (15, 32). RCOFF values allow reliable identification of the zone diameter ranges in which isolates should be subjected to additional testing and/or which may be used to define an intermediate zone in the process of CBP setting.

The in vitro activity of an antibiotic against a specific resistotype may be deduced from the AUC value of the corresponding ROC curve. AUC values of ≤0.70 indicate high in vitro activity of the antibiotic, which is little affected by the resistance mechanism, while AUC values of ≥0.90 indicate low in vitro activity of the antibiotic and the presence of an effective resistance mechanism. AUC values of third-generation cephalosporins for ESBL- and AmpC-positive strains were ≥0.90, indicating low in vitro activity independent of the species analyzed (Fig. 2 and Table 2; see also Fig. S2 in the supplemental material). These AUC values are well in line with the current treatment recommendations and with results of previous studies showing poor therapeutic outcomes for treatment of AmpC- or ESBL-positive Enterobacteriaceae with cephalosporins (41–44). The low AUC values for meropenem (AUC of ≤0.70) indicate high in vitro activity against all resistotypes; resistotypes and wild types show similar distributions of inhibition zone diameters (Fig. 2 and Table 2; see also Fig. S3 in the supplemental material). This finding is corroborated by the results of clinical reports recommending carbapenems as the treatment of choice for infections with ESBL- and AmpC-positive Enterobacteriaceae (44–46). AUC values of >0.70 and <0.90 indicate a moderate level of in vitro activity of an antibiotic against a specific resistotype. Thus, this range represents a gray zone where treatment outcome is likely to be uncertain. Good examples are ESBL and beta-lactam inhibitor combinations. The AUC values of piperacillin-tazobactam for ESBLs were always >0.70 and <0.90 (Fig. 2 and Table 2; see also Fig. S4 in the supplemental material). Of note, the therapeutic use of beta-lactam combinations against ESBL-positive Enterobacteriaceae is still a matter of debate (47–53). To conclude, for any resistotype-drug combination, AUC thresholds which reflect in vitro activity and, at least in the case of beta-lactams, are in agreement with the results of clinical outcome studies may be determined.

One limitation of this study was the restriction of genetic investigations to isolates that had inconsistent results in the phenotypic AmpC, ESBL, and carbapenemase confirmation assays. The phenotypic detection of AmpC, ESBL, and carbapenemase was performed according to EUCAST guidelines and by using established algorithms (27–30). Another limitation was the use of data from a single hospital for resistotype-specific RCOFF determination, resulting in a potential bias toward types of BSBL, ESBL, and AmpC enzymes that are endemic to the area. However, previous studies at our institution have shown a high prevalence of CTX-M-type I ESBL strains (69.4%). This observation correlates well with epidemiological studies that demonstrated a global dominance of CTX-M-type I ESBL enzymes (54–57). Likewise, the analysis of AmpC enzymes at our institution showed a high prevalence (91.7%) of the CIT family in our region, which is consistent with reports from the European, North American, and the Asia-Pacific regions showing a high prevalence of CMY AmpC enzymes (CIT family) (55, 57, 58). Thus, despite a possible impact of local epidemiology on the RCOFF values, the IMM RCOFF values for AmpC and ESBLs are likely to be representative for other geographic areas.

ROC curve analysis of inhibition zone diameter distributions enables statistical determination of cutoff values (RCOFFs and ECOFFs) and leads to reproducible results. ROC curve analysis results in a broad and reliable set of epidemiological and resistant-population cutoff data for critical species-drug combinations, for which ECOFFs are currently not available from EUCAST (such as several beta-lactams and E. cloacae). Quantitative in vitro antimicrobial activity levels can be derived from the AUC values, which appear to correlate with the results of the therapeutic outcome studies.