Unacceptably High Error Rates in Vitek 2 Testing of Cefepime Susceptibility in Extended-Spectrum-β-Lactamase-Producing Escherichia coli

Abstract

While a lack of concordance is known between gold standard MIC determinations and Vitek 2, the magnitude of the discrepancy and its impact on treatment decisions for extended-spectrum-β-lactamase (ESBL)-producing Escherichia coli are not. Clinical isolates of ESBL-producing E. coli were collected from blood, tissue, and body fluid samples from January 2003 to July 2009. Resistance genotypes were identified by PCR. Primary analyses evaluated the discordance between Vitek 2 and gold standard methods using cefepime susceptibility breakpoint cutoff values of 8, 4, and 2 μg/ml. The discrepancies in MICs between the methods were classified per convention as very major, major, and minor errors. Sensitivity, specificity, and positive and negative predictive values for susceptibility classifications were calculated. A total of 304 isolates were identified; 59% (179) of the isolates carried bla_CTX-M, 47% (143) carried bla_TEM, and 4% (12) carried bla_SHV. At a breakpoint MIC of 8 μg/ml, Vitek 2 produced a categorical agreement of 66.8% and exhibited very major, major, and minor error rates of 23% (20/87 isolates), 5.1% (8/157 isolates), and 24% (73/304), respectively. The sensitivity, specificity, and positive and negative predictive values for a susceptibility breakpoint of 8 μg/ml were 94.9%, 61.2%, 72.3%, and 91.8%, respectively. The sensitivity, specificity, and positive and negative predictive values for a susceptibility breakpoint of 2 μg/ml were 83.8%, 65.3%, 41%, and 93.3%, respectively. Vitek 2 results in unacceptably high error rates for cefepime compared to those of agar dilution for ESBL-producing E. coli. Clinicians should be wary of making treatment decisions on the basis of Vitek 2 susceptibility results for ESBL-producing E. coli.

INTRODUCTION

The accuracy of the methods used to evaluate the MIC of a given pathogen is a vital component in providing optimal antimicrobial therapy for patients with serious infections. Previous investigations have revealed poor agreement and high error rates for cefepime as determined by automated susceptibility testing (AST) (1–6). Of particular concern are infections caused by extended-spectrum-β-lactamase (ESBL)-producing Enterobacteriaceae. However, the Clinical Laboratory and Standards Institute (CLSI) no longer recommends the routine use of confirmatory ESBL testing to establish resistance (7, 8). The CLSI now supports the use of phenotypic determination of MICs to establish clinical resistance, often tested using ASTs (9, 10).

In 2010, the MIC breakpoints for all cephalosporin antibiotics were lowered, except those for cefepime (7, 8). Due to concerns about clinical failures at elevated MICs (≥8 μg/ml), the 2014 update to the CLSI interpretive criteria includes a reduced cefepime susceptibility breakpoint for Enterobacteriaceae (≤2 μg/ml), with dose-dependent susceptibility defined at an MIC of 4 μg/ml to 8 μg/ml and resistance defined at an MIC of ≥16 μg/ml (11). The purpose of this study was to compare cefepime MICs determined by an AST method, Vitek 2, to a reference standard method, agar dilution, for ESBL-producing Escherichia coli confirmed by phenotype and genotype. In so doing, we hope to provide a framework for clinicians to use to evaluate the predictive performance of cefepime MICs by Vitek 2 for ESBL-producing E. coli.

(Portions of this article were presented as a poster at the 53rd Interscience Conference on Antimicrobial Agents and Chemotherapy, Denver, CO, USA, 10–13 September 2013.)

MATERIALS AND METHODS

Isolate identification and collection.

Isolates were obtained from frozen stores (i.e., isolates were suspended in Mueller-Hinton broth [Becton, Dickinson, Sparks, MD] with 15% glycerol and frozen at −80°C) at Northwestern Memorial Hospital. These isolates were originally obtained from clinical cultures of blood, sputum, urine, and wound specimens. The isolates were obtained as sequential nonduplicative community and nosocomial E. coli isolates, originally collected between January 2003 and July 2009. For this analysis, isolates were subcultured a minimum of two times before being tested for susceptibility. Susceptibility testing was performed in two manners. Automated susceptibility was determined using Vitek 2 (bioMérieux, La Balme-les-Grottes, France) AST cards GN-09 (12) and GN-25, and reference susceptibility was determined using standard agar dilution. Organism susceptibility was interpreted according to the CLSI guidelines in place during the study period (MIC breakpoint, 8 μg/ml) (13–19). Classification was also assessed according to the revised cefepime susceptibility breakpoint of 2 μg/ml and the susceptible dose-dependent MIC categories of 4 to 8 μg/ml (11).

Phenotypic identification of ESBLs.

Isolates with elevated MICs (≥2 μg/ml) for any 1 of the 3 drugs, aztreonam, ceftazidime, and ceftriaxone, were tested for ESBL production by using the CLSI-recommended double-disk method with disks containing cefotaxime (30 μg), cefotaxime plus clavulanic acid (30 μg and 10 μg, respectively), ceftazidime (30 μg), and ceftazidime plus clavulanic acid (30 μg and 10 μg, respectively) (BD Microbiology Systems, Cockeysville, MD). Isolates were considered ESBL positive when the addition of clavulanic acid resulted in a ≥5-mm increase in the zone diameter for either antimicrobial agent, as previously described (12).

Agar dilution.

Confirmed ESBL-producing E. coli underwent additional susceptibility testing using agar dilution methods, as recommended by the CLSI (20) and executed as previously described. In brief, isolates were subcultured a minimum of 24 h prior to MIC testing. Mueller-Hinton agar (Remel, Lenexa, KS) was prepared according to the manufacturer's specifications, as previously described (21). Laboratory grade standard powder of cefepime (Sigma Chemical Company, St. Louis, MO) was obtained and reconstituted according to the manufacturer's recommendations. Cefepime solution was freshly prepared or was prepared and frozen at −20°C and given a 6-month expiration date. Autoclaved isolate replicators delivered approximately 1 to 2 μl (or 10⁴ CFU) of ESBL-producing E. coli on cefepime-impregnated agar. Quality control was performed for all tests using methods recommended by the CLSI (20).

Detection of bla_CTX-M, bla_TEM, and bla_SHV genes by PCR.

Amplification of the genes encoding β-lactamases was carried out with generic PCR using the primers PAN TEM-F (5′-AGGCACCTATCTCAGCGA-3′), PAN TEM-R (5′-CCGTGTCGCCCTTATTCC-3′), PAN SHV-F (5′-ATTTGTCGCTTCTTTACTCGC-3′), PAN SHV-R (5′-TTTATGGCGTTACCTTTGACC-3′), PAN CTX-M-F (5′-GGATATCGTTGGTGGTGCCATA-3′), and PAN CTX-M-R (5′-TTTGCGATGTGCAGTACCAGTAA-3′). Amplification was performed with a GeneAmp thermal cycler (Life Technologies Corporation, Carlsbad, CA). The PCR conditions were 34 cycles of denaturation at 95°C for 40 s, annealing at 52°C for 30 s, and extension at 68°C for 30 s for bla_CTX-M, 34 cycles of denaturation at 95°C for 30 s, annealing at 55°C for 30 s, and extension at 68°C for 1 min for bla_TEM, and 34 cycles of denaturation at 95°C for 30 s, annealing at 52°C for 30 s, and extension at 68°C for 1 min for bla_SHV. Full genomic sequencing was not performed.

Definitions.

Vitek 2 performance was defined in relation to agar dilution in terms of very major errors or false susceptibility, major errors or false resistance, and minor errors or misclassification of intermediate results for cefepime MICs based on categorical agreement (22). Essential agreement was considered when the MICs obtained with the reference method and Vitek 2 were identical or within ±1 log₂ dilution of each other (23). Scattergrams for agar dilution to Vitek 2 were created (i) for all isolates and (ii) for each cohort of isolates by genotype to compare discrepancy rates (24). The primary susceptibility breakpoint used in this study was 8 μg/ml, as previously defined, and it was used to calculate the rate of very major, major, and minor errors. We also assessed the impact of lowered susceptibility breakpoints (2 and 4 μg/ml).

Statistical analysis.

Descriptive statistics were calculated for very major, major, and minor error rates using candidate cefepime cutoff values of 8, 4, and 2 μg/ml. A confusion matrix was created by comparing each agar-determined susceptible MIC against each Vitek 2-determined susceptible MIC at a cefepime MIC cutoff of 8, 4, or 2 μg/ml. The corresponding sensitivities, specificities, positive predictive values (PPVs), and negative predictive values (NPVs) were calculated, along with 95% confidence intervals (CIs) for each value (25). Minimization of misclassification was assessed by comparing the area under the receiver operating characteristic curve (ROC area) at the MIC cutoff values of 8, 4, and 2 μg/ml. Error rates were compared according to genotype and resistance phenotype using the Chi-square test or Fisher's exact test as appropriate. Significance was set at a two-sided α value of <0.05 for significance. Initial odds ratios (ORs) and 95% CIs for the associations were calculated using bivariate logistic regression. All data analyses were performed using Intercooled Stata, version 13 (Stata Corp, College Station, TX).

RESULTS

Of the 330 isolates that were initially tested by agar dilution, 304 isolates contributed both an evaluable Vitek 2 and an agar dilution MIC to the analysis. A total of 26 isolates did not have an MIC recorded from Vitek 2 and were thus ineligible for the analysis. Of the evaluable isolates, 17 (5.6%) were isolated from blood cultures, 24 (7.9%) were isolated from respiratory cultures (sputum or bronchoalveolar lavage specimens), 234 (77%) were isolated from urine cultures (free or catheterized specimens), and 29 (9.5%) were isolated from wound specimen cultures. The distribution of PCR-identified ESBL genotypes in the sample was 59% (n = 179) bla_CTX-M, 47% (n = 143) bla_TEM, and 4% (n = 12) bla_SHV. Multiple ESBL genotypes were observed in 29.3% (n = 89) of the isolates. Combined expression of bla_CTX-M and bla_TEM occurred in 82 isolates, combined expression of bla_SHV and bla_TEM occurred in 6 isolates, and combined expression of all three genotypes occurred in 1 isolate.

Error rates by category.

The categorical classifications for all isolates are displayed in Table 1. Substantial differences were not seen for genotype cohorts (data not shown). Essential and categorical agreements between Vitek 2 and the reference agar dilution at a breakpoint of 8 μg/ml were 59.2% (180/304 isolates) and 66.8% (203/304 isolates), respectively. Categorical agreements fell to 64.1% and 62.5% at breakpoints of 4 and 2 μg/ml, respectively. Essential agreement was unchanged irrespective of the breakpoint.

TABLE 1.

Agar dilution versus Vitek 2 MICs for 304 phenotypic ESBL-producing E. coli isolates^a

Vitek 2 MIC (μg/ml)	No. of isolates with agar dilution MIC (μg/ml) of:
	1 (S)b	2 (S)	4 (S)	8 (S)	16 (I)c	32 (R)d	64 (R)	Total
1 (S)	28	15	13	5	2	0	0	63
2 (S)	3	11	21	26	13	2	0	76
4 (S)	3	1	5	8	7	3	1	28
8 (S)	3	1	1	5	15	12	2	39
16 (I)	0	0	0	0	1	6	8	15
32 (R)	0	0	0	0	7	3	0	10
64 (R)	2	1	1	4	15	23	27	73
Total	39	29	41	48	60	49	38	304

^aAxes truncated at an MIC of 1 μg/ml, as Vitek 2 does not provide MICs of <1 μg/ml. The primary analysis of error rates utilized a breakpoint MIC of 8 μg/ml. Additional analyses were conducted according to susceptibility breakpoints of 4 μg/ml and 2 μg/ml.

^bS, susceptible.

^cI, intermediate.

^dR, resistant.

Vitek 2 classification errors for susceptible, intermediate, and resistant isolates according to the MIC breakpoints of 8, 4, and 2 μg/ml are displayed in Fig. 1. At a breakpoint of 8 μg/ml, very major, major, and minor errors occurred in 20 cases (23%), 8 cases (5.1%), and 73 cases (24%) for Vitek 2 using the denominator of isolates within the corresponding reference category. Categorical very major errors fell to 19% at an MIC breakpoint of 4 μg/ml but increased to 24.6% at a breakpoint of 2 μg/ml. The rate of very major errors as assessed using CLSI intermethod agreement analysis at a breakpoint of 8 μg/ml was similar to that of the categorical agreement method (Table 2). Very major errors were observed in 20 cases (21.3%), major errors were observed in 8 cases (26.7%), and minor errors were observed in 30 (31.9%) and 15 cases (50%) above and below, respectively, the intermediate category in the intermethod agreement analysis.

FIG 1.

Error rates of Vitek 2 compared to those of agar dilution for cefepime MICs.

TABLE 2.

CLSI intermethod validation for cefepime testing using agar dilution or Vitek 2 at a breakpoint of 8 μg/ml

MIC	Total no. (%) of isolates	No. (%) of isolates with:
		Very major error	Major error	Minor error
≥2 dilutions above breakpoint	94 (30.9)	20 (21.3)	0 (0)	30 (31.9)
1 dilution above or below breakpoint	180 (59.2)	0 (0)	0 (0)	28 (15.6)
≤2 dilutions below breakpoint	30 (9.86)	0 (0)	8 (26.7)	15 (50)

Sensitivity analysis.

The sensitivities, specificities, positive predictive values, and negative predictive values of Vitek 2 to correctly predict susceptibilities defined by agar dilution at MIC cutoff values of 8, 4, and 2 μg/ml are displayed in Table 3. As the MIC cutoff value was lowered from 8 to 2 μg/ml, there was a decrease in sensitivity from 94.9% (95% CI, 90.2% to 97.8%) to 83.8% (95% CI, 72.9% to 91.6%) and an increase in specificity from 61.2% (95% CI, 52.8% to 69.1%) to 65.3% (95% CI, 58.8% to 71.3%). Similarly, as the MIC cutoff was lowered from 8 to 2 μg/ml, there was a decrease in the positive predictive value of susceptible classification (by Vitek 2) from 72.3% (95% CI, 65.7% to 78.3%) to 41% (95% CI, 32.7% to 49.7%) and an increase in the negative predictive value of susceptible classification (resistant or intermediate) from 91.8% (95% CI, 84.5% to 96.4%) to 93.3% (95% CI, 88.4% to 96.6%). The ROC areas for the cutoff values of 8, 4, and 2 μg/ml were 0.718, 0.787, and 0.745, respectively.

TABLE 3.

Performance measures for Vitek 2 determination of cefepime susceptibility compared with those of cefepime agar dilution

Cefepime MIC cutoff (μg/ml)	Sensitivity (% [95% CI])	Specificity (% [95% CI])	Positive predictive value (% [95% CI])	Negative predictive value (% [95% CI])	ROC area
8	94.9 (90.2–97.8)	61.2 (52.8–69.1)	72.3 (65.7–78.3)	91.8 (84.5–96.4)	0.718
4	91.7 (84.9–96.2)	65.6 (58.5–72.3)	59.9 (52–67.4)	93.4 (87.9–97)	0.787
2	83.8 (72.9–91.6)	65.3 (58.8–71.3)	41 (32.7–49.7)	93.3 (88.4–96.6)	0.745

Error rates by category and resistance genes.

At an MIC cutoff value of 8 μg/ml, isolates carrying the bla_CTX-M gene were associated with categorical very major, major, and minor error rates of 16.9% (11/65 isolates), 7.1% (5/70 isolates), and 29.6% (53/179 isolates), respectively. Isolates carrying the bla_TEM gene were associated with categorical very major, major, and minor error rates of 20.9% (9/43 isolates), 4.2% (3/71 isolates), and 23.8% (34/143 isolates), respectively. Isolates carrying the bla_SHV gene were associated with categorical very major, major, and minor error rates of 0% (0/1 isolate), 0% (0/10 isolates), and 16.7% (2/12 isolates), respectively. Of isolates expressing multiple ESBL genes, the categorical very major, major, and minor error rates were observed to be 18.8% (6/32 isolates), 5.6% (2/36 isolates), and 28.1% (25/89 isolates), respectively. The only association of a genotype with error classification categories occurred for bla_CTX-M with a minor error at a cutoff value of 8 μg/ml (OR, 2.2; 95% CI, 1.2 to 3.9).

DISCUSSION

We observed poor agreement between the Vitek 2 and agar dilution methods. The observed categorical very major error rate for cefepime of 23% was much greater than the acceptable FDA standard of <3% for n = 300 isolates. Our study is unique in that, to our knowledge, it is the largest collection of ESBL-producing E. coli isolates tested with an AST and a reference standard method that also provides clinicians with performance metrics that can be used in clinical decision making. Finally, this study provides the first analysis of the predictive performance of Vitek 2 for ESBL-producing E. coli isolates at a cefepime breakpoint of 2 μg/ml.

From a clinical perspective, the PPV of a Vitek 2 classification of susceptibility compared to that of an agar dilution at an MIC cutoff value of 8 μg/ml is low and falls dramatically as the MIC cutoff is lowered from 8 to 2 μg/ml. The NPV for a Vitek 2 classification of resistance compared to that of an agar dilution at an MIC cutoff value of 8 μg/ml increased from 91.8% to 93.3% as the MIC cutoff was lowered from 8 to 2 μg/ml. Based on our findings, lower cefepime breakpoint cutoffs (4 μg/ml and 2 μg/ml) reduced the PPV and increased the NPV for a given cefepime MIC as determined by Vitek 2. The ROC area attempts to balance sensitivity and specificity, with an increasing area under the curve reflecting improved balance. The actual breakpoint must also be set with consideration for pharmacokinetic-pharmacodynamic data on the likelihood of target attainment, clinical outcomes, and minimization of severe misclassification (very major error). Our data suggest that lowering the cefepime susceptibility breakpoint from 8 to 2 μg/ml will increase the very major error (false susceptibility) rate to 24.6% and increase the major error (false resistance) rate to 10.3% for ESBL-producing E. coli. As the CLSI has emphasized the use of MICs over confirmatory testing for determining ESBL status, our data suggest that one should be cautious when interpreting any MIC from Vitek 2 for a given ESBL-producing E. coli isolate as being truly susceptible.

Other studies have also found discrepancies between the Vitek 2 and reference standard methods. Jang et al. evaluated the error rates of Vitek 2 against those of broth microdilution for cefepime susceptibility (3). The authors found a combined categorical very major error rate of 27.4% for cefepime with Vitek 2 for E. coli and Klebsiella pneumoniae isolates. Specifically, the Vitek 2 produced very major error rates of 35.7% (20/56 isolates) and 18% (9/50 isolates) for E. coli and K. pneumoniae isolates, respectively. We observed a categorical very major error rate of 23%, which is similar to those observed for the E. coli isolates in their sample.

Using Vitek 2 and a reference microdilution method, Cantón et al. evaluated 92 Enterobacteriaceae isolates, including 86 confirmed ESBL-producing organisms and 6 inhibitor-resistant TEM-carrying isolates (1). The authors found that Vitek 2 produced a categorical very major error rate of 13.3% (4/30 isolates) for cefepime. We observed a marginally higher 23% categorical rate of very major errors among our isolates.

Limitations to this study should be considered. First, our results can only be interpreted for Vitek 2 technology; however, the Vitek 2 system is widely used in health care systems and is approved by the FDA for susceptibility testing. Second, our study examined only the error rates for ESBL-producing E. coli, and our findings may not be generalizable to other ESBL-producing organisms. Third, wide generalizability of our findings is limited by the predominance of bla_CTX-M-type ESBLs in our sample, which have a higher prevalence in the community than in the health care environment (3). Likewise, the current analysis may not have been adequately powered to assess AST interactions with other genotypes (bla_TEM or bla_SHV), and the use of a generic primer may have underestimated the level of association between these genotypes and the error rates. Fourth, our data may be limited in that our reference standard was not a broth-to-broth comparison against Vitek 2 but rather a comparison of agar to broth. However, we do not think that this comparison had a major impact on our results, as the CLSI has determined agar dilution to be an acceptable standard for MIC determination (20).

Vitek 2 appears to be associated with very high error rates and may not be a reliable clinical tool for making predictions of the susceptibility of ESBL-producing Enterobacteriaceae to cefepime. Lowering the cefepime susceptibility breakpoint from 8 to 2 μg/ml increased the very major error rate to 24.6% and the major error rate to 10.3%. More investigation into the observed discrepancies in MICs and the impact on clinical decision making is needed to determine strategies for improving the ability of Vitek 2 to predict susceptibility to cefepime in ESBL-producing Enterobacteriaceae. Certainly, failing to identify isolates that are resistant to cefepime may lead to unintended and preventable treatment failures (i.e., increased mortality, prolonged length of stay, and reinfection). We suggest that, until additional studies are conducted or improved methodologies are developed to explain or reduce the observed error rates, clinicians exercise extreme caution in interpreting susceptible cefepime MICs from Vitek 2 as truly susceptible for ESBL-producing E. coli.