Accuracies of β-Lactam Susceptibility Test Results for Pseudomonas aeruginosa with Four Automated Systems (BD Phoenix, MicroScan WalkAway, Vitek, and Vitek 2)

Abstract

Contemporary clinical isolates and challenge strains of Pseudomonas aeruginosa were tested by four automated susceptibility testing systems (BD Phoenix, MicroScan WalkAway, Vitek, and Vitek 2; two laboratories with each) against six broad-spectrum β-lactams, and the results were compared to reference broth microdilution (BMD) and to consensus results from three validated methods (BMD, Etest [AB Biodisk, Solna, Sweden], and disk diffusion). Unacceptable levels of error (minor, major, and very major) were detected, some with systematic biases toward false susceptibility (piperacillin-tazobactam and imipenem) and others toward false resistance (aztreonam, cefepime, and ceftazidime). We encourage corrective action by the system manufacturers to address test biases, and we suggest that clinical laboratories using automated systems should consider accurate alternative methods for routine use.

Concern about the accuracy of commercial automated systems when testing Pseudomonas aeruginosa has been long-standing and is featured in two contemporary presentations (14, 16). Highly elevated rates of error have been reported for β-lactam agents tested by the MicroScan WalkAway, Vitek, and Vitek 2 instruments (1, 2, 4, 5, 9, 13, 16, 17) compared to results with reference methods (6, 7); errors ranged from false resistance (major error) to false susceptibility (very major error) results. The most recent comprehensive study (16) also described unacceptable levels (15) of minor interpretive errors for aztreonam (28 to 31%) and cefepime (18 to 32%) when testing the three commonly used commercial automated systems (MicroScan WalkAway, Vitek, and Vitek 2). The most serious very major errors were detected for piperacillin-tazobactam (19 to 27%) (16), confirmed by results (10.0% very major errors) reported by Jorgensen et al. (14), in which a minor error rate of 23.6% for cefepime when testing 55 P. aeruginosa isolates was also noted.

Studies of the BD Phoenix system (BD Diagnostic Systems [BDDS], Sparks, MD) have included relatively few P. aeruginosa strains (only 63), and most evaluations used interpretive criteria other than those of the Clinical and Laboratory Standards Institute (CLSI; formerly the National Committee for Clinical Laboratory Standards [NCCLS]) (8, 10-12). Thus, our knowledge of the ability of the BD Phoenix to accurately predict resistance among P. aeruginosa isolates remains incomplete. For these reasons and concerns, a multicenter investigation was organized to evaluate the accuracies of four automated susceptibility testing methods for testing P. aeruginosa. The study was performed in seven laboratories as follows: for BD Phoenix, Arkansas Children's Hospital and St. Luke's Regional Laboratories; for MicroScan WalkAway, Wayne State University Detroit Medical Center and Medical University of South Carolina; for Vitek, Emory University-Centers for Disease Control and Loyola University Medical Center; and for Vitek 2, St. Vincent's Hospital-Manhattan and Emory University-Centers for Disease Control.

These participating laboratories processed 30 strains of P. aeruginosa, representing local contemporary clinical isolates (15 strains) and a selected challenge set (15 strains) that included approximately equal distributions of isolates that were susceptible and resistant to the antipseudomonal β-lactams. The antimicrobial susceptibilities of this challenge set were not revealed to the participating laboratories, and testing was performed in a blinded fashion. The antimicrobial agents tested included aztreonam, cefepime, ceftazidime, imipenem, piperacillin, and piperacillin-tazobactam. The organisms were tested at the laboratories using their routine automated system according to procedures and reporting protocols recommended by the manufacturers (bioMerieux, Durham, NC; Dade MicroScan Inc., West Sacramento, CA; and BD Diagnostic Systems, Sparks, MD). The specific product cards or panels/software programs utilized were NMIC-107 and 112/V5.15A for BD Phoenix, NEGMIC30 or NC32/LabPro 2.01 for MicroScan WalkAway, GNS-122/WSVTK-R10.01 for Vitek, and AST-GN09/WSVT2-R04.02 for Vitek 2. The comparison methods, also tested at each participating location, used reference frozen-form panels produced under good manufacturing practices by TREK Diagnostics (Cleveland, OH), Etest (AB Biodisk, Solna, Sweden), and the disk diffusion method (BDDS, Sparks, MD) using the CLSI reference method or the method recommended by the manufacturer (6-8, 15). The agar diffusion method (disk diffusion and Etest) results have previously been validated by Burns et al. for testing P. aeruginosa (3). Quality control was assured via concurrent testing of CLSI-recommended strains, and all presented results were associated with acceptable quality-control test results (8).

Data were analyzed by comparing the results from each automated system to those produced by the reference broth microdilution test (6, 8, 15) as well as to the consensus categorical results of the reference broth microdilution and agar diffusion (Etest and disk diffusion) methods (3, 6-8, 15, 18). The origins of the errors (laboratory or organism subset) were also assessed, and acceptable performance was defined by intermethod error criteria found in CLSI M23-A2 (15). A significant bias toward susceptibility or resistance was defined as a shift of ≥10% in the perceived rate of susceptibility of the entire population (60 results per method or system) when using the commercial product compared to results from the categorical consensus (16).

Table 1 lists the results (error rates) after comparing the automated system categorical test results to the results of the CLSI reference test (6) and the consensus of three validated methods (3, 6-8). Unacceptable levels of intermethod error (15) were encountered using both applied comparative analyses (Table 1). In testing the BD Phoenix system, one participant did not test piperacillin (panel NMIC-112), so only 30 of 60 test results were recorded for that agent tested alone, but 60 results were available for analysis for piperacillin-tazobactam, a carbapenem, two cephalosporins, and the one monobactam tested. Unacceptably (15) elevated rates of minor errors (16.7 to 36.7%) were identified with aztreonam, cefepime, and ceftazidime, regardless of the reference result utilized for analysis. These results were consistent between the laboratories and the organism subsets (recent clinical or challenge strains; data not shown). Previous publications evaluating automated systems have also reported very high minor error rates for the β-lactams and low overall “categorical agreement” (75.8 to 84.8%) when these systems were tested against P. aeruginosa (10, 11). For the results of aztreonam (not approved for testing in this instrument by the U.S. Food and Drug Administration) and cefepime testing, the minor error level was combined with a systematic trend toward false resistance (Table 2). In contrast, imipenem and piperacillin with or without tazobactam had error rates that were generally acceptable or were only marginally elevated (>3% major errors [false resistance]) (Table 1).

TABLE 1.

Types of intermethod errors produced when testing 30 P. aeruginosa isolates by four commercial automated systems in seven laboratories^a

System and antimicrobial agent (no. of strains tested)	Percentage of indicated type of error
	Compared to BMD resultb			Compared to consensus resultc
	Very major	Major	Minor	Very major	Major	Minor
BD Phoenix
Aztreonam (60)g	0.0	1.7	33.3d	0.0	1.7	36.7d
Cefepime (60)	0.0	1.7	18.3d	0.0	1.7	18.3d
Ceftazidime (60)	1.7d	0.0	18.3d	1.7d	0.0	16.7d
Imipenem (60)	0.0	0.0	3.3	0.0	0.0	1.7
Piperacillin (30)e	0.0	6.7d	NAf	0.0	3.3d	NAf
Piperacillin-tazobactam (60)	1.7d	6.7d	NAf	1.7d	5.0d	NAf
MicroScan WalkAway
Aztreonam (60)	0.0	3.3d	21.7d	0.0	3.3d	23.3d
Cefepime (60)	0.0	3.3d	48.3d	0.0	3.3d	45.0d
Ceftazidime (60)	1.7d	6.7d	23.3d	0.0	6.7d	20.0d
Imipenem (60)	0.0	1.7	11.7d	1.7d	1.7	10.0
Piperacillin (60)	10.0d	3.3d	NAf	15.0d	3.3d	NAf
Piperacillin-tazobactam (60)	5.0d	1.7	NAf	10.0d	0.0	NAf
Vitek
Aztreonam (60)	0.0	3.3d	18.3d	0.0	5.0d	31.7d
Cefepime (60)	1.7d	0.0	36.7d	1.7d	0.0	36.7d
Ceftazidime (60)	1.7d	0.0	20.0d	1.7d	3.3d	16.7d
Imipenem (60)	8.3d	0.0	13.3d	6.7d	0.0	10.0
Piperacillin (60)	0.0	8.3d	NAf	0.0	6.7d	NAf
Piperacillin-tazobactam (60)	15.0d	5.0d	NAf	15.0d	5.0d	NAf
Vitek 2
Aztreonam (60)	1.7d	0.0	28.3d	0.0	0.0	33.3d
Cefepime (60)	0.0	0.0	13.3d	1.7d	0.0	16.7d
Ceftazidime (60)	3.3d	0.0	23.3d	1.7d	0.0	21.7d
Imipenem (60)	6.7d	0.0	25.0d	5.0d	0.0	26.7d
Piperacillin (60)	5.0d	0.0	NAf	6.7d	0.0	NAf
Piperacillin-tazobactam (60)	21.7d	1.7	NAf	20.0d	0.0	NAf

^aBD Phoenix system results from Arkansas Children's Hospital, Little Rock, AR, and St. Luke's Regional Laboratories, Kansas City, MO; MicroScan WalkAway results from Medical University of South Carolina, Charleston, SC, and Wayne State University Detroit Medical Center, Detroit, MI; Vitek 2 results from St. Vincent Hospital-Manhattan, New York, NY, and Emory University-CDC, Atlanta, GA; and Vitek data from Loyola University Medical Center, Maywood, IL, and Emory University-CDC, Atlanta, GA.

^bBMD, broth microdilution reference method results from CLSI M7-A7 (6).

^cConsensus of broth microdilution, disk diffusion (7) and E-test (AB Biodisk, Solna, Sweden) categorical results.

^dUnacceptable levels of error (15).

^eOne laboratory only.

^fNA, not applicable because of no CLSI (8) intermediate category criteria.

^gTesting of aztreonam against P. aeruginosa in this instrument has not been approved by the U.S. Food and Drug Administration.

TABLE 2.

Analyses of error trends for systematic bias in four automated systems when testing six broad-spectrum β-lactams against contemporary P. aeruginosa isolates (60 total test results for each agent/system)

System and antimicrobial agent (no. of errors)	No. of automated system error results (categorical trend)		Net trend (%)
	More susceptible	More resistant
BD Phoenix
Aztreonam (23)	0	23	38.3a
Cefepime (12)	1	11	16.7a
Ceftazidime (11)	7	4	5.0
Imipenem (1)	0	1	1.7
Piperacillin (1)	0	1	3.3b
Piperacillin-tazobactam (4)	1	3	3.3
MicroScan WalkAway
Aztreonam (16)	5	11	10.0a
Cefepime (29)	0	29	48.3a
Ceftazidime (16)	3	13	16.7a
Imipenem (8)	5	3	3.3
Piperacillin (11)	9	2	11.7a
Piperacillin-tazobactam (6)	6	0	10.0a
Vitek
Aztreonam (22)	2	20	30.0a
Cefepime (23)	2	21	31.7a
Ceftazidime (13)	2	11	15.0a
Imipenem (10)	8	2	10.0a
Piperacillin (4)	0	4	6.7
Piperacillin-tazobactam (12)	9	3	10.0a
Vitek 2
Aztreonam (20)	3	17	23.3a
Cefepime (11)	8	3	8.3
Ceftazidime (14)	9	5	6.7
Imipenem (19)	18	1	28.3a
Piperacillin (4)	4	0	6.7
Piperacillin-tazobactam (12)	12	0	20.0a

^aResults shown in bold have significant testing bias as defined by a ≥10% net trend (≥six occurrences) toward susceptibility or resistance when compared to consensus results (broth microdilution, disk diffusion, and Etest categories) (6-8, 15).

^bBased upon 30 test results.

For the MicroScan WalkAway system, the errors also occurred equally between the two participant centers and between the tested organism populations (challenge and clinical strains). Markedly elevated rates of minor errors were observed for aztreonam (21.7 to 23.3%), cefepime (45.0 to 48.3%), and ceftazidime (20.0 to 23.3%), regardless of the reference result selected for comparative analysis (Table 1). More-serious errors of false resistance were also noted for the same agents with the MicroScan WalkAway system, and false-susceptible results were detected at rates of 5.0 to 15.0% for piperacillin with or without tazobactam. Among the antimicrobials showing unacceptable levels of minor or serious interpretive errors, a net systemic trend toward false-resistance MicroScan WalkAway results was detected for aztreonam (10.0%), cefepime (48.3%), and ceftazidime (16.7%) (Table 2). The skew toward resistance is particularly pronounced for cefepime (threefold greater than for ceftazidime), and this example of bias is illustrated in Table 3. In contrast, the piperacillin and piperacillin-tazobactam results for the MicroScan WalkAway trended toward false susceptibility at a net level of 10.0 to 11.7% (Table 2).

TABLE 3.

Example of a systematic trend toward resistance as observed for the MicroScan WalkAway when testing P. aeruginosa strains against cefepime (60 results from two medical centers^a)

Consensus category (MIC; μg/ml)	No. of strains in MicroScan WalkAway category (MIC; μg/ml)
	Susceptible (≤8)	Intermediate (16)	Resistant (≥32)
Susceptible (≤8)	19	14	2
Intermediate (16)	0	3	13
Resistant (≥32)	0	0	9

^aResults were recorded at Wayne State University Detroit Medical Center, Detroit, MI, and Medical University of South Carolina, Charleston, SC.

The results for both Vitek systems are summarized in Tables 1 and 2. High minor error levels (ideal limit of ≤10%) were noted for both automated systems when testing aztreonam (18.3 to 33.3%), cefepime (13.3 to 36.7%), ceftazidime (16.7 to 23.3%), and imipenem (13.3 to 26.7%). More-serious very major errors (false-susceptible) were detected at rates severalfold greater than the acceptable limit (15) (≤1.5%) for piperacillin-tazobactam when using the Vitek (15.0%) and Vitek 2 (20.0 to 21.7%) systems, while lesser rates of serious errors (false susceptibility or resistance) were encountered with Vitek and Vitek 2 when testing piperacillin alone (5.0 to 8.3%). High very-major-error rates were also encountered for both Vitek systems when testing imipenem (5.0 to 8.3%). The error rates for each Vitek system were comparably distributed between tested organism populations and testing centers (data not shown). Elevated minor error rates can be considered acceptable for organism populations having MICs distributed near the breakpoint concentrations. However, the analysis of the minor errors in such cases should reflect equal variations of 1 log₂ dilution step toward susceptibility and resistance, e.g., without systematic unidirectional bias. Table 2 shows the trends toward greater susceptibility or resistance for Vitek or Vitek 2 categorical interpretations among the strains having intermethod errors. Clear trending toward false resistance was observed for Vitek (aztreonam, cefepime, and ceftazidime) and Vitek 2 (aztreonam); this was most marked for nearly one-third of the aztreonam and cefepime results. In contrast, a false susceptibility bias for ≥10% of the tested strains was noted for imipenem (8 to 18 occurrences) and piperacillin-tazobactam (9 to 12 occurrences) with both systems.

Concerns about the accuracy of antimicrobial susceptibility testing for P. aeruginosa have been long-standing and have recently been highlighted by intermethod comparisons of commercial automated systems (14, 16, 17) and the piperacillin-tazobactam false-susceptibility results reported to external quality assurance programs such as those of the College of American Pathologists (9). Automated systems have not performed at acceptable levels of accuracy with some antimicrobial agents (1, 2, 4, 5, 9, 13, 14), while manually read reference tests (broth microdilution or agar dilution) and agar diffusion methods have functioned as reliable tests producing comparable categorical results (3, 6-8, 18). The data from a multicenter experiment presented here illustrate the high level of discord between the challenged automated systems (BD Phoenix, MicroScan WalkAway, Vitek, and Vitek 2) and the recommended/validated susceptibility methods (3, 6, 18). These systematic errors of automated system origin result in documented false-susceptibility or -resistance trends among the β-lactam antipseudomonal agents, thereby leading to choices of inappropriate therapeutic agents for individual patients and also, potentially, to misdirected empirical treatment guidelines or formulary decisions by medical centers.

Clinical laboratories should be aware of these interpretive problems with the automated systems in testing P. aeruginosa and seek alternative, validated methods for routine use (3, 4). Agar diffusion methods (disk diffusion and Etest) (3, 7) are accurate when compared to the results generated by the CLSI (6) reference methods with MIC endpoints read manually (18). An appropriate choice of method for P. aeruginosa testing may differ from methods used for other commonly tested pathogen groups (Enterobacteriaceae, staphylococci, enterococci, etc). Furthermore, these intermethod discords for the automated systems may be more widespread and not limited to the results for P. aeruginosa, as noted by Tenover et al. (19) in testing the rapidly emerging, epidemic, KPC-enzyme-producing Klebsiella pneumoniae.