Validation of Available Extended-Spectrum-Beta-Lactamase Clinical Scoring Models in Predicting Drug Resistance in Patients with Enteric Gram-Negative Bacteremia Treated at South Texas Veterans Health Care System

ABSTRACT

Extended-spectrum-beta-lactamase (ESBL)-producing Enterobacteriaceae are increasingly common; however, predicting which patients are likely to be infected with an ESBL pathogen is challenging, leading to increased use of carbapenems. To date, five prediction models have been developed to distinguish between patients infected with ESBL pathogens. The aim of this study was to validate and compare each of these models to better inform antimicrobial stewardship. This was a retrospective cohort study of patients with Gram-negative bacteremia treated at the South Texas Veterans Health Care System over 3 months from 2018 to 2019. We evaluated isolate, clinical syndrome, and score variables for the five published prediction models/scores: Italian “Tumbarello,” Duke, University of South Carolina (USC), Hopkins clinical decision tree, and modified Hopkins. Each model was assessed using the area under the receiver operating characteristic curve (AUROC) and Pearson correlation. One hundred forty-five patients were included for analysis, of which 20 (13.8%) were infected with an ESBL Escherichia coli or Klebsiella spp. The most common sources of infection were genitourinary (55.8%) and gastrointestinal/intraabdominal (24.1%), and the most common pathogen was E. coli (75.2%). The prediction model with the strongest discriminatory ability (AUROC) was Tumbarello (0.7556). The correlation between prediction model score and percent ESBL was strongest with the modified Hopkins model (R² = 0.74). In this veteran population, the modified Hopkins and Duke prediction models were most accurate in discriminating between Gram-negative bacteremia patients when considering both AUROC and correlation. However, given the moderate discriminatory ability, many patients with ESBL Enterobacteriaceae (at least 25%) may still be missed empirically.

INTRODUCTION

According to the Centers for Disease Control and Prevention (CDC), extended-spectrum-beta-lactamase (ESBL)-producing Gram-negative bacilli are increasingly common in the community and hospital settings, affecting as many as 197,400 patients annually in the United States alone (1). ESBL-producing Enterobacteriaceae have been classified by the CDC as “serious threats” to health care systems due their clinical and economic impact, incidence, transmissibility, and limited treatment options (2). Specifically, ESBL bacteremia is associated with increased mortality and delayed appropriate antimicrobial therapy (3). The empirical use of piperacillin-tazobactam has been associated with increased mortality in patients with ESBL Gram-negative bacteremia compared to that in patients with empirical use of a carbapenem (4); thus, carbapenems are considered the optimal antibiotic therapy in these patients. Recent studies, however, have shown that the overuse of carbapenems as a result of increasing ESBL prevalence has led to the development of multidrug-resistant bacteria, further limiting therapeutic options (5).

Due to the significant risks associated with both suboptimal therapy and overuse of carbapenems, it is critical to identify risk factors for ESBL bacteremia and start appropriate therapy as soon as possible. Several tools have been designed and validated to direct the empirical use of carbapenems in an attempt to improve patient outcomes and optimize antimicrobial use (6–10). This study retrospectively compared several scoring systems to identify the most accurate model for predicting ESBL infection in a veteran population.

RESULTS

A total of 145 hospitalized patients with Gram-negative bacteremia were included for analysis, of which 20 (13.8%) pathogens were ESBL positive. The most common sources of infection were genitourinary (55.8%) and gastrointestinal and intraabdominal infection (24.1%) (Table 1). The most common organisms were Escherichia coli (n = 109; 75.2%), Klebsiella pneumoniae (n = 33; 22.8%), Klebsiella oxytoca (n = 2; 1.4%), and Klebsiella aerogenes (0.1%) There were no significant differences in ESBL status by age, source of infection, pathogen, or Charlson comorbidity index score.

TABLE 1.

Characteristics of patients included and sources

Characteristic	No. (%) of patients		P value
	ESBL positive (N = 20)	ESBL negative (N = 125)
Age ≥70 yrs	9 (45.0)	61 (48.8)	0.7520
Source of infection	0.1887
Urinary		12 (60.0)	69 (55.2)
Gastrointestinal/intraabdominal		2 (10.0)	33 (26.4)
Pulmonary		1 (5.0)	7 (5.6)
Skin		3 (15.0)	4 (3.2)
Multiple sources		0 (0.0)	4 (3.2)
Unknown		2 (10.0)	8 (6.4)
Organism	0.6012
E. coli		17 (85.0)	92 (73.6)
K. pneumoniae		3 (15.0)	30 (24.0)
K. oxytoca		0 (0.0)	2 (1.6)
K. aerogenes		0 (0.0)	1 (0.8)
Charlson comorbidity index score of ≥4	15 (75.0)	88 (70.4)	0.6699

The prediction model with the strongest discriminatory ability (area under the receiver operating characteristic curve [AUROC]) was Tumbarello (0.7556), followed by Duke (0.7532), modified Hopkins (0.7402), University of South Carolina (USC) (0.7138), and Hopkins (0.5932) models (Table 2). While significant discriminatory ability was found for all prediction models, the AUROCs indicated fair to moderate discrimination. The correlation between prediction model score and percentage of patients with isolates that were ESBL positive was strongest with the modified Hopkins model (R² = 0.74), followed by Duke (R² = 0.65), USC (R² = 0.34), and Tumbarello (R² = 0.12) models.

TABLE 2.

Discrimination of ESBL-producing pathogens for each prediction score

Score	AUROC (95% CI)a
USC	0.7138 (0.6456–0.7820)
Hopkins	0.5932 (0.5221–0.6643)
Modified Hopkins	0.7402 (0.6736–0.8068)
Duke	0.7532 (0.6875–0.8189)
Tumbarello	0.7556 (0.6901–0.8211)

^aCI, confidence interval.

DISCUSSION

Several prediction models have been developed and validated at their respective institutions in predicting the risk of ESBL among patients with Enterobacteriaceae bloodstream infections (6–10). However, there are limited data regarding the generalizability of these findings at other institutions, and modifications of the model may be required to improve performance and simplify their use (7). In our institution, the modified Hopkins and Duke prediction models were able to best predict ESBL-producing pathogens. However, both demonstrated moderate discriminatory ability, which in clinical practice would still lead to a significant number of missed ESBLs. This may be due to different epidemiological factors such as migration and travel patterns that may be considered crucial for some models (i.e., Hopkins model) but may not be applicable to all regions (9, 10). Therefore, it may be useful for institutions to develop their own validation scores and tools in order to predict outcomes for their specific populations, especially at hospitals such as the VA, that serve a niche population.

The potential difficulties with applicability of ESBL risk scoring tools across heterogenous institutions and patient populations highlight the need for rapid diagnostic testing to identify ESBL-producing pathogens. A recent study by Cwengros and colleagues demonstrated that rapid identification of CTX-M using the Verigene system (Technopath) outperformed three ESBL scoring tools in identifying ceftriaxone-resistant Enterobacteriaceae bloodstream infection (BSI) (11). Our findings further suggest that external application of these scoring tools has limited efficacy.

Additional challenges to the development of predictive scoring systems include the changes in the epidemiology and heterogenicity of ESBL enzymes over time and the dissemination of different ESBL-producing strains, with a few that may be more fit than others to spread through communities (8). Furthermore, even though there is growing evidence that outcomes of treatment of ESBL-associated bacteremia may improve with the use of carbapenems, there is paucity of data regarding the impact of scoring systems on clinical outcomes (i.e., even if the scoring systems have enough discrimination, we need further evidence that their use impacts outcomes).

Limitations in our study include that data were collected retrospectively from a single center with a limited sample size and that the study utilized manual chart review for risk score data collection and focused on a veteran population. However, it highlights the need to validate any such scoring tool in a specific population prior to implementation.

Conclusion.

In this veteran population, the modified Hopkins and Duke prediction models were most accurate in discriminating between Gram-negative bacteremia with and without an ESBL-producing pathogen when considering both AUROC and correlation. However, given the moderate discriminatory ability, many patients with ESBL-producing Enterobacteriaceae (at least 25%) may still be missed empirically when using these scores. Differences in the accuracy of these models in our patients was likely due to our specific patient population and variable epidemiological factors used by each model as risk factors. Further studies are needed to improve these models for application to antimicrobial stewardship and future plans may include further adapting current risk scoring systems or developing a model specific for the veteran population.

MATERIALS AND METHODS

This was a retrospective cohort study of consecutive patients admitted to the South Texas Veterans Health Care System, San Antonio, TX. Patients were included if they were hospitalized in a 3-month period with one or two positive blood cultures for E. coli, Proteus spp., or Klebsiella spp. between 2018 and 2019. If the patient had more than one isolate, only the first was included (index isolate). Patients were categorized into ESBL-positive or ESBL-negative groups based on a VITEKVR 2 ESBL screen (bioMérieux, Inc., Durham, NC, USA) and manual confirmatory disks.

We reviewed the five available published scoring models or algorithms that have been proposed to predict the likelihood of ESBL bacteremia in patients admitted to the hospital setting: the model by Tumbarello et al. (6), developed in Italy, which was later validated and adapted at Duke University (7), the model developed by Augustine et al. at the University of South Carolina (USC) (8), the Hopkins clinical decision tree (9), and the modified Hopkins score (10). These scoring tools looked at patient demographics, comorbidities (some including the Charlson comorbidity score [12]), prior health care exposure within the past 12 months, β-lactam or macrolide use within 6 months to 30 days, gastrointestinal (GI)/genitourinary (GU) procedures, history of colonization with a multidrug-resistant organism (MDRO) or ESBL-producing microorganisms, etc. (Table 3). All variables were collected for each patient through manual data extraction by two team members. To support increased accuracy, a uniform abstraction guideline document was developed, and a random sample of 10% of the charts was reviewed by an infectious disease trained physician and a pharmacist to validate the data set.

TABLE 3.

Clinical scoring systems

System	Variables measured
USC model	No. of outpatient GI/GU procedures within 1 mo before infection within 30 days
	No. of prior infections or colonization with ESBL-producing pathogen within 12 mo
	No. of prior courses of β-lactams and/or fluoroquinolones used within 3 mo
Hopkins	Age, ≥43 yrs
	Recent hospitalization in an ESBL high-burden region
	History of ESBL colonization/infection in the prior 6 mo
	Chronic indwelling vascular hardware
	≥6 days of antibiotic exposure in the prior 6 mo
Modified Hopkins score	Demographic data
	Preexisting medical conditions: COPDa, emphysema, ventilator dependency
	Healthcare exposure: length of stay in healthcare or post-acute care facility in prior 6 mo; hospitalization in another country in the prior 6 mo
	No. of wks of antibiotic therapy with Gram-negative activity in prior 6 mo
	Presumptive source of bacteremia (e.g., catheter or pneumonia)
	Indwelling hardware (orthopedic hardware, central vascular catheter, Foley catheter, or GI feeding tube)
	MDRb organism colonization or infection (MDR Pseudomonas aeruginosa, MDR Acinetobacter baumannii, ESBL-producing Enterobacteriaceae, carbapenem-resistant Enterobacteriaceae, vancomycin-resistant Enterococcus species, and methicillin-resistant Staphylococcus aureus) in the prior 6 mo
Tumbarello model	Age, >70 yrs
	Healthcare exposure: recent hospitalization, ≤12 mo; transfer from another health care facility
	β-Lactam and/or fluoroquinolone treatment ≤3 mo before admission
	History of urinary catherization, ≤30 days
	Charlson comorbidity index score, >4
Duke Tumbarello model	Healthcare exposure: previous hospitalization, ≤12 mo; transfer from another healthcare facility/hospital, ≤30 days
	Antibiotic therapy with β-lactam and/or fluoroquinolone therapy of ≤3 mo
	History of urinary catherization, ≤ 30 days
	Immunosuppression ≤3 mo prior to hospitalization

^aCOPD, chronic obstructive pulmonary disease.

^bMDR, multidrug resistant.

This project was submitted as a quality improvement project to the IRB at UT Health San Antonio and was considered to be nonregulated research.

Statistical methods.

Data and statistical analyses were conducted using JMP Pro 14 (SAS Corp., Cary, NC). First, clinical characteristics of our cohort were compared between ESBL-positive and ESBL-negative groups using the chi-square test. Next, we assigned patients in our own cohort ESBL risk scores based on the previously described scoring models. Then, the ability of each risk score (continuous independent variable) to differentiate between patients with and without an ESBL-producing pathogen (nominal dependent variable) was assessed using logistic regression to generate the area under the receiver operating characteristic curve (AUROC). Finally, Pearson correlation was used to assess the correlation between each risk score and the percentage of patients with an ESBL bacteremia. In addition to discriminatory ability of each model, correlation data help inform whether there is a relationship between increasing risk score and increasing percentage of patients with an ESBL infection as a means of further model validation.