The Rapid Prediction of Carbapenem Resistance in Patients With Klebsiella pneumoniae Bacteremia Using Electronic Medical Record Data

Abstract

Background

The administration of active antibiotics is often delayed in cases of carbapenem-resistant gram-negative bacteremia. Using electronic medical record (EMR) data to rapidly predict carbapenem resistance in patients with Klebsiella pneumoniae bacteremia could help reduce the time to active therapy.

Methods

All cases of Klebsiella pneumoniae bacteremia at Mount Sinai Hospital from September 2012 through September 2016 were included. Cases were randomly divided into a “training set” and a “testing set.” EMR data from the training set cases were reviewed, and significant risk factors for carbapenem resistance were entered into a multiple logistic regression model. Performance was assessed by repeated K-fold cross-validation and by applying the training set model to the testing set. All cases were also reviewed to determine the time to effective antibiotic therapy.

Results

A total of 613 cases of Klebsiella pneumoniae bacteremia were included, 61 (10%) of which were carbapenem-resistant. The training and testing sets consisted of 460 and 153 cases, respectively. The regression model derived from the training set correctly predicted 73% of carbapenem-resistant cases and 59% of carbapenem-susceptible cases in the testing set (sensitivity, 73%; specificity, 59%; positive predictive value, 16%; negative predictive value, 95%). The mean area under the receiver operator characteristic curve of the K-fold cross-validation repeats was 0.731. Patients with carbapenem-resistant infections received active antibiotics significantly later than those with susceptible infections (40.4 hours vs 9.6 hours, P < .0001).

Conclusions

A multiple logistic regression model using EMR data can generate rapid, sensitive predictions of carbapenem resistance in patients with Klebsiella pneumoniae bacteremia, which could help shorten the time to effective therapy in these cases.

The early initiation of effective antibiotic therapy is a critical step in the successful treatment of gram-negative bacteremia. The use of inadequate initial antibiotics has been associated with an increased risk of death [1, 2]; therefore, patients with gram-negative bacteremia typically receive broad-spectrum antibiotics until susceptibilities are available. However, empiric antibiotic regimens may still be ineffective against carbapenem-resistant Enterobacteriaceae (CRE), and as a result, active treatment for CRE bacteremia is often delayed [3–5]. This delay may contribute to the high mortality associated with CRE infections [6].

Although treating CRE bacteremia with inappropriate antibiotics is associated with poor outcomes, the overuse of antibiotics active against CRE may also be harmful. Given the relatively low prevalence of CRE infections [7], routinely administering CRE therapy to all patients with gram-negative bacteremia may cause unnecessary toxicity and promote widespread resistance to critical last-line antibiotics [8, 9].

Identifying the bacterial species growing in a blood culture can help guide early antibiotic choices, and rapid species identification is now possible in many laboratories [10]. However, antibiotic susceptibility results may not be available for several days. Therefore, a method is needed to predict carbapenem resistance in patients with gram-negative bacteremia after the species is identified but before more definitive susceptibility results are available. Such a prediction tool would be especially useful when treating Klebsiella pneumoniae bacteremia, as this is the species of Enterobacteriaceae most likely to be carbapenem-resistant [3, 7].

We have developed a multiple logistic regression model using electronic medical record (EMR) data to generate immediate predictions of carbapenem resistance in patients with Klebsiella pneumoniae bacteremia.

METHODS

All cases of Klebsiella pneumoniae bacteremia at Mount Sinai Hospital from September 2012 through September 2016 were reviewed. This study was approved by the Institutional Review Board at the Icahn School of Medicine at Mount Sinai.

EMR data for each case were obtained from an institutional data warehouse and compiled into a database using Structured Query Language (SQL). Demographic data, antibiotic administration records, microbiology results, hospital location, and length of stay information were all included. Information from admissions to other institutions was not available, and only data from our single center were included.

All unique cases of Klebsiella pneumoniae bacteremia during the study period were included. Unique cases were defined as patients with 1 or more blood cultures positive for Klebsiella pneumoniae during a single 2-week period. Patients with more than 1 blood culture positive for Klebsiella pneumoniae within a 2-week period were considered to be a single case, and data were collected from the first date that a positive blood culture was collected. Patients with recurrent bacteremia, defined as more than 1 blood culture positive for Klebsiella pneumoniae with more than 2 weeks of negative blood cultures between infections, were included more than once. Each recurrence was considered a new case, and data were collected from the first day of each recurrence.

Carbapenem resistance was defined as an imipenem minimum inhibitory concentration of ≥2 μg/mL. Susceptibility testing for other carbapenems was inconsistently performed during the study period; therefore, resistance to other carbapenems was not considered. Blood cultures were initially collected and stored in the BACTEC system (Franklin Lakes, NJ, USA). Bottles with bacterial growth were plated onto traditional culture media, and automated bacterial species identification and antimicrobial susceptibility testing were performed using the VITEK 2 system (Marcy l’Etoile, France).

The time to active antibiotics was defined as the time between culture collection and administration of the first dose of an antibiotic to which the isolate was susceptible. This time was defined as 0 for patients already receiving an active empiric antibiotic when the positive culture was collected. Automated susceptibility testing for noncarbapenem antibiotics was also performed using the VITEK 2 system. Susceptibility to these antibiotics was determined based on the most recent breakpoints recommended by the Clinical Laboratory Standards Institute at the time of culture collection.

Variable Selection and Model Development

The model development and validation workflow are summarized in Figure 1. Cases were randomly divided into a “training set” and a “testing set,” with the same proportion of imipenem-resistant cases in each group. The training set consisted of 75% of all cases and was used to develop the logistic regression algorithm. The testing set consisted of the remaining 25% of cases and was used for validation of the model. Baseline demographic variables were compared between the training and testing sets to evaluate for any introduced bias, and no significant differences were found.

Figure 1.

Workflow of the logistic regression model development and validation.

Possible risk factors for carbapenem resistance were queried from the training set and analyzed via univariate methods, such as the Fisher exact test to compare categorical variables and the t test to compare means. Potential independent variables were considered for inclusion in the regression model only if they (1) were biologically plausible and (2) existed within the EMR as objective data fields. Free text fields and data requiring a subjective opinion or manual input were not considered.

The hospital unit where each patient was located at the time of culture collection was classified as either high risk for carbapenem resistance, low risk, or the emergency department (ED). High-risk units were defined as intensive care units (ICUs) and acute medical and surgical floors. Acute floors and ICUs were grouped together because CRE rates in these units were similar. Low-risk units were all other hospital units excluding the ED.

Antibiotic exposures were quantified as “antibiotic-days.” One antibiotic-day was defined as each day that a distinct antibiotic was given. Patients receiving more than 1 distinct antibiotic on a given day could accrue more than 1 antibiotic-day per calendar day.

Colonization with imipenem-resistant Klebsiella pneumoniae was defined as growth of this organism in any prior culture, including blood, urine, respiratory, and wound cultures. Rectal swabs for multidrug-resistant gram-negative rods were collected sporadically during the study period, and these were also considered; however, all patients with positive surveillance cultures also had at least 1 other prior clinical specimen with growth of imipenem-resistant Klebsiella pneumoniae.

Factors that were significantly associated with imipenem resistance on univariate analysis (P < .2) were entered into a multiple logistic regression model in a forward stepwise approach using SPSS and R. The accuracy of potential models was assessed by comparing the area under the receiver operator characteristic curve (AUROC) of each model. Variables were retained in the model if they resulted in a higher AUROC even if they were not associated with a P value of <.05. The final ROC curve for the model was also used to select a threshold for positivity of the regression output that emphasized sensitivity of the algorithm over specificity.

The logistic regression model created from the training set was applied to the testing set using the threshold for positivity defined above. A confusion matrix was generated from the classification of the testing set data, and performance characteristics were calculated.

Repeated K-fold Cross-Validation

Repeated K-fold cross-validation was performed on the training set using the same variables included in the model. The training set cases were divided into 10 equal folds, with 9 folds used to generate a logistic regression algorithm and the 10th fold used to test the predictive accuracy of the algorithm. This process was repeated for each of the 10 folds, and then the entire 10-fold cross-validation was repeated 500 times. An AUROC for each of the 5000 repeats was reported.

To compare the results of the cross-validation with random chance, a “random model” was created that consisted of random permutations of the dependent variable used in the logistic regression equation. This process was repeated 5000 times, and an AUROC was reported for each of these repeats. The mean AUROCs of the cross-validation model and random model were then compared.

RESULTS

A total of 613 cases of Klebsiella pneumoniae bacteremia were identified in 540 unique patients. The rate of imipenem resistance was 10% (61 cases in 53 unique patients). The training and testing sets consisted of 460 and 153 cases, respectively. Baseline demographic information and potential risk factors for imipenem resistance from the 460 training set cases are presented in Table 1.

Table 1.

Baseline Patient Characteristics of the Training Set

	Imipenem- Resistant (n = 46)	Imipenem- Susceptible (n = 414)	P Value
Mean age, y	63	57	.0652
Age >60 y, No. (%)	32 (70)	211 (51)	.0192
Male, No. (%)	29 (63)	227 (55)	.3485
Hospital unit, No. (%)			<.0001
High risk	32 (70)	155 (37)
Low risk	2 (4)	90 (22)
ED	12 (26)	169 (41)
Mean LOS, d	18.6	19.0	.9475
Inpt days in past 5 y	153	106	.0178
Mean ICU days during current admission	23.4	15.4	.1795
Any ICU days during current admission, No. (%)	27 (59)	186 (45)	.0869
Colonization with imi-R Klebsiella, No. (%)	10 (22)	7 (1)	<.0001
Prior CDI, No. (%)	6 (13)	58 (14)	1.0000
Mean ABX-days in the previous:
2 y	75.9	39.6	.0004
1 y	51.4	33.6	.0330
30 d	14.8	8.4	.0011
Mean carbapenem-days in the previous:
2 y	8.6	3.7	.0029
1 y	6.6	3.4	.0227
30 d	3.3	0.8	<.0001

Abbreviations: ABX-days, days of antibiotics; CDI, Clostridium difficile infection; ED, emergency department; ICU, intensive care unit; Inpt, inpatient; imi-R, imipenem-resistant; LOS, hospital length of stay before infection.

Risk factors for imipenem resistance that were included in the logistic regression model were (1) prior colonization with imipenem-resistant Klebsiella pneumoniae, (2) hospital unit, (3) total inpatient days in the previous 5 years, (4) total days of oral or parenteral antibiotics in the past 2 years, and (5) age >60 years. The logistic regression coefficients and P values are listed in Table 2.

Table 2.

Multiple Logistic Regression Model Variables and Coefficients

	B	OR	SE	Wald	df	P Value
Imi-R Klebsiella colonization	2.873	17.697	0.708	16.491	1	<.001
Locationa				7.689	2	.021
Location (high risk)	0.730	2.075	0.381	3.671	1	.055
Location (low risk)	–1.002	0.367	0.784	1.636	1	.201
Age >60	0.771	2.161	0.367	4.414	1	.036
ABX-days in past 2 y	0.009	1.009	0.003	6.897	1	.009
Inpatient days in past 5 y	–0.004	0.996	0.002	3.544	1	.060
Constant	–3.144	0.043	0.430	53.543	1	<.001

Abbreviations: ABX-days, days of antibiotics; B, regression coefficient; df, degrees of freedom; Imi-R, imipenem-resistant; OR, odds ratio.

^aEmergency department was considered the reference location.

The logistic regression model created from the training set generated an ROC curve with an AUROC of 0.755. When applied to the testing set using a threshold of 0.08, the model correctly predicted 73% of the imipenem-resistant cases while incorrectly labeling 41% of the susceptible cases as resistant (sensitivity, 73%; specificity, 59%; positive predictive value [PPV], 16%; negative predictive value, 95%) (Table 3). The AUROC of the model when applied to the testing set was 0.721.

Table 3.

Performance of the Model on the Testing Set Data

		Predicted Resistance
		Imi-R	Imi-S	% Correct
Observed resistance	Imi-R	11	4	73.3
	Imi-S	56	82	59.4
Overall % correct				60.8

Abbreviations: Imi-R, imipenem-resistant; Imi-S, imipenem-susceptible.

Ten-fold cross-validation was performed on the training set data and repeated 500 times. The mean AUROC of the 5000 cross-validation repeats was 0.731, whereas the mean AUROC of the random model repeats was 0.488 (difference in means, 0.243; P < .001) (Figure 2).

Figure 2.

Histogram showing the area under the receiver operating characteristic curve for each K-fold cross-validation repeat and each random model repeat. Abbreviations: AUROC, area under the receiver operating characteristic curve.

Time to Active Therapy

All 613 cases were reviewed to determine the time to active antibiotics. Of patients with imipenem-resistant Klebsiella pneumoniae bacteremia, 33% never received active antibiotics, compared with 3% of those with imipenem-susceptible infections (P < .0001). Among patients who did receive active antibiotics, the mean time to effective therapy was significantly greater for patients with imipenem-resistant infections (40.4 hours vs 9.6 hours, P < .0001). Only 44% of imipenem-resistant cases were appropriately treated within the first day, compared with 84% of susceptible infections (P < .0001).

DISCUSSION

This study demonstrates that a multiple logistic regression model using EMR data can generate immediate, sensitive predictions of carbapenem resistance in patients with Klebsiella pneumoniae bacteremia. Of patients who received appropriate antibiotics, the mean time to active therapy was more than 30 hours greater in patients with carbapenem-resistant infections, which underscores the need for rapid identification of carbapenem resistance in these cases.

Several new diagnostic tests may significantly shorten the time required to identify CRE infections [11–16]. However, some novel CRE diagnostics still require several hours after culture positivity for a result to return, whereas others have a faster turnaround time but only test for a limited number of resistance genes, which may not always correlate with a resistant phenotype [17]. In patients with septic shock, each hour without active antibiotic therapy has been associated with an increased risk of death [18]. Therefore, even when new molecular methods of CRE detection gain more widespread use, an algorithm that can immediately predict carbapenem resistance could still complement genotypic testing and impact patient outcomes.

In practice, clinicians may attempt to estimate a bacteremic patient’s risk for CRE infection by considering known risk factors, which include prior CRE colonization, previous antibiotic exposures, critical illness, renal failure, advanced age, recent surgery, hospital and ICU length of stay, and the presence of a central venous catheter [19–26].

Given the complexity of considering all these variables for each patient, prior studies have attempted to create clinical prediction models for gram-negative resistance. Logistic regression [24, 25, 27–31] and clinical decision trees [32, 33] have previously been used to predict patients at high risk for resistant gram-negative infections. However, some published models have been limited by the inclusion of few CRE cases, a focus on predicting colonization as opposed to infection, or a lack of robust validation methods. Others have incorporated variables that require manual data entry, such as travel history, or a subjective opinion, such as the suspected source of infection, which may impair the speed and reproducibility of predictions.

A major advantage of our model is that it includes only variables that exist as objective fields in the EMR. This approach could allow for the rapid generation of automated predictions of carbapenem resistance as soon as Klebsiella pneumoniae is identified in a blood culture, without the need for manual input or interpretation. Another benefit of this model is that it generates an automated review of prior microbiology data for each patient. Although clinicians would likely choose empiric antibiotics active against CRE for a bacteremic patient with a history of CRE infection, in practice, prior cultures may sometimes be overlooked. By automatically encorporating all prior culture data, our algorithm could help prevent this type of oversight. This model could potentially be integrated directly into the EMR to provide real-time clinical predictions.

Another strength of our model is that it is supported by 2 different validation methods. The K-fold cross-validation and the performance of the model when applied to the testing set both support its out-of-sample validity. Additionally, the significantly higher mean AUROC of the cross-validation repeats, when compared with the random model, indicates that our findings are not due to chance or sampling error.

The major limitation of this model is its relatively low PPV. The low PPV is likely driven by the low prevalence of carbapenem resistance in the study population, as well as our decision to choose a threshold for positivity that emphasizes sensitivity over specificity. Prospectively incorporating more data into the model in future years could potentially improve the specificity and PPV.

The single-center nature of this study is another drawback. Data were collected from only 1 institution, but patients may have been admitted to other hospitals and been exposed to other risks factors for CRE that were not included in the model.

Another limitation is that this model is not directly generalizable to other institutions. The inclusion of hospital unit as a variable precludes comparisons with other hospitals, and because the prevalence of CRE varies drastically between institutions and geographic locations, the model developed at our tertiary medical center in New York City may not be applicable to hospitals in other locations. Specifically, the PPV of the model might be lower if used in a population where CRE infections are less prevalent.

This study shows that a logistic regression model using EMR data can generate rapid, sensitive predictions of carbapenem resistance in patients with Klebsiella pneumoniae bacteremia. Because effective treatment is often delayed in cases of CRE bacteremia, and this delay may contribute to the high mortality associated with these infections, our model could potentially help improve outcomes by quickly identifying patients at risk for CRE infection. Further research is needed to determine how this model could best be applied to patient care.

Financial support. This work was supported by the New York State Department of Health Empire Clinical Research Investigator Program.