Prospective Implementation of a Risk Prediction Model for Bloodstream Infection Safely Reduces Antibiotic Usage in Febrile Pediatric Cancer Patients Without Severe Neutropenia

Abstract

PURPOSE

Management of febrile pediatric patients with cancer with an absolute neutrophil count of 500/µL or greater is unclear. The Esbenshade Vanderbilt (EsVan) risk prediction models have been shown to predict bloodstream infection (BSI) likelihood in this population, and this study sought to prospectively validate and implement these models in clinical practice.

METHODS

Data were prospectively collected on febrile pediatric patients with cancer with a central venous catheter from April 2015 to August 2019 at a single site, at which the models (EsVan: 2015 to 2017; EsVan2: October 2017 to 2019) were initially developed and subsequently implemented for clinical management in well-appearing nonseverely neutropenic individuals. It was recommended that patients with low BSI risk (< 10%) be discharged home without antibiotics, those with intermediate BSI risk (10%-39.9%) be administered an antibiotic before discharge, and those with high BSI risk (> 40%) be admitted on broad-spectrum antibiotics. Seven-day outcomes were then collected and EsVan models were prospectively validated and C-statistics estimated.

RESULTS

In 937 febrile, nonsevere neutropenia episodes, frequencies of low-, intermediate-, and high-risk episodes were 88.9%, 8.6%, and 2.3% respectively. BSI incidence was 4.2% (39 of 937). Within risk groups, low-risk BSI incidence was 1.9% (16 of 834) with BSI incidence of 13.6% and 54.5% for intermediate- and high-risk episodes, respectively. Empirical intravenous antibiotics were administered in 21.1% of low-risk episodes at presentation and at 7 days postpresentation, 72.3% of episodes never required intravenous antibiotics. There were no deaths or clinical decompensations attributable to antibiotic delay. For BSI detection, EsVan and EsVan2 models applied to the new cohort achieved C-statistics of 0.802 and 0.824, respectively.

CONCLUSION

Prospective, real-time clinical utilization of the EsVan models accurately predicts BSI risk and safely reduces unnecessary antibiotic use in febrile, nonseverely neutropenic pediatric patients with cancer.

INTRODUCTION

Pediatric patients with cancer are immunosuppressed from their disease and/or receipt of cytotoxic therapies. Many require central venous catheters (CVCs) for receipt of therapy, but the use of CVCs also increases the risk of infection. Fever is often the presenting sign of a potential bacterial bloodstream infection (BSI). For febrile pediatric oncology patients presenting with severe neutropenia (absolute neutrophil count [ANC] < 500/µL), management guidelines are well established, including empirical antibiotics¹; however, there are no established guidelines for the management of febrile pediatric patients with cancer who present with an ANC ≥ 500/µL.² Studies in this population have been largely descriptive, including a systematic review.^2-9 Whereas some small studies demonstrated the safety of withholding empirical antibiotics in well-appearing, nonneutropenic fever patients,^4,9 a common management strategy for such patients remains empirical treatment with a third-generation cephalosporin, often ceftriaxone, with discharge home; however, the sensitivity of ceftriaxone to isolated BSI from such cases can be < 45%⁵ and the need for empirical antibiotics has been unclear. To address these limitations, our team in 2015 developed a clinical risk prediction model, the Esbenshade Vanderbilt (EsVan) model, using clinical and laboratory measurements obtainable at fever presentation to the Monroe Carrel Junior Children’s Hospital at Vanderbilt (MCJCH).⁵ The EsVan model effectively classifies patients according to their predicted risk.⁵ The model has been retrospectively validated at 6 independent centers.⁶ Building on previous data, refined, modified versions of the initial model—EsVan2 and EsVan2b—were developed in 2017 and validated, which have demonstrated even better performance in classifying patients across multiple data sets.⁶ On the basis of these data, these models were adopted and have been implemented into daily clinical practice at MCJCH. We report the safety and efficacy of using these models prospectively on a newly created prospective data set as well as to further critically re-evaluate the variables used in the model to try to further simply and improve the model.

CONTEXT

• Key Objective

• To demonstrate if a risk prediction model for predicting bloodstream infections in pediatric patients with cancer without severe neutropenia safely identifies those who do not require initial empirical antibiotics.

• Knowledge Generated

• This study evaluated the prospective use of the publicly available Web-based Esbenshade Vanderbilt models in 937 episodes of pediatric patients with cancer presenting with non-neutropenic fever at a single institution and identified 88.9% of these as low risk. In the low-risk group, empirical antibiotics were only administered in 21.1% of cases and 72.3% required no intravenous antibiotics within 7 days of presentation, with no deaths or clinical decompensations attributable to antibiotic delay. The models also showed good discrimination when applied to the prospective data set with C-statistics greater than 0.8.

• Relevance

• Real-time clinical use of the Esbenshade Vanderbilt models accurately predicts bloodstream infection risk and safely reduces unnecessary antibiotic use in febrile, nonseverely neutropenic pediatric patients with cancer and can aid clinical decision making.

METHODS

EsVan Models

Using 12 clinical variables in patients with fever and ANC ≥ 500/µL, the EsVan model was designed to predict BSI risk, thereby informing clinical management. In 2017, it was simplified once to the EsVan2 model to remove 2 variables (patient location at presentation and cancer diagnosis) less clinically reliable for BSI, and, furthermore, to account for the fact that occasionally absolute monocyte count (AMC) is not readily available at presentation, a version of the model without it was created (EsVan2b).

Development and internal and external validations of the EsVan and EsVan2 risk prediction models for BSI have been previously reported^5,6 and were converted into a Web-based module for ease of use.^5,10 A description of the variables and how EsVan models have changed over time is provided in Table 1.

TABLE 1.

Summary of EsVan Model Development: Over Time, Variables Have Been Removed From the Model That Were Less Specific for Bloodstream Infection

For this study, there is now a newly created prospective data set, and it was felt that the variables used in the EsVan models could again be revaluated to ensure that they are reliably predicting for each individual. Although knowing an ANC is ≤500/µL is critical to patient management, with increased use of extended-release PEG-filgrastim, an elevated ANC is not uncommon soon after chemotherapy administration, and thus the precise ANC elevation may be less predictive of BSI. Upper respiratory symptoms can be caused by noninfectious causes, such as seasonal allergies, and similarly not predict BSI well. Thus, using the 2015 retrospective data set, a new model, EsVan3a, was created by removing these 2 variables from the EsVan2b model. We then validated it on the 5-study retrospective cohort and the new prospective Vanderbilt cohort.^5,6 Furthermore, believing that the prospectively collected data may be more accurate than those retrospectively collected, we also newly created the EsVan3b model using the prospective Vanderbilt cohort as the derivation data set and the same variable set as with EsVan3a. We then used the previously published study cohorts to validate these new models.

The model development and validation followed the same approaches previously described.^5,6 In brief, logistic regression models were developed. To assess the potential model overfitting, statistics from 300-iteration bootstrap approaches were reported. Harrell C-index was reported as the model discrimination measurement. Model calibration was assessed graphically with calibration plots. All statistical analyses were performed using R software version 3.5.¹¹ The Data Supplement contains additional statistical description.

Prospective Implementation of the EsVan and EsVan2 Models

To prospectively implement these models, whenever a well-appearing pediatric patient with cancer or histiocytosis presented with fever, a CVC, and an ANC ≥ 500/µL, clinical data were entered into a Web-based module. Using a 1-way API function, these data were then recorded directly into a Vanderbilt REDcap database, a secure, Web-based software platform for electronic data capture.¹² After a minimum of 7 days after the initial fever, investigators assessed the medical chart to record patients’ outcome data. Exclusion criteria for using the models included a history of allogeneic transplant, ≤ 30 days post–auto-transplant, and current treatment with intravenous empirical antibiotics at fever presentation.

Data collection was approved by the Vanderbilt Institutional Review Board. This implementation study included data captured between April 1, 2015, and August 22, 2019. In addition, all potentially eligible records were abstracted from the medical record to identify all additional eligible episodes that were not directly recorded at the time of fever.

Risk predictions were initially generated using the EsVan model from April 2015 to September 2017, and then the EsVan2 model from October 2017 to present, which provided better prediction performance across data sets.⁶ Before October 2017, when the AMC was not immediately available at the time of prediction, the AMC was imputed using the Web-based module as the median AMC of the data set (450/µL) to provide the prediction. However, starting in October 2017 when the AMC was not available, the Web-based module used the EsVan2b model (same variables as with the EsVan2 model but without AMC) to generate the prediction. The risk category assigned in this analysis was based on the prediction generated at the time of the episode.

Risk Classification and Recommended Management

Low risk.

Episodes with a predicted risk < 10% were classified as low risk and given the recommendation to obtain a blood culture, and, if clinically stable, to be discharged to home or local family accommodation unless already an inpatient, with instructions for reassessment immediately with worsening symptoms or every 24 hours if still febrile. For patient safety, the following additional criteria were used to define patients who should receive antibiotics despite a low risk score; these included:

1. Decreasing ANC from previous check and < 1,000/µL

2. Patients < 1 year of age

3. Patients with hematologic malignancy not in remission

4. Concern for sepsis (ill appearing)

5. Symptoms suggestive of other non-BSI bacterial processes requiring antibiotics independent of their BSI prediction

6. Treating provider clinical judgment.

Intermediate risk.

Episodes with a predicted risk of 10% to 39.9% were classified as intermediate risk and given the recommendation to administer a dose of an empirical antibiotic, either intravenous cefepime if reassessment occurred in < 12 hours or intravenous levofloxacin. Those < 5 years of age were administered a second dose of oral levofloxacin for 12 hours after the first dose. Similarly, patients were instructed to be reassessed immediately with worsening symptoms or every 24 hours if still febrile.

High risk.

Those patients with a predicted risk ≥ 40% were recommended intravenous cefepime with admission until cultures were negative for > 48 hours.

Study Definitions for Fever and BSI

Fever was defined per the Infectious Diseases Society of America as a temperature ≥ 38.0°C for more than 1 hour or ≥ 38.3°C once.¹³ Blood cultures of confirmed pathogens were classified as BSI with one positive culture; however, for common commensal organisms, 2 positive cultures were required.^14,15 For prediction analysis, commensals determined to be contaminants were reclassified as non-BSI. High-risk BSI was defined as isolation of a gram-negative or Staphylococcus aureus BSI.^5,6 Standard practice at MCJCH is to obtain blood cultures from each lumen of multilumen CVCs as well as to obtain 2 separate blood cultures from other central lines.

RESULTS

Implementation of the EsVan and EsVan2 Models

There were 937 episodes meeting inclusion criteria in 331 unique patients. Overall BSI rate was 5.1% (48 of 937) and the true BSI rate—minus contaminants—was 4.2% (39 of 937). Using the EsVan model predictions (EsVan: April 2015 to September 2017; EsVan2: October 2017 to August 2019), 88.9% of episodes (834 of 937) were classified as low risk (< 10% risk), 8.6% (81 of 937) as intermediate risk (10%-39.9%), and 2.3% (22 of 937) as high risk (≥ 40%). The characteristics of the cohort comparing previous cohorts used for EsVan model development and validation are listed in Table 2. Significant differences in the current cohort include a reduction in BSI rate, a reduction in external dwelling catheters (24%-14.1%), and a decrease in shaking chills (10%-5.2%). The model was used 80.6% of the time (755 of 937).

TABLE 2.

Patients Characteristics by Study Cohort

Low-Risk Group

Among the 834 episodes classified as low risk by model, BSI rate was 2.8% (23 of 834) and the true BSI rate was 1.9% (16 of 834). Details of the BSIs isolated are listed in Table 3. In addition, the rate of high-risk organism BSI was 1.0% (8 of 834). In those with predicted low risk but who were assessed as ill appearing by the treating provider, BSI and true BSI rates were both 16.7% (3 of 18). Empirical intravenous antibiotics were administered 21.1% of the time on day 1 and an additional 8.8% (n = 73) required oral antibiotics (Fig 1). By 7 days postpresentation, 72.3% of episodes (603 of 834) never needed intravenous antibiotics and 62.4% (520 of 834) required no intravenous or oral antibiotics. Among 834 episodes classified as low risk, 124 episodes received antibiotics for protocol-based reasons. The remaining 711 episodes were eligible for no empirical antibiotics, and among those, 7.3% (n = 52) received antibiotics at presentation and another 7.6% (n = 54) received antibiotics within 7 days. Thus, the remaining 84.8% (603 of 711) did not receive any intravenous empirical antibiotics within 7 days of presentation. There was 1 death in a patient with a refractory sarcoma with lung metastases whose risk was classified as low but who died of disease progression with or without pneumonia 5 days after presentation with fever. No bacterial pathogen was isolated from the blood, but it is not possible to know if a bacterial process in the lung was present. There was one additional death from disseminated varicella. Of note, there were no deaths, requirements for intensive care, or severe clinical decompensations that resulted from delay or withholding empirical antibiotics in this group. There were 7 true BSIs in which the patient was discharged without antibiotics and 85.7% of patients (6 of 7) were afebrile on reassessment and in all cases well appearing.

TABLE 3.

Description of Organisms Isolated

FIG 1.

Flow diagram of the implementation of Esbenshade Vanderbilt (EsVan) models and outcomes. ANC, absolute neutrophil count; BSI, bloodstream infection; ICU, intensive care unit; IV, intravenous; PO, oral.

Intermediate-Risk Group

Among the 8.6% (n = 81) intermediate-risk episodes, the true BSI rate was 13.6% (11 of 81) with no contaminant organisms identified, and 63.6% (7 of 11) were high-risk organism BSIs (Table 2). The clinical management is described in Figure 1. In the 8 patients, despite intermediate-risk classification, the model was not used clinically and antibiotics were held despite elevated risk and the BSI rate was 37.5% (3 of 8). However, in all 3 BSI cases, patients were well appearing on reassessment. No deaths or decompensations were noted from a bacterial source, but there was 1 death from respiratory syncytial virus.

High-Risk Group

Twenty-two cases (2.3%) had a predicted risk > 40%. Among those, the BSI rate was 59.1% and the true BSI rate was 54.5%, with 91.7% (11 of 12) being high-risk organisms (Table 2). All recovered well after the prescribed management (Fig 1).

Validation of Previous EsVan Models on the Prospective Vanderbilt Cohort

All 3 EsVan models performed similarly well on the prospective Vanderbilt cohort. On detecting BSI status, the EsVan, EsVan2, and EsVan2b models achieved an area under the curve (AUC)/C-index of 0.802, 0.824, and 0.818, respectively (Table 3). Consistent with previous validations results, the models performed even better in detecting high-risk BSIs, achieving an AUC/C-index of 0.837, 0.855, and 0.858, respectively.⁶ More detailed model performance on other study cohorts are listed in Table 4. The calibration curves of these models on the prospective Vanderbilt cohort are presented in Figure 2.

TABLE 4.

Summary of Model Performance/Discrimination (C-statistic) on Study Cohorts

FIG 2.

Calibration plots of previous Esbenshade Vanderbilt (EsVan) models on the newly created prospective Vanderbilt cohort. (A) EsVan model. (B) EsVan 2 model. (C) EsVan2b model. BSI, bloodstream infection; ROC, receiver operating characteristic.

New EsVan Models: EsVan3a and EsVan3b

For the newly developed EsVan3a and EsVan3b models, the model fitting details and effect measurements of all the predictors are presented in the Data Supplement. The EsVan3a model achieved an AUC/C-index of 0.885 on the retrospective Vanderbilt cohort (model development cohort). The bootstrap-corrected AUC is 0.878, which indicates no evidence of overfitting. When applied to the 5-study validation cohort and the prospective Vanderbilt cohort, the EsVan3a achieved an AUC/C-index of 0.724 and 0.817, respectively (Table 4). The EsVan3a model performed even better in detecting high-risk BSI (Table 4). The EsVan3b model created from the prospective data set achieved an AUC/C-index of 0.840 on the prospective Vanderbilt cohort. The bootstrap-corrected AUC is 0.827, which indicates no evidence of overfitting; on detecting high-risk organism BSI, it achieved an AUC/C-index of 0.863. When applied to the retrospective Vanderbilt cohort and the 5-study cohort, the EsVan3b model achieved an AUC/C-index of 0.877 and 0.724, respectively. Both models calibrated well, indicating accurate risk predictions (Data Supplement).

Prediction Consistency Among the EsVan Models

Using the prospective Vanderbilt data set, all versions of the EsVan models similarly and effectively identified those at < 10% predicted risk for BSI (EsVan: 88.8%; EsVan2: 90.1%; EsVan2b: 89.2%; EsVan3a: 90.6%; EsVan3b: 90.7%). The true BSI rates were also similar among the low-risk group identified by the models (EsVan: 2.0%; EsVan2: 1.8%; EsVan2b: 1.8%; EsVan3a: 1.9%; EsVan3b: 1.8%). There was more variance in the number of episodes classified as high risk (> 40%), with the older version of the model EsVan predicting more episodes in this category (n = 22) versus the new model EsVan3b (n = 12). However, the new model more specifically identified BSI in those who were predicted to be high risk (75.0% EsVan3b v 54.5% EsVan). A summary of how each version of the model classified episodes is provided in the Data Supplement.

DISCUSSION

A systematic approach to the management of fever in pediatric oncology patients in the absence of severe neutropenia is needed to balance patient safety and antibiotic stewardship. Therefore, this is an ideal clinical situation in which to implement and evaluate risk prediction modeling. We have developed the EsVan models to meet this need and have demonstrated that they effectively identify those at low risk of symptomatic BSI, as well as those at higher risk. Use of this model in clinical practice has demonstrated a notable decrease in empirical antibiotics without increasing adverse events. Moreover, with the identification of a higher risk, patients can receive appropriate broad-spectrum antibiotics upfront, to which the organisms are initially susceptible. The publicly available Web-based module allows real-time use of the model to aid decision making and ease in a busy clinical practice.

Sometimes fever is the only presenting sign of BSI, and no modeling strategy will predict with 100% accuracy all BSI; however, all patients who were low risk and discharged home without empirical antibiotics were still well appearing and, in many cases, afebrile when they returned after BSI. This is consistent with Bartholomew et al⁴ and Wu et al,⁹ who also demonstrated the safety of sending well-appearing febrile patients without severe neutropenia home without empirical antibiotics.

No model can effectively predict all types of possible bacterial infections, and potential sources should continue to be treated independently of a BSI model. A decision to withhold antibiotics should always be deferred to the treating provider, and this study gives evidence that patients who are ill appearing by provider opinion but low risk by model did have an elevated observed BSI rate (16.7%; 3 of 18 episodes). In this study, the BSI rate was less than 2% for those who had a predicted risk of BSI < 10%, met model use criteria, and were administered antibiotics anyway. This indicates there may be additional patients who could avoid empirical antibiotics. The BSI rate was high for those with a predicted risk > 40%, and as this is only 2% of the cohort, it remains reasonable to standardly admit these patients and provide empirical antibiotics that will cover high-risk BSI organisms.

Antibiotic resistance is a mounting problem in pediatric oncology as these patients will have continued risk for invasive infections and will need to have sensitive antibiotics available. Antibiotics increase health care costs and are not fully benign medications as they can be associated with adverse effects. Thus, there are multiple advantages to appropriate stewardship.

All versions of the EsVan model effectively identify those at risk; however, continuing to improve and reduce the variables in the model as clinically indicated will increase the chances of generalizability to other pediatric oncology populations. Several variables (inpatient status, acute lymphoblastic leukemia diagnosis, ANC, AMC, and upper respiratory symptoms) have been removed in the new versions of the model as the directionality of these variables is not always reliable. Despite their removal, the new models still perform well over all 3 data sets and additional multicenter validation is ongoing. As the EsVan3b model was developed with prospectively collected data and requires fewer variables, we think it may eventually be the most user-friendly version; however, we do not recommend its use in clinical practice, pending additional external prospective validation.

There are some limitations to model implementation. It has only been studied in those with pediatric cancer or histiocytosis and only applied to febrile patients with a CVC in place. This study had few patients < 1 years of age or > 21 years of age and excluded those who were post allogeneic transplantation. Thus, additional study is needed to know the safety of using the model in those populations. There were 3 deaths in this cohort unrelated to a bacterial BSI, and this demonstrates that severely morbid events can occur in this population that a model will not predict and so the clinical status of patients always needs close evaluation.

BSI estimates are stable over 3 data sets and so are likely to be useful in other clinical pediatric oncology practices; however, caution should be used in environments with different bacterial flora or sensitivity patterns than what is described in this study without first evaluating its reliability locally, particularly in countries outside of the United States.

In summary, the prospective implementation of the EsVan model has been successful at a single institution and the data suggest that it may be safely implemented in clinical practice. It has reduced empirical intravenous antibiotic for the nearly 90% of episodes that have less than 10% predicted risk from more than 95% to only 27.8% without evidence of adverse outcomes. Furthermore, it identified those at higher risk who required empirical antibiotics other than ceftriaxone, along with hospital admission. The EsVan models are now undergoing external prospective validation.

AUTHOR CONTRIBUTIONS

Conception and design: Adam J. Esbenshade, Zhiguo Zhao, Daniel E. Dulek, Debra L. Friedman

Collection and assembly of data: Adam J. Esbenshade, Alania Baird, Emily A. Holmes

Data analysis and interpretation: Adam J. Esbenshade, Zhiguo Zhao, Alania Baird, Daniel E. Dulek, Ritu Banerjee, Debra L. Friedman

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

Prospective Implementation of a Risk Prediction Model for Bloodstream Infection Safely Reduces Antibiotic Usage in Febrile Pediatric Cancer Patients Without Severe Neutropenia

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/jco/authors/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).

Daniel E. Dulek

Research Funding: Viracor, Eurofins

No other potential conflicts of interest were reported.