Design, Implementation, and Validation of a Pediatric ICU Sepsis Prediction Tool as Clinical Decision Support

Abstract

Background  Sepsis is an uncontrolled inflammatory reaction caused by infection. Clinicians in the pediatric intensive care unit (PICU) developed a paper-based tool to identify patients at risk of sepsis. To improve the utilization of the tool, the PICU team integrated the paper-based tool as a real-time clinical decision support (CDS) intervention in the electronic health record (EHR).

Objective  This study aimed to improve identification of PICU patients with sepsis through an automated EHR-based CDS intervention.

Methods  A prospective cohort study of all patients admitted to the PICU from May 2017 to May 2019. A CDS intervention was implemented in May 2018. The CDS intervention screened patients for nonspecific sepsis criteria, temperature dysregulation and a blood culture within 6 hours. Following the screening, an interruptive alert prompted nursing staff to complete a perfusion screen to assess for clinical signs of sepsis. The primary alert performance outcomes included sensitivity, specificity, and positive and negative predictive value. The secondary clinical outcome was completion of sepsis management tasks.

Results  During the 1-year post implementation period, there were 45.0 sepsis events per 1,000 patient days over 10,805 patient days. The sepsis alert identified 392 of the 436 sepsis episodes accurately with sensitivity of 92.5%, specificity of 95.6%, positive predictive value of 46.0%, and negative predictive value of 99.7%. Examining only patients with severe sepsis confirmed by chart review, test characteristics fell to a sensitivity of 73.3%, a specificity of 92.5%. Prior to the initiation of the alert, 18.6% (13/70) of severe sepsis patients received recommended sepsis interventions. Following the implementation, 34% (27/80) received these interventions in the time recommended, p  = 0.04.

Conclusion  An EHR CDS intervention demonstrated strong performance characteristics and improved completion of recommended sepsis interventions.

Background and Significance

Sepsis is an uncontrolled inflammatory reaction caused by an infection and is a leading cause of morbidity and mortality in pediatrics. 1 Pediatric sepsis-associated mortality has decreased from 97% 2 in 1966 to less than 10% by 2005. 3 However, despite improvements in mortality, care comes at a significant burden with pediatric severe sepsis hospitalizations costing more than 7 billion dollars in 2016 and accounting for one-fifth of total hospital costs for U.S. children. 4 Timely fluid resuscitation and antibiotic administration has been shown to decrease sepsis mortality, 5 6 7 leading to an emphasis on early recognition and treatment of sepsis. 8 To improve the early recognition and care of children with sepsis, the improving pediatric sepsis outcomes (IPSO) collaborative 9 was formed. IPSO is a quality improvement collaborative of 56 children’s hospitals focused on improving timely recognition and treatment of pediatric sepsis. IPSO guidelines recommend the use of trigger tools to help with sepsis recognition, and several studies in emergency departments have shown improved sepsis recognition with their implementation. 10 11 12 13 In addition, an automated approach within the pediatric emergency department demonstrated equal accuracy and significantly improved time to identification of sepsis over manual tools, 14 while adult inpatient units with electronic surveillance found improved outcomes, including mortality. 15 Despite this promise, sepsis triggering tools vary greatly between and even within institutions, and no study has established evidence sufficient to promote a standardized screening tool. 16 17 18 19 Furthermore, no sepsis triggering tool is specifically recommended for the pediatric intensive care unit (PICU) and the use of complex proprietary tools is infeasible for many PICUs serving as a primary barrier to implementation.

Prior to this work, the PICU at Cincinnati Children's Hospital Medical Center (CCHMC) did not have a standardized method of identifying patients at high-risk for sepsis. In response to this problem, a group of clinicians at CCHMC developed a two-step paper tool on clinical decision support (CDS) to improve the recognition of sepsis as part of a larger quality improvement effort to improve pediatric sepsis recognition and treatment within the PICU. This paper tool was affixed to blood culture bottles and requested that nurses screen all patients with a fever and an ordered blood culture for signs of altered perfusion, a marker of sepsis. While this was a significant advance, the paper form was often ignored or discarded by the bedside nurse without completion of a sepsis evaluation. Partnering with clinical informatics expertise within the PICU, a project team was formed to develop and implement a real-time CDS intervention to identify patients at high risk for sepsis. Complex prediction models, including promising artificial intelligence and machine learning approaches, have been developed for sepsis prediction. 20 21 22 However, despite the promise of these more sophisticated tools, there lie many challenges in their implementation including provider trust, algorithm and system maintenance, and cost. Alternatively, prior successful CDS interventions that relied only on standard EHR functionality without significant additional documentation have shown promise, 14 23 and we sought to emulate their success within the PICU setting. Our objective was to improve real-time identification of PICU patients with sepsis through an automated electronic health record (EHR)-based CDS intervention.

Methods

Setting and Participants

The PICU is a quaternary care pediatric medical/surgical ICU with 35 beds and averages over 2,500 admissions per year in each of the past 5 years with an average daily census of 28 patients. The cardiac surgical ICU and neonatal ICU were excluded. For our primary analysis, we evaluated the predictive ability of the EHR-based CDS intervention over 1 year from May 2018 to May 2019. For our analysis of intervention performance, we defined a patient as being sepsis positive if he or she either meet the IPSO definition of severe sepsis while admitted to the PICU or if he or she was treated as sepsis by the care team as determined by physician chart review ( Table 1 ). Treated as sepsis by the care team was defined as (1) documentation of concern for sepsis following a bedside huddle by the multidisciplinary team or (2) use of the sepsis order set and (3) at least 48 hours of intravenous (IV) antibiotics. In-depth chart review included a review of all clinical notes, laboratories, and vital sign trends of at least the 48 hours pre- and postmeeting IPSO severe sepsis criteria. If the patient did not meet the IPSO severe sepsis definition or was not treated by the care team as sepsis (defined above) then the patient was considered to be sepsis negative. If a patient was determined to have an episode of sepsis but did not have a CDS intervention fire prior to the sepsis episode, then he or she was determined to be CDS intervention negative. The primary data source for the analysis was the epic EHR data warehouse using a standard SQL query (Epic Systems, Verona Wisconsin, United States).

Table 1. Sepsis definitions used for cohort evaluation.

Sepsis definitions
Sepsis positive	• Met the IPSO definition of severe sepsis • Treated as sepsis by the care team (documentation of concern for sepsis following a multidisciplinary bedside huddle OR use of the sepsis algorithm/order set AND at least 48 hours of IV antibiotics)
IPSO severe sepsis	• Blood culture • Two fluid boluses • Antibiotic administration
IPSO severe sepsis plus manual chart review	• IPSO definition • Manual chart review with:  – Diagnosis of sepsis or septic shock OR  – High clinical suspicion resulting in use of the sepsis algorithm or order set

Abbreviations: IPSO, improving pediatric sepsis outcomes; IV, intravenous.

For our secondary analysis, we evaluated the impact of the CDS intervention on completion of recommended sepsis therapies including timely evaluation of lactate and administration of fluids and antibiotics. To complete this analysis, we performed a prospective cohort study of all patients admitted to the PICU from May 8, 2017 to May 8, 2019, with May 8, 2017 to May 8, 2018 providing preimplementation data and May 9, 2018 to May 8, 2019 providing postimplementation data. All patients presenting to the PICU during the study period were included.

Ethical Issues

The purpose of this work was to measure and improve the quality of existing care practices, as per our IRB's existing guidance regarding projects of this nature, it did not require review via the Institutional Review Board.

Planning the Intervention

We sought to streamline our current workflow through improved standardization and reliability via an EHR-based automated CDS intervention. In optimal practice, clinicians maintain a low threshold to consider sepsis in pediatric patients and then use a more structured approach for identified patients to determine risk and action thresholds for treating the condition. To mimic this stepwise workflow while accounting for the low incidence of true sepsis events, 8.0% per the literature, 24 we selected a two-stage screen similar to the pilot paper tool.

During the first stage, we utilized the EHR to continually screen for patients that met highly sensitive but broad criteria as follows: (1) fever (temperature >38.5°C) or hypothermia (temperature <35.0°C) plus, (2) an order for a blood culture placed within 6 hours of the abnormal temperature reading. These criteria were extrapolated from published data 25 and agreed upon by all PICU attending physicians at CCHMC. Any time the criteria were met for a patient, the automated CDS intervention fired and an interruptive alert prompted nursing to complete the second stage of screening, a bedside perfusion screen ( Fig. 1 ) adapted from our institutional sepsis guidelines. If there was a concern for sepsis based upon the results of this second stage of screening, the nurse bedside assessment, nurses were asked to activate a sepsis huddle, a bedside gathering between the nurse and provider to facilitate ordering of necessary diagnostic tests (including a lactate level) and administering appropriate antibiotics and intravenous fluids in a timely manner. Sepsis huddles as a quality improvement intervention were a process already in place in our PICU to improve adherence to sepsis treatment targets prior to initiation of the CDS intervention. During this huddle, patients were determined by the multidisciplinary team to either have a high likelihood of sepsis resulting in activation of the local sepsis algorithm and categorization of sepsis positive in our analysis, or were determined to not have concerns for sepsis. Critical care physicians, clinical informaticians, critical care nurses, and respiratory therapists were all involved in determining the user goals of the CDS intervention.

Fig. 1.

Two-stage sepsis interruptive CDS alert that is triggered when a patient admitted to the PICU has a temperature derangement of <35 or >38.5°C and has had a blood culture collected within 6 hours. CDS, clinical decision support PICU, pediatric intensive care unit.

Prior to full-scale implementation, we performed iterative testing and revision. We implemented the EHR-based CDS intervention in a testing environment starting on April 23, 2018, meaning the underlying rules were applied to real patient data but frontline providers were not exposed to the alert. During this trial period, the alert triggered 38 times over a 2-week period with all rules correctly identifying hyperthermia or hypothermia associated with a blood culture order within 6 hours of the abnormal temperature reading. We also did two-unit-wide chart reviews on April 24, 2018 and April 26, 2018 to review every patient within the PICU and found no patients who had hypothermia or hyperthermia within 6 hours of a blood culture who were not flagged by the alert during this preimplementation test period. The CDS intervention was deployed in a production environment on May 8, 2018. Unit wide education was provided prior to and throughout the first 2 weeks of deployment of the CDS intervention. The clinical sepsis champions, comprised of providers, nurses, respiratory therapists, and a quality improvement advisor, were responsible for monitoring compliance and providing feedback to providers and nurses.

Methods of Evaluation

This was a prospective cohort study to evaluate the test characteristics of an automated EHR-based CDS intervention and, secondarily, its performance at improving the completion of recommended sepsis therapies through an interruptive nursing alert. The primary outcomes for alert performance included sensitivity, specificity, positive predictive value and negative predictive value. We also included the number needed to screen (NNS), a value that is analogous to number needed to treat. 26 This number signifies the number of patient episodes labeled as at risk for sepsis by the CDS intervention for each patient episode that met criteria for being sepsis positive ( Table 1 ). NNS is calculated by taking the inverse of the positive predictive value. This number adds additional information to clinicians by allowing for a simple to understand estimate of the number of alerts per patient correctly identified. 26

The secondary outcome assessed was the clinical outcome of completion of sepsis management tasks within the recommended time, lactate checked within 60 minutes, antibiotics delivered within 60 minutes, and intravenous fluid bolus administered within 20 minutes. We hypothesized that standardized recognition of sepsis using the CDS intervention and subsequent evaluation by the nurse and possible bedside sepsis huddle, would lead to improve timeliness of recommended sepsis interventions. Completion of tasks pre- and post-CDS intervention were evaluated using a two-tailed t -test.

Results

From May 9, 2018 to May 8, 2019 during our postimplementation study period, there were 486 episodes of sepsis positive patients over 10,805 patient days in PICU patients, with a sepsis event rate of 45.0 events per 1,000 patient days. Of the 486 total episodes of sepsis, there were 138 episodes of IPSO severe sepsis, with a severe sepsis event rate of 12.8 events per 1,000 patient days. The EHR-based CDS intervention interruptive sepsis alert fired a total of 852 times during this same 1-year period. The sepsis alert triggered for 7.9% (852/10,805) of daily encounters leading to a nurse sepsis perfusion screen. 38.8% of these nurse perfusion assessments generated a concern for sepsis and lead to a sepsis huddle. Of the 486 episodes of sepsis positive, 62 sepsis episodes were identified prior to PICU admission, for a total of 424 episodes of sepsis within the PICU. The sepsis alert identified 392 of the remaining 424 episodes accurately leading to a sensitivity of 92.5%, a specificity of 95.6%, a positive predictive value of 46.0%, a negative predictive value of 99.6% and a number needed to screen of two patients for every one true episode of sepsis. Of the 392 episodes accurately identified by the sepsis alert, the nurse selected “concern for sepsis” 152 times leading to evaluation by the care team, “known sepsis” 228 times prompting no evaluation by the care team, and were inaccurate only 12 times in selecting “no concern for sepsis” when the patient in fact met criteria for sepsis ( Fig. 2 ).

Fig. 2.

Performance of the clinical decision support intervention to identify patients with known sepsis or new concern for sepsis. Red boxes are misclassified patients. IPSO, improving pediatric sepsis outcomes; PICU, pediatric intensive care unit.

Examining only the subset of events that met IPSO severe sepsis criteria ( Table 1 ) within the PICU ( n  = 76), the sepsis alert triggered in 44 out of 76 episodes. This resulted in a lower sensitivity of 57.9%, a specificity of 92.5%, a positive predictive value of 5.2% and a negative predictive value of 99.7% ( Table 2 ). The NNS for IPSO severe sepsis was 19-patient episodes for each true episode of IPSO classified severe sepsis. The in-depth chart review was conducted for the 32 patients who met IPSO severe sepsis criteria within the PICU but for whom the sepsis alert did not fire prior ( Supplementary Appendix A , available in the online version). The most common reason for the alert not being triggered was that no temperature dysregulation was recorded ( Fig. 3 ). Due to the use of a pragmatic definition for severe sepsis within the IPSO collaborative, the IPSO severe sepsis definition is known to identify patients not just with severe sepsis but with other types of shock or critical illness. 9 Upon chart review, 16 patients did not meet criteria for severe sepsis by manual chart review ( Table 1 ) and excluding these patients improved the sensitivity and negative predictive value of the sepsis alert to 73.3 and 99.8% respectively ( Table 2 ). Two-by-two tables of alert performance are included in Supplementary Appendix B (available in the online version).

Table 2. Test characteristics of the interruptive sepsis alert based on cohort definitions.

	Patients with sepsis ( n  = 424)	Patients with IPSO severe sepsis ( n  = 76)	Patients with IPSO severe sepsis confirmed by chart review ( n  = 60)
Sensitivity in % (95% CI)	92.5 (89.5–94.8)	57.9 (46.0–69.1)	73.3 (60.3–83.9)
Specificity in % (95% CI)	95.6 (95.2–96.0)	92.5 (92.0–93.0)	92.5 (92.0–93.0)
Positive predictive value in % (95% CI)	46.0 (43.7–48.3)	5.2 (4.3–6.3)	5.2 (4.4–6.0)
Negative predictive value in % (95% CI)	99.7 (99.6–99.8)	99.7 (99.6–99.8)	99.8 (99.8–99.9)
Positive likelihood ratio (95% CI)	20.9 (19.0–22.9)	7.7 (6.3–9.4)	9.8 (8.3–11.5)
Negative likelihood ratio (95% CI)	0.08 (0.06–0.11)	0.5 (0.4–0.6)	0.3 (0.2–0.4)
Number needed to screen	2	19	19

Abbreviations: CI, confidence interval; IPSO, improving pediatric sepsis outcomes.

Fig. 3.

In-depth chart review of 32 patients who met IPSO severe sepsis criteria within the PICU but for whom the sepsis interruptive alert did not fire. PICU, pediatric intensive care unit.

Prior to the initiation of the CDS intervention, 18.6% (13/70) of IPSO severe sepsis patients received the following three recommended interventions fluid bolus within 20 minutes, lactate level obtained within 1 hour, and antibiotics given within 1 hour. Following the initiation of the EHR-based CDS intervention, 34% (27/80) of patients received these three interventions in the time recommended, p  = 0.04 ( Fig. 4 ).

Fig. 4.

Completion of recommended sepsis interventions per and post implementation of an interruptive sepsis alert as part of a clinical decision support intervention.

Discussion

A novel and simple two-stage automated EHR-based CDS intervention to improve sepsis recognition directed at bedside nurses, had excellent test characteristics with high sensitivity and specificity and a low NNS for identifying sepsis positive patient episodes. The CDS intervention required screening for sepsis by nurses in 7.9% of daily encounters over a 1-year period, with only 1.4% of all encounters requiring bedside provider assessment through a sepsis huddle. Importantly, this low burden of nursing and provider assessment included a significant number of patients (more than 50%) who were already identified as being sepsis positive by the provider team. While it would have been possible to develop exclusion criteria for the alert to prevent a trigger in children with a diagnosis of sepsis or already on antibiotics, we chose to include these patients, as we believed the ongoing reevaluation, assessment, and perfusion screening by the bedside nurse was important for meeting goal directed therapy and prompting a change in antibiotics subsequent to a change in patient status. Framed another way, the total number of alerts over 1 year was on average 2.3 per day for the entire 35-bed unit. Given this alert frequency and an average daily census of 28, a nurse caring for two patients per shift (as is common within our PICU) would see one alert every six shifts on average. This represents an acceptable alert burden given the morbidity and mortality ramifications of missed sepsis.

The positive predictive value noted here for prediction of sepsis positive is higher than that published previously for pediatric sepsis alerts, 27 28 29 and this is likely due to the inclusion of patients with known sepsis. The NNS to identify a patient episode with new or current sepsis of two and with new severe sepsis of 19 is feasible within the PICU setting given that screening only requires a quick bedside perfusion assessment by the nurse. Although our interruptive sepsis alert captured the majority of patients with sepsis, it is important to note that it did miss 32 patients who met criteria for severe sepsis by IPSO. Half of these patients (16) did not meet criteria for severe sepsis by manual chart review. The IPSO severe sepsis definition can be met by ICU patients without infection, for example, postoperative hemorrhage or anaphylactic shock, and therefore manual chart review is required to remove such patients from the analysis. The remaining 16 patients who were missed underscore that a vital sign based screen is not sufficient to capture all patients. Review of these cases found that most of the patients who were not captured by the alert did not have temperature dysregulation at all or to a sufficient degree to meet our thresholds. While adding additional criteria of certain medical therapies that regulate temperature or broadening fever thresholds could increase detection of these patients, it would also likely to increase the false positive rate significantly. Based on that, we have focused on nonautomated tools to improve identification for these patients through other quality improvement methods.

The use of two-stage alerts for sepsis screening has been previously studied in both the adult intermediate care area and pediatric emergency department settings. 28 30 In adult patients, use of a two-step nurse-driven paper tool demonstrated high sensitivity and specificity, although it did not significantly change the care of patients with sepsis. 30 Our intervention is similar to a successful two-stage alert followed by a huddle as described in a pediatric emergency department. 28 We believe that this approach emphasizes CDS best practices 31 32 by fitting into the workflow of the bedside nurses and limiting interruptions. Bedside nurses at our institution were highly accurate in their identification of high-risk patients, only incorrectly classified 12 patients out of 852 alerts as no concern for sepsis. In addition, through the use of the quick bedside nurse screening, the interruption to physician workflow was limited with only 1.4% of encounters requiring additional evaluation by the provider team during a sepsis huddle.

Limitations

There are multiple limitations to this study. First, this is a single center study, thus findings may not be generalizable to other institutions given our unique patient population and the small sample size included in this work. Second, this CDS intervention was just one component of a large quality initiative to improve sepsis care within the PICU which may have contributed to the improvement in the patient-orientated quality metric as we assessed. Due to this significant limitation, we did not include additional clinical outcomes in terms of decrease in missed or delayed sepsis recognition or improvement in patient outcomes. In the future, we hope to introduce this simple CDS intervention into other pediatric ICUs with robust sepsis quality improvement protocols in place to see if it is additive to basic educational and nurse-based interventions. Third, due to rarity of pediatric severe sepsis, prevalence reported 7 to 8%, and pediatric sepsis mortality, 14.4% for severe sepsis cases, 24 33 we were not adequately powered to evaluate patient outcomes related to severe sepsis in this study. Future expansions of this work through the IPSO collaborative will allow us to see the impact on relevant patient outcomes including mortality, length of stay, and hospital charges. Fourth, we utilized a combined definition of sepsis including both cases identified as severe sepsis by the accepted IPSO collaborative definition but also an intention to treat definition including patients who were treated as sepsis positive. This likely overestimates the performance of this sepsis alert to predict sepsis, further supported by the lower test characteristics when evaluating those who met criteria for IPSO severe sepsis. However, our occurrence of sepsis events in this study is 4.5%, which is below the 8% commonly reported in the literature, 24 supporting that we were not significantly overestimating the sepsis prevalence with our intention to treat approach. Since we used an event rate, these were not independent units of evaluation limiting the conclusions on patient level performance.

This CDS intervention provides a feasible targeted solution for a low prevalence high-risk event. It highlights the components of effective CDS, has a reasonable alert burden, and resides within a system of ongoing improvement, all key components for automated alerts in acute care pediatrics. 34 As more complex screening mechanisms are developed for rare- and high-risk events, the intersection with the clinical team and the assessment for truth are of significant importance. We continue to review our missed cases and work to optimize both our automated and nonautomated means of identification to improve pediatric sepsis outcomes.

Conclusion

Using a simple two-stage interruptive alert increased the reliability of our prior workflows for the recognition and management of pediatric sepsis. We demonstrated strong performance characteristics and an improvement in recommended sepsis interventions. Although prediction for severe sepsis was lower, it remains a reasonable and feasible automated CDS tool to add to ongoing pediatric sepsis quality improvement efforts. Future efforts will focus on testing this alert in other PICUs and ongoing work to improve detection of all patients with severe sepsis through automated and manual methods.

Clinical Relevance Statement

An automated CDS intervention to improve sepsis identification is a much-needed tool to improve pediatric sepsis patient outcomes. Findings of this work contribute to the growing research efforts to evaluate the use and impact of health information technology systems, and specifically clinical decision support systems, on patient outcomes.

Multiple Choice Questions

1. What was the goal of the bedside nurse perfusion screen?

   1. To determine if the correct antibiotics were ordered.

   2. To screen patients for altered perfusion associated with sepsis and if positive activate a sepsis huddle.

   3. To improve nursing satisfaction.

   4. To reduce unnecessary antibiotics.

   Correct Answer: The correct answer is option b. Nurses screened patients for signs of altered perfusion which can be a sign of sepsis. If positive, the nurse would then activate a multidisciplinary sepsis huddle to facilitate ordering of necessary diagnostic tests and therapeutic interventions.

2. Define the number needed to screen.

   1. The percentage of alerts that are followed by an outcome within a certain number of hours.

   2. Proportion of actual positives that are correctly identified as such.

   3. Probability that subjects with a negative screening test truly don't have the disease.

   4. The number of patients that it is necessary to further screen (or evaluate) to detect one outcome. It is a direct measure of the cost-efficiency of each alert.

   Correct Answer: The correct answer is option d. The number needed to screen, or number needed to evaluate, is the number of patients that it is necessary to further screen (or evaluate) to detect one outcome. It is a direct measure of the cost-efficiency of each alert. Due to the low prevalence of pediatric sepsis, the use of receiver operator characteristic curve, C -statistic, likelihood ratio, or specificity is not adequate for evaluation. 26

Funding Statement

Funding None.