Performance of a Clinical Decision Support Tool to Identify PICU Patients at High-Risk for Clinical Deterioration

Abstract

OBJECTIVE

To evaluate the translation of a paper high-risk checklist for pediatric intensive care unit (PICU) patients at risk of clinical deterioration to an automated clinical decision support tool.

DESIGN

Retrospective, observational cohort study of an automated clinical decision support tool, the PICU Warning Tool, adapted from a paper checklist to predict clinical deterioration events in PICU patients within 24-hours.

SETTING

Two quaternary care medical-surgical PICUs—The Children’s Hospital of Philadelphia and Cincinnati Children’s Hospital Medical Center.

PATIENTS

The study included all patients admitted from July 1, 2014 through June 30, 2015, the year prior to the initiation of any focused situational awareness work at either institution.

INTERVENTIONS

We replicated the predictions of the real-time PICU Warning Tool by retrospectively querying the institutional data warehouse to identify all patients that would have flagged as high-risk by the PICU Warning Tool for their index deterioration.

MEASUREMENTS

The primary exposure of interest was determination of high-risk status during PICU admission via the PICU Warning Tool. The primary outcome of interest was clinical deterioration event within 24 hours of a positive screen. The date and time of the deterioration event was used as the index time point. We evaluated the sensitivity, specificity, positive and negative predictive value of the performance of the PICU Warning Tool.

MAIN RESULTS

There were 6233 patients evaluated with 233 clinical deterioration events experienced by 154 individual patients. The positive predictive value of the PICU Warning Tool was 7.1% with a number needed to screen of 14 patients for each index clinical deterioration event. The most predictive of the individual criteria were elevated lactic acidosis, high mean airway pressure, and profound acidosis.

CONCLUSIONS

Performance of a clinical decision support translation of a paper-based tool showed inferior test characteristics. Improved feasibility of identification of high-risk patients using automated tools must be balanced with performance.

Graphical Abstract

INTRODUCTION

In children’s hospitals, most cardiopulmonary resuscitation (CPR) events occur in pediatric intensive care units (PICUs).¹ Many children who require CPR die during or shortly after the event, and those who survive are often left with new disabilities.^2–6 Identifying the children most likely to require CPR is important because evidence based interventions currently exist to improve their outcomes. Examples include “just in time” CPR training, team based simulation, and bundles for tracheal intubation.^7–14

Our conceptual framework for identifying and improving outcomes for the PICU patients most likely to require CPR is based on the principles of risk mitigation through situational awareness.¹⁵ Situational awareness is the ability to monitor and recognize cues that increase the awareness of what is happening around you, integrate information to develop a comprehensive picture of the current state, and extrapolate forward to determine if the knowledge obtained will adversely influence the situation immediately or in the near future.^16,17 The improvement of situational awareness within the PICU and development of mitigation plans to prevent CPR events is predicated on the accurate early identification of high-risk patients.

The baseline method used to identify PICU patients most likely to require CPR at the study hospitals was based on clinician intuition alone; patients that were of concern to attending physicians were identified followed by discussion of these patients twice daily at situational awareness huddles. This approach had a poor positive predictive value and a problematic number needed to screen of 37–45 patients for each clinical deterioration event, defined as a cardiac arrest or a code bell activation with response of the unit code team. Number needed to screen (NNS) is analogous to number needed to treat. This number signifies the number of patients labeled as high-risk (potentially leading providers to institute mitigation plans or receive just in time training) for each patient who experienced a clinical deterioration event. As a first step to improve the identification of high-risk PICU patients, we developed a paper checklist of factors with excellent sensitivity and specificity and a NNS of 6 patients for each clinical deterioration event.¹⁸ Despite the accuracy of the paper checklist, we found that a major barrier to implementation was the time needed to manually screen the criteria for each patient.

Due to this implementation barrier, we shifted our focus to adapting the paper checklist into an automated electronic health record (EHR)-based clinical decision support tool, the PICU Warning Tool. Our aim was to examine the retrospective test characteristics of the PICU Warning Tool with a pragmatic approach to the tradeoffs between predictive accuracy and the effort required to implement.

MATERIALS AND METHODS

Study Design

We performed a retrospective observational cohort study to evaluate the performance of a replicated version of the PICU Warning Tool to predict clinical deterioration events in PICU patients within a 24-hour window following a positive high-risk screen. The Institutional Review Boards of CCHMC and CHOP approved this study.

Setting

The study PICUs are tertiary care pediatric medical/surgical ICUs with 55 and 35 beds respectively and combined yearly admissions of over 6000 patients/year with a baseline clinical deterioration rate of 2–3% of all PICU patients. The cardiac surgical ICU and neonatal ICU were excluded.

Selection of the Cohort

The study included all patients admitted to the PICU at The Children’s Hospital of Philadelphia (CHOP) and Cincinnati Children’s Hospital Medical Center (CCHMC) from July 1, 2014 through June 30, 2015, the year prior to the initiation of any focused situational awareness work at either institution. No patients were excluded.

The primary exposure of interest was determination of high-risk status during PICU admission via the PICU Warning Tool. The primary outcome of interest was clinical deterioration event within 24 hours of a positive screen. The date and time of the deterioration event was used as the index time point. Only the initial deterioration event was included for each patient as the goal of the PICU Warning Tool was to establish initial situational awareness for high-risk patients. This also accounted for multiple sampling.

Identification of High-Risk Patients

The primary exposure of interest was high-risk patient status as determined by the automated PICU Warning Tool. The original paper-based checklist, on which the PICU Warning Tool is based, included 15 equally weighted single parameter high-risk patient factors that predicted clinical deterioration events.¹⁸ We began the process with a review of the paper checklist to determine if the items were computable and translatable into an automated clinical decision support tool. Some criteria were excluded because the data were not reliably recorded as discrete elements, or were not documented at all, such as “provider intuition.” While documentation fields could be added to the EHR to capture these missing elements, we chose to build a system without requiring any additional clinical documentation in order to meet the clinical decision support best practice of integration into existing workflows.¹⁹ Due to these limitations, we excluded five of the original criteria and modified others (Table 1). We added a criterion for severe cardiac dysfunction in an attempt to capture an additional high-risk population that was lost due to the excluded criteria; we have further described this process and its adherence to clinical decision support best practices in a prior manuscript.²⁰

Table 1:

Review of original paper checklist criteria and translation to PICU Warning Tool

	Criterion	Definition or adaptation* in PICU Warning Tool, or reason for exclusion
Unchanged	High Mean Airway Pressure	Mean airway pressure sustained greater than 20 cm H20 for at least 2 hours
	Extracorporeal Membrane Oxygenation (ECMO)	Any patient receiving veno-venous or veno-arterial ECMO (up to 24 hours post decannulation)
Adapted	Pulmonary Hypertension	Most recent nitric oxide gas value > 5 parts per million AND   • Active pulmonary hypertension on problem list OR   • Received medication indicating pulmonary hypertension in prior 24 hours
	Use of two vasoactive medications OR high dose single drug	Medication Administration Record within the last 6 hours of:   • Any dose of phenylephrine, dobutamine, or vasopressin   • Any two of the following: dopamine, epinephrine, or norepinephrine   • Epinephrine or norepinephrine with dose>0.1 mcg/kg/min   • Dopamine with dose >5 mcg/kg/min
	Traumatic Brain Injury	• Sustained elevated intracranial pressure > 20 for most recent 2 hours • 3% saline or mannitol bolus in prior 6 hours for high ICP
	PICU Initiated Renal Replacement Therapy	First 5 hours on CRRT and first 5 hours off of CRRT
	Potassium >7.0	Two consecutive potassium laboratory values >7 mmol/L (resulted with the last 24 hours)
	pH<7.1	Most recent pH laboratory value (resulted within the past 24 hours)<7.1
	Lactate>10	Most recent lactate laboratory result >10 mmol/L
Added	Severe Cardiac Dysfunction	• Mixed venous saturation<60% without 2 consecutive values >70% within the last 24 hours OR • Milrinone infusion within the previous 6 hours
Excluded	Stage 1 Hypoplastic Left Heart Repair in Past 24 hours	HLH patients not cared for in the study PICUs
	Recent life-threatening event requiring code team activation or cardiac arrest	No standardized EHR documentation of events
	Intubation/Extubation of known difficult or critical airway	Difficult or critical airway is reliably captured at both institutions, intubation not documented until post procedure.
	Provider Intuition	Concern on the part of an attending physician that a patient is high-risk regardless of criteria met cannot be captured in an automated fashion.
	Anticipated tracheal intubation of known diagnosis of myocarditis	Tracheal intubations are only documented post procedure.

^*Adaptations noted in italics

We replicated the predictions of the real-time PICU Warning Tool by retrospectively querying the institutional data warehouse to identify all patients that would have flagged as high-risk by the PICU Warning Tool, as well as the specific criteria that caused the patient to be identified as high-risk. The EHR was screened hourly for patients meeting high-risk criteria per the established definitions. Patients who met any single criteria were considered high-risk patients.

Identification of Patients who Experienced Clinical Deterioration

The outcome of interest was clinical deterioration event within 24 hours of a positive high-risk screen by the PICU Warning Tool. Clinical deterioration was defined as a cardiac arrest or a code bell activation with response of the unit code team. These events were identified through multiple means including the use of a quality improvement database in place at the time of the study that tracked CPR and respiratory emergency events (CHOP), an electronic log of code button presses with manual chart review (CCHMC), and a review of all code sheets documented during that time period (CHOP, CCHMC).

Data Analysis

We evaluated the sensitivity, specificity, positive and negative predictive value of the overall performance of the PICU Warning Tool to retrospectively identify patients who would experience clinical deterioration within 24 hours of a positive screen. A second sensitivity analysis evaluated the test characteristics of the PICU Warning Tool to predict a clinical deterioration event at any point during a patient’s PICU admission following a positive high-risk flag. Secondary analysis included standard diagnostic test characteristics of individual components of the EHR-based PICU Warning Tool including likelihood ratios, sensitivity, specificity, positive predictive value, and negative predictive value of each component within the combined data set from both institutions and evaluated separately. Categorical variables between institutions were compared using the chi-square test or Fisher’s exact test.

RESULTS

A total of 6233 patients were evaluated between the two centers. There was a total of 233 clinical deterioration events experienced by 154 individual patients (2.5%). At CCHMC, 3.7% of patients (83 out of 2241 total patients) experienced a clinical deterioration event and 21.6% (485 out of 2241 total patients) met high-risk criteria during their PICU stay. At CHOP, 1.8% of patients (71 out of 3992 total patients) experienced clinical deterioration and 9.6% (385 out of 3992 total patients) met high-risk criteria during their PICU stay (Figure 1). Both the number of clinical deterioration events and the percent of patients meeting high-risk criteria were significantly different at the two institutions (p<0.0001 for both).

Figure 1:

Cohort Identification

The percent of patients meeting individual high-risk criteria was also different between centers for three of the criteria—pulmonary hypertension, high mean airway pressure, and profound acidosis (Table 2). The highest frequency criteria was vasoactive shock at both centers followed by profound acidosis and high mean airway pressure.

Table 2:

Percent of patients meeting individual high-risk criteria by center

Criteria	% of High Risk Patients who Met Criteria (N)*)	% of CCHMC High Risk Patients who Met Criteria (N)*)	% of CHOP High Risk Patients who Met Criteria (N)*)	P-Value
ECMO	1.1% (10)	0.6% (3)	1.8% (7)	0.10
Hyperkalemia	2.6% (23)	1.9% (9)	3.6% (14)	0.12
Renal Replacement Therapy	4.4% (38)	3.9% (19)	4.9% (19)	0.47
Pulmonary Hypertension	4.7% (41)	3.3% (16)	6.5% (25)	0.03
Lactic Acidosis	5.3% (46)	4.1% (20)	6.8% (26)	0.08
Intracranial Hypertension	12.6% (110)	12.0% (58)	14.5% (56)	0.28
Severe Cardiac Dysfunction	25.7% (224)	26.4% (128)	24.9% (96)	0.62
High Mean Airway Pressure	26.6% (231)	21.0% (102)	33.5% (129)	<0.01
Profound Acidosis	30.0% (261)	34.0% (165)	24.9% (96)	0.004
Vasoactive Shock	48.2% (419)	50.3% (244)	45.5% (175)	0.16

^*Total number of high risk patients=870, CCHMC high risk patients=485, CHOP high risk patients=385

The overall positive predictive value of the EHR-based tool was 7.1% (95% CI: 5.9–8.6%) with a NNS of 14 patients (95% CI: 12–17) for each index clinical deterioration event. The tool performed with a PPV of 8.0% and a NNS of 13 at CCHMC as compared to CHOP with a PPV of 6.0% and a NNS of 17. All of these test characteristics were inferior to the original paper-based tool with a positive predictive value of 17.3% and an NNS of 6 (Table 3).

Table 3:

Test Characteristics of the electronic PICU Warning Tool at each center and overall as compared with the original paper based tool.

Test Characteristics	CHOP	CCHMC	Overall	Paper Checklist
Sensitivity (95% CI)	32.4% (21.8–44.6%)	47.0% (35.6–58.3%)	40.3% (32.5–48.5%)	100.0%
Specificity (95% CI)	90.8% (89.8–91.6%)	79.3% (77.6–81.0%)	86.7% (85.8–87.6%)	97.7%
PPV (95% CI)	6.0% (4.3–8.3%)	8.0% (6.4–10.0%)	7.1% (5.9–8.6%)	17.3%
NPV (95% CI)	98.7% (98.4–98.9%)	97.5% (97.0–98.0%)	98.3% (98.1–98.5%)	100%
NNS (95% CI)	17 (12–23)	13 (10–16)	14 (12–17)	6

The predictive value of the individual criteria varied. The most predictive of the individual criteria was elevated lactic acidosis with a positive likelihood ratio (LR) of 6 (95% CI: 2.6–13.9), high mean airway pressure had a positive LR of 4.8 (95% CI: 3.3–7.0) and profound acidosis had a positive LR of 3.6 (95% CI: 2.4–5.5) (Figure 2). The least predictive were ECMO and hyperkalemia which did not accurately predict any clinical deterioration events followed by intracranial hypertension with a positive LR of 1 (95% CI: 0.2–2.9). Pulmonary hypertension only predicted clinical deterioration at CHOP and renal replacement therapy only predicted clinical deterioration events at CCHMC.

Figure 2:

Positive likelihood ratios with confidence limits for individual criteria and overall for prediction.

Sensitivity Analysis

To predict clinical deterioration at any point following a positive screen during the patient’s PICU admission, the PICU Warning Tool had improved test characteristics with a sensitivity of 60.4% (95% CI: 52.2% to 68.2%), specificity of 87.2% (95% CI: 86.3–88.0%), positive predictive value 10.7% (9.4%−12.1%), negative predictive value of 98.9% (95% CI:98.6–99.1%), and a number needed to screen of 10 (95% CI: 9–11).

DISCUSSION

Using data from two institutions, we evaluated the performance of an automated clinical decision support tool, the PICU Warning Tool, derived from a previously published single center paper checklist. The key findings of this study were that, with adaptation from paper to automated to ease screening, the sensitivity decreased from 100% to 40% to predict clinical deterioration within 24 hours of a positive high-risk screen, and the NNS increased from 6 to 14. As a practical example, using the PICU Warning Tool rather than the paper checklist, a 30-bed PICU with yearly admissions of 2000 patients would have on average five high-risk patients per week requiring mitigation planning and just in time training. However, this would only train the teams of 40% of patients who would experience clinical deterioration within 24 hours and 60% of patients who would experience clinical deterioration during their PICU admission. Using the paper checklist, this same PICU would have on average one high-risk patient per week requiring just in time training and would train 100% of the teams caring for patients who experience clinical deterioration. While this example may overestimate the performance of the paper checklist in a real-life setting, as it was only studied over a three-month period at a single center, it does raise important concerns about the often unstudied conversion from a paper based clinical decision support tool to an automated one.

To summarize, in adapting the tool from paper to electronic, we eliminated the workload burden of manual screening but markedly reduced the tool’s sensitivity and more than doubled the NNS. Depending on the recommended action associated with screening positive, this could increase providers’ workload burden of developing mitigation plans or delivering just in time training, limiting its feasibility and acceptability. More standard situational awareness improvements around structured communication tools²¹, standardized handoffs²², and huddles^16,23 are likely feasible for most centers with a NNS of 14 patients for each clinical deterioration event. However, more intensive training centered around individual at-risk patients^5,8,24 and team based simulations^12–14 may not be feasible at this NNS. It is important to note that although we have focused on individual level patient identification and mitigation, prior work demonstrates that with improved situational awareness and a proactive approach to identify high-risk patients, improvements can be seen in prevention of high-risk events at a systems level.^23,25–27

This difference in test characteristics identified here demonstrates the importance of evaluating the translation from paper to automated clinical decision support tools. Optimizing the human-automation interaction may lead to improved outcomes in this context. Prior literature demonstrates that automation does not replace the human activity but changes it in planned and unplanned ways.²⁸ To the extent to which the clinical decision support automation provides the right information at the right time, it can decrease the team workload and improve the team situational awareness.²⁹ While the performance of the automated tool may be inferior to that of the paper tool, our use of clinical decision support best practices^19,30,31 and our implementation plan for effective automation design³², may lead to improved patient outcomes despite the higher NNS.

We speculate that the decrease in sensitivity and increase in NNS is likely related to the removal of components that are known to be predictive including high-risk intubations³³ and prior clinical deterioration events.³⁴ In addition, provider intuition (i.e. identification of high-risk patients by physician judgement) has been shown to be more predictive than early warning scores alone when completed by attendings.³⁵ As the original evaluation of the paper checklist did not include evaluation of the individual criteria, it was not possible for us to know a priori the impact of removing these elements. It is reasonable to study the addition of these criteria to the future prospective implementation of this tool to see if we are able to improve the sensitivity of the PICU Warning Tool using a combined automated screening component and a manual entry component from those criteria identified during huddle.

This work has several limitations. First, the retrospective evaluation of this automated clinical decision support tool was completed at only two centers, and the frequency of deterioration events at these centers was low. External validation in a larger number of pediatric centers is important prior to widespread adoption. Second, this data was collected retrospectively. Although designed to mimic the real-time queries of the prospective tool, there may have been missed indications due to documentation errors such as incorrect documentation of a patient’s mean airway pressure or diagnosis in the problem list. Lastly, we only evaluate the first clinical deterioration event of each patient. It is possible the predictability of the tool changes with subsequent events.

We will next implement and study the PICU Warning Tool as a prospective high-risk patient identifier on its impact on clinical deterioration rate. To improve the performance of the tool, we plan to add the missing criteria through manual identification to investigate if the sensitivity of the PICU Warning Tool increases using combined automated screening and manual entry of criteria identified twice daily during safety huddle. As we decrease the prevalence of clinical deterioration events, the positive predictive value will fall and the NNS of the tool will rise. It will therefore be imperative that we track additional outcomes including situational awareness among team members, use of mitigation plans, and quality of resuscitation response in addition to predictive ability.

Conclusions

An automated clinical decision support translation of a paper checklist for PICU patients at risk for clinical deterioration demonstrates reduced accuracy in prediction with a NNS of 14 rather than 6 despite improved feasibility. We speculate that improved feasibility of identification of high-risk patients using automated EHR warning tools may allow for risk mitigation and just in time preparation and that test characteristics may improve with incorporation of more provider and procedural concerns.