Structured Abstract
Background and Objectives
Delays in septic shock diagnosis cause preventable mortality in children. Evidence is limited around early recognition strategies. The hypothesis was that clinical decision support (CDS) based on machine-learning predictive models would increase the proportion of children receiving septic shock treatment prior to shock onset.
Methods
CDS was implemented in a prospective, stepped-wedge, cluster randomized trial in four pediatric Emergency Department (EDs) over five 10-week periods. The CDS used models identifying children who did not yet have shock but were predicted to be at high risk, based on EHR data at arrival and after two hours. Providers received CDS; effectiveness was evaluated in patients 60 days-18 years with concern for sepsis. The primary outcome was antibiotic and bolus within one hour of sepsis suspicion. Secondary outcomes were time to antibiotic, hypotensive septic shock. Implementation outcomes were evaluated in qualitative interviews.
Results
Of 200,354 ED encounters from 3/16/22–3/1/23, 1331 encounters met inclusion criteria (979 intervention, 352 control arms). Antibiotic and bolus within one hour occurred in 39.0% patients in the intervention arm versus 38.9% in the control arm (aOR: 1.07 [0.61 – 1.88]). There was no difference in outcomes of shock (aOR: 1.12 [0.53 – 2.46]) or antibiotic timeliness (aHR: 0.85 [0.63 – 1.16]). Providers reported the CDS felt valuable and unobtrusive (adoption); six months after the trial, EDs continued to use the CDS (maintenance).
Conclusions
Implementing predictive CDS that infrequently alerted was feasible and acceptable. It did not change the proportion of patients with suspected sepsis who progressed to hypotensive shock.
Article Summary:
In a cluster-randomized trial in four Emergency Departments, this study implemented decision support to predict septic shock and measured its effect on treatment and outcomes.
Introduction
Septic shock is a leading cause of death and morbidity in children, with mortality of 8–15%, and declines in quality of life for 35% of survivors.1–3 Septic shock is a type of sepsis, or critical infection, in which hypotension and cardiovascular dysfunction require time-sensitive resuscitation that depends upon timely diagnosis. Treatment delivered in a delayed fashion is less effective; for each hour of unrecognized shock the odds of death more than double.4
Although advances have been made in sepsis treatment, improving early diagnosis remains elusive. The 2020 Surviving Sepsis Campaign’s pediatric guidelines stated that “high-quality trials on pediatric sepsis recognition are lacking, and data are not sufficient to suggest any particular screening tool.”5 Existing studies surrounding sepsis diagnosis in children consist of pre-/post-implementation comparisons, and their effectiveness in changing care quality and patient outcome have had mixed results.6,7 There have not been prospective controlled trials of sepsis diagnostic tools in children older than neonates.6
This study addressed this gap in knowledge about pediatric septic shock prediction tools. A Clinical Decision Support (CDS) alert was developed based on previously validated early diagnostic models that leverage clinical data in the Electronic Health Record (EHR) to predict septic shock in children in the emergency setting.7, 8 The objective of this trial was to test whether implementing CDS for prediction of septic shock in four Emergency Departments (EDs) would increase delivery of guideline-concordant septic shock care. Implementation outcomes were evaluated using the Practical Implementation Sustainability Model (PRISM) and Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) frameworks.9, 10
Methods
Design
This was a prospective, stepped-wedge cluster randomized trial to test the implementation of a Clinical Decision Support (CDS) tool for prediction of septic shock in four EDs. The site was the unit of randomization. CDS was activated silently at the start of the trial, to track CDS triggering, without being seen by providers. During each 10-week period, one site crossed over to the intervention of live (visible to providers) CDS. All ED providers (physicians, nurse practitioners and physician assistants) received the CDS intervention. Effectiveness outcomes were measured at the patient level. Adoption was assessed in qualitative interviews of providers who had seen the CDS. Maintenance was assessed at the site level.
This trial was registered on CT.gov (NCT05065333) prior to starting enrollment, and approved by the Colorado Multiple Institutions Review Board (#21–2916) with a waiver of consent. A Data Safety Monitoring Committee conducted an interim review for safety after 6 months. The Consolidated Standards of Reporting Trials extension for stepped wedge cluster randomized trials reporting guideline was followed.11
Setting
This trial was conducted in four EDs within a pediatric healthcare network in Colorado, sharing a single EHR. The EDs saw >200,000 ED encounters annually, with site volumes from 35,000 to 75,000. All sites had inpatient pediatric capacity; two sites had a Pediatric Intensive Care Unit (PICU) on site. All EDs were staffed by Pediatric Emergency Physicians and pediatric nurses, with pediatricians, nurse practitioners and physician assistants. Trainees worked at all sites.
Sepsis quality improvement programs had been active for >10 years. There was no screening tool for sepsis; instead, any physician, nurse, or other clinician could initiate sepsis evaluations at the first suspicion for sepsis. Protocols advised sepsis evaluations for confirmed sepsis/organ dysfunction, central lines or immunosuppression, and any concerning history or exam (Supplemental Figure 1). An orderset or page was used to start the sepsis evaluation consisting of expedited blood culture, blood count, lactate and intravenous access, and, at the provider’s discretion, recommended antibiotics, additional labs and intravenous fluid in the order set.12, 13
Dates
The study period was March 16, 2022 to March 1, 2023.
Participants
Providers were the recipients of the CDS interventions. Effectiveness was evaluated in ED patients who were >60 days and <18 years, and had a sepsis evaluation initiated. Exclusion criteria were leaving against medical advice, and transfer to a non-included hospital.
Interventions
This study team had previously developed and validated early diagnostic models that leveraged clinical data in the Electronic Health Record (EHR) to predict septic shock.7, 8 These models used EHR data at ED arrival and two hours later to produce an estimated risk for hypotensive septic shock within 24 hours, in patients who did not yet have shock. The arrival model had 10 variables, including vital signs and patient characteristics (arrival modality, oncologic comorbidity, central line). The two hour model had 19 variables, comprised of vital signs, patient characteristics and laboratory results (Figure 1).
Figure 1.
Clinical workflow and screenshot of the clinical decision support tool, which triggered based on predictive models of septic shock.7, 8
The predictive models were designed to identify patients at high risk for shock among patients in whom clinicians initially had suspicion for sepsis, an approach known as “assistive monitoring,” rather than a ‘sniffer.’ This approach was chosen because of limitations described in prior CDS trials, including high false positive rates and alert fatigue.14, 15 The models were built to assess risk in real time in ED patients undergoing a sepsis evaluation, using the Cognitive Computing Platform of the EHR, Epic (Verona, WI). Patients who had hypotension triggered an existing ‘hypotension alert’ CDS (Figure 1); hypotensive patients did not trigger the predictive CDS because they already had shock. If risk exceeded thresholds previously developed to provide 90% sensitivity, a CDS alert, the “High risk for septic shock” Best Practice Alert would be shown to ED providers (Figure 1). If a sepsis evaluation started after 2 hours, the CDS would calculate risk using data available until 2 hours. The high-risk CDS fired once for each provider until they dismissed it in their EHR, but did not recalculate or re-fire once dismissed.
Crossover happened on the first day of each 10-week period, with informaticians activating the CDS at that site. ED providers were educated about the CDS prior to the study start in meetings; an email was sent to all providers the night before activation at their site, and the CDS was designed to be intuitively understood by providers in the moment when triggered.
Outcome Measures
The primary effectiveness outcome was receipt of septic shock care concordant with the 2020 Surviving Sepsis Campaign guidelines for pediatric shock.5 Concordant care was a binary outcome, defined as intravenous antibiotics and intravenous fluid bolus administration within 60 minutes of suspicion for sepsis. Sepsis suspicion was the earlier of: sepsis page, sepsis orderset, or intravenous antibiotic order. Secondary outcomes were time to first intravenous antibiotics, and septic shock within 24 hours after ED arrival. Time to antibiotics was a time-to-event outcome measured in minutes from the time of sepsis suspicion to the start of intravenous antibiotic treatment. Shock was defined as systolic hypotension and vasoactive or ≥30 ml/kg (≥1500 mL for patients weighing ≥50 kg) intravenous bolus fluid administration.
A balancing outcome measure was the number of patients receiving intravenous antibiotics during ED care. 30-day in-hospital mortality was measured for safety monitoring, but the trial was not powered for this outcome. In addition to effectiveness outcomes, adoption, implementation, and maintenance outcomes were assessed using the RE-AIM framework.9
Sample Size
The unit of analysis for this study was included ED encounters. The study period necessary to achieve power was based on historic volumes, using the pwr16 package for R. Based on existing patterns, we estimated guideline-concordant septic shock care would be delivered to 25% of encounters with suspected sepsis.17 In the literature, pediatric sepsis quality interventions increased concordant early care to 50%.18–20 With an estimated effect of the alert of increasing concordant care from 25% to 50%, using a significance of 0.05, we anticipated >90% power with a sample size of 2054, expected in 50 weeks.
The unit of randomization was the ED site, with a randomization scheme to optimize power.21, 22 Using a simulation based on historic volumes, power was calculated for each possible order; the final order was randomly selected from a subset of the 12 best orderings. All sites agreed to participate prior to being notified of their allocated order.
Blinding
The outcome was assessed from existing clinical data in the secure EHR Data Universe. Data analysis was classified following pre-specified outcome definitions by the statistical analyst who was not blinded to arm allocation. ED providers were not blind to the arm to which their site was allocated due to the nature of the intervention.
Statistical Methods
To test the primary hypothesis, an odds ratio was calculated and was considered significant if the 95% confidence interval did not contain 1.0. Primary and secondary outcomes were analyzed under a generalized linear model framework using fixed effects to adjust for stepped-wedge design.23 A logit link and Bernoulli distribution were assumed for binary outcomes while a proportional hazard model was used to analyze time to receipt of antibiotics. All models included an effect for intervention, categorical indicators of transition time and site. Up to five additional pre-specified covariates were included if they were imbalanced between the two arms of the trial. Covariates considered had previously been described as affecting pediatric sepsis care quality and outcomes: patient race/ethnicity (a social determinant that has been associated with delays in septic shock treatment), arrival modality, surge period, triage level, oncological comorbidity, central line, and hospitalization in the prior year. Surge period was when the hospital ran a crisis command center to manage historic respiratory virus surge volumes (10/26/22–12/9/22). An absolute standardized mean difference of 0.1, or greater, was considered an indication of imbalance. A univariable Fisher’s exact test was applied to associations with 30-day mortality due to the rareness of this outcome. Significance of effects was assessed at the 5% level, with all tests structured as two-sided hypotheses. SAS software version 9.4 (SAS Institute, Cary, NC) and R version 4 (R Foundation for Statistical Computing, Vienna, Austria) was used for statistical analysis. A subgroup analysis was pre-planned for encounters identified as high-risk by triggering the arrival or two-hour CDS.
Qualitative Methods
We identified providers who saw the CDS during the trial. We purposively sampled all sites and provider groups. Interviews were conducted virtually using audio/video communication (Zoom, San Jose, CA). After verbal consent, providers participated in semi-structured interviews with a qualitative analyst. Interviews were audio recorded, professionally transcribed verbatim, and spot-checked for accuracy. The interview guide and coding domains were developed deductively using the PRISM implementation framework.24 Transcripts were iteratively reviewed to refine domains and achieve reconciliation; 20% were double-coded. Transcripts were coded in Atlas.ti, version 8 using deductive domains to capture key points. Matrix analysis was used to summarize and analyze coded data across interviews, by domain. Interviews were conducted until thematic saturation within each site, as well as among sites, was reached. The analytic team made a determination that thematic saturation had been reached when there was no new content being identified for at least three interviews in a row, with repetitious interviews.25 After themes were identified, member checking was performed in two semi-structured focus groups that were conducted by a qualitative analyst, recorded, professionally transcribed and analyzed.
Results
Four ED sites received the CDS intervention in a randomized order, including 1331 encounters (979 intervention; 352 control) meeting inclusion criteria [Figure 2]. Characteristics of encounters and illness severity was similar across arms, with 207 (21.1%) encounters resulting in ICU admission in intervention and 68 (19.3%) control arms [Table 1].
Figure 2.
Study flow and structure of the study
2a) Cluster-Randomized patient inclusion diagram
2b) Included ED encounters by cluster. Red cells represent encounters during a usual care (control) arm, blue cells represent encounters when the site was in the CDS intervention arm.
Table 1.
Characteristics of included ED patient encounters with concern for sepsis, by arm. P-values calculated by chi-square or Kruskal-Wallis tests.
| Characteristic | Clinical Decision Support Arm (n=979) | Usual Care Arm (n=352) | p value |
|---|---|---|---|
| Sex | 0.79 | ||
| Male, No. (%) | 501 (51.2) | 183 (52.0) | |
| Female, No. (%) | 478 (48.8) | 169 (48.0) | |
| Age, years, median [IQR] | 5.6 [8.3] | 5.7 [9.2] | 0.2 |
| Race | |||
| White, No. (%) | 646 (66.0) | 242 (68.8) | <0.001 |
| Unspecified1, No. (%) | 183 (18.7) | 66 (18.8) | |
| Black/African American | 106 (10.8) | 16 (4.5) | |
| More than one Race | 44 (4.5) | 28 (8.0) | |
| Hispanic Ethnicity, No. (%) | 305 (31.2) | 110 (31.3) | 0.97 |
| Oncologic Comorbidity, No. (%) | 177 (18.1) | 76 (21.6) | 0.15 |
| Indwelling Central Line Present, No. (%) | 241 (24.6) | 101 (28.7) | 0.13 |
| Hospitalized last year, No. (%) | 463 (47.3) | 171 (48.6) | 0.68 |
| Arrival via Emergency Medical Services, No. (%) | 128 (13.1) | 36 (10.2) | 0.16 |
| Triage Level 1, No. (%) | 54 (5.5) | 17 (4.8) | 0.4 |
| Triage Level 2, No. (%) | 728 (74.4) | 248 (70.5) | |
| Triage Level 3, No. (%) | 172 (17.6) | 78 (22.2) | |
| Triage Level 4, No. (%) | 24 (2.5) | 9 (2.6) | |
| Triage Level 5, No. (%) | 1 (0.1) | 0 (0) | |
| Hypotension prior to sepsis suspicion, No. (%) | 65 (6.6) | 24 (6.8) | 0.91 |
| Hypotension time after arrival in patients with hypotension prior to sepsis suspicion, minutes, median [IQR] | 13.0 [21.0] | 13.5 [26.0] | 0.69 |
| Lactate, mmol/L, median [IQR] | 1.98 [1.20] | 1.50 [1.30] | 0.43 |
| Encounter During Surge Capacity Time Period,2 No. (%) | 190 (19.4) | 15 (4.3) | <0.001 |
Unspecified includes categories comprising <1% of the study population (Native Hawaiian/Pacific Islander, American Indian/Alaska Native, Asian), and those who registered as “Declined,” “Other,” “Unknown,” reflecting patient race registration practices at time of the study. Race was considered as a covariate in analysis due to literature reporting associations between race as a social determinant of health and care delivery and outcome in sepsis.
Surge period defined by the time when the hospital crisis command center was operating due to historic respiratory virus surge volumes (10/26/22–12/9/22)
Among encounters evaluated by the CDS, only those calculated to be at high risk for septic shock resulted in the CDS seen by the provider. In the intervention arm, 475 (48.5%) triggered the ‘high risk for septic shock’ CDS alert seen by providers; in the control arm, 171 (48.5%) triggered the silent high risk CDS. These 646 triggers occurred in the context 200,354 patient encounters across all four sites during the 50 weeks of the trial, representing 0.3% of ED encounters.
In the intervention arm, 382 (39.0%) encounters received concordant care, the primary outcome, versus 137 (38.9%) in the control arm [Table 2]. Encounters in the intervention arm were not more likely to receive concordant care (aOR: 1.07 [0.61 – 1.88]). All outcomes were adjusted as pre-specified for covariates time epoch and site, and covariates which significantly differed between arms: race, crisis surge period, oncologic comorbidity, triage level. There was no difference in the secondary outcome of shock in the CDS arm compared to control (aOR: 1.12 [0.53 – 2.46]) or antibiotic timeliness (aHR: 0.85 [0.63 – 1.16]) [Table 2]. Time-to-antibiotics differed by site and did not show significant change after crossover to active decision support [Figure 3]. Hospital course outcomes were similar between arms [Supplemental Table 1].
Table 2.
Implementation and Effectiveness Outcomes in a RE-AIM Framework9
| Component | Measure | Results | |||
|---|---|---|---|---|---|
| Reach | Number, characteristics of encounters in which CDS triggers | 1331 encounters CDS triggered 646 encounters with CDS shown to provider due to high risk calculation (Table 1) | |||
| Effectiveness | CDS Intervention Arm (n=979) | Usual Care Control Arm (n=352) | Unadjusted Comparison in CDS vs. Usual Care Arms | Adjusted Comparison * | |
| Primary | Guideline-concordant shock care, No. (%) | 382 (39.0) | 137 (38.9) | 1.00 [0.78 – 1.29] Odds Ratio |
1.07 [0.61 – 1.88] Odds Ratio |
| Secondary | Septic Shock, No. (%) | 124 (12.7) | 39 (11.1) | 1.16 [0.80 – 1.72] Odds Ratio |
1.12 [0.53 – 2.46] Odds Ratio |
| Secondary | Time to antibiotics, minutes, Median [Quartile 1 – Quartile 2] |
47 [28–90] n=703 |
41 [27–74] n=235 |
0.80 [0.64 – 1.02] Hazard Ratio |
0.85 [0.63 – 1.16] Hazard Ratio |
| 30-day mortality, No. (%) | 7 (0.7) | 1 (0.3) | Fisher’s exact p-value 0.69 |
Not calculated due to low numbers | |
| Balancing: received antibiotics, No. (%) | 703 (71.8) | 235 (66.8) | 1.27 [0.97–1.65] Odds Ratio |
1.22 [0.69–2.12] Odds Ratio |
|
| Adoption | Willingness to go live measured in time to go live; days with requests to turn off or failure of CDS in internal EHR monitoring systems | All sites went live at midnight on the scheduled date. No requests to stop CDS, no failure of CDS. | |||
| Adoption | Qualitative data from providers | See Table 3 | |||
| Implementation | Proportion of encounters with recommended action followed after triggering alert (all organ dysfunction labs measured) | 312 (31.9%) in intervention arm; 59 (16.8%) in control | |||
| Maintenance | N (%) of sites choosing to continue to keep the CDS tool active 6 months post-trial | 4 (100%) | |||
Adjusted for time epoch, ED site, and the following covariates significantly differing between arms: race, crisis surge period, oncologic comorbidity, triage level.
Figure 3.
Minutes to antibiotics by site and study period. Dots represent median and whiskers represent upper and lower quartile times for each site during each 10-week study period.
A planned subgroup analysis was performed in encounters which were assessed as high-risk by the model, triggering CDS. Outcomes in this subgroup were not significantly different between arms [Supplemental Tables 2, 3]. In a post-hoc analysis of patients who did not have antibiotics ordered before CDS alerted the clinician, time to antibiotic in 162 patients in the CDS arm was 105 [IQR: 63–163] minutes, and in the 35 patients in the control arm it was 106 [IQR: 57–136] minutes.
Implementation, Adoption, and Maintenance
Organ dysfunction laboratories were measured in 16.8% of encounters in the control arm and 31.9% in the CDS arm. Adoption was assessed in qualitative analysis. Thematic saturation was identified after 33 interviews. Four themes were identified [Table 3]: 1) the CDS felt accurate and valuable because sepsis was a high priority, 2) the most important function of the CDS was its help in prioritizing and making providers pause, 3) alert fatigue was common in the physical and electronic ED environment, however this CDS did not cause fatigue because it alerted rarely and was important, 4) the CDS worked well because it integrated well with existing sepsis care pathways.
Table 3.
Themes with exemplar quotes and corresponding PRISM implementation framework domain indicated.10 Parentheses indicate the study ID number of the subject quoted, as well as role. PEM: Pediatric Emergency Medicine attending physician (n=19), Peds: General Pediatrician (n=3), APP: Advanced Practice Provider (Nurse Practitioner or Physician Assistant, n=3), Trainee: Resident or Fellow (n=8)
|
Theme 1: Alert felt accurate and valuable (Intervention)
|
| Subtheme: It felt accurate |
| It felt like a good reminder to “keep your eyes open” (14, PEM) |
| It did feel accurate. Yeah, they had vital sign abnormalities or risk factors in their history where sepsis was probably our number one diagnosis on our differential, so yeah, it was very accurate. (8, trainee) |
|
Subtheme: Preserved provider autonomy, allowed quick movement past alert in EHR |
| The nice thing is it doesn’t force me to click anything. It’s not a hard stop, which I like, so I don’t have to X out of a window if I need to order my cultures and labs and everything. (17, trainee) |
| Once I knew that the order set was easily accessible through the BPA, it saved me like four extra button clicks and was easier to navigate through Epic… I think just efficiency and pulling the right things up with all of the orders clicked already, the ones that you want, was helpful. (12, trainee) |
| I do appreciate for this particular BPA—I just wanna click the button to be like, I saw this, go away. There are other BPAs that ask more of me, like indicate why you’re not doing this. (11, PEM) |
| What I wouldn’t wanna see is did you start a pressor. Right? You get what I’m saying?… There is a baseline knowledge that is expected, and the way the BPA options are currently laid out is respectful of the fact that we do have some knowledge base. (23, PEM) |
|
Theme 2: Prioritization and Cognitive Pause (Adoption) |
| Subtheme: Most important function was to prioritize high-risk patients, trigger a cognitive pause |
| It certainly got my attention, it got nurses’ attention and definitely resulted in more rapid assessment of the patient, more repeat follow-up. (5, Peds) |
| Probably just gave me more confidence that I was thinking about the right thing. Okay. Perfect. I think that I was already likely doing the labs and interventions that were appropriate. I think I was, oh, look, this agrees that this is a high-risk patient. (24, PEM) |
| It’s something that makes people stop, kind of question, look at vitals again, and then alert me. Or even, when I see it, just making sure that, okay, this is an alert for potentially high—or potentially high risk for sepsis. What labs have I ordered? What was I initially concerned about? Does that involve all the appropriate labs to evaluate for sepsis?” (6, PEM) |
|
Subtheme: Common actions were additional labs, transport, involving PEM |
| I was already doing a lot of what it was requiring or suggesting, but being an APP, I don’t always consult the PEM, and that was a new feature to get more people involved right away. I thought that was nice.” (7, NP) |
| We do not have a PICU. We don’t have ECMO. We don’t have blood. We don’t have a lot of those things. Our job is to diagnose critical illness sooner and think about transport sooner. An alert like this is useful for us so that we are not sitting on a patient. (5, Peds, speaking about a non-tertiary pediatric hospital site) |
| I think it’s good… it does alert other providers to let me know so that I can more quickly come to the patient’s bedside and order the appropriate labs and testing. (16, PEM) |
|
Theme 3: ED, EHR environment: Alert fatigue but not from this BPA (Organizational characteristics) |
| There’s pressure to keep churning through the patients, and so I think the BPA is helpful in saying basically, “Stop, you need to pay attention to this patient and do something.” It’s very easy to get distracted, you have multiple things vying for your attention at a time, and so knowing what to pay closer attention to is helpful. (13, PEM) |
| We are getting much, much busier, things like vital signs and sepsis will get missed more often. I think having alerts that—having a BPA that alerts us to a potentially sick patient is a good thing. (16, PEM) |
| In-your-face alerts that are hard to miss for things that are life-threatening are important for the patient, for me medicolegally, and for the hospital system. (21, PEM) |
| It wasn’t like it popped up and it was like, “Oh, God. Does this have an alert for everything?” It was like, “Oh, yeah. Sepsis is something that you’d want to get notified about.” (14, PA) |
| The BPA triggers in red, and I feel like the alert fatigue—there’s a lot of BPAs or things that trigger in yellow. I often ignore the things that alert in yellow, I have to be honest, but, because this one alerts in red, I feel like, it seems more important, so I don’t think people will ignore it (19, PEM) |
| There are other alerts—what I have found is the alerts that I have most fatigue for are the ones that I don’t trust. This one, I’m not getting all the time, and so it’s not giving me that much fatigue. (20, PEM) |
|
Theme 4: Interaction with existing sepsis systems (Implementation Infrastructure) |
| I think the whole culture of “is this sepsis” has really taken flight, and I think a lot of people—you know, there’s tons of buy-in to that. (4, PEM) |
| The policy of how we do Sepsis Yellows and Sepsis Stats as an institution probably makes it more useful… if we had seen the BPA, it’s easy enough to call a Sepsis Yellow. There are order sets in place for us to initiate work up, and then it kinda just like flows into that. (8, trainee) |
Abbreviations and local terms used by participants: BPA: Best Practice Alert, the name for a Clinical Decision Support tool in Epic; PICU: Pediatric Intensive Care Unit; ECMO: Extra-corporeal membranous oxygenation; FYI: For Your Information, a type of alert in Epic; Sepsis Yellow and Stat: local names for sepsis pathways
Six months after trial completion, 4/4 sites chose to keep the clinical decision support active, the maintenance measure.
Discussion
In this stepped wedge cluster randomized trial, predictive decision support did not change outcomes of guideline-concordant care, shock, or time to antibiotics. The CDS studied alerted ED providers at the time that they first suspected sepsis to patients at the highest risk for shock. There are no prior prospective controlled trials of CDS in pediatric sepsis in the ED. These findings are similar to prior trials of CDS for sepsis diagnosis in adults in the ED, which have not demonstrated improved patient outcomes.14, 15, 26, 27 CDS for sepsis in adults has demonstrated modest changes in process measures, with one trial in adult inpatients changing outcomes.28, 29
These results were seen despite the goal of the predictive model to address limitations of past trials of decision support in sepsis. Prior trials of ‘sniffer’ decision support, which sought to find sepsis before providers, described alerts coming to providers after they had already made the decision to treat sepsis.30 This model and CDS were designed to provide additional support to differentiate patients after providers were concerned but when they had not finalized treatment and disposition decisions. A second problem described previously was sepsis alerts with a high false positive rate.14 Sepsis alerts historically have been designed to be highly sensitive, due to the potentially fatal diagnosis, but models have not been able to produce high sensitivity without producing low positive predictive values (PPV), from 5–10%. This model had a PPV of 20% in a retrospective validation, however in the prospective trial, the PPV was only 12% (Supplemental Table 1), which may have contributed to the results.
The CDS was attempting to accomplish a difficult task – to give antibiotics and fluids more quickly to patients who needed them, while not over-treating or increasing unnecessary antibiotic use. While Surviving Sepsis recommends fluid and antibiotic administration within one hour of septic shock, in this study, the outcome measure was administration within one hour of suspicion for sepsis, in patients who did not yet have shock, a more challenging measure than Surviving Sepsis recommendations.5 The CDS suggested actions but did not force them. While this supported provider autonomy and individualized patient decisions, and is aligned with guidance for use of predictive models in clinical care, it made it less likely to cause immediate changes in clinical management.31 In addition, this trial was conducted at sites with sepsis quality improvement in place for nearly ten years at the start of the trial.12 In these high-performing settings with pre-existing timely care and low mortality baseline, it may have been difficult to detect an effect of CDS. Future research is needed to test whether CDS for pediatric sepsis might be more effective in different settings such as general EDs, or with a different population or trigger point in the workflow.
Some objectives of the CDS design were realized. It minimized alert fatigue in adoption outcomes, and triggered in <1% of all ED patients. Organ dysfunction laboratories were measured more often in the CDS arm [Table 3]. There are not accepted standards for how often to check laboratories in children with infection and concern for sepsis. This modest increase aligned with the goals of this CDS to prompt additional close, individually tailored evaluation of patients, but did not improve outcome measures. The proportion of antibiotics administered was a balancing measure; no significant change in antibiotic use was aligned with the alert’s design.
Observational studies of frequently-alerting CDS in pediatric sepsis, have described no effects on process or outcome measures.32 The effectiveness of such CDS has not been tested in prospective trials with control groups, highlighting the importance of trials such as this one.
Limitations included that the study sites were all connected to a pediatric health care system with mature sepsis quality systems, limiting generalizability. The projected sample size was not achieved during the designated study period, increasing the risk of a Type 2 error for effectiveness outcomes. The trial was developed for prior definitions of shock, which differ from the newer Phoenix criteria for septic shock.33
In cluster-randomized trials, demographics of each arm commonly differ, because sites contribute patients to each arm based on order of randomization. We observed differences between arms in race and proportion of patients enrolled during the surge period; these variables were included as covariates when calculating adjusted odds of the outcome.
Median time to antibiotics by site, shown in Figure 3, suggested that contextual differences among sites influenced timeliness of antibiotics more strongly than decision support. A full exploration of contextual factors was not within the scope of this trial, but differences in ED nursing and pharmacy staffing may have influenced site-level differences. Prior studies have shown associations between unit census and time to antibiotics in sepsis, along with other contextual factors, including capacity strain, cognitive, systemic biases.34, 35 CDS has potential to mitigate these factors, but did not do so in this study.
Providers reported that the predictive septic shock CDS felt useful in risk-stratifying in a busy ED, and did not cause alert fatigue. Participants felt that well-designed decision support could mitigate some challenges to diagnosing and treating septic shock in the ED.
Conclusion
This trial demonstrated feasibility of implementing CDS based on machine learning predictive models in septic shock. In the setting of a mature sepsis program, there was no effect of this CDS on the proportion of patients who received guideline-concordant care. The CDS and study were designed to test whether predictive CDS could improve timely care in those who needed it, without increasing resource utilization in patients who did not. Providers reacted favorably to the CDS which did not mandate specific clinical treatments; they felt that it served as a useful backup. The EDs in this study chose to continue to use the CDS after the study period concluded. Future research building on these successful features of the CDS implementation should test whether modifications to the predictive models, timing, and setting can improve the effectiveness results.
Supplementary Material
Supplemental Figure 1. Children’s Hospital Colorado ED Sepsis Pathway13
What’s Known on This Subject:
Delays in septic shock diagnosis cause preventable morbidity and mortality in children. Models and machine learning have been proposed solutions, but prospective trials of predictive decision support in pediatric septic shock are lacking, and alert fatigue has been a concern.
What this Study Adds:
Implementation of decision support based on machine learning models to predict septic shock was feasible. It did not change the proportion of patients with suspected sepsis who progressed to hypotensive shock in Emergency Departments with pre-existing high-quality sepsis care.
Acknowledgments:
The authors wish to acknowledge the late Diane Fairclough, DrPH, Professor of Biostatistics in the Colorado School of Public Health, for her invaluable mentorship and contributions to the design of this study from its inception. The authors gratefully acknowledge the members of the Data Safety Monitoring Committee, Darcy Thompson, MD MPH, Joseph Grubenhoff, MD MSCS, and Jan Leonard, MSPH.
Funding:
Support for this work was provided by the Agency for Healthcare Research and Quality K08 HS025696 (PI: Scott). AHRQ did not have a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.
Abbreviations:
- BPA
Best Practice Alert, the name for a Clinical Decision Support tool in Epic
- CDS
Clinical Decision Support
- ECMO
Extra-corporeal membranous oxygenation
- FYI
For Your Information, a type of alert in Epic
- ED
Emergency Department
- EHR
Electronic Health Record
- PICU
Pediatric Intensive Care Unit
- PRISM
Practical Implementation Sustainability Model
- RE-AIM
Reach Effectiveness Adoption Implementation Maintenance
Footnotes
Conflicts of Interest Disclosure: The authors have no conflicts of interest relevant to this article to disclose.
Clinical Trials Registration: ClinicalTrials.gov NCT05065333 https://clinicaltrials.gov/study/NCT05065333
Data Sharing Statement:
Deidentified individual participant data (including data dictionaries) will be made available, in addition to study protocols, the statistical analysis plan and code. The data will be made available upon publication to researchers who provide a methodologically sound proposal for use in achieving the goals of the approved proposal, upon approval from the Colorado Multiple Institutional Review Board. Proposals should be submitted to halden.scott@cuanschutz.edu.
References
- 1.Tan B, Wong JJ, Sultana R, et al. Global Case-Fatality Rates in Pediatric Severe Sepsis and Septic Shock: A Systematic Review and Meta-analysis. JAMA Pediatr. 2019;173(4):352–362. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Zimmerman JJ, Banks R, Berg RA, et al. Trajectory of Mortality and Health-Related Quality of Life Morbidity Following Community-Acquired Pediatric Septic Shock. Critical care medicine. 2020;48(3):329–337. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Hartman ME, Linde-Zwirble WT, Angus DC, et al. Trends in the epidemiology of pediatric severe sepsis. Pediatric critical care medicine : a journal of the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. 2013;14(7):686–693. [DOI] [PubMed] [Google Scholar]
- 4.Carcillo JA, Kuch BA, Han YY, et al. Mortality and functional morbidity after use of PALS/APLS by community physicians. Pediatrics. 2009;124(2):500–508. [DOI] [PubMed] [Google Scholar]
- 5.Weiss SL, Peters MJ, Alhazzani W, et al. Surviving Sepsis Campaign International Guidelines for the Management of Septic Shock and Sepsis-Associated Organ Dysfunction in Children. Pediatric critical care medicine : a journal of the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. 2020;21(2):e52–e106. [DOI] [PubMed] [Google Scholar]
- 6.Cherian J, Segal J, Sharma R, et al. Patient Safety Practices Focused on Sepsis Prediction and Recognition: Rapid Review. Making Healthcare Safer IV: A Continuous Updating of Patient Safety Harms and Practices. Rockville (MD)2023. [PubMed] [Google Scholar]
- 7.Scott HF, Colborn KL, Sevick CJ, et al. Development and Validation of a Model to Predict Pediatric Septic Shock Using Data Known 2 Hours After Hospital Arrival. Pediatric critical care medicine : a journal of the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. 2021;22(1):16–26. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Scott HF, Colborn KL, Sevick CJ, et al. Development and Validation of a Predictive Model of the Risk of Pediatric Septic Shock Using Data Known at the Time of Hospital Arrival. J Pediatr. 2020;217:145–151 e146. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American journal of public health. 1999;89(9):1322–1327. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008;34(4):228–243. [DOI] [PubMed] [Google Scholar]
- 11.Hemming K, Taljaard M, McKenzie JE, et al. Reporting of stepped wedge cluster randomised trials: extension of the CONSORT 2010 statement with explanation and elaboration. Bmj. 2018;363:k1614. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Scott HF, Kempe A, Deakyne Davies SJ, et al. Managing Diagnostic Uncertainty in Pediatric Sepsis Quality Improvement with a Two-Tiered Approach. Pediatr Qual Saf. 2020;5(1):e244. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Lockwood JM, Scott H, Wathen B, et al. Sepsis Pathway: Initial Management. https://www.childrenscolorado.org/globalassets/healthcare-professionals/clinical-pathways/sepsis.pdf. Published 2021. Updated 1/5/2021. Accessed 12/12/2024. [Google Scholar]
- 14.Austrian JS, Jamin CT, Doty GR, et al. Impact of an emergency department electronic sepsis surveillance system on patient mortality and length of stay. Journal of the American Medical Informatics Association : JAMIA. 2018;25(5):523–529. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Semler MW, Weavind L, Hooper MH, et al. An Electronic Tool for the Evaluation and Treatment of Sepsis in the ICU: A Randomized Controlled Trial. Critical care medicine. 2015;43(8):1595–1602. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Champely S pwr: Basic Functions for Power Analysis. R package version 13–0. https://CRAN.R-project.org/package=pwr. Published 2020. [Google Scholar]
- 17.Greenwald E, Olds E, Leonard J, et al. Pediatric Sepsis in Community Emergency Care Settings: Guideline Concordance and Outcomes. Pediatric emergency care. 2021;37(12):e1571–e1577. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Paul R, Melendez E, Stack A, et al. Improving adherence to PALS septic shock guidelines. Pediatrics. 2014;133(5):e1358–1366. [DOI] [PubMed] [Google Scholar]
- 19.Larsen GY, Mecham N, Greenberg R. An emergency department septic shock protocol and care guideline for children initiated at triage. Pediatrics. 2011;127(6):e1585–1592. [DOI] [PubMed] [Google Scholar]
- 20.Cruz AT, Perry AM, Williams EA, et al. Implementation of goal-directed therapy for children with suspected sepsis in the emergency department. Pediatrics. 2011;127(3):e758–766. [DOI] [PubMed] [Google Scholar]
- 21.Harrison LJ, Wang R. Power calculation for analyses of cross-sectional stepped-wedge cluster randomized trials with binary outcomes via generalized estimating equations. Statistics in medicine. 2021;40(29):6674–6688. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials. 2007;28(2):182–191. [DOI] [PubMed] [Google Scholar]
- 23.McNeish D, Kelley K. Fixed effects models versus mixed effects models for clustered data: Reviewing the approaches, disentangling the differences, and making recommendations. Psychol Methods. 2019;24(1):20–35. [DOI] [PubMed] [Google Scholar]
- 24.Trinkley KE, Kahn MG, Bennett TD, et al. Integrating the Practical Robust Implementation and Sustainability Model With Best Practices in Clinical Decision Support Design: Implementation Science Approach. J Med Internet Res. 2020;22(10):e19676. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Hennink MM, Kaiser BN, Marconi VC. Code Saturation Versus Meaning Saturation: How Many Interviews Are Enough? Qual Health Res. 2017;27(4):591–608. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Downing NL, Rolnick J, Poole SF, et al. Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation. BMJ quality & safety. 2019;28(9):762–768. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Hooper MH, Weavind L, Wheeler AP, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit*. Critical care medicine. 2012;40(7):2096–2101. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Leisman DE, Deng H, Lee AH, et al. Effect of Automated Real-Time Feedback on Early-Sepsis Care: A Pragmatic Clinical Trial. Crit Care Med. 2024;52(2):210–222. [DOI] [PubMed] [Google Scholar]
- 29.Arabi YM, Alsaawi A, Alzahrani M, et al. Electronic Sepsis Screening Among Patients Admitted to Hospital Wards: A Stepped-Wedge Cluster Randomized Trial. JAMA : the journal of the American Medical Association. 2024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Ginestra JC, Giannini HM, Schweickert WD, et al. Clinician Perception of a Machine Learning-Based Early Warning System Designed to Predict Severe Sepsis and Septic Shock. Critical care medicine. 2019;47(11):1477–1484. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.de Hond AAH, Leeuwenberg AM, Hooft L, et al. Guidelines and quality criteria for artificial intelligence-based prediction models in healthcare: a scoping review. NPJ Digit Med. 2022;5(1):2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Eisenberg MA, Freiman E, Capraro A, et al. Outcomes of Patients with Sepsis in a Pediatric Emergency Department after Automated Sepsis Screening. J Pediatr. 2021;235:239–245 e234. [DOI] [PubMed] [Google Scholar]
- 33.Sanchez-Pinto LN, Bennett TD, DeWitt PE, et al. Development and Validation of the Phoenix Criteria for Pediatric Sepsis and Septic Shock. JAMA : the journal of the American Medical Association. 2024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Ginestra JC, Kohn R, Hubbard RA, et al. Association of Unit Census with Delays in Antimicrobial Initiation among Ward Patients with Hospital-acquired Sepsis. Ann Am Thorac Soc. 2022;19(9):1525–1533. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Bowman A, Peltan ID. What’s Taking So Long? Known Unknowns, Capacity Strain, and Hospital-acquired Sepsis. Ann Am Thorac Soc. 2022;19(9):1453–1454. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplemental Figure 1. Children’s Hospital Colorado ED Sepsis Pathway13
Data Availability Statement
Deidentified individual participant data (including data dictionaries) will be made available, in addition to study protocols, the statistical analysis plan and code. The data will be made available upon publication to researchers who provide a methodologically sound proposal for use in achieving the goals of the approved proposal, upon approval from the Colorado Multiple Institutional Review Board. Proposals should be submitted to halden.scott@cuanschutz.edu.