Clinical Decision Support Aiming to Accelerate Triage and Time to Dextrose-Containing IV Fluids in the ED for Children with Severe Metabolic Conditions

Abstract

Objectives

This study aimed to describe human-centered design of clinical decision support (CDS) for children with metabolic diseases at high risk of rapid decompensation, assess the influence of CDS on care processes and patient outcomes, and share insights from the implementation.

Methods

A CDS was designed in collaboration with pediatric genetics experts to provide accelerated triage and recommend dextrose-containing fluids for patients metabolic conditions. Formative usability testing was conducted with emergency department (ED) nurses and providers. Pre- and post-intervention data on triage, alert acceptance, and order set usage, as well as clinical outcomes such as time to dextrose fluids, intensive care unit (ICU) admission rates, and length of stay, were compared.

Results

Provider alert acceptance was at 39%. Following CDS implementation, nurse triage at Emergency Severity Index (ESI) <3 (ESI 1 or ESI 2 to escalate patients with metabolic conditions to a higher severity) increased from 84 to 98%. Time to dextrose-containing fluids from patient rooming to administration decreased significantly from 101 to 82 minutes ( p  = 0.006) for all patients with metabolic conditions, and from 110 to 88 minutes ( p  = 0.029) for those admitted to the hospital. However, the median time from arrival to fluids administration saw a non-significant reduction from 114 to 102 minutes ( p  = 0.07). ICU admission rates remained stable pre- and post-intervention (13% vs. 14%; p  = 0.60), and there was no significant change in total length of stay.

Conclusion

The CDS, developed through a user-centered design approach, improved appropriateness of triage acuity rates and reduced the time from rooming to administration of dextrose-containing fluids for children with metabolic diseases at risk of rapid decompensation. The study did not demonstrate a significant change in ICU admissions or length of stay, possibly due to increased patient load and external factors. The findings emphasize the importance of usability testing and clinician-centered design for effective CDS integration.

Background and Significance

Patients with certain metabolic diseases are at risk of rapid decompensation when caloric intake is reduced because at least one of their usual metabolic compensatory mechanisms for fasting does not function. For example, children with medium chain acyl-coA dehydrogenase deficiency (MCADD) who have limited caloric intake even for several hours can develop life-threatening hypoketotic hypoglycemia. 1 Thus, when children contract gastroenteritis or another common viral illness, they may initially present with just vomiting or diarrhea, similar to otherwise healthy children, but without prompt intervention may suddenly deteriorate. Timely interventions such as dextrose-containing fluids can reduce the risk of deterioration and ICU admission by providing a direct calorie source. 2 At initial presentation to the emergency department (ED) and prior to overt deterioration, patients with metabolic conditions may appear relatively stable. They are therefore difficult to triage appropriately and distinguish from other healthy children with similar common symptoms (e.g., vomiting) but at lower risk of rapid deterioration. Thus, a mechanism to facilitate early identification, triage, and rapid management of patients with metabolic conditions may prevent decompensation from severe acidosis, hyperammonemia, or hypoglycemia and reduce ICU admissions and length of stay.

In our large, tertiary healthcare system, we developed a clinical decision support (CDS) in the electronic health record (EHR) to support (1) identification and triage of patients with metabolic conditions at risk for rapid deterioration in the ED and (2) provision of dextrose-containing fluids with the ultimate goal of reducing ICU admissions and length of stay.

Objectives

1. Describe human-centered design of CDS to improve care of children with metabolic disease presenting to the ED.

2. Determine the influence of the CDS on care process and patient outcome measures.

3. Enumerate lessons learned from this CDS implementation.

Methods

Setting and Ethics

This study was done at a large pediatric health system in the Southeastern United States with three freestanding Children's hospitals on a single enterprise instance of Epic Systems© as its electronic health record (EHR). The Children's Healthcare of Atlanta institutional review board deemed this study non-human subjects research as a quality improvement initiative (approval no.: STUDY00000393).

CDS Design

We first developed a list of metabolic disease conditions that merit accelerated triage and dextrose-containing fluids in consultation with pediatric genetics experts (W.R.W. and Abdulrazak Alali, MD). For each disease, we identified a set of diagnosis codes to inform CDS logic. A genetics counselor (H.D.) also reviewed known metabolic disease patients from a manually curated list to ensure diagnoses were populating the problem list. For each metabolic disease group, we identified early interventions and laboratory evaluation recommendations. For example, for patients with β-ketothiolase deficiency experiencing catabolism, the recommended rapid evaluation would include point of care glucose, complete metabolic panel, urine ketones, and blood gas and the recommended rapid treatment would include 10% dextrose-containing IV fluids and a dose of IV carnitine if the ED stay were prolonged. We developed a candidate CDS prototype that would (1) fire an interruptive alert on chart open for the nurse doing triage recommending expedited rooming (Emergency Severity Index [ESI] level 2), (2) fire an interruptive alert on chart open for the ED provider and direct the provider to an order set containing pre-determined evaluation and treatment specific for each metabolic disease.

Next, we performed formative usability testing using a think-aloud protocol described in detail elsewhere. 3 Briefly, we created three scenarios with genetics experts and documented the ideal practice: Case A: A patient with a history of methylmalonic acidemia (MMA) coming in for a complaint of ear pain and diagnosis of acute otitis media who does not require IV fluids; Case B: A patient with a similar history of MMA coming in with a complaint of vomiting thought to be due to an infectious GI problem requiring IV fluids and labs; and Case C: A patient with a history of MMA coming in listless who requires immediate IV fluids. We then built the CDS prototype in a test version of the EHR and approached ED nurses and providers when feasible based on ED workload. After providing psychological safety, the clinical scenarios were read to the participant, and we observed responses such as triage acuity selection and order placement. We also obtained qualitative feedback at the end of each session. We iteratively changed the design between participants based on learnings from each session until no new input was identified with two consecutive participants.

Implementation and Evaluation

The CDS alerts were implemented in the background in June 2020 (i.e., firings showing up in the database, but not visible tor nurses or providers) initially to check for accurate performance. The charts where the background alert fired were reviewed, and alert logic was adjusted iteratively until the alert was consistently firing on the patient population as designed. 4 After implementation of the CDS, we queried the EHR to identify ED visits for patients with a problem list or visit diagnosis of a metabolic disease identified for the cohort. We analyzed care process measures focused on nurse triage, provider alert acceptance, and order set usage with outcome measures including time to dextrose-containing IV fluids, ICU admission rate, and length of stay ( Table 1 ). We compared these outcomes before and after intervention. We also explicitly analyzed the subset of patients who were admitted since admission suggests that the patient's condition was likely severe enough to merit rapid treatment as opposed to a patient with a metabolic condition coming in for a minor unrelated complaint. Comparisons for median values were done using Mood's median test and for dichotomous variables using Χ ² .

Table 1. Interventions and associated measures.

Intervention	Measure type	Measure	Measure definition
Nurse alert	Process	Acuity assignment	Proportion of all metabolic ED encounters assigned ESI ≤2 pre/post
	Process	Time to rooming	Median arrival-to-room time among all metabolic ED encounters pre/post
Provider alert	Process	Provider alert acceptance	Proportion of alert firings in which providers opened the order set
Order set	Process	Order set usage	Proportion of metabolic ED encounters where order set was used
Order set and alert	Process	Time to dextrose-containing fluids	Median time from arrival to administration of dextrose-containing fluids (minutes)
	Outcome	ED to ICU admission rate	Among metabolic ED encounters, percentage of patients admitted to the ICU
	Outcome	Length of stay	Among metabolic patients who were admitted, their median length of stay (minutes)

Abbreviations: ED, emergency department; ESI, Emergency Severity Index; ICU, intensive care unit.

Results

CDS Design

We performed two sessions of formative usability testing, one with five ED nurses who performed triage and one with ED providers including two pediatric residents and four pediatric emergency medicine attendings. We identified several design flaws and aimed to mitigate them between participants ( Table 2 ). In our final design, we created:

Table 2. CDS design issues and changes.

Issue identified	Design Change	Rationale
Participants accepted order set recommendations and ordered fluid in Case A when these were not required.	Made orders into a panel that encouraged providers to first make decision on need for IV fluids based on the patient's presenting complaint, clinical appearance, and hydration status (see Supplemental Fig. S2 ).	Enable decision-making based on what users know—providers are more likely to know the patient's clinical appearance and oral intake status than knowing if fluids are required for specific conditions.
Participants placed fluids as 1x Maintenance Rate instead of 1.5x Maintenance Rate (MR).	Provided rate reference: fluid-specific instructions asking them to order D10 fluids at 1.5x MR.	Reduce reliance on physician memory.
Participants voiced uncertainty about disease-specific workups and labs required for patients.	Included a list of all relevant labs specific to the disease after consultation with experts.	Reduce reliance on physician memory. Allow recognition instead of recall.
Missing orders in the Order Set.	(1) Change order to Venous iSTAT for gas. (2) Added normal saline bolus. (3) Changed carnitine dosing instructions to single dose.	Include all relevant orders in the order set and default order details matching ED requirements.
Participants placed duplicate orders as there was a pre-selected diagnosis (order panel) and they re-selected the diagnosis in the master list of metabolic conditions.	Included only either pre-selected diagnosis or master list of diagnosis.	Design to eliminate errors.
Participants felt forced to open order set out of concern for not being able to get valuable information before they were ready. They did not notice acknowledgment reason (“Remain active for me”) applicable to this concern.	Included acknowledgment reason “Yet to see the patient, show again later.”	Match design with users' mental model.

Abbreviations: CDS, clinical decision support; ED, emergency department.

1. An interruptive alert for ED nursing ( Supplementary Fig. S1 , available in the online version only) that would fire on chart open for patients with a metabolic disease on their problem list and no ESI level entered.

2. An interruptive alert for ED providers ( Supplementary Fig. S2 , available in the online version only) that would fire on chart open for patients with a metabolic disease on their problem list. This alert aimed to:

   1. Help the provider identify the presence of metabolic disease.

   2. Provide guidance for when to initiate rapid evaluation.

   3. Direct the provider to the order set if rapid evaluation was needed.

3. A dynamic order set that would default open to the appropriate order group based on the specific metabolic disease in the problem list. For example, if the patient had β-ketothiolase deficiency, opening the order set would immediately open the appropriate section ( Supplementary Fig. S3 , available in the online version only). Within the specific disease group, the provider would first select if the patient met criteria for rapid treatment (i.e., ill appearing, vomiting, lethargic, or poor oral intake). If the provider selected that the patient did not meet criteria, the order set provided text guidance and offered a genetics consult. If the provider did select that the patient met criteria, the disease-specific order group would show up with rapid lab and treatment recommendations all default-checked.

   Additionally, in case of inaccurate documentation of the problem list diagnosis or for patients with known metabolic disease without a problem list entry (e.g., recently moved), we also created a full list of metabolic diseases, allowing the provider to make a manual selection and see the recommended care ( Supplementary Fig. S4 , available in the online version only).

A total of six physicians (four attendings, two residents) participated in the formative testing. We made six design changes based on formative testing. The design changes and rationality for the changes are shown in Table 2 . Participants gave positive feedback that (1) the alert language was clear and appropriately conveyed the risk and also when the risk would not apply, (2) the choice architecture of the order set that made it clear what decisions needed to be made based on the patient's condition, and (3) the presence of a master list of potentially relevant diagnoses with diagnosis-specific treatment recommendations.

Background Implementation

During the background validation phase, we identified that the alert was firing more frequently than initially anticipated ( Fig. 1 ). On chart reviews we identified that the alert was firing for premature infants with no metabolic conditions. Further review of the alert grouper revealed a typo in the diagnosis code. The grouper list had more than 350 diagnosis codes, and one of the included codes was 13 8 3869 (Premature less than 24 weeks of gestation) instead of 13 5 3869 (Mitochondrial Complex 3 deficiency nuclear type 1). This error was rectified, and the alert was implemented in the foreground on September 16, 2020.

Fig. 1.

Alert firings during background implementation.

CDS Evaluation

Pre- and post-intervention process and outcome metrics are shown in Table 3 . There were 536 eligible encounters prior to intervention and 971 encounters post intervention. Nursing triage for eligible patients at ESI <3 increased from 84% (442/536) to 98% (958/971) ( p  < 0.01). The median time to rooming unexpectedly increased from 34 to 50 minutes ( p  < 0.01) in the same time period. Of note, the ED volumes also generally increased during the post-intervention period due to COVID-19 pandemic and other respiratory virus surges that affected children later in the pandemic. From April 2020 until implementation on September 16, 2020, there were between 7,000 and 12,000 ED visits per month, where this ranged from 15,000 to 27,000 ED visits per month in the post-intervention period. The provider alert acceptance rate was 39% (1,613/4,369). Post-intervention, ICU admission rate was similar to pre-intervention,13% (70/536) vs. 14% (136/970), respectively ( p  = 0.60).

Table 3. Pre- and post-intervention comparison of process and outcome measures.

Measure	Measure definition	Pre-intervention (September 16, 2019 to September 15, 2020)	Post-intervention (September 16, 2020 to July 15, 2022)	P value
Acuity assignment	Proportion assigned ESI ≤2 pre/post	84% (442/536)	98% (958/971)	<0.01
Time to rooming	Median arrival to time to room time	34 minutes	50 minutes	<0.01
Provider alert acceptance —open orderset from the alert	Proportion of alert firings in which providers opened the orderset	NA	39% (1,613/4,369)
Orderset usage	Proportion of ED encounters who were at risk of metabolic decompensation where orderset was used	NA	39% (391/1,003)
Time to dextrose-containing fluids	Median time from arrival to administration of dextrose-containing fluids (min)	114 minutes (IQR 64–250 minutes)	102 minutes (IQR 61–178 minutes)	0.07
	Median time from arrival to administration of dextrose-containing fluids (minutes) among those with admission	124 minutes (IQR 65–269 minutes)	108 minutes (IQR 59–190 minutes)	0.13
	Median time from roomed to administration of dextrose-containing fluids (min)	101 minutes (IQR 48–241 minutes)	82 minutes (IQR 41–159 minutes)	0.006
	Median time from roomed to administration of dextrose-containing fluids (min) among those with admission	110 minutes (IQR 48–258 minutes)	88 minutes (IQR 41–172 minutes)	0.029
ED to ICU admission rate among admitted patients	Percentage of admitted patients who went to the ICU	13% (70/536)	14% (136/971)	0.66
Total length of stay	Among patients who got admitted their median length of stay (min)	3,912 minutes	4,322 minutes	0.11

Abbreviations: ED, emergency department; ICU, intensive care unit; IQR, interquartile range.

The median time to dextrose-containing fluids from arrival to administration decreased from 114 to 102 minutes post-intervention ( p  = 0.07). The time to dextrose fluids from arrival to administration among those admitted also decreased from 124 minutes (IQR 65–269 minutes) to 108 minutes (IQR 59–190 minutes) ( p  = 0.13). To further delineate this, we also looked at time to dextrose fluids from patient room time to administration time, which would not be affected by the longer arrival to room times due to higher ED volumes. This median time for all patients with metabolic disease decreased from 101 minutes (IQR 48–241 minutes) to 82 minutes (IQR 41–159 minutes) ( p  = 0.006). Among metabolic patients admitted to the hospital, the median time from room to administration of dextrose-containing fluids decreased from 110 minutes (IQR 48–258 minutes) to 88 minutes (IQR 41–172 minutes) ( p  = 0.029). The median total length of stay among those who were admitted was similar pre- (3,912 minutes) and post-intervention (4,322 minutes ( p  = 0.11)).

Discussion

In this study of human-centered design of CDS to improve treatment for children with metabolic disease at risk of rapid decompensation, we found that (1) although appropriate nurse triage rates improved, time to actual rooming increased in the setting of globally increased patient load in the ED and (2) although dextrose-containing fluids were provided more quickly, there was no change in ICU admission rates or total length of stay. This decrease in time to fluids is particularly remarkable given that during the post-intervention phase, time to room increased and a provider's opportunity to order medications starts only after rooming. Overall, the CDS was well received with reasonably high provider alert acceptance and order set usage rates.

Importance of Usability Testing of CDS

Our application of human-centered design with in-situ usability testing helped identify critical design issues. 3 We made several key modifications to the CDS design including order set structure, content, and representation based on feedback from usability testing. In the absence of formal usability testing, order set developers may not sufficiently understand how users will interact with the system to complete tasks, leading to an inadequate application of the “five rights,” inadequate support for users' cognitive needs, and misalignment between workflow and order set design. 5 Human-centered design informed by simulated testing can lead to substantial design adjustments that ultimately improve adherence to recommended practices while reducing the user's cognitive workload. 6 7

Need to Address Clinician's Cognitive Needs

In our initial candidate design, we found that clinicians ordered fluids in cases when it was not required. Our adjustments were based on recognizing that providers were more likely to know a patient's clinical appearance and oral intake status rather than knowing if fluids are required for specific metabolic conditions. Thus, our updated design matched the providers' mental model by first asking about ill appearance and risk of catabolism, then creating disease-specific panels that reduced clinician's cognitive burden. 8

Need to Test CDS Performance in Background

Our background implementation helped identify that the alerts were firing more frequently due to a typographical error in the value set for diagnosis codes that refer to metabolic conditions. Had we not checked for this error and implemented directly, this might have led to unnecessary alert burden, poor CDS adherence, and loss of trust.

Limitations

This was a single site study limiting generalizability of the influence of CDS for patients with metabolic conditions in the ED on outcomes. However, we feel the lessons learned specifically on design choices of CDS and the process for formatively evaluating them can be applied more generally to CDS development. Our CDS improved time from rooming to administration of dextrose-containing fluids but did not significantly improve time to dextrose-containing fluids from arrival to the ED. It is unclear if there had not been an increase in rooming time globally whether the improvement from the CDS alone could have reduced ICU admission rates and length of stay or if this CDS approach is unable to have enough of an effect on time to dextrose-containing fluids to change those outcome measures. Inclusion of a control group in the study design instead of just a pre–post comparison might have allowed us to better distinguish this. There was also a wide variation in IQR of our times to dextrose-containing fluids, suggesting that other factors such as how busy the ED is, staffing level, patient factors, or other unmeasured variables likely affect time to treatment. A larger sample or a longer study may help address these fluctuations but may also be infeasible given the relative rarity of children with severe metabolic disease. Finally, the CDS as designed relies on the patients having a metabolic condition added to their problem list. For patients who are new to the healthcare system or without history in our health system, no alerts would fire although ED providers could seek out the order set on their own. Better interoperability and capacity to trigger these kinds of warnings from problem list diagnoses at other health systems could help mitigate this challenge.

Conclusion

Iteratively developed CDS with formative usability testing resulted in improvement in several process measures associated with caring for children with metabolic conditions at risk of rapid decompensation presenting to the ED. This work highlights the importance of formative testing for technical, workflow, and usability issues to identify problems and fix them before implementation. However, we did not demonstrate improvements in clinical outcomes, which may be due to secular trends in patient load in the ED or due to lack of sufficient effect from the CDS. This finding demonstrates the benefits of implementing in a quasi-experimental or ideally randomized design. It also shows that adherence to recommended practices alone may not improve outcomes and that tracking the outcome metrics that ultimately matter to patients and families (e.g., ICU admission, length of stay) is critical to achieve the potential of CDS. Future work such as streamlining use of emergency letters for specific initial management or workflows that identify these patients even further upstream prior to coming to the ED may yield a stronger influence on outcomes. Additionally, multicenter trials may improve statistical power and allow CDS intervention to be compared to control conditions.

Clinical Relevance Statement

This study provides important lessons for CDS alert and order set development, implementation and evaluation including the need for usability testing, background performance evaluation prior to CDS implementation, and consideration to providers' mental models in CDS design.

Multiple-Choice Questions

1. Which of these are important to interruptive alert design and implementation process to avoid frequent inappropriate firings?

   1. Alert firing should be monitored post-implementation.

   2. Alert firing should be monitored in background during silent implementation.

   3. Alert content should be validated with clinical experts.

   4. Alert logic should restrict firings to once per encounter.

   Correct Answer : The correct answer is option b. Monitoring alert firing in the background during silent implementation helps identify and rectify issues before the alerts are visible to users, thus avoiding frequent inappropriate firings. This allows for adjustments in alert logic to ensure accurate performance.

2. Implementing alerts with inappropriate value sets can lead to

   1. Alert burden

   2. Poor CDS adherence

   3. Loss of trust

   4. All of the above

   Correct Answer : The correct answer is option d. Implementing alerts with inappropriate value sets can lead to alert burden, poor CDS adherence, and loss of trust among users. Each of these consequences can negatively impact the effectiveness and acceptance of the clinical decision support system.