Understanding Clinical Decision Support Failures in Pediatric Intensive Care Units via Applied Systems Safety Engineering and Human Factors Problem Analysis: Insights From the DISCOVER Learning Lab

Abstract

Objectives:

Children receiving care in pediatric intensive care units (PICUs) are vulnerable to decompensation and diagnostic error due to the complex and dynamic nature of pediatric critical illness. In the PICU, the few clinical decision support (CDS) tools that have been implemented to support diagnostic accuracy (i.e., the ability to detect the presence of a condition) have not led to an increase in clinician adoption of desired practices nor demonstrated clear clinical benefit.

Methods:

The DISCOVER Learning Lab analyzed workflow and failure modes in diagnosing and managing clinical decompensation in the PICU, using systems safety engineering and human factors to examine intersections with established CDS. Methods employed included qualitative interviews, workflow mapping, immersive virtual reality (VR) systems testing via a digital twin environment, and a failure modes effect analysis.

Results:

Workflow mapping and qualitative interviews revealed barriers to communication, workflow inefficiencies, and limited access to up-to-date clinical information during critical events in the PICU. The immersive VR systems testing elucidated how PICU staff members currently interact with CDS tools and how various tools could better integrate into or influence clinical workflows. Critical failure modes were identified with corresponding opportunity areas for intervention.

Conclusions:

The application of a systems safety engineering and human factors approach to problem analysis, partnered with novel use of immersive VR and digital twin technology, led to valuable insights into common failure modes and potential opportunity areas to improve diagnostic accuracy and care delivery in a quaternary referral center PICU.

Clinical decision support (CDS) tools are built to improve health care delivery and outcomes to enhance the accuracy and speed of medical decisions by coupling pertinent patient information with targeted clinical knowledge.¹ Implementation of CDS tools into high-acuity clinical contexts without fully understanding how providers and staff interface with these tools can lead to unintended patient safety consequences. These consequences can include disrupted workflows and alert fatigue, all of which can result in indirect patient harm.² In the pediatric intensive care unit (PICU), the few CDS tools that have been implemented have not led to an increase in clinician adoption of desired practices³ nor demonstrated clear benefit for patient safety.⁴ The use of active disruptive alerts around sepsis recognition or acuity scores has been found by the majority of clinicians to be burdensome to their workflow and to not provide useful information, and thus they were not adopted into clinical practice, allowing for no impact on clinical care or outcomes.⁴ We hypothesize that CDS tools in health care have limited or unanticipated impact because they are implemented without a systems safety engineering approach.

Systems safety engineering is the process required to design, integrate, operate, and manage safety-critical systems.⁵ In most industries, substantial resources are devoted to designing, deploying, and evaluating new products.⁶ We currently lack this infrastructure in medicine. We developed the DISCOVER Patient Safety Learning Lab, using a systems safety engineering methodology, with the goal to envision, develop, prototype, and test design–informed CDS tools.

A typical PICU clinician encounters multiple CDS tools daily. Yet, in many cases, robust assessments of efficacy before implementation and comprehensive surveillance of their performance post-implementation do not occur.⁷ These CDS tools are deployed as “fixes” to address the growing complexity and acuity of patient care in modern medicine. However, there is minimal attention to design, inadequate infrastructure for testing, and a paucity of clinical research informatics investment to support and sustain these efforts. For example, the majority of alerts provided by CDS tools are bypassed, leading to minimal to no real-world impact on patient care.⁸ In addition, the associated alert fatigue and workflow disruptions have been identified as key drivers of clinician burnout.⁹ The result is a failure of CDS tools to live up to their promise.²

This end state leaves health care systems ill-equipped to effectively leverage emerging tools like predictive algorithms for the purposes of improving patient care.¹⁰ Currently, no guidance exists on how to implement such systems in the pediatric critical care environment. The DISCOVER Patient Safety Learning Lab seeks to address this gap by employing a design-informed approach, incorporating systems safety engineering and human factors, to develop guidance on the implementation of CDS tools in the PICU context.

This manuscript describes a comprehensive and innovative approach to conducting the problem analysis phase of the systems safety engineering model, spanning traditional approaches of stakeholder interviews and shadowing experiences to the novel use of a digital twin environment experienced through immersive virtual reality (VR). Our focus was on defining the current workflow and failure modes during the diagnosis and subsequent acute management of patient clinical decompensation in the PICU and characterizing how these workflows intersect with established CDS.

METHODS

Human Centered Design Thinking

Our study team, funded by the Agency for Healthcare Research and Quality (AHRQ), partnered with the LiveWell Collaborative, a leading design research group, to implement a design thinking approach to problem analysis. The problem analysis phase lays the foundation for understanding complex system issues and is crucial for establishing a clear, actionable framework to guide CDS implementation.^11,12 Our team-based approach to defining the problem consisted of 4 activities: (1) qualitative interviews with key stakeholders, (2) workflow mapping, (3) patient and family feedback, and (4) immersive VR systems testing via a digital twin environment (Fig. 1).

FIGURE 1.

Problem analysis overview.

Key Stakeholder Qualitative Interviews

Prior research has identified 4 principles for the successful integration of continuous predictive analytics into clinical care.¹³ First, clinicians must understand the science behind the algorithm; second, they need to trust the data inputs; third, it must integrate with the electronic health record; and last, it must be embedded within optimal clinical workflows. Using this framework, we developed a semi-structured interview guide to ask stakeholders about: (1) standard routines and sources of patient information during the care of critically ill patients; (2) barriers encountered during both routine or scheduled events such as rounds, and unscheduled or emergency events like unplanned admissions or patient decompensation; (3) their knowledge and understanding of currently available CDS in the PICU; (4) use of currently available CDS; and (5) ideas for improving or enhancing workflows supported by CDS. The LiveWell team, composed of a faculty lead (M.D./M.F.A.) and industrial and communication design students (BS/MDes), conducted qualitative interviews with 14 PICU staff members (2 attending physicians, 3 fellow physicians, 3 advanced practice providers, 3 registered nurses, and 3 respiratory therapists). They recruited participants via email, with interviews lasting for 1 hour each. In addition, the team interviewed 2 non-staff members: a former PICU patient and a parent of a current PICU patient. A core interview guide was developed in collaboration with the study team, with all interviewers trained on its use. All interviews were audio recorded and transcribed via Otter.ai (Otter.ai Inc.) for subsequent review. Transcriptions were coded by the LiveWell team within Google Sheets (Google LLC) as a team to ensure consistency in interpretation and organized into groups using deductive thematic analysis based on the tools used, successes, challenges, and opportunities at each phase of clinical care. Identified themes underwent subsequent member checking and triangulation with the study team. Following this, the team conducted a Failure Mode and Effects Analysis (FMEA) to examine the causes and effects of specific events within the process. These insights were then transferred into the Miro collaboration platform (RealtimeBoard Inc.), where they were color-coded and tagged according to the identified categories. Once categorized and tagged, the insights were digitally affinity mapped within Miro, leading to the emergence of key themes.

Workflow Mapping

We began with workflow mapping in a 48-bed quaternary care PICU in an urban free-standing children’s hospital to delineate the prevailing workflows used in the care of critically ill children. The observations spanned 26 hours of shadowing real-life clinical care in the unit (both day and night shifts). In addition, the LiveWell team observed 9 hours of interprofessional manikin-based simulations for low-frequency but high-stakes events (i.e., cardiac arrest with cannulation to extracorporeal life support). The team carefully documented the workflows of both scheduled and unscheduled events, including admissions, escalations of care, and clinical deterioration events across multiple clinical roles. Particular attention was paid to the use of communication methods and CDS tools. The team identified barriers or perceived pain points and validated these observations in real time (when active patient care was not in process) with the involved staff. These barriers were validated during the previously described key stakeholder interviews to explore generalizability. The utilized observation tools were grounded in the SEIPS 2.0 model—a conceptual framework that helps to understand patient safety in health care—with an eye toward design-informed approaches to visualizing workflows as journey maps.¹⁴ Drafts of the workflow maps were presented to PICU staff members and patient/family representatives (1 former PICU patient and 1 current parent of a PICU patient) during 2 workshops where feedback was solicited and incorporated into the finalized maps.

Immersive VR Systems Testing via a Digital Twin Environment

To assess gaps in CDS implementation using systems safety engineering, it is essential to evaluate clinicians’ interactions with these tools under realistic conditions. A simulated environment provides insights into real-world behaviors by testing clinicians’ readiness to adjust their practice based on CDS output. Digital twinning¹⁵ allows for creation of an exact virtual replica of the PICU, including relevant virtual patients, access to physical data (i.e., patient exam, present medications/equipment, and real-time vital sign monitor), as well as electronic data pertinent to the incorporated CDS (i.e., in-room dashboards or waveforms) to assess the impact of CDS tools on clinician behavior. Our team previously developed and implemented immersive VR training for frontline health care providers using a digital twin, advancing from previously relied upon low-fidelity or manikin-based exercises that lacked realism and flexibility.^15–21 Unlike traditional methods, VR allows seamless trailing of multiple CDS tools in a virtual environment without requiring significant physical resources, and importantly, with no risk to patients.

A digital twin of the PICU, previously developed for clinical onboarding,¹⁵ was adapted for this work. Within the digital twin, participants can interact with a virtual patient tailored to a specific condition, with dynamic vital signs synced to clinical interventions. In addition, a digital twin of the electronic health record (EHR) was replicated for the clinical scenario, allowing participants to interact with the EHR throughout the simulation as they normally would. The patient scenario was selected due to the potential for diagnostic error and the potential for different data sources and CDS to influence the recognition of the patient’s clinical needs. The clinical case and scenario description are included in Appendix 1, Supplemental Digital Content 1, http://links.lww.com/JPS/A707.

We recruited 9 unique interprofessional clinical bedside staff trios (bedside nurse, respiratory therapist, and physician/advanced practice provider) to participate in the simulations. The sessions occurred within the PICU, and the participants were recruited through convenience sampling during clinical shifts. Following onboarding to the goal of the session and orientation to the digital twin environment and EHR, participants were provided the clinical scenario and encouraged to engage in their normal workflows for caring for this patient.

During the simulation, clinical staff trios had access to all currently available sources of information, including current CDS tools, that would be available in the PICU. In our local setting, the in-room CDS is Epic Monitor (EPIC), designed to be a centralized dashboard from which clinicians can easily access all of the data included within the EHR. Information can be arranged per institution or context preferences, but consists of listed vital ranges over a period of time, recent labs, links to imaging, and flow sheet values including patient ins and outs (Fig. 2). No direct clinician guidance on treatment pathway or patient acuity scores is displayed.

FIGURE 2.

Epic monitor view for simulation.

A study team member observed and documented the clinician’s workflow on a journey map, focusing on the sequence of their actions, their sources of information, tools used, and when/how they engaged in interprofessional communication. Immediately following the simulation, the study team members reflected on the observed workflow with the simulation participants to gain clarity on what informed the workflow and identify drivers of behavior. In the second phase of the simulation, participants were introduced to 3 examples of CDS visualizations to discuss the tools’ anticipated impact on their workflow and ability to rapidly reach a working diagnosis: (1) EPIC Monitor (EPIC), (2) Inadequate Oxygen Delivery Index (IDO₂) (Etiometry Inc.), and (3) Continuous Monitoring of Event Trajectories (CoMET, AMP3D Inc.). Participants then returned to the digital twin environment to experience these additional CDS approaches in a clinical setting (Fig. 3). The discussion continued within the digital twin environment, focusing on strategies where the multiple types of CDS and optimization of their use could enhance the diagnostic and care process.

FIGURE 3.

Simulation review process.

The team synthesized findings across the 9 simulations. First, role-specific synthesis was performed to identify core aspects of the current workflow for each role type (i.e., respiratory therapy, nursing, and nurse practitioner/physician). Second, we synthesized observations by role and overlaid them to develop a generalized workflow for each role, characterizing the current state of care. Third, the discussions on various CDS approaches and their potential impact on workflow were used to generate CDS workflow hypotheses for each of the presented tools. Lastly, based on these findings, the team established key design parameters to inform next steps in the project. These parameters included specific functional requirements for CDS tools, workflow integration strategies, and modifications to the clinical environment to optimize CDS.

Failure Mode and Effects Analysis and Identification of Opportunity Areas

Using a failure mode and effects analysis (FMEA),²² a risk assessment tool, the teams identified common failure modes and latent safety threats during the assessment and management of patient clinical decompensation within the PICU, with access to currently available CDS within our unit and EHR. Data were incorporated from interviews, observations, and simulations following established best practices of simulation-based clinical systems testing.²³

RESULTS

Workflow Mapping and Qualitative Interviews

The PICU workflow mapping resulted in key insights to better understand the scope of the problem. First, a single highly detailed “Day in the Life” map was created (excerpt seen in Fig. 4, full map in Supplemental Materials Appendix 2, Supplemental Digital Content 2, http://links.lww.com/JPS/A708). This map highlights several high-risk periods, including simultaneous morning handoffs, daytime rounds with their substantial communication burden, and unplanned events in the evening and overnight hours with a smaller staff footprint. Second, to visualize the workflow of the PICU clinical staff, a video was created depicting 24 hours in the PICU. The video demonstrated the various care roles and the impact of routine or scheduled events, such as rounds, and unscheduled or emergency events, like unplanned admissions or patient decompensation, on the movement of staff throughout the day (Video of PICU Day in the Life, https://youtu.be/BJNqqk9PRhY). Workflow mapping and qualitative interviews identified several themes during critical events in the PICU, including (1) significant barriers to communication, (2) workflow inefficiencies, and (3) limited access to up-to-date clinical information during critical events in the PICU.

FIGURE 4.

Excerpt of workflow map including roles, actions, and communication methods (for full map, see supplemental materials, Appendix 2, Supplemental Digital Content 2, http://links.lww.com/JPS/A708).

Immersive VR Systems Testing

Within the digital twin environment, we found that throughout the scenarios, users transitioned their attention between the EHR, physical assessment of the patient, and the vital sign monitor. No user ever interacted with the in-room CDS (EPIC Monitor) as confirmed by study staff observations, debriefing with the participants, and review of VR headset eye tracking data. Figure 5 visualizes the workflow for each team member as they worked to establish the patient's diagnosis and care plan. Specifically, physicians and advanced practice providers parsed through multiple data points to determine trajectory, often involving time in the EHR before entering the patient room to establish a mental framework to subsequently confirm with the patient exam. Nurses spent the majority of their time in the room, with vital sign changes and patient exam prompting entry into the EHR for documentation, less for information gathering. Finally, respiratory therapists relied primarily on in-room patient assessments and prompting by the nurse or provider to come to the bedside to discuss care plans, with EPIC again for documenting rather than information gathering.

FIGURE 5.

Summary of systems testing by interprofessional role and key findings.

CDS workflow hypotheses were generated based upon participant discussion as to the impact the 3 presented CDS tools would have on their workflows. The physicians and advanced practice providers endorsed a need to parse through multiple time points of data to assess a patient’s trajectory. The ability to view trends, in partnership with up-to-date results, was identified as a crucial need that would impact care. The access to CDS with the above characteristics led to changes in the hypothesized workflow that enhanced the efficiency of care and presence at the bedside. Meanwhile, a CDS tool that gave a quick view into a patient's status and trajectory was perceived to be valuable for placement outside the patient room (or a central location) to prompt a patient assessment. For nurses, much of the discussion focused on supporting new nurses' situation awareness as well as avoiding information loss with shift change. To that end, a combination of visualized trends and patient risk status for deterioration helps to bridge both of those needs. Similar to the providers, access to trends over time at the bedside provides context for clinical assessments, to prompt questions when deviations from baseline (or the prior shift) are noted. In addition, a global status/trajectory visualization outside the room would prompt assessments by nursing leadership and support staff. Finally, respiratory therapists’ workflow was primarily driven by task accomplishment. While there was some endorsement that access to patient trends in the room would be helpful to anchor clinical assessments, the main driver of workflow changes was perceived to be outside-the-room visualizations of global status/trajectory to prompt a reassessment outside of scheduled respiratory treatments.

These findings are summarized in Figure 6 as a visual guide to the findings of the workflow hypothesis generation. Integrating the key learnings from the entire problem analysis phase, the team created high-level CDS digital design recommendations. These recommendations included: intuitive use, predictive alerts, seamless integration with existing systems, and data transparency. Figure 6A leverages ongoing use of a vital sign monitor within the room with the expansion of visualized patient trends—a hypothesized workflow enhancement for providers and nurses. Figure 6B visualizes an in-room display that allows easy access to EHR data without the need to leave the patient room—a hypothesized workflow enhancement for all team members. Figure 6C visualizes the implementation of a hallway display outside the patient room that is visually simple and provides a summative risk score for quick evaluation—a hypothesized workflow enhancement for providers, support nurses, and respiratory therapists. And lastly, Figure 6D visualizes a CDS tool in a central location for overall unit situation awareness—a hypothesized workflow enhancement for unit leadership and support staff across professions. All of these visualizations would include transparent and understandable data, EHR integration, and predictive alerts, supporting all interprofessional roles.

FIGURE 6.

Digital design recommendations for CDS. A, CDS design within the room with patient trends. B: CDS design within the room with access to EHR data. C, CDS design with a hallway display of summative risk score. D, CDS design in a central location of unit situation awareness.

Identification of Failure Modes

PICU workflow analysis in the form of an FMEA revealed multiple failure modes as well as identified key opportunity areas. Failure modes included CDS failures in both accurate prediction and anticipated response, information failures with an emphasis on barriers to real-time data with concurrent alarm fatigue, diagnostic inaccuracy and lack of shared situation awareness among PICU staff and patients/families, failure to develop or implement mitigation plans, reassessment failures in both timing and accuracy, and competing clinical team demands in the form of documentation requirements, information overload through excessive digital communication, or staffing challenges.

These failure modes informed several opportunity areas: (1) enhanced CDS tools with high accuracy that are embedded in clinician workflows that enhance shared situation awareness of patient status and trajectory; (2) improved alert saliency through tiered notifications; (3) implementation of standardized and reliable engagement of patients and families; (4) expanded team redundancy for periods of high acuity/complexity; (5) use of bedside mitigation tools for high-risk patients; and (6) standardization of the approach to patient reassessment. In summary, these opportunities can be described as approaches to streamline and enhance digital communication, role clarity, resource allocation, clinical status transparency, and team-level situation awareness in the pursuit of diagnostic excellence - correct diagnosis in a timely manner with the fewest resources while maximizing patient experience. All of these are vital to inform CDS tool development and application.

DISCUSSION

An extensive and innovative problem analysis phase of a systems science engineering and human factors approach successfully identified common failure modes and potential opportunity areas to improve patient safety in a quaternary referral center PICU. Through enhanced understanding of clinical workflows and the intersection with CDS, the study’s approach allowed for the identification of critical issues in information gathering and information integration and leveraged co-identification of opportunity areas with direct input from PICU staff and patient family stakeholders. The findings underscore the power of systems science engineering to enhance diagnostic safety.

Workflow mapping and qualitative interviews revealed significant barriers to communication, workflow inefficiencies, and access to up-to-date clinical information during critical events in the PICU. The immersive VR environment further elucidated how members of the clinical staff currently interact with CDS tools and how various tools could better integrate into or influence clinical workflows. These findings support the established literature that CDS tools have not been effectively integrated into PICU clinical workflows, resulting in limited clinician adoption and minimal impact on patient outcomes, despite massive investment.⁴ In addition, our findings support the published challenges associated with CDS integration, such as alert fatigue and workflow disruptions.^4,24 However, our study extends the literature on several fronts. First is the use of a systems safety engineering lens.⁵ Second, the specific focus on challenges within the context of pediatric critical care is novel, adding insights into the failure modes of current CDS tools in high-acuity environments. Finally, our approach provides a framework for supporting future CDS development and implementation using immersive VR and digital twin technologies.

There are broad implications of this work outside the PICU at a single center. The identified barriers, pain points, and workflow disruptions emphasize the need for CDS tools that are aligned with clinical and diagnostic processes rather than adding layers of complexity. Incorporating systems safety engineering and human factors approaches during the design phase of CDS tool development is essential.^25,26 Designing CDS tools with an understanding of site-specific clinician roles, responsibilities, and common workflow disruptions can enhance their relevance and utility.^1,27 This study underscores the need for dedicated resources and infrastructure to support the design, implementation, and evaluation of CDS tools. Investment in digital health technologies, such as digital twin environments and immersive VR, can help to bridge the gap between tool development and real-world applicability.²⁸

There are limitations of our approach to consider. Although immersive VR provides a unique and realistic environment for systems testing, it is currently unclear whether VR can fully replicate the complex dynamics of a live clinical setting. Our study sample size was limited to a single PICU environment, which may limit the generalizability of findings to other settings. In addition, the findings may be specific to pediatric critical care environments and may not directly translate to other clinical settings without adaptation of the CDS tools and evaluation methods. However, the methodology used for this study can be applied broadly to determine site-specific CDS tool needs.

Next steps for our learning lab will be continuing the systems science engineering approach to develop novel interventions to improve CDS uptake and use, evaluate the impact of these design-informed tools on diagnostic accuracy, and develop a prospective implementation guide for CDS tool development and implementation. Additional future research should include the use of systems safety engineering and digital twin approaches in other critical care and non-critical care environments to validate the findings. Finally, applying the findings from this work to new CDS tools that leverage artificial intelligence and machine learning for predictive analytics will also be a key future direction.

CONCLUSIONS

The DISCOVER Learning Lab’s application of a systems safety engineering and human factors approach to problem analysis, partnered with novel use of immersive VR and digital twin technology, led to valuable insights into common failure modes and potential opportunity areas to improve diagnostic accuracy and care delivery in a quaternary referral center PICU. This study provides a comprehensive evaluation of CDS tool uptake in the PICU, with the findings and insights offering a pathway to more effective CDS tool design to support diagnosis in critically ill children. Learnings from the problem analysis will lead to design-informed CDS tools that will enhance clinician engagement and improve diagnostic outcomes, leading to timely and appropriate management in a high-acuity environment. In addition, learnings from our approach will provide a framework for supporting future CDS development and implementation.