A Clinician-Centered Explainable Artificial Intelligence Framework for Decision Support in the Operating Theatre

INTRODUCTION

The integration of Artificial Intelligence (AI) into clinical decision support systems (CDSS) marks a significant advancement in the pursuit of enhanced patient care and operational efficiency in high-stakes environments, such as the operating room (OR) [1]. However, the complexity and often “black box” nature of these AI systems pose substantial challenges for human-AI teaming, where explainability is crucial for clinicians to understand and trust the AI’s recommendations. The field of explainable Artificial Intelligence (xAI) offers promising methods and techniques to make AI decisions more explainable and interpretable, yet the adaptation of these methods to meet the specific needs of clinicians remains underexplored [2]. Our proposed clinician-centered framework, xAI-SURG, seeks to bridge this gap by aligning xAI approaches with the complex decision-making processes inherent in the OR. By evaluating the xAI-SURG framework through the lens of perfusionists’ decision-making during cardiac surgery, we provide a use-case, showcasing the importance of tailored xAI that not only aligns with but also enhances clinician expertise and patient safety and outcomes in the OR.

MATERIALS AND METHODS

Study Design and Setting:

This was a cross-sectional study conducted via online videoconferencing through semi-structured interviews. All sessions were audio recorded. The study protocol was approved by the local Institutional Review Board (IRB) and all participants completed an informed consent process.

Population:

Participants of this study were perfusionists with more than 10 years of experience, working in a cardiac surgery OR, in the United States.

Procedures:

Semi-structured interviews were conducted for 90 minutes with each participant via Zoom videoconferencing application, facilitated by two human factors experts. During the interviews, we applied Cognitive Task Analysis (CTA) which is a broad set of methods used to describe and explain the mental processes and strategies that people use to perform complex tasks. The goal of CTA is to gain a deep understanding of how cognitive skills, such as perception, memory, judgment, and decision-making, contribute to task performance, especially in specialized and skilled domains [3]. Our CTA protocol explored the expert perfusionist’s decision-making process during Goal-Directed Perfusion (GDP), which is a strategy used during cardiopulmonary bypass (CPB) procedures, aiming to optimize the perfusion parameters to meet the metabolic needs of the patient’s body during surgery and avoid metabolic acidosis. This strategy is focused on achieving specific physiological goals to improve patient outcomes and reduce the risk of complications that can be associated with CPB. The GDP approach involves careful monitoring and control of various intraoperative factors such as Oxygen delivery (DO2), Pump flow rates, Hemoglobin levels, Temperature management, and Blood Pressure control [4].

Data Synthesis:

Based on qualitative analysis of the interview findings and through iterative discussions among the research team, we elucidated the perfusionists’ decision-making during DO2 monitoring as part of the GDP strategy (Goal of a DO2 > 280). We captured the problems and pitfalls that may occur during this complex task, their causes, problem-solving, and mitigation strategies, decision points, critical communications, and possible courses of action, as well as the myriad input information needed to reach a decision, and factors influencing perfusionists’ decision-making during GDP. Based on this analysis we build a CTA Map illustrating the decision-making process of expert perfusionists when implementing GDP (Fig. 1).

Fig. 1.

Cognitive Task Analysis (CTA) Map of Perfusionists’ Decision-Making during Goal-Directed Perfusion (GDP) in Cardiac Surgery.

Framework Development:

To develop the xAI-SURG framework, we build on the well-established Recognition-primed decision-making (RPDM) model [5] that describes how experts make quick, effective decisions in complex situations without comparing options. Experts’ prior experience is leveraged to recognize patterns and similarities to past situations, which then informs their decision-making. In high-stakes environments, such as the OR, where perfusionists must make rapid decisions during CPB procedures, integrating RPDM with the detailed cognitive task analysis (CTA) map can enhance our understanding of the underlying human factors entailed in clinical decision-making processes. By integrating the CTA Map with the RPDM model, we developed an interactive framework for the design, development, and evaluation of xAI approaches for CDS in the OR.

RESULTS

Eight expert perfusionists were included in this study, with ages ranging from 37 to 59 years, only 1 female, and experience in the OR ranging from 13 to 33 years working as perfusionists. Fig. 1 displays the CTA Map and its components related to GDP in cardiac surgery. The map lays out the complex interplay of causes, inputs, and actions that perfusionists must navigate to maintain optimal oxygen delivery (DO2) to the patient’s tissues during CPB. From the map, we can infer that the perfusionist’s ultimate goal is to keep the DO2 above 280 ml/min/m². To achieve this, perfusionists monitor a range of variables and make decisions based on:

• Causes: Such as factors that can impair venous drainage, such as airlock, and wrong cannula size, which may lead to the nadir of DO2 falling below 280 ml/min/m², prompting immediate action.

• Inputs: Critical information provided by the surgeon, visual cues, and a variety of clinical measurements like hematocrit levels, arterial flow, body surface area, and patient’s blood pressure. Perfusionists must integrate this data to assess the patient’s perfusion status and make decisions at regular intervals.

• Actions: Potential interventions, such as administering crystalloid, alerting the surgeon, adjusting flow rates, or transfusing. These actions are contingent on the dynamics of the surgical procedure and the physiological needs of the patient at any given moment.

DISCUSSION

In this study, we propose the xAI-SURG framework and its potential to be integrated into AI-based surgical decision support systems in the OR. The framework, underpinned by a detailed CTA map and enriched with RPDM principles, has been shown to align closely with the cognitive workflow of expert perfusionists. The use of xAI techniques, such as feature importance and case-based reasoning, may provide a transparent decision-making process that is both interpretable and actionable, which is essential in the complex environment of cardiac surgery. Such a system has the potential to support OR teams by reinforcing their intuitive judgment processes with data-driven, explainable AI insights, potentially leading to improved patient safety and outcomes [6].

The discussion of our results must consider the broader implications of AI in healthcare. By grounding AI recommendations in the practical and experiential knowledge base of clinicians, the xAI-SURG framework is interpretable and explainable by design and may contribute to significant steps toward realizing the full potential of AI as a collaborative teammate in the OR. This framework respects the expertise of clinicians and supports their naturalistic decision-making processes, which is critical in situations where rapid and accurate decisions can have profound impacts on patient safety and quality of care. It also points to the need for a careful balance between AI-driven recommendations and human input, ensuring that AI supports but does not overshadow clinicians’ expertise. Looking forward, the integration of xAI into clinical practice in the OR will require ongoing evaluation and refinement. As we continue to develop and test the xAI-SURG framework, special attention must be paid to the adaptability of the system to individual clinician needs and the various surgical contexts. Future research should focus on longitudinal studies to assess the impact of the xAI-SURG framework on clinical outcomes and to explore its application across different surgical specialties and roles, and among junior clinicians. Moreover, further development of the user interface will be crucial to ensure that the system is user-friendly and enhances, rather than disrupts, clinical workflows without causing cognitive overload.