Artificial intelligence in cardiothoracic surgery

Abstract

The tremendous and rapid technological advances that humans have achieved in the last decade have definitely impacted how surgical tasks are performed in the operating room (OR). As a high-tech work environment, the contemporary OR has incorporated novel computational systems into the clinical workflow, aiming to optimize processes and support the surgical team. Artificial intelligence (AI) is increasingly important for surgical decision making to help address diverse sources of information, such as patient risk factors, anatomy, disease natural history, patient values and cost, and assist surgeons and patients to make better predictions regarding the consequences of surgical decisions. In this review, we discuss the current initiatives that are using AI in cardiothoracic surgery and surgical care in general. We also address the future of AI and how high-tech ORs will leverage human-machine teaming to optimize performance and enhance patient safety.

Artificial intelligence in surgery

The term artificial intelligence (AI) has a range of meanings, from specific forms of AI, such as machine learning, to the more far-fetched idea of AI that meets criteria for consciousness and sentience. AI systems range from those that seek to model human reasoning to solve a problem, to those that exclusively use large datasets to generate a framework to answer the problem of interest, to those that attempt to incorporate elements of human reasoning but do not require accurate modeling of human processes. Machine learning (ML) is a family of statistical and mathematical modeling techniques that uses a variety of approaches to automatically learn and improve the prediction of a target state, without explicit programming (e.g. Boolean rules). Different methods, such as Bayesian networks, random forests, deep learning, and artificial neural networks, each use different assumptions and mathematical frameworks for data input, and learning occurs within the algorithm.¹

AI is increasingly important for surgical decision making to help address diverse sources of information, such as patient risk factors, anatomy, disease natural history, patient values and cost, and assist surgeons and patients to make better predictions regarding the consequences of surgical decisions.² For instance, a deep learning model was used to predict which individuals with treatment-resistant epilepsy would most likely benefit from surgery.^{1, 3} AI platforms can provide roadmaps to aid the surgical team in the operating room, reducing risk and making surgery safer.^{4, 5} In cardiothoracic surgery, previous studies have developed machine learning algorithms that can outperform standard operative risk scores in predicting intrahospital mortality after cardiac procedures.⁶

In addition to planning and decision making, AI may be applied to change surgical techniques. Remote-controlled robotic surgery has been shown to improve the safety of interventions where clinicians are exposed to high doses of ionizing radiation and make surgery possible in anatomic locations not otherwise reachable by human hands.^{7, 8} As autonomous robotic surgery improves, it is likely that surgeons will in some cases oversee the movements of robots.⁹

Surgical data science

With the emergence of novel technologies and their incorporation into the operating room (OR), alongside the enormous amount of data generated through patient surgical care, a new scientific discipline called surgical data science (SDS) was created. The main goal of SDS is to improve the quality of interventional healthcare and its value by capturing, organizing, processing and modeling data.¹⁰ Within SDS, complex data can emerge from different sources, such as patients; operators involved in delivering care; sensors for measuring patient and procedure-related data; and domain knowledge. Built upon SDS, promising applications of AI and ML have been developed with the ultimate goal of supporting surgical decision-making and improving patient safety.¹¹ Differently from more traditional data modeling approaches which are mostly based on regression techniques, SDS leverages machine learning techniques that can learn relationships between data features without much input from the human modeler. In unsupervised machine learning, for example, there are no pre-existing labels/annotations, requiring minimum human supervision. Previous supervised and unsupervised machine learning techniques have been used to assess physician competence in a variety of settings.¹²

A good example of how SDS can be used for quality improvement initiatives is the OR black box system.¹³ This analytic platform allows the capture and integration of a wide variety of intraoperative data (e.g. audio, video, physiological parameters), enabling both human- and AI-based metrics. Recent studies have been using this platform to investigate technical and non-technical surgical performance and their relationship with patient outcomes.^{14, 15} More recently, few studies have been able to demonstrate the feasibility and validity of machine learning algorithms to early predict intraoperative complications, such as hypotension and hypoxemia in both noncardiac^{5, 16} and cardiothoracic surgery.^{6, 17}

Another data-driven application of AI in surgery concerns the assessment of intraoperative performance at both individual and team levels. Current gold standard assessments of intraoperative technical and non-technical skills are based on observation and rating by experts. Although these methods are widely used, there are many limitations related to the inherent subjectivity of these tools, suboptimal inter-rater reliability and limited reproducibility and scalability. The use of AI, especially computer vision, offers a promising opportunity to automate, standardize and scale performance assessment in surgery, including cardiothoracic surgery. Prior investigations have documented the reliability of video-based surgical motion analyses for assessing laparoscopic performance in the operating room as compared to the traditional time-intensive, human rater approach.¹⁸ Azari et al. compared expert surgeon’s rating assessments to computer-based assessments of technical skills (e.g. suturing, knot tying) including fluidity of motion, tissue handling and motion economy.¹⁹

Augmented cognition in the OR

The tremendous and rapid technological advances that humans have achieved in the last decade have definitely impacted how surgical tasks are performed in the OR. As a high-tech work environment, the contemporary OR has incorporated novel computational systems to the clinical workflow, aiming to optimize processes and support the surgical team.²⁰ In addition to generating an enormous amount and variety of data, which can be used for developing predictive machine learning models, this complex computational-based environment has also enabled the augmentation of human cognition at both individual and team levels.²¹ In this high-tech OR, cognition is extended outside individuals’ minds throughout the entire surgical team, incorporating not only human agents but also non-human systems that are involved during the course of surgery.

Cardiothoracic surgery is a perfect example of how AI can be used to support surgical care through cognitive augmentation. The cardiothoracic OR is a high-risk high-stakes environment, where multiple specialized professionals interact with each other, coordinate tasks as a team, and use a variety of equipment, technological devices and interfaces to effectively care for complex patients in need of surgical treatment.²² By functioning as a complex socio-technical system, the cardiothoracic team performs tasks in a coordinated way, requiring cognitive abilities that are beyond each individual team member’s performance. Since each team member in isolation does not have control of the team performance as a whole, cognitive activities are emergent processes of teamwork rather than individual tasks.²³

Existing AI systems are able to collect, process and make sense of information gathered in the OR.²⁴ However, an important requirement for these systems is the ability to understand and adapt their algorithms based on real-time contextual information, enabling them to provide context-aware assistance.^{25, 26} Predictive accuracy is also important. In order to be able to support and guide team cognitive tasks, an AI system should anticipate future states by using past and current information from the OR’s human and non-human systems.²⁷ In terms of cognitive augmentation in the OR, cognitive state monitoring and human activity recognition are essential features of an AI system. To monitor cognitive states at both individual and team levels, physiological metrics such as heart rate variability (HRV), electroencephalography (EEG) and near-infrared spectroscopy (NIRS) are the most used, since they allow real-time objective measures of cognitive load.²⁸ Figure 1 displays the cardiac surgeon’s cognitive load, indexed by HRV (LF/HF ratio) during different steps of a cardiac procedure.

Figure 1.—

Surgeon’s cognitive load as measured by the HRV parameter: low frequency (LF)/high frequency (HF) ratio captured by a wearable physiological sensor.

Computer vision in surgery

In the realm of human activity recognition, computer vision is a promising AI method that can be used for surgical task segmentation and team dynamics monitoring. To encompass all the advances and future potentials of the use of AI to enhance cognition in the OR, a new interdisciplinary field called “cognitive surgery” or “cognition-guided surgery” has recently been created.²⁵ Computer vision is a branch of AI that extracts and processes data from images and videos and provides the machine understanding of this data.²⁹ In several fields, computer vision technologies are able to achieve human-level performance, and in certain cases even exceed human abilities.^29-31

In surgery, the main applications of computer vision are related to surgical workflow segmentation,^{32, 33} instrument recognition and detection,³⁴ and image-guided surgical interventions.³⁵ However, a new area for applying computer vision in the OR, especially in team-based complex procedures, such as cardiothoracic surgery, is in the understanding of individual and team behaviors. Other fields of medicine and psychology already are using automated body position and movement tracking to investigate human non-verbal behaviors.³⁶ In surgery, most of the applications of this technology involve tracking surgeon’s gestures and hands motion to extract objective metrics of technical psychomotor skills.³⁷ However, recent studies have explored the use of position and motion data generated by computer vision applications to measure team dynamics and coordination in the OR. Team centrality and team proximity are examples of behavioral metrics investigated in these studies (Figure 2).^37-39

Figure 2.—

Computer vision system extracting human body position and motion in the cardiac operating room.

Autonomous robotic surgery

Robotic technology is going to change the face of surgery in the near future. Robots are expected to become the standard modality for many common procedures, including coronary bypass and abdominal surgery. Autonomous and semi-autonomous modes are increasingly being investigated and implemented in surgical procedures, automating various phases of the operation. The complexity of these tasks is also shifting from the low-level automation early medical robots to high-level autonomous features, such as complex endoscopic surgical manoeuvres and shared-control approaches in stabilized image-guided beating-heart surgery. Future progress will require a continuous interdisciplinary work, with breakthroughs such as nanorobots entering the field. Autonomous robotic surgery is a fascinating field of research involving progress in artificial intelligence technology.^{9, 40}

Machine-learning-empowered instrumentation for robotic-assisted surgery is the object of intense investigation. For ML surgical skill learning, expert knowledge is typically supplied by experienced surgeons. Implicit imitation learning is a form of supervised learning, which is usually concerned with accelerating reinforced learning through the observation of an expert mentor. The agent observes the state transitions of the experts’ actions and uses the information extracted from these observations to update its own states and actions. The mentor (surgeon) and the agent may have identical or different action capabilities, or identical or different reward structures. Several methods that have been developed for modeling human movement could be used to learn the state and actions of the expert. Human skill has been modeled from sets of recorded data using hidden Markov models, neural networks, and fuzzy nets.⁴¹

Human-machine teaming in the OR

As computational systems become ubiquitous, and our workplace is full of computer-based devices and networks, new forms of interaction, communication, and coordination have surged.⁴² The way computer-based systems are designed and operated in the cardiothoracic OR plays a critical role in workflow efficiency, clinicians’ cognitive load and, ultimately, surgical performance.²²

When AI systems are integrated within a complex OR environment, the opportunity for human-machine teaming emerges, creating novel cognitive engineering opportunities that have the potential to enhance patient safety and improve clinical outcomes in complex team-based surgery.

Conclusions

Recent research has aimed to develop intelligent machine teammates, developing innovative computational algorithms and expanding the use of human cognitive models for AI.⁴³ This research has produced novel forms of human-machine teaming in healthcare applications, transportation, manufacturing assembly lines and defense.^{44, 45}

In cardiac surgery, novel systems have been developed aiming to integrate clinicians’ physiological data, as a proxy for cognitive performance, with patient data and OR medical devices.^{46, 47} Some studies have attempted to optimize surgical coordination and team communication by using a data-driven approach that integrates human and non-human agents to enhance safety and mitigate errors in the cardiothoracic OR.^{48, 49}