Artificial Intelligence-Assisted Surgery: Potential and Challenges

Abstract

Background

Artificial intelligence (AI) has recently achieved considerable success in different domains including medical applications. Although current advances are expected to impact surgery, up until now AI has not been able to leverage its full potential due to several challenges that are specific to that field.

Summary

This review summarizes data-driven methods and technologies needed as a prerequisite for different AI-based assistance functions in the operating room. Potential effects of AI usage in surgery will be highlighted, concluding with ongoing challenges to enabling AI for surgery.

Key Messages

AI-assisted surgery will enable data-driven decision-making via decision support systems and cognitive robotic assistance. The use of AI for workflow analysis will help provide appropriate assistance in the right context. The requirements for such assistance must be defined by surgeons in close cooperation with computer scientists and engineers. Once the existing challenges will have been solved, AI assistance has the potential to improve patient care by supporting the surgeon without replacing him or her.

The Potential of Artificial Intelligence in Surgery

Artificial intelligence (AI) has recently achieved considerable success in domains such as object detection, speech recognition, or natural language processing [1]. Especially deep learning (DL) techniques have been responsible for such breakthroughs, and they have experienced a renaissance due to the massive increase in computational power and data availability [2]. DL is a subset of machine learning, which itself is part of the all-encompassing concept of AI. DL methods are based on artificial neural networks, which are inspired by neurons in a biological brain. DL refers to the concept of training specific tasks based on a large amount of data, learning from them and making predictions about these specific tasks through flexible adaptation to new data.

Recently, several success stories have been published in the medical domain based on DL for image classification, such as prediction of cardiovascular risk based on retinal images [3], skin lesion classification [4], or breast cancer detection based on mammograms [5]. However, in surgery, AI has not yet leveraged its full potential, due to several challenges that are specific to this discipline. Unlike in the aforementioned examples, which are focused on the analysis of static images, surgery consists of procedural data in a dynamic environment including the patient, different devices and sensors in the operating room (OR), and the OR team, as well as domain knowledge such as clinical guidelines or experience from previous procedures [6]. Furthermore, DL methods require large amounts of annotated data for training, which is especially challenging in surgery. In addition, real-time capability is required of machine learning methods if they should be used during an operation.

Surgical data science is a newly emerging field that has the aim of “improving the quality of interventional healthcare and its value through capturing, organization, analysis, and modeling of data” [6], especially using AI-based methods. Any potential for surgery can be identified along the surgical treatment path, such as decision support, context-aware assistance, and cognitive robotics (Fig. 1).

Fig. 1.

Potential of artificial intelligence (AI) in surgery based on a sensor-enhanced operating room (SensorOR).

The remainder of this review will focus on data-driven methods and technologies as a prerequisite for different AI-based assistance functions in the OR. Potential effects of AI use in surgery will be highlighted, concluding with ongoing challenges to enabling AI for surgery.

Enabling AI-Assisted Surgery

The following paragraphs provide a short introduction to the basic technologies and concepts needed as a prerequisite for AI-assisted surgery. This includes (1) access to comprehensive data in a sensor-enhanced OR (SensorOR) as well as (2) enrichment of those data with surgical knowledge by annotation to make them usable for (3) machine learning methods that provide AI assistance to the surgeon.

SensorOR

Today's operating theaters are characterized by a multitude of information sources. For example, preoperative image and planning data provide information about the positions of tumors and the planned course of the operation, various medical devices (e.g., a suction-irrigation system, operating light, and anesthesia monitor) provide regular status reports and intraoperative imaging devices (endoscope, ultrasound, etc.) provide data about the patient and current processes in the OR. In their entirety, these heterogeneous sensors provide the information necessary to infer the actual course of the operation and to provide proper assistance at the right time. This is subsumed under the term “context-aware assistance” [7], which avoids an information overflow and decreases the cognitive load, in particular in an already stressful and complex environment such as the OR. A prerequisite for providing such assistance is a SensorOR in which all devices are connected to collect their data (Fig. 1).

Data Annotation

In order for AI to learn from collected data, the original raw data has to be enriched with additional knowledge through annotation. The most common forms of annotation are classification (e.g., which organs are visible in an image), semantic segmentation (e.g., which pixels belong to which organ in an image), and numerical regression (e.g., the size of an object). The process of data annotation is often time-consuming, especially for large data sets. Annotation of surgical data requires expert knowledge, which can be expensive to muster and is often a bottleneck. In addition, only a fraction of the data is digitally available, and there are no standard acquisition and annotation protocols. The data has to be representative of the task to be learned, preferably from multiple centers and accessible in comprehensive open data registries, highlighting questions regarding privacy and confidentiality.

Several approaches to reducing the annotation effort have been proposed, such as active learning [8, 9, 10], where only the most informative data points are selected and then are annotated, as well as crowdsourcing, where the “wisdom of the crowd” can be utilized for certain clinical tasks [11, 12]. A promising pathway for overcoming the lack of annotated data is to generate realistic synthetic images based on a simple simulation by using generative adversarial networks [13] (Fig. 2).

Fig. 2.

Left: example of generating synthetic images based on simple laparoscopic simulations. Middle: generative adversarial networks translate these images so they look like real laparoscopic images. These images, along with their generated labels, can be used without further annotation effort. Right: machine-learning-based detection of the surgical phase and prediction of the duration of the remaining procedure during laparoscopic cholecystectomy.

To ensure a consistent vocabulary for annotation, ontologies are used. Ontologies are widely used in the medical domain − for instance, for clinical terms [14], for modeling surgical knowledge [15], to identify risks across medical processes [16], or for surgical processes [17, 18].

In other AI domains [19], there are many open data sets that can be used to develop, evaluate and compare different machine learning algorithms. While access to data sources is crucial also in surgery, only few public annotated data sets for different applications such as surgical phase detection [20, 21], surgical training [22, 23], and segmentation [21, 24] exist [7]. The Endoscopic Vision Challenge [25], an initiative that supports the availability of new public data sets for the systematic comparison of algorithms, has hosted challenges in surgical vision.

Machine Learning

Machine learning algorithms can operate in a supervised or an unsupervised manner. While in both cases the algorithms rely on data to learn, in supervised learning, the data have to be annotated [26]. DL, a popular example of supervised learning [2], is the current state of the art for many applications, such as surgical data analysis. Methods for DL have the benefit that they can be “pretrained,” meaning that a method used to solve one problem, e.g., transforming grayscale laparoscopic images into color images, can be retrained to solve a different task, e.g., segmenting laparoscopic tools [27]. This makes it possible to retain and apply knowledge from the former task to improve upon the later task.

Such methods can be used for workflow analysis through automatic segmentation of procedures into phases or surgical actions. These methods rely, for example, on data from tool usage [28, 29] or robotic kinematics [30] or, by using DL, directly on camera data, such from an endoscope [20, 31, 32] or 3D camera [32]. DL methods for determining surgery duration (Fig. 2) [33, 34, 35] and for predicting surgical tool usage [36] have also been developed.

Of further importance are methods for semantic segmentation, especially surgical scene analysis, which provides information on relevant structures, such as tools and organs, during surgery. Until recently, most research has focused on surgical tools [37], though the broader topic of analyzing the entire surgical scene is becoming more popular [38]. One application of semantic segmentation has been used for measurements during surgery with a stereo-endoscope [39].

AI has the potential to enhance soft-tissue navigation where the risk and target structures are highlighted during surgery. DL methods have been used in data-driven registration and deformation models that estimate non-rigid deformations of structures inside an organ when given only the displacement of the partial organ surface [40, 41, 42], a prerequisite for soft-tissue navigation.

Potential Effects of AI in Surgery

The abovementioned novel AI technologies developed by computer scientists will have tremendous effects on surgical practice. The effects can be categorized into the fields of decision support, context-aware assistance, and cognitive robotics [6].

Decision support systems that gather patient information in order to provide a clinical recommendation are commercially available and under scientific investigation in internal medicine [43], but they show only marginal change in clinical practice. Novel AI-based systems can predict future acute kidney injury based on data from hundreds of thousands of patients [44]. Similarly, AI can be used to predict circulatory failure in the intensive care unit [45]. Another system for intraoperative blood pressure management has shown its benefit in supporting the anesthesiologist in a randomized controlled trial [46]. In surgery, however, we still lack these kinds of assistance system. For example, in oncological liver surgery, one would like to have a system that combines patient information (laboratory results, imaging, comorbidities, and previous surgeries) with evidence-based information (scientific publications) and surgical experience (the clinical course of previously treated patients) to avoid post-hepatectomy liver failure or to choose the optimal multimodal treatment. Similarly, AI may help to improve systems of predictive analytics that estimate survival after pancreatoduodenectomy for pancreatic cancer [47] or secondary effects of surgery, such as incisional hernia [48].

Context-aware assistance systems will provide this additional information for the right patient at the right point in time. To achieve this, the surgical workflow may be optimized by modelling the process [49] to define important steps during the operation and recognize them using machine learning, such as has been shown by Twinanda et al. [20] for laparoscopic cholecystectomy. This may also allow the detection of dangerous deviations from the optimal workflow. Extended information could be provided to predict the remaining duration of a surgery to optimize the organizational workflow and treat more patients. Bodenstedt et al. [34] demonstrated a way of online procedure duration prediction using unlabeled endoscopic video data and surgical device data in a laparoscopic setting. Relevant information may also be obtained by means of AI such as computer vision for detecting structures at risk such as the cystic duct and common bile duct during laparoscopic cholecystectomy, as demonstrated by Tokuyasu et al. [50].

AI may also change practice in robot-assisted surgery towards cognitive surgical robotics. Today, clinically used surgical robots are mere telemanipulators without any autonomous activity. In research, robotic systems have been developed for situation-aware automatic needle insertion [51]. Another system has shown its superiority over humans in performing bowel anastomosis on a porcine bowel [52]. However, even these robotic systems do not understand the surgical scene and do not adapt to the surgical workflow. This is why surgical workflow analysis and an understanding of the surgical scene have to be developed and validated towards a level of robustness where they can be used as an information source for cognitive surgical robots. Only then can auxiliary tasks such as control of the laparoscopic camera and stretching tissue, or even certain major surgical tasks such as an anastomosis, be performed by a cognitive robot. Using AI, such a cognitive robot will understand its environment and may even learn from experience to improve its performance over time.

Conclusions

AI-assisted surgery enables objective data-driven decision-making and will have a strong impact on how surgery is performed in the future. Clinical implications arise from the aim of assistance, but not replacement of surgical expertise. Decision support systems will recommend, not make, a decision. Cognitive robots will perform autonomous actions only as requested by surgeons. Less experienced surgeons will need this support more often than experienced surgeons. However, in the end, all AI assistance has to improve patient outcomes to be effective.

While machine learning methods are evolving and further achievements are to be expected, there exist numerous challenges in the surgical domain that have hampered any significant impact so far. Challenges arise regarding data, methods, devices, and integration. Machine learning methods require large amounts of labeled training data, which are difficult and expensive to acquire considering the need for high-quality annotations. Furthermore, such methods need to be robust and accurate, as well as able to deal with heterogeneous data sources and high variability, since the course of surgery greatly depends on the patient and the OR team, which makes it difficult to predict anomalies that are not represented in the training data. To reduce the amount of training data and to address diversity, DL approaches could be combined with semantic knowledge to incorporate medical background knowledge and context. Up to now, such methods have behaved like a “black box,” which is critical for such a high-risk domain as surgery. Research towards explainable AI can overcome the black box paradigm and make decisions more transparent and traceable by humans [53]. Another important aspect is related to the devices and their integration; for instance, to enable data-driven online analysis in the OR, devices have to be connected and accessible (SensorOR), which can only be achieved in collaboration with the device manufactures.

In summary, AI-assisted surgery has the potential to improve patient care if the aforementioned challenges are addressed by all stakeholders including clinicians, engineers, patients, and industry.

Conflict of Interest Statement

The authors have no conflicts of interest to declare.

Funding Sources

Funded by the German Research Foundation (DFG, Deutsche Forschungsgemeinschaft) as part of Germany's Excellence Strategy − EXC 2050/1 − Project ID 390696704 − Cluster of Excellence “Centre for Tactile Internet with Human-in-the-Loop” (CeTI) of TU Dresden.

Author Contributions

All authors drafted and reviewed the manuscript.