The application of augmented reality in robotic general surgery: A mini-review

Abstract

In robotic surgery, surgical planning and surgical navigation represent two crucial elements, allowing surgeons to maximize surgical outcomes while minimizing the risk of complications. In this context, an emerging imaging technology, namely augmented reality (AR), can represent a powerful tool to create an integration of preoperative 3D models into the live intraoperative view, providing an interactive visual interface rather than a simple operative field. In this way, surgeons can be guided by preoperative imaging during the operation. This makes the surgical procedure more accurate and safer, leading to so-called “precision surgery”. This article aims to provide an overview of developments in the application of AR in robotic general surgery. The integration of this imaging technology in this surgical field is showing promising results. The main benefits include improved oncological outcomes and reduced occurrence of complications. In addition, its application may also be important for surgical education. However, we are still in the initial phase of the experience and some important limitations remain. Moreover, to our knowledge, to date, reports in the literature regarding the integration of AR in robotic general surgery are still very limited. To improve its application, close collaboration between engineers, software developers, and surgeons is mandatory.

1. Introduction

In the last few decades, in the field of general surgery, minimally invasive approaches have been increasingly used.

In this context, laparoscopic surgery has amply demonstrated its advantages, such as reduced postoperative pain and shorter recovery, with comparable results compared with open surgery in terms of complications and oncological outcomes. However, laparoscopy, especially regarding challenging surgical procedures, has some disadvantages, such as limited maneuverability. Laparoscopic instruments are rigid and can be opened and closed to grasp or cut, allowing five degrees of freedom. In this field, improvements in instrumentation have lagged behind clinical developments [1,2,3,4,5,6].

More recently, robotic surgery has been able to overcome some of the limitations of the laparoscopic approach [7,8,9,10,11,12,13]. The robotic platforms allow the combination of the multiple advantages of laparoscopy and the dexterity of the open approach, also offering better visualization and ergonomics. Concerning robotic instruments, they are equipped with a miniaturized wrist that allows them to achieve seven degrees of freedom [14,15].

In consideration of its advantages, the use of the robot in surgery is gradually extending to almost all surgical specialties [16,17,18,19]. The main obstacle to the wide diffusion of robotic surgery is represented by the high costs, both direct (related to the robotic platforms and necessary instrumentation) and indirect (related to operative times, which are longer) [20].

In robotic surgery, surgical planning and surgical navigation represent two crucial elements, allowing surgeons to maximize surgical outcomes while minimizing the risk of complications. In this context, an emerging imaging technology, namely augmented reality (AR), can represent a powerful tool to create, by means of dedicated software, which can be supported by robotic platforms, an integration of preoperative 3D models into the live intraoperative view, providing an interactive visual interface rather than a simple operative field. In this way, surgeons can be guided by preoperative imaging during the operation [21,22,23,24,25,26,27]. This makes the surgical procedure more accurate and safer, leading to so-called “precision surgery”. The emerging concept of “precision medicine”, in this case “precision surgery”, aims to personalize treatment for each individual patient with each specific disease [28,29].

This article aims to provide an overview of developments in the application of AR in robotic general surgery.

2. General principles on AR

The most common definition of AR, enunciated by Azuma in 1997, states that in AR “3D virtual objects are integrated into a 3D real environment in real-time” [30]. In fact, this imaging technology allows to improve the perception of the real world by superimposing virtual images on it.

It is important to distinguish AR from two other similar but different imaging technologies: virtual reality (VR) and mixed reality (MR). VR transports users to a completely different and immersive environment. Unlike AR (and even MR), this imaging technology isolates users from the real world, providing an entirely artificial experience. MR makes a step forward from AR by combining the virtual and physical worlds in a way that enables virtual objects to interact with the real environment. Through this imaging technology, users not only see the digital content but also can manipulate and interact with it as if it is part of their surroundings [23].

An AR environment can be experienced by means of various technologies, such as computer monitors, mobile displays (smartphone or tablet screens), head-mounted displays, and the recently introduced AR goggles (e.g., Google Glass and Microsoft Hololens) [23].

From a historical point of view, AR dates back to the invention of VR in the 1960s. In those years, Sutherland introduced the concept of “Ultimate Display”, which stands for the simulation of a synthetic environment similar to the actual reality [23,31]. Subsequently, Milgram and Kishino introduced the “virtuality continuum” concept, which created the connection between the real world and a completely virtual one [23,32].

Over the last decades, AR has been used in various areas to improve visual feedback from information systems, becoming a popular multidisciplinary research topic. The continuous developments in the field of technology (represented, for example, by advanced cameras, faster computers, and novel algorithms) increasingly motivate researchers to broaden the application areas of AR, overcoming its limitations. Sensing errors and registration issues are considered the most important challenges of AR technology [23,33].

In recent years, several applications of AR have been developed in the field of robotics. In this context, AR can act as a new tool for information exchange with autonomous systems, enhancing the effectiveness of the interaction between humans and robots. Robots are progressively becoming omnipresent in everyday life, expanding their conventional use in industry to other areas, including the field of medicine [23].

3. AR and robotic surgery

In medicine, and in particular in the field of surgery, the application of AR existed before the introduction of robotic surgery.

In the early 2000s, AR was applied in laparoscopic operations, in various surgical branches. In this context, collaboration between engineers, software developers, and surgeons has led to progressive developments in this field of minimally invasive surgery [34,35,36,37,38,39].

However, robotic surgery represents the area in which AR may have the best application. In fact, although this imaging technology may also be used during laparoscopy, in robotic surgery surgeons are able to interact directly with the computer (robotic platform), greatly facilitating the integration [23,24,40].

3.1. Usefulness during the surgical procedure

During robotic surgery, as mentioned above, AR can provide valuable help, even to already experienced surgeons. In fact, when the preoperative 3D reconstruction of the surgical target is accurately superimposed on the surgical field, the interpretation of intraoperative anatomy is considerably facilitated. This simplifies decision-making processes during the operation and improves operative precision, allowing safer surgery [21,22,23,41,42,43].

In the literature, there are some studies reporting the application, in different surgical specialties, of AR in robotic surgery. Liu et al. [44] described this integration in transoral robotic surgery to perform base-of-tongue tumor resection. In craniomaxillofacial surgery, this application was documented by Lin et al. [45]. Wang et al. [46] reported an AR navigation system in oral and maxillofacial surgery. In robotic mandibular plastic surgery, the integration of AR was described by Zhou et al. [47]. Roberts et al. [26] and Porpiglia et al. [48] documented the use of AR technology during urological robot-assisted procedures (such as prostatectomy and partial nephrectomy). In the field of neurosurgery, various applications of AR in robotic systems are reported [49].

In relation to the different surgical specialties, it is important to emphasize that the accuracy of virtual image superposition is completely different if this imaging technology is applied in surgical procedures involving rigid and immobile structures or in operations involving mobile and deformable structures [36,50].

3.2. Usefulness in surgical education

The integration of AR into robotic platforms can also potentially be very useful for surgical education [50,51].

This imaging technology may represent a useful tool for residents or less-experienced surgeons to better identify anatomical structures, thus making it easier to perform operations. Moreover, for the same reason, AR can be used to train surgeons in new procedures [50,51].

It is important to emphasize that through the application of AR in robotic surgery, instead of passive observation of the operations, real-time trainee teaching is possible [21,50,51].

In laparoscopic surgery, AR-based surgical training appears to accelerate the acquisition of simple skills (such as suturing) and even reduce the learning curve in more complex operations [21,52,53].

In robotic surgery, similar applications in surgical education are lacking, with only a few experiences reported [21,54,55].

3.3. Limitations

Although the advantages related to the application of AR in robotic surgery are clear, its use in clinical practice is still very limited [21,22,56,57,58].

The main limitation hindering the wide extension of this integration, particularly in some surgical specialties, is represented by the continuous misalignment of 3D models with respect to the real surgical field. Factors causing the misalignment include the interference due to surgical instruments, the mobility and deformability of some organs, and the interference caused by breathing movements [21,22,58].

Current research in this field is focused on the development, by employing more efficient deep learning algorithms, of an automated overlay system that allows the automatic anchoring of virtual models to the real organs. However, significant improvements have not yet been achieved [21,22,40,58,59,60].

4. AR in robotic general surgery

The application of AR technology can provide valuable help in the field of general surgery. For example, in a cholecystectomy, where bile duct injury, associated with anatomical distortion due to aberrant anatomy or inflammation, may be a disastrous complication, this imaging technology, with an intraoperative overlay of safe dissection areas, can be a very useful tool for improving surgical outcomes [61].

However, the application of AR in robotic general surgery is still very limited compared to other surgical fields, such as urology, otolaryngology, neurosurgery, maxillofacial surgery, and orthopedics [21,22,23,62,63,64]. This difference is mainly due to the mobility and deformability of the target organs, as described above.

The branch of general surgery in which the application of this imaging technology is developing most is robotic liver surgery. In this context, AR may allow to overcome the widely described limitations in performing liver resections with this approach. In fact, despite the enhanced video resolution and the sense of depth obtained through the 3D visualization, robotic liver surgery has some of the disadvantages of laparoscopic surgery compared to the open approach [22,23,65,66].

The main issue in this context is certainly represented by the absence of tactile feedback, which helps the surgeon to orient himself during parenchymal transection. In this regard, the integration of AR into robotic platforms can be useful for the accurate and safe identification of vascular structures, thus helping in the mapping of resection plans [22,67].

Moreover, from an oncological point of view, the application of AR technology helps to localize the tumor [22,68]. An intraoperative CT or ultrasound guidance can also be utilized, however, with the drawback of radiation, the constant shifting of instruments, or the occasional difficulty in locating lesions [22].

Another benefit of the application of AR in robotic liver surgery is port positioning, which represents a crucial step of the operation. The projection of a virtual image of the liver on the surface of the skin before beginning the surgical procedure enables accurate targeting of the liver structures and the lesion, allowing optimal port placement. This makes surgical maneuvers easier [22,67].

From a general point of view, regarding the indications for the application of AR in robotic surgery, as this integration is at an early stage of development, they are not yet well defined. However, we believe that once the limitations are overcome, the indications for its application will expand more and more. In fact, the application of this imaging technology in robotic general surgery has theoretically unlimited potential and can lead to a considerable simplification of surgical procedures.

5. Conclusions

In the era of “precision medicine”, robotic surgery represents an excellent field for the application of AR in clinical practice.

The integration of this imaging technology in robotic general surgery is showing promising results. The main benefits include improved oncological outcomes and reduced occurrence of complications. In addition, its application may also be important for surgical education.

However, we are still at the initial phase of the experience and some important limitations remain, due to the limited spread of robotic platforms and the still insufficient technological development. Moreover, to our knowledge, to date, reports in the literature regarding the integration of this imaging technology in robotic general surgery are still very limited. To improve its application, close collaboration between engineers, software developers, and surgeons is mandatory.