Laparoscopic Liver Resection with Augmented Reality: A Preclinical Experience

Abstract

Introduction: Intraoperative imaging, such as ultrasound, provides subsurface anatomical information not seen by standard laparoscopy. Currently, information from the two modalities may only be integrated in the surgeon's mind, an often distracting and inefficient task. The desire to improve intraoperative efficiency has guided the development of a novel, augmented reality (AR) laparoscopic system that integrates, in real time, laparoscopic ultrasound (LUS) images with the laparoscopic video. This study shows the initial application of this system for laparoscopic hepatic wedge resection in a porcine model.

Materials and Methods: The AR system consists of a standard laparoscopy setup, LUS scanner, electromagnetic tracking system, and a laptop computer for image fusion. Two liver lesions created in a 40-kg swine by radiofrequency ablation (RFA) were resected using the novel AR system and under standard laparoscopy.

Results: Anatomical details from the LUS were successfully fused with the laparoscopic video in real time and presented on a single screen for the surgeons. The RFA lesions created were 2.5 and 1 cm in diameter. The 2.5 cm lesion was resected under AR guidance, taking about 7 minutes until completion, while the 1 cm lesion required 3 minutes using standard laparoscopy and ultrasound. Resection margins of both lesions grossly showed noncoagulated liver parenchyma, indicating a negative-margin resection.

Conclusions: The use of our AR system in laparoscopic hepatic wedge resection in a swine provided real-time integration of ultrasound image with standard laparoscopy. With more experience and testing, this system can be used for other laparoscopic procedures.

Introduction

The current paradigm of using preoperative radiographic images for intraoperative surgical guidance involves looking away from the surgical field to review anatomical information on a separate and often far away computer screen. This process can be imprecise and ergonomically taxing. A proposed solution to this problem is to bring imaging data into the operative field. This can be achieved with augmented reality (AR), which overlays information from a second imaging modality onto what is visualized by the laparoscope. While still an evolving technology, the use of AR in the operating room has been described for gynecologic, splenic, hepatobiliary, renal, and adrenal surgeries.^1–5 These experiences utilize the fusion of preoperative computed tomography (CT) or magnetic resonance (MR) images with the laparoscopic camera view to generate the AR image overlay.

The use of laparoscopic ultrasound (LUS) is another possible imaging source for laparoscopic AR. In contrast to CT and MR, LUS imaging provides real-time imaging data that can be obtained throughout the course of an operation. Given its ease of use, low cost, and lack of ionizing radiation, our group has researched AR with LUS for many years. We previously described the development of our second-generation electromagnetic (EM) tracking-based AR system.⁶ The system obtains location coordinates by tracking a 1 mm diameter sensor wire affixed to both the laparoscope camera and the ultrasound probe. This provides the relative location and orientation needed for the AR system to register the ultrasound image to the ultrasound probe tip when shown in the laparoscopic video.

Designed and developed by a team of biomedical engineers, interventional radiologists, and minimally invasive surgeons, our AR system merges live LUS images with conventional laparoscopic video in real time. We believe that this system is close to being suitable in a real operating room. In this article, we describe our early experience using our laparoscopic AR system with EM tracking for laparoscopic hepatic wedge resection in a porcine liver tumor model. Furthermore, we aim to show our system's ease of use and its ability to guide a laparoscopic surgeon in achieving a negative-margin resection.

Materials and Methods

Laparoscopic AR system overview

As seen in Figure 1, the AR system includes a standard laparoscopic vision system (Image 1 Hub; KARL STORZ, Tuttlingen, Germany), an LUS scanner (Flex Focus 700; BK Ultrasound, Analogic, Peabody, MA), an EM tracking system with a tabletop field generator (Aurora; Northern Digital, Waterloo, Ontario, Canada), and a laptop computer for image fusion. Mechanical tracking mounts containing EM sensors were designed in-house to be mounted on the imaging tip of the LUS probe (4-Way Laparoscopic 8666-RF; BK Ultrasound, Peabody, MA), as well as the handle of the laparoscope (10-mm 0°).⁶ These mounts allow fixed geometric relationships between the LUS imaging tip and laparoscope camera lens and the EM sensor.

FIG. 1.

Laparoscopic AR system with (A) EM field generator, (B) LUS, (C) standard laparoscopy tower, and (D) laptop computer for image fusion. AR, augmented reality; EM, electromagnetic; LUS, laparoscopic ultrasound.

Radiofrequency ablation lesion creation

Animal procedures were approved by the Institutional Animal Care and Use Committee (IACUC no. 00030557), and animals were treated in accordance with PHS Policy on Humane Care and Use of laboratory Animals, the National Institutes of Health Guide for the Care and Use of Laboratory Animals, and the Animal Welfare Act. After successful anesthesia and intubation, a 40-kg female Yorkshire swine was placed supine. Liver lesions were created using percutaneous radiofrequency ablation (RFA) by our interventional radiologist (K.S.). For each lesion, a RFA needle (2.25 mm in diameter and 15 cm in length) was inserted 5–10 cm deep through the abdominal skin into the liver parenchyma. A total of two lesions were made in separate lobes of the liver. The ablation was monitored with ultrasound, which showed the development of a signal void secondary to the thermal injury. Each ablation took ∼6–8 min. The animal was awakened from anesthesia and returned to its holding cage. The animal was monitored for 3 days and given diet ad-lib until the day of laparoscopic surgery.

Porcine hepatic wedge resection

On postablation day 3, laparoscopic hepatic wedge resection of the liver lesions was performed. The swine was placed under general anesthesia and intubation per standard protocol. The animal was positioned supine over the EM tracking tabletop field generator attached to the surgical table. Ultrasound calibration was performed 1 day before the surgery using the PLUS library, a publically available software tool for ultrasound image tracking calibration.⁷ This is a process to determine the coordinate transformation from the ultrasound image plane to the EM sensor (mounted on the LUS transducer tip) three-dimensional coordinate system. This provides the coordinates needed for the AR system to position the ultrasound image in the tracking space. In addition, using a fast calibration method developed by our team, a second process known as laparoscope calibration was performed immediately before the surgery started.⁸ Briefly, this process determines the laparoscope camera intrinsic parameters and the coordinate transformation from camera lens coordinates to that of the EM sensor. The AR system then maps the three-dimensional spatial coordinates to the two-dimensional coordinates of the image captured by the laparoscope, thus correctly fusing the ultrasound image to the laparoscope image.

After the calibration, a 12-mm periumbilical trocar was placed for the laparoscope, while another 12-mm trocar was placed in the right upper quadrant for the LUS probe (Fig. 2). Two 5-mm trocars were placed in the left upper quadrant as working ports. Pneumoperitoneum was achieved at 15 mmHg. Sharp resection of the first lesion was performed under AR guidance, while the second lesion was resected under standard laparoscopy with ultrasound imaging shown only on the ultrasound system. Electrocautery was not used during the operation to prevent thermal distortion to the RFA margins. Time for resection of each lesion was recorded. The lesions were retrieved through a laparotomy incision after euthanasia of the animal. The resected tissues were preserved in 10% formalin and analyzed the next morning to check for complete removal of coagulated liver parenchyma.

FIG. 2.

Operative setup of laparoscopic hepatic wedge resection with AR in a porcine model. The laparoscopy screen shows AR image of fused LUS image with laparoscope video. Tracking mounts (containing EM sensors) on the laparoscope and tracking wire of the laparoscopic ultrasound probe are highlighted in red. AR, augmented reality; EM, electromagnetic; LUS, laparoscopic ultrasound.

Results

Laparoscopic hepatic wedge resection with AR was completed without complications. Intraoperative LUS was able to visualize the hepatic lesions created from preoperative RFA. With EM tracking, articulation of the LUS probe tip was unrestricted and allowed for continuous imaging of the liver during the course of resection with AR. Anatomic details from ultrasound imaging were able to be fused with the laparoscopic images in real time and presented on a single screen (Fig. 3). The diameter of the two lesions created by RFA was 2.5 and 1 cm. The former was resected under AR guidance, while the latter was resected with standard laparoscopy and LUS. The time for laparoscope calibration was 3 minutes. The time to complete the resection under AR was 7 minutes, while the time to complete resection without AR was 3 minutes. Gross assessment of the resected lesions shows margins without ablated liver parenchyma in the lesion resected with AR (Fig. 4) and without AR (Fig. 5).

FIG. 3.

Fusion of (A) laparoscopic camera image with (B) LUS image displayed as a single image (C). The AR system utilizes the EM sensor coordinates to position the ultrasound image at the ultrasound probe tip seen by the laparoscope. AR, augmented reality; EM, electromagnetic; LUS, laparoscopic ultrasound.

FIG. 4.

RFA lesion resected with AR guidance. (A) Full and (B) Cross-sectional views of resected lesion. Green dashed line shows margin of coagulated liver parenchyma; blue dashed line shows interior margin of normal liver parenchyma. AR, augmented reality; RFA, radiofrequency ablation.

FIG. 5.

RFA lesion removed with standard laparoscopy. (A) Full and (B) Cross-sectional views of resected lesion. Green dashed line shows margin of coagulated liver parenchyma; blue dashed line shows interior margin of normal liver parenchyma. RFA, radiofrequency ablation.

Discussion

We report our first preclinical experience using our second-generation laparoscopic AR system based on EM tracking and intraoperative LUS for hepatic wedge resection in a porcine model. Our system fused the laparoscopic video with ultrasound images in real time onto the primary laparoscopy screen. This provided continuous imaging guidance for the resection of the liver lesion models, allowing the surgeon to correct for depth and direction of resection without distracting focus from the primary laparoscopy screen.

Unlike some of the other systems developed, ours does not require the use of preoperative imaging for a complete AR experience.^1,3,4,9 With preoperative images, the anatomical details of organs, vessels, nerves, and tumors are processed into individual three-dimensional structures. To create the AR overlay, these segmented structures are then matched onto the respective physical structures visualized by the laparoscope during a process known as registration. This fusion process is technically difficult given the need to account for tissue motion and deformation. An obvious problem also arises when tissue is mobilized and permanently changed during surgery. Once surgeons begin operating on the organ or structure of interest, such AR overlay becomes outdated. This limits the effectiveness and reliability of the intraoperative AR experience. Ultimately, the benefits of having AR for operative guidance are lost.

In contrast, the LUS provides, in real time, subsurface anatomical information based on where the ultrasound transducer is placed. For the AR fusion, the ultrasound image is shown at the transducer tip in the laparoscopic video. The process is performed through the EM tracking coordinates rather than using specific anatomical structures as relative fixation points for the image fusion. This method provides an AR experience with continuously updated radiographic information in real time, while avoiding the need for special correction of tissue movements and deformation.

From this experience, we demonstrate that it is feasible to perform a successful hepatic wedge resection using AR with LUS as real-time guidance for laparoscopic liver surgery. The operating surgeons were able to utilize the AR system to adjust the direction of dissection without difficulty. The time for resection was measured starting after initial localization of the simulated lesion and ending after the lesion was completely separated from the remainder of the liver parenchyma. The lesion resected with use of AR was larger and took more time than resection of the second lesion by standard laparoscopy. As this was our first in vivo experience with the EM tracking-based system, we believe that the additional 4 minutes needed to complete the resection can be attributed to both the larger size of the lesion and the surgeons' learning curve in using the system. Some time was spent to fine-tune the AR images to optimize use during resection. For example, in the fusion of ultrasound images onto the laparoscopic video, there was a need for gain adjustment on the AR display due to the brightness created by the laparoscopy light. With more experience using this device, we expect that the total time needed for a hepatic wedge resection will decrease.

As there is not an easily achievable model for creating tumors in the porcine liver, lesions created by RFA thermal ablation were thought to be an appropriate alternative. Thermal ablation created a volume of coagulated liver parenchyma that became firmer by the day of laparoscopic resection. This mimicked the mechanical difference one would have noticed when resecting neoplastic liver tissue from normal liver parenchyma. Due to the lack of described methodology in creating ablation lesions by RFA in the porcine liver, we obtained lesions of different sizes, 2.5 cm versus 1 cm in diameter. This was not realized until the day of resection. With the creation of the lesions through thermal ablation, the lesions were sharply resected so as not to cause thermal changes to the liver tissue from cautery. As seen in Figures 4 and 5, we noted only noncoagulated liver tissue at the margins. We believe this indicates that, at least on the macroscopic level, a negative margin was achieved with the resection under AR guidance. In addition, a larger rim of normal appearing parenchyma is seen around the lesion removed with standard laparoscopy. We believe that laparoscopic resection with AR can provide a smaller margin of resection. In the future we aim to improve our tumor models, making more uniformly sized lesions. The width of resection margins can then be compared as another quantitative assessment of the resection accuracy and precision from using our AR system.

In conclusion, we have developed an AR system with EM tracking that integrates in real-time intraoperative LUS imaging data with laparoscopic video. Through this preclinical experience, we show the feasibility of using our system in laparoscopic hepatic wedge resection. This AR system is capable of presenting LUS and laparoscopy data on a single screen through AR to achieve resection with negative margins. Our preclinical experience shows the capabilities of our system for complex laparoscopic liver surgery, and with more experience, it will be possible to adapt our AR system for clinical use.

Disclosure Statement

R.S. and W.P. are cofounders of IGI Technologies. All other authors have no conflicts of interest or financial ties to disclose.