An augmented reality and high-speed optical tracking system for laparoscopic surgery

Abstract

While minimally invasive laparoscopic surgery can help reduce blood loss, reduce hospital time, and shorten recovery time compared to open surgery, it has the disadvantages of limited field of view and difficulty in locating subsurface targets. Our proposed solution applies an augmented reality (AR) system to overlay pre-operative images, such as those from magnetic resonance imaging (MRI), onto the target organ in the user’s real-world environment. Our system can provide critical information regarding the location of subsurface lesions to guide surgical procedures in real time. An infrared motion tracking camera system was employed to obtain real-time position data of the patient and surgical instruments. To perform hologram registration, fiducial markers were used to track and map virtual coordinates to the real world. In this study, phantom models of each organ were constructed to test the reliability and accuracy of the AR-guided laparoscopic system. Root mean square error (RMSE) was used to evaluate the targeting accuracy of the laparoscopic interventional procedure. Our results demonstrated a registration error of 2.42 ± 0.79 mm and a procedural targeting error of 4.17 ± 1.63 mm using our AR-guided laparoscopic system that will be further refined for potential clinical procedures.

INTRODUCTION

Laparoscopy is a widely used technique in minimally invasive surgery that requires only a few small incisions through the patients’ abdomen for the insertion of a laparoscope and fine surgical instruments.¹ In the United States, prostate cancer is the most common cancer diagnosed among men and ranks as the third most significant contributor to cancer-related mortality in men.² Laparoscopic radical prostatectomy is routinely employed in patients with locally advanced prostate cancer. Although laparoscopic surgery offers many advantages over open surgery, the surgeon relies on limited depth perception from the monocular laparoscope and reduced tactile feedback during the operation.³ Given its high disease burden and significant quality of life implications of prostate surgery, it is important to continue exploring and improving current treatment approaches.

Augmented reality (AR) is a technology that can overlay virtual information on the real-world environment and can improve the safety and efficacy of interventions.^4–5 Three-dimensional anatomy and the location of subsurface lesions can be shown in real time using virtual overlays. While plenty of studies have proven the feasibility and usability of AR-guided surgical systems, current limitations include the need for specific eye calibration, the lack of robust, deformable image registration techniques, the high cost of AR-based navigation systems, and the workflow disruptions it can cause.^4,5 Many applications for AR-guided interventions have been studied and show to potentially increase the accuracy of biopsies. Our group previously developed an AR-guided biopsy system for soft tissue and lung lesions.⁶ This system was able to enhance the accuracy of the procedure in a phantom study. In addition, our group also created an AR-assisted visualization system for prostate biopsy using Microsoft HoloLens 2.⁷ The AR headset was utilized to display lesions, surrounding organs, and ultrasound images in real time. In another study, our group implemented an AR-system with deformable registration for holographic overlaying of renal lesion enhancement.⁸ An targeting error of 2.9 ± 1.5 mm was achieved in a phantom study. Francesco et al. integrated AR into robot-assisted radical prostatectomy.⁹

The purpose of this study is to develop a framework for AR-guided laparoscopic surgery. We tested this framework’s performance on an anatomically simulated phantom that includes intraprostatic lesions and adjacent anatomic structures. Furthermore, we performed anatomical model registration in laparoscopic video feed and projected those models to HoloLens 2 during operation to improve surgeon ergonomics.

MATERIALS AND METHODS

System Overview

Our system uses a patient’s MRI data to generate a virtual representation of the patient’s internal organs. A doctor identifies important structures in this model that are relevant to the surgery, such as major blood vessels and nerves in and near organs; these structures are given a special relevance through the use of selective view filters or different colors in the virtual model. Retroreflective markers are used to identify patient positioning in real time with motion tracking hardware and software during the surgical procedure; the resulting information is used to provide a surgeon an enhanced perspective of the important structures inside a patient by overlaying models of relevant structures in their real positions through the use of AR technology in both camera feeds and a surgeon’s view of a patient. Figure 1 illustrates the components of this AR-guided system.

Figure 1.

Overview of the proposed AR-guided system for laparoscopic surgery. The system is composed of an AR headset, a motion tracking camera system, and surgical instruments.

Model Fabrication

As shown in Figure 2(a–b), a computer model of a prostate with two intra-prostatic lesions (targets) was created from a publicly accessible MRI dataset.^10,11 This was combined with a pelvis model that was acquired from the National Institutes of Health (NIH) 3D portal,¹² as illustrated in Figure 2(c), and ascending colon and appendix, sigmoid colon, and rectum that were obtained from the model exchange site Thingiverse.¹³ The lesions are spheroidal with a diameter of 5 mm and 10 mm, respectively. The components were scaled and arranged so the positioning and size of organs was accurate to the pelvic cavity for an adult human male of approximately average size, using the MRI scans used to model the prostate as additional reference for accuracy.

Figure 2.

Workflow of the AR-guided laparoscopy system. (a) The MR image of the prostate and surrounding organs for 3D model construction. (b) Imported MR images of each organ in 3D Slicer for segmentation. (c) Segmented 3D models of those organs. (d) 3D-printed phantom models of those organs. (e) AR view of superimposed virtual models onto the phantom models. (f) Hologram registration of the prostate with lesion onto real-time video feed displayed on HoloLens 2.

The virtual model was used to generate a phantom model of the same dimensions using fused deposition 3D-printing with polylactic acid (PLA). A rigid phantom generated from a virtual model was selected to ensure accuracy between the virtual and real models for the purpose of error reduction. Unexpected errors can be introduced by using a physical model that is not the same size, dimension, or shape as its virtual representation when measuring the error of a system. For a similar reason, the model design contains sufficient complexity over anatomical accuracy, particularly with organs of less relevance to the procedure that was tested. The phantom organs were contained in a plastic box as shown in Figure 2(d), to which infrared markers were affixed for use with the tracking system. A virtual model was rendered of the box to complete a virtual representation of the complete phantom. The tools used to perform the experiment also received virtual models. Of note, the plastic box was initially clear but made opaque with sheets of paper because the translucence and specular nature of the material interfered with the accuracy of the tracking system by reflecting and refracting the positions of the tracking markers when viewed through the plastic by the infrared cameras.

Implementation

The AR-guided software was developed in Unity to stream virtual models and surgical guidance information to the HoloLens 2 headset with live positional data imported from the motion capture software to adjust the locations and orientations of virtual models in real time to reflect changes in reality. A customized high-speed infrared motion capture camera system was applied to localize phantom models and surgical instruments in real time. This real-time tracking is critical for improving the accuracy of hologram registration. When users put on the headset, they are able to see a virtual monitor containing the laparoscopic video feed with the overlaid virtual models as well as an overlay of the surgical instruments with a guidance beam at its tip. The beam was designed to be green when a surgical path is clear and become red when the path is obstructed, directed from the end of the virtual laparoscopy camera rod to the target site to aid in guiding a surgeon to the target location. Users can also see the positions of virtual models and their tools through the opaque elements of the subject. To increase the precision of hologram location registration, a QR code calibration method¹⁴ was used to set a mutual origin point between Unity’s interpretation of the virtual landscape and the HoloLens 2’s proprioception of its own position in space, which is necessary for displaying overlayed images correctly in a viewer’s perspective, as seen in Figure 2(e). Of note, the image captured in the figure was taken using the on-board location positioning cameras embedded in the HoloLens 2. These cameras do not adjust for the difference between rendering position given the perspective of a user’s eyes or the lens of a camera, so the visible offset between the virtual model and the phantom model is caused by the image capture method.

The holographic remoting feature in Mixed Reality Toolkit (MRTK) was deployed to enable us to use a more powerful computing system than the native HoloLens 2 device while streaming holograms from Unity to the AR headset. Open Broadcaster software (OBS) was used to capture the real-time laparoscopic video feed, which was also streamed to Unity using Network Device Interface (NDI).¹⁵ The hologram registration to the laparoscopic video feed was performed manually within Unity by changing the position of a virtual camera embedded in the laparoscope virtual model until virtual models of the test phantom were aligned with the video feed. This method was used because the focal distance of the laparoscope used for the experiment is unknown. Figure 2(f) depicts an example of the overlaid hologram on the laparoscopic video feed. There are three major virtual components on the overlaid video feed. Arrow 1 (yellow) shows a surgical guidance path, Arrow 2 (red) shows an overlaid target on the real-time laparoscopic video feed.

RESULTS

We conducted two experiments to test the image registration accuracy and procedural targeting accuracy, respectively. We developed the testing software used for each experiment. First, virtual markers were created and placed on the virtual models in the related position as retroreflective markers we placed on phantom models as shown in Figure 3. To measure distance between both the virtual and physical markers, we deployed the optical tracking probe kit, which is attached with retroreflective markers to identify the location of each marker. Root mean square error (RMSE) was used as the metric for accuracy evaluation.

Figure 3.

The AR viewpoint (red dots are considered as virtual markers) in three different orientations.

Hologram Registration Experiment

In order to measure the error of hologram registration, the plastic box was attached with four retroreflective markers as shown in Figure 1(d). Four virtual markers were then created on the box’s model in Unity corresponding to the locations of the retroreflective markers. During the experiment, an operator utilized the AR-system and placed a probe’s tip at both virtual and retroreflective markers on three separate trials for each marker. The positional data of each marker was extracted from the motion capture system and the distances between the position of the probe’s tip at the virtual marker and the position of the probe’s tip at the retroreflective markers for each retroreflective-virtual marker pair are listed in Table 1. The average hologram registration error was 2.42 ± 0.79 mm across all markers and trial runs.

Table 1.

Hologram registration errors (in mm).

Marker Number	Trial 1	Trial 2	Trial 3
1	1.80	3.29	3.19
2	2.05	2.57	2.00
3	1.89	0.87	2.39
4	3.34	3.53	2.09

Average: 2.42 ± 0.79 mm.

Simulated Prostatectomy Experiment

Since the prostate model was too small to attach retroreflective markers, we employed ten fiducial markers using an ink marker. These ink markings were placed near the two prostatic lesions. Virtual markers were defined at the corresponding location of the ink markings in the testing software. An operator then wore the HoloLens 2 headset to visualize the virtual markers and retroreflective markers on the real-time laparoscopic video feed as shown in Figure 4. Similar to our experiment above, the probe kit, attached with a surgical guidance path, was utilized in the real-time video feed to localize the position of and calculate the distance between each fiducial-virtual marker pair. The average procedural targeting error was 4.17 ± 1.63 mm across all markers and trial runs, as depicted in Table 2.

Figure 4.

The view from HoloLens 2 during the AR-guided laparoscopic procedure.

Table 2.

Procedural targeting error results (in mm).

Marker Number	Trial 1	Trial 2	Trial 3
1	3.93	5.66	4.12
2	5.42	3.2	3.77
3	5.07	6.08	3.54
4	4.04	8.13	4.16
5	3.72	7.39	3.06
6	2.69	3.72	2.76
7	6.06	4.15	2.73
8	4.06	4.27	2.66
9	1.64	5.38	2.83
10	0.92	6.65	3.17

Average: 4.17 ± 1.63 mm

DISCUSSION AND CONCLUSION

AR technologies show a variety of potentials to improve image-guided interventions by overlaying more anatomical information, for example the location of tumors, vessels, and nerves, onto patients during operation. Motion tracking software can be used to enhance the accuracy of AR technologies in interventional applications that require a high precision of positional accuracy. To this end, we developed a novel augmented reality guided laparoscopic system which has potential to not only enhance the reliability of laparoscopic procedures but also improve ergonomics of operators. We achieved a satisfactory accuracy for hologram registration accuracy that can be used for AR-guided procedures, including robot-assisted laparoscopic prostatectomy.⁹ Our AR-guided laparoscopic system only uses a monocular camera, while the endoscope for robot-assisted laparoscopy uses a binocular camera. Lin et al.¹⁶ described an AR-guided system called HoloNeedle that achieved a needle placement accuracy of 8.15 ± 0.4 mm when employing rigid body registration. In this study, we are able to achieve a smaller average procedural targeting error of 4.17 mm.

To further refine the system, a potential solution is to develop or find more precise and automated methods of hologram registration. The pipeline of preoperative data (e.g., MRI) to surgical procedures should be as automated as possible and introduce minimal error to the process. The user interface of our implementation can be further improved and would benefit from more attention, as would the control scheme, which could be improved to acknowledge foot gestures or verbal commands. The system serves to augment and enhance a surgeon’s view of information. Surgeons still need to make the ultimate decision of actions for such procedures. The AR-guided laparoscopic system can have immediate applications in many laparoscopic procedures.