Follow the light: projector-based augmented reality intracorporeal system for laparoscopic surgery

Abstract.

A projector-based augmented reality intracorporeal system (PARIS) is presented that includes a miniature tracked projector, tracked marker, and laparoscopic ultrasound (LUS) transducer. PARIS was developed to improve the efficacy and safety of laparoscopic partial nephrectomy (LPN). In particular, it has been demonstrated to effectively assist in the identification of tumor boundaries during surgery and to improve the surgeon’s understanding of the underlying anatomy. PARIS achieves this by displaying the orthographic projection of the cancerous tumor on the kidney’s surface. The performance of PARIS was evaluated in a user study with two surgeons who performed 32 simulated robot-assisted partial nephrectomies. They performed 16 simulated partial nephrectomies with PARIS for guidance and 16 simulated partial nephrectomies with only an LUS transducer for guidance. With PARIS, there was a significant reduction [30% ($p<0.05$)] in the amount of healthy tissue excised and a trend toward a more accurate dissection around the tumor and more negative margins. The combined point tracking and reprojection root-mean-square error of PARIS was 0.8 mm. PARIS’ proven ability to improve key metrics of LPN surgery and qualitative feedback from surgeons about PARIS supports the hypothesis that it is an effective surgical navigation tool.

1. Introduction

In 2015, there were 61,560 new cases of kidney cancer diagnosed in the United States.¹ According to the American and European Urological Association guidelines,² a partial nephrectomy should be offered to all patients who have T1 tumors (i.e., tumor is $<7\mathrm{cm}$ across and is only in the kidney), whereas patients with tumors that are over 7 cm across should be offered a radical nephrectomy. A partial nephrectomy is the removal of the entire cancer tumor while attempting to preserve the maximum amount of healthy kidney tissue.³ The preservation of the healthy kidney tissue results in improved health outcomes.³ Matched patients who underwent partial nephrectomy rather than radical nephrectomy (i.e., removal of the whole kidney) had equivalent long-term cancer outcomes and a lower risk of developing chronic renal insufficiency and proteinuria.⁴ For a partial nephrectomy, the standard of care is a 5-mm negative margin.⁵ Laparoscopic partial nephrectomies (LPNs) are considered to be technically challenging because the surgeon has a limited field of view and effectively no tactile feedback. Furthermore, it is challenging for the surgeon to excise the entire tumor while also minimizing the excision of healthy tissue. Tumor size, tumor position, and surgeon experience are important factors in determining whether a patient is offered an LPN. Smaller and more peripheral tumors make for a simpler partial nephrectomy surgery and are more likely to be removed by LPN. Surgeons that have higher kidney cancer surgical volumes and practice in larger cities conduct a higher proportion of nephrectomies as LPNs.⁶ Between 2002 and 2013, the percentage of nephrectomies that were performed in the USA as LPNs increased from 2% to 17%.⁶ With more effective surgical navigation technology available that reveals to the surgeon the tumor outline and tumor depth, it is likely that more surgeons will opt for LPNs and that their patients will benefit accordingly.

Planning and executing a successful LPN surgical approach is particularly challenging for endophytic (inward growing) kidney tumor resections. LPNs for endophytic tumors have a 47% complication rate, five times higher than that for exophytic (outward growing) tumors.⁷ As defined by Herrell et al.,⁸ the ideal approach for endophytic tumors is to start as close as feasible to the tumor and excise straight down from the organ surface along the orthographic projection of the tumor. This ideal approach minimizes the excised tissue, while maintaining a negative margin. A negative margin indicates that the entirety of the tumor was removed and that the tumor was not visible in the excised specimen. The ideal excision specimen for a spherical tumor is a cylinder normal to the surface of the kidney. In reality, tumors are only approximately ellipsoidal.

When surgeons are interested in visualizing the underlying anatomy of the kidney during an LPN, they commonly use laparoscopic ultrasound (LUS). However, using an LUS effectively during surgery is challenging and complicated and mostly performed by expert surgeons. When using an LUS transducer during an LPN, surgeons put the LUS transducer on the kidney, interpret the LUS image on the ultrasound image display screen, create a mental image of the tumor, mentally register the LUS with the laparoscopic view, and mark the kidney tissue surface using electrocautery. When surgeons remove the LUS transducer, they have to recall the ultrasound images and their poses relative to the kidney throughout the surgery.

To assist in surgical navigation, there has been significant research and development in augmented reality (AR). Herrell et al.⁸ used the information from a preoperative computer tomography (CT) scan of a phantom tumor model to provide intraoperative guidance during robotic surgery. Bernhardt et al.⁹ recently published a comprehensive review about AR in laparoscopic surgery. There have been important advances in surface reconstruction using structured light,¹⁰ surface tracking, and surface registration^11–13 as well as advances in AR visualization techniques. The five AR visualization techniques for depth perception of blood vessels that were studied by Wang et al.¹⁴ are transparent overlay, virtual window, random-dot mask, transparent mask, and ghosting. Jayarathne et al.¹⁵ developed a real-time three-dimensional (3-D) ultrasound reconstruction and visualization strategy for laparoscopy, in which the user views a 3-D ultrasound volume through a transparent window in the surface image. With regard to AR for partial nephrectomy, Hughes-Hallett et al.¹⁶ published a review on the topic. Examples of video-based AR for partial nephrectomy include the following: display of semitransparent segmented preoperative CT using real-time 3D-CT to stereoscopic video registration,¹⁷ display of intraoperative cone-beam CT images of the kidney using radio-opaque needles for tracking and registration,¹⁸ display of an AR overlay of the ultrasound image during the intraoperative ultrasound scan of the kidney using image-based tracking of an ultrasound transducer,¹⁹ display of a 3-D tumor model during the planning and execution stage of the surgery using image-based tracking of an optical marker on the kidney,²⁰ and showing the surgeon how to access the kidney tumor with an incision tool.²¹ Another type of AR for surgical guidance is projector-based AR guidance with the projector outside²² or inside the patient²³ for open and laparoscopic surgery, respectively. Projector-based AR guidance, rather than video-based AR guidance, is of interest because it does not introduce latency into the video-based display being used by the surgeon. Furthermore, it could provide structured light to improve surface reconstruction and project the display information directly onto the surgical scene, resulting in a natural blending of the augmented and real worlds. Prior work with the projector inside the patient explored the display of blood vessels during laparoscopic surgery.²³ This work incorporates intraoperative ultrasound imaging for real-time tracking and a display of the kidney tumor by the projector inside the patient.

The development of the projector-based augmented reality intracorporeal system (PARIS) for image-guided laparoscopic surgery made possible the projector-based AR described already (Fig. 1). The purpose of PARIS is to improve the efficacy and safety of LPN surgery and render LPN more accessible to surgeons so as to make LPN, not radical nephrectomy, the most common surgery for T1 tumors. In surgical experiments on phantom kidney models, PARIS was compared to standalone LUS. While the use of PARIS is proposed for all types of nonrobotic and robotic minimally invasive surgery, in this work, it is specifically evaluated in the context of robot-assisted LPNs with the da Vinci Surgical System (Intuitive Surgical Inc., Sunnyvale). PARIS includes a projector that displays an outline of the orthographic tumor outline on the surface of the kidney.

Fig. 1.

Overview of PARIS. The DART is the white object that is above the tumor in (a) and (b). (a) The conceptual diagram shows a kidney, an underlying tumor, and how the outline of the tumor is projected onto the surface of the kidney. An orthogonal projection of the tumor is shown. The dashed green lines represent the orthographic lines from the tumor in the direction of the projector. (b) The surgeon’s view shows a picture of what the surgeon actually views—a projection of the tumor onto the kidney phantom surface. The outline of the projection is the outline of the orthographic projection of the tumor onto the kidney.

The sections in this paper include Secs. 2, 3, and 4. Section 2 includes a description of intraoperative tumor model generation, the augmented visualization provided by PARIS, verification and validation tests to determine the accuracy of PARIS, and the user study conducted to test PARIS for LPN on kidney phantoms.

2. Materials and Methods

2.1. Materials

PARIS consists of multiple devices that could be used during partial nephrectomy. PARIS includes a tracked marker called the dynamic augmented reality tracker (DART),²⁴ projector called the Pico Lantern,²³ and miniature LUS transducer. The DART is placed onto the kidney and anchored in place with its own barbed legs. The Pico Lantern is used for both surface reconstruction and augmentation of the surgical scene. The LUS transducer is used to generate a 3-D model of the tumor. The DART is used as a means for tracking the position of the kidney and accurately creating an AR display of the 3-D model of the tumor. Both the DART and LUS transducer are optically tracked in real time in 6 degrees-of-freedom using only image analysis of the laparoscopic video feed—no external tracking hardware is used. A diagram of PARIS is shown in Fig. 1.

The Pico Lantern is a PicoPro projector (Celluon Inc., Seoul, Republic of Korea) with an attached KeyDot^®, a marker with an asymmetric grid of circles (Key Surgical, Eden Prairie). Using OpenCV, the KeyDot^® is tracked optically with 6 degrees-of-freedom relative to a laparoscope.¹⁹ No external tracking hardware is required. Images are projected directly onto the surgical scene via laser raster scanning, and the Pico Lantern has a focus range from 3 cm to infinity. There is some latency for the projected images. However, direct projection implies that unlike video-based AR, no modifications are made to the video pipeline displaying the surgical scene to the surgeon. Compared to our previous work,²³ which included a Pico Lantern projector with VGA resolution ($640\times 480$) and 15 lumens of brightness, this paper presents a projector prototype [Fig. 2(e)] with increased resolution ($1920\times 1080$), 30 lumens brightness, and with wireless capability. It does not require a dedicated port as it can be placed through the skin incision with a cable beside the trocar or it can be controlled wirelessly. The prototype used in this paper has not been miniaturized as was done previously.²³ However, it shares a similar underlying architecture and component sizing; therefore, the miniaturization techniques and engineering applied previously in Ref. 23 may be applied here, too. In another previous work, we developed the DART, a navigation aid with barbed legs made in either plastic for one-time use or stainless steel for repeated use, with a KeyDot^® optical marker on it.²⁴ With dimensions of $8.5\times 8.5\times 13\mathrm{mm}$, the DART can be inserted into a 12-mm trocar and picked up in a repeatable manner by the da Vinci Pro-Grasp™ (Intuitive Surgical Inc., Sunnyvale). The LUS transducer is a 10-MHz, 28-mm linear array designed for robot-assisted minimally invasive surgery; additionally, it has a KeyDot^® marker fixed to it for tracking.²⁵ Cylindrical Super Soft Plastic (M-F Manufacturing, Fort Worth) kidney phantoms, with color, elasticity, and size similar to those of human kidneys, were used. Each of the kidney phantoms had a height of 6 cm and diameter of 15 cm and a Young’s modulus of 15 kPa, consistent with human kidneys.²⁶ Spherical inclusions with a 10- to 20-mm diameter are randomly placed at a depth of $\sim 10$ to 30 mm in the phantom. PARIS is evaluated with the da Vinci Si^® surgical system (Intuitive Surgical Inc., Sunnyvale).

Fig. 2.

(a) Pick-up LUS transducer with KeyDot^®, (b) plastic 3-D printed DART, (c) metal 3-D printed DART, (d) original Pico Lantern prototype—a miniature projector for laparoscopic surgery, and (e) Celluon PicoPro used in the experiments.

2.2. Tumor Model Generation

Using PARIS requires two major steps: first, an accurate model of the tumor must be generated; this is followed by the AR visualization of the tumor model. In the first step, the model is generated by tracking the LUS transducer and DART simultaneously as done by Singla et al.²⁰ A 3-D tumor model is generated in the DART coordinate system by registering the two-dimensional LUS ultrasound images to the DART, reconstructing the 3-D volume, and then performing manual segmentation. The 3-D volume reconstruction method tests the proximity of each registered ultrasound plane to a coarse discretization of the scanned volume. Where intersections take place, each voxel value within the respective regions is assigned in accordance with the ultrasound image texture at the closest perpendicularly projected point within a specified distance threshold. Using ITK-Snap’s manual segmentation and interpolation features, it takes $<5\min$ to perform the segmentation and 3-D reconstruction of the tumor model.²⁷ The segmentation is integrated into the surgical workflow by conducting the segmentation on the surgical navigation computer in the operating room as soon as the data are collected. Automatic segmentation could be incorporated in the future. Deformable models and machine learning^28,29 are likely to be used to develop effective automatic segmentation.

2.3. Augmented Reality Visualization

In the second step, the surgeon uses the Pico Lantern and AR visualization to visualize the tumor and determine the ideal excision approach. The goal is complete tumor removal while minimizing the amount of healthy kidney tissue excised. The Pico Lantern first displays a checkerboard pattern on the kidney to provide extra corresponding features, which improve the quality of the kidney surface reconstruction. Surface reconstruction is achieved via semiglobal block matching (SGBM)³⁰ using OpenCV. It is similar to the projector-enhanced stereo surface reconstruction described previously.²³ Next, PARIS displays the orthographic projection of the tumor model as a dense point set on the surface of the kidney. To perform the orthographic projections, the rays are projected in parallel from the tumor toward the Pico Lantern. The projected image is determined by the intersection of the rays with the surface. In this manner, the reconstructed surface is used to predistort the image and account for the phantom kidney surface when projecting an image onto the nonplanar surface.³¹

The excision approach is the vector that the surgeon follows to excise the tumor. The ideal approach, as described by Herrel et al.,⁸ is normal to the surface at the point closest to the tumor. As the Pico Lantern has a flexible cable, the surgeon can use it as an “eye in the hand” and move it to a variety of positions to visualize the surface projection of the tumor from different perspectives. Ultimately, the surgeon places the tracked Pico Lantern in a position that is unused by the laparoscope and aligns the Pico Lantern such that it is parallel with and centered on the ideal approach vector. This is achieved by manually positioning the Pico Lantern so it is normal to the surface and the center of the projection image intersects the tumor centroid. The surgeon then uses the projected tumor outline and the line between the projector and center of the projected image to identify the ideal approach and plan the LPN. Throughout, the orthographic projection of the tumor in the direction of the Pico Lantern is displayed on the kidney surface.

2.4. Verification and Validation

The laparoscope was calibrated using OpenCV. The projector calibration was performed as done previously.³² The ultrasound calibration was achieved geometrically via a careful alignment of the axes of the KeyDot^® to the axes of the LUS transducer.¹⁹ The pixel-to-millimeter ratio was determined by the known width and resolution of the LUS linear array and the axial depth and resolution of the ultrasound image. After ultrasound calibration, the LUS transducer has a relative root-mean-square (RMS) point reconstruction accuracy of 0.9 mm estimated over 10 ultrasound images covering a working volume of $16\times 10\times 19\mathrm{mm}$.²⁴

It was previously demonstrated by a biomechanical simulation using ANSYS (ANSYS, Pittsburgh, Pennsylvania) that the local deformation between the DART and the centroid of the tumor owing to the pressure of the LUS transducer on the kidney surface is submillimeter.²⁰

To evaluate the benefit of using the Pico Lantern to improve surface reconstruction, the surface density was measured. The surface density is defined as the percentage of the kidney’s surface that is successfully reconstructed by SGBM within the region of interest in the laparoscope view. The surface density of an ex vivo kidney, with and without the projection of extra features, is compared among 12 unique laparoscope and projector poses. The mean surface density change is reported.

To evaluate the accuracy of the Pico Lantern’s augmentations, the point reprojection error was measured. The point reprojection error is the distance between a point on the DART and its transformed equivalent as projected onto the scene. This captures the error in the tracking of the two KeyDots^® and the error in the laparoscope and projector calibration models. To measure it, the Pico Lantern was moved to five poses, and for each pose, the DART was placed at 10 poses, $\sim 80\mathrm{mm}$ from the laparoscope. The RMS error is reported.

For the phantoms used in the user study, the segmented tumor volumes are compared to the ground truth values measured during phantom construction. Additionally, a phantom was sectioned in half, and the segmented ultrasound volume of the exposed part of the tumor is projected onto the cut surface. The Hausdorff distance and mean RMS distance between the contours of the actual tumor and the projected tumor were measured for five laparoscope and projector poses. The mean RMS distance is defined as the tumor localization error.

2.5. User Study

A novice urologist (second year resident) and an expert urologist (with over 10 years of surgical experience) completed partial nephrectomies on kidney phantoms using either PARIS or only LUS imaging. The RENAL nephrometry score³³ was 12 for all the tumors, indicating a challenging resection task. Prior to the start of the user study, each surgeon was permitted a practice surgery with each of PARIS and only LUS technique. In an equal amount of time for the user study, the novice completed 12 simulated surgeries, and the expert completed 20 simulated surgeries. After each simulated surgery, the surgeons answered a five-point Likert scale questionnaire and provided open-ended qualitative feedback.

For each simulated surgery, the margin status was recorded as negative, microscopic, or gross. A negative margin indicates that the entirety of the tumor was removed and that the tumor was not visible in the excised specimen. A microscopic margin indicates that the tumor was removed with exposure, while a gross margin indicates that a visible part of the tumor remained in the kidney phantom.

Additionally, the mean and standard deviation of excision time, tumor volume, total excised specimen volume, adjusted excised specimen volume, and depth of cut are reported for all the user study surgeries that had negative margins. The excision time is defined as the time it takes the surgeon from the moment the surgeon cuts into the kidney phantom to the time when the entire tumor is removed. The total excised specimen volume is the volume of the tissue that is excised by the surgeon minus the tumor volume. The tumor volume is subtracted so that the variable size of the tumor does not affect the results. The adjusted excised specimen volume is the total excised specimen volume minus the volume of the top layer of parenchyma that sits above the tumor. The volume of the top layer of parenchyma and the tumor is subtracted so that the variable depth and size of the tumors do not affect the results. The volumes are determined from the specimen weights and known density. The depth of cut is the thickness of the parenchyma tissue below the tumor in the excised specimen. This is determined by US imaging of the excised specimens.

Finally, the metrics used in this work to quantify the deviation from the ideal excision include cross-sectional analysis of the specimens. Each excised specimen was sectioned at increments of 5 mm, and the cross section with the largest tumor diameter is analyzed. The tumor and the full cross section of the specimens were segmented, and the RMS distance between their centroids and the Hausdorff distance between their segmented contours is reported. For all the metrics, a two-tailed paired $t$-test was performed to assess statistical significance with a $p$-value of 0.05. The Holm–Bonferroni correction is used to account for multiple comparisons.

3. Results

3.1. Verification and Validation

The projected patterns improved surface density by an absolute mean of $15.4\%\pm 8.3\%$. An example of how the projected pattern improved surface reconstruction is shown in Fig. 3.

Fig. 3.

Surface reconstruction with (a) no projected texture and (b) with projected texture. The blue box shows the region on which the extra checkerboard features are projected. The black areas in the images indicate where the surface reconstruction failed.

The Pico Lantern’s point reprojection error was 0.8-mm RMS. During the data collection, the Pico Lantern was moved over a range of $32\times 9\times 11\mathrm{mm}$. The mean ground truth and tumor model volumes were $2.6\pm 0.7{\mathrm{cm}}^3$ and $4.2\pm 1.4{\mathrm{cm}}^3$, respectively. This corresponds to an overestimation of the radius of the tumor by $1.4\pm 1.4\mathrm{mm}$. Given that there is an error in the tumor model estimation, it is better that the tumor model volume be larger, rather than smaller, than the actual tumor volume. As shown in Fig. 4, for the projection of the tumor onto the actual tumor, the mean Hausdorff distance and RMS distance between the tumor contour and projection contour were 3.9 and 1.7 mm, respectively.

Fig. 4.

(a) The phantom was cut in half to expose the black-colored tumor, which is indicated by a blue arrow. (b) A video-based AR display of the 3-D tumor model superimposed on the laparoscopic view. (c) A projector-based AR display of the 3-D tumor model superimposed on the laparoscopic view.

3.2. User Study

The quantitative results from the 32 simulated LPNs are summarized in Table 1. The mean and standard deviation ($\mathrm{avg}\pm \mathrm{stdev}$) of each metric are listed. The results of the trials of LUS only and PARIS are presented. The asterisk indicates the statistical significance ($p<0.05$) of PARIS compared to LUS. Positive margin cases were excluded from quantitative analysis. In summary, the novice surgeon had $5/6$ negative margins with both PARIS and LUS, and the expert surgeon had $10/10$ and $8/10$ negative margins with PARIS and LUS, respectively. Furthermore, when using PARIS, both the surgeons excised a statistically significant less amount of healthy tissue ($p<0.05$), as measured via the adjusted excised specimen volume. As examples, the cross sections of the expert surgeon’s first four excised specimens with each of PARIS and only LUS are shown in Fig. 5. As reported in Table 1, the Hausdorff distance between the tumor and complete cross section of the excised specimen was reduced for both the surgeons when they used PARIS.

Table 1.

Quantitative results of simulated partial nephrectomies. The mean and standard deviation (avg ± stdev) of each metric are listed. The results of the trials of only LUS and PARIS are presented. Positive margin cases were excluded from quantitative analysis.

Metric (unit)	Surgeon and visualization type
	Novice surgeon ($n=12$)		Expert surgeon ($n=20$)
	LUS ($n=6$)	PARIS ($n=6$)	LUS ($n=10$)	PARIS ($n=10$)
Negative margins	$5/6$	$5/6$	$8/10$	$10/10$
Positive margins	1 gross	1 microscopic	2 gross	0
Excision time (s)	$579\pm 155$	$469\pm 152$	$199\pm 31$	$207\pm 40$
Tumor volume (${\mathrm{cm}}^3$)	$2.8\pm 0.7$	$2.6\pm 0.7$	$2.6\pm 0.8$	$2.4\pm 0.9$
Total excised specimen volume (${\mathrm{cm}}^3$)	$34\pm 2$	$21\pm 5$*	$27\pm 5$	$22\pm 5$
Adjusted excised specimen volume (${\mathrm{cm}}^3$)	$26\pm 3$	$17\pm 3$*	$20\pm 4$	$14\pm 4$*
Depth cut under tumor (mm)	$11\pm 6$	$12\pm 3$	$11\pm 4$	$10\pm 4$
Centroid distance (mm)	$5.1\pm 1.5$	$4.1\pm 1.8$	$4.4\pm 1.9$	$2.9\pm 1.2$
Hausdorff distance (mm)	$19.3\pm 0.8$	$13.3\pm 3.7$*	$18.0\pm 2.2$	$11.0\pm 1.7$*

^*Indicates statistical significance ($p<0.05$) of LUS compared to PARIS.

Fig. 5.

Cross sections of excised specimens from the first four phantoms from each of (a) PARIS and (b) only LUS user studies with the expert surgeon. The black inclusion is the simulated kidney cancer lesion. The centroids and contour drawings, conducted via manual segmentation of the tumors, and the complete tissue cross sections are shown.

The surgeons reported that they would prefer to use PARIS over the standalone ultrasound imaging method partly because the AR provides persistent guidance by projecting the tumor’s outline. Conventionally, the surgeon observes a limited cross section of the tumor during the ultrasound scan. The surgeons considered the drawbacks of PARIS to be that the Pico Lantern is an extra piece of hardware, there is an absence of guidance during the surgical excision of the tumor, and the surgical light must be set to a moderate light intensity to prevent the washing out of the projected image. After each resection, the surgeons answered a five-level Likert scale questionnaire wherein the response of 1 indicates “strongly disagree” and 5 indicates “strongly agree.” A comparison of LUS and PARIS results indicates that the surgeons felt more confident ($3.3\pm 1.1$ versus $5.0\pm 0.0$) and had a better spatial understanding ($3.5\pm 0.8$ versus $4.6\pm 0.5$) in the case of PARIS. All these results favor PARIS over LUS visualization.

Singla et al.²⁰ conducted a nearly identical user study, in which the same expert surgeon used an AR guidance system that provided guidance during the execution stage of the surgery with computer graphic overlays. Both this user study and the Singla et al.’s user study were shaped by and benefited from early feedback on study design from the same clinical collaborator. When the expert surgeon used the AR guidance system in Singla et al.’s prior work²⁰ instead of PARIS, the excision time, total excised specimen volume, and adjusted excised special volume were effectively similar. However, there was a statistically significant reduction in the depth of cut under the tumor, from $10\pm 4\mathrm{mm}$ to $3\pm 2\mathrm{mm}$, when the expert surgeon used the AR guidance system instead of PARIS. This is important because it reduces the risk of the surgeon entering the collecting system of the kidney. The AR guidance system provides AR navigation during the execution stage of the surgery. Therefore, it is reasonable that there was a reduction in the depth of cut under the tumor. There is scope for combining the PARIS described in this paper with the AR guidance system.²⁰ That is likely to provide the optimum guidance for the planning and execution stage of the partial nephrectomy.

4. Discussion

This work presents, from an engineering and clinical user study perspective, a completely integrated intracorporeal AR system for intraoperative guidance in soft tissue surgery. Given the 5-mm margin of safety for partial nephrectomy, the overall tumor localization error (1.7-mm RMS) is small enough to consider PARIS as acceptable for guidance.

From the engineering perspective, it is demonstrated that the integration of the two components, vision-based tracking and PARIS projector augmentation, is feasible. The integration of PARIS with the da Vinci system requires read-only access to the video feed, which will ease dissemination. PARIS includes an improved Pico Lantern prototype as compared to previous work.²³ This prototype provides higher resolution, is brighter, has wireless capability, and is automatically tracked in real time using optical tracking computer vision techniques. These additional features enable ease of use; however, the fundamental concept of a tracked projector for AR display remains unchanged.

Several projection strategies were considered. They included the orthographic or perspective projection of the tumor in the projector and/or laparoscopic coordinate systems. The perspective projections were unintuitive because the perspective projection of the tumor on the organ surface covers an area that is smaller than the actual tumor. Thus, an orthographic surface projection is the only option if the surgeon wishes to achieve a “cylindrical-approach” to the dissection of the tumor. Furthermore, a display in the laparoscopic coordinate system and field of view proved to be confusing to the surgeon because the laparoscope is generally oriented at a shallow angle to the tissue surface.

The user study was intended to test the overall concept of PARIS in the clinical setting. The main outcome from the user study presented herein is that there was a statistically significant reduction of 30% in the amount of healthy tissue excised when the surgeon used PARIS. The implications for a reduction in healthy kidney tissue excised are critical because the preservation of the kidney tissue has been established to improve health outcomes.^4,6 Furthermore, strategies to save healthy kidney tissue may include the provision of continuous guidance throughout the planning and execution of the surgery. Second, the surgeons that used PARIS felt more confident and had a better self-reported spatial understanding of the underlying anatomy. While initial results were positive, further ex vivo and in vivo studies with additional challenges, such as bleeding, smoke, and specular reflection, are required. Maier-Hein et al.³⁴ conducted a comparative study on the effect of smoke and blood on the surface reconstruction accuracy of eight state-of-the-art passive and active reconstruction methods. They determined that structured light surface reconstruction exhibited optimum performance in the presence of smoke and that smoke generally increased the median RMS stereo surface reconstruction error by 1 to 6 mm. Furthermore, the assumption of submillimeter local deformation between DART and tumor must be validated in vivo.

If guidance tools, such as PARIS, continue to improve key surgical metrics when tested in vivo and is approved for clinical use, it could potentially improve LPN outcomes and increase the percentage of kidney cancer surgeries that are conducted as partial nephrectomies. PARIS aids the surgeon to localize the tumor and plan the surgery, thus with less chance of missing a part of the tumor, the surgeon can consider bigger and more complex tumors for LPN surgery. It is noteworthy that based on the key surgical metrics of total excised specimen volume (${\mathrm{cm}}^3$) and adjusted excised specimen volume (${\mathrm{cm}}^3$), the novice surgeon with PARIS performed as well as the expert surgeon with only LUS. This data supports the suggestion that surgeons, and in particular novice surgeons, who have access to guidance tools, such as PARIS, would be more likely to perform LPNs. This would imply faster adoption of LPNs and that more of the 61,560 patients per year who develop kidney cancer in the USA would benefit from partial nephrectomies. Furthermore, a new avenue for AR in laparoscopic surgery, with numerous interesting applications, will have been developed.

Here are a few important observations from the user study. Each surgeon required a practice trial before she/he was able to use PARIS effectively for guidance. PARIS is intended as an adjunct and not as a replacement for standard practice. With the mobile Pico Lantern, the surgeon can dynamically explore tumor visualization from a wider range of angles, benefit from valuable dynamic visual cues, and align the Pico Lantern with the ideal surgical approach. It also became apparent that there are potential trade-offs associated with PARIS. The extra DART, which is meant to be inserted into the kidney and removed with the kidney tumor specimen, and the Pico Lantern hardware increase the complexity of the surgery. However, the advantage of the DART is that it provides a means of tracking the tumor throughout the surgery so that the Pico Lantern can provide persistent guidance from the projector—a feature that the surgeons found beneficial. A second advantage was that the surgeons observed that PARIS generated a clear image that blended effectively with the phantom surface and that the depiction of the resection line was natural and intuitive. This compares favorably with video-based AR overlays, which generally appear to float above the surgical scene. Finally, PARIS does not interfere with the video-based surgical display.

In conclusion, PARIS can be integrated into the existing laparoscopic surgical workflow. It improves surface reconstruction and provides meaningful surgical guidance information resulting in statistically significant improvements in key surgical metrics and outcomes in kidney phantom LPNs.

Biographies

Philip Edgcumbe is an MD and PhD student at the University of British Columbia (UBC). He received his BASc degree in engineering physics and PhD in biomedical engineering from UBC. He is the author of five peer-reviewed publications. He is an innovator, entrepreneur, scientist, and doctor in training. In 2014, he was the recipient of the Outstanding Young Scientist award at the MICCAI conference. He is a Singularity University alumnus; he also led the development of the Alzheimer’s XPRIZE incentive research competition.

Rohit Singla is a research assistant at the UBC. He received his BASc degree in computer engineering and his MASc degree in biomedical engineering from the UBC in 2015 and 2017, respectively. He is the recipient of engineers in Scrubs fellowship, NSERC-CGSM, and two best paper awards. His current research interests include ultrasound-guided interventions, augmented reality, software systems, obstetric anesthesia, and urology.

Philip Pratt is a senior research fellow in the Department of Surgery and Cancer, Imperial College, London. He received his BSc degree in mathematics and his PhD in artificial intelligence from Imperial College in 1991 and 1997, respectively. He is the author of more than 50 peer-reviewed publications and undertakes a very active research program in the fields of image-guided surgery and autonomous surgical robotics. He has successfully translated new technology and software into clinical practice in the operating theater.

Caitlin Schneider is a postdoctoral fellow at the UBC. She received her BSc degree in biomedical engineering from Johns Hopkins University in 2009 and her MASc and PhD degrees in electrical engineering from the UBC in 2011 and 2017, respectively. She has authored many papers in the field of ultrasound and surgical robotics. One of her current research interests is ultrasound elastography.

Christopher Nguan is a full-time urology surgeon-scientist appointed by the UBC who works at Vancouver General Hospital in Vancouver, Canada. His urology practice centers around kidney surgery and transplantation with particular research interests in integrating advanced technologies in surgery, such as robotics and mixed reality.

Robert Rohling is a professor with a joint appointment in electrical and computer engineering and mechanical engineering at the UBC. He is also currently the director of the Institute for Computing, Information and Cognitive Systems. His research is in the field of biomedical engineering with specialization in medical ultrasound.

Disclosures

The authors have no conflicts of interest to report.