A Safe Framework for Quantitative In Vivo Human Evaluation of Image Guidance

Abstract

Goal: We present a new framework for in vivo image guidance evaluation and provide a case study on robotic partial nephrectomy. Methods: This framework (called the “bystander protocol”) involves two surgeons, one who solely performs the therapeutic process without image guidance, and another who solely periodically collects data to evaluate image guidance. This isolates the evaluation from the therapy, so that in-development image guidance systems can be tested without risk of negatively impacting the standard of care. We provide a case study applying this protocol in clinical cases during robotic partial nephrectomy surgery. Results: The bystander protocol was performed successfully in 6 patient cases. We find average lesion centroid localization error with our IGS system to be 6.5 mm in vivo compared to our prior result of 3.0 mm in phantoms. Conclusions: The bystander protocol is a safe, effective method for testing in-development image guidance systems in human subjects.

I. Introduction

Image-guided surgical (IGS) systems provide medical imaging information to surgeons in real time during surgery. They typically provide a 3D map of anatomy and indicate the location of the surgeon's tool on it. This assists the surgeon in understanding unseen subsurface anatomical relationships, without requiring incisions to expose them. Creation of IGS systems has been enabled over the past few decades by the advancement of computational power and three-dimensional imaging technology, which IGS combines [1]. IGS systems employ patient images, typically derived from tomographic data, and use tracking technology to measure tool locations [2], combined with registration algorithms to align images with the physical patient [3]. Images datasets can be either segmented (e.g. [4]) or non-segmented (e.g. [5]).

IGS can be deployed in either open surgery or in a minimally invasive surgical (MIS) setting. It can increase surgical accuracy and speed, since it spares the surgeon from the complex task of building a 3D mental model from a series of 2D images, and from mentally registering that model to patient anatomy. In this way, IGS increases the surgeon's situational awareness in the face of complicated anatomy and limited field of view [6]. Such systems have been applied to surgeries in the brain, lung, colon and other areas of the body (see e.g. [7], [8], [9]), and they can also be deployed in robotic MIS settings [10], [11], [12], [13].

IGS systems were originally designed to work in bone or in places like the brain or sinuses that are surrounded by bone. In these settings, IGS is well developed and accuracy can be quantified based on landmarks that are rigidly coupled to one another (see e.g. [14], [15], [16]). However, soft tissue image guidance is newer [17] and accuracy is more challenging to evaluate, especially in vivo.

These challenges arise because of tissue deformation and other changes in tissue morphology during surgery [18]. These factors mean that any attached fiducials or anatomical features used for quantitative evaluation may move relative to one another intraoperatively, making quantitative system accuracy assessment challenging. Another confounding factor is that even people with the same disease can differ anatomically. This makes matching of subjects highly challenging, meaning that large patient numbers are typically required to obtain statistically significant results (see e.g. [19]). Having a large number of patients in a study also usually means that a large number of surgeons must also participate, and different surgeons can set up and use systems in different ways – especially early in the development process for a new IGS system. Similarly, a variety of metrics might be of interest, including hitting a desired target, resecting a tumor completely, or avoiding complications following surgery. Furthermore, by definition, the first time a new IGS display is tried on a human subject, it is not fully tested, and hence any errors can be dangerous to the patient.

Because of these many confounding factors, the accuracy of soft tissue IGS systems has mainly been assessed in phantoms [20]. In prior work, we employed phantoms to assess multiple aspects of our IGS system including initial touch-based registration [10], periodic re-registration [21], and system impact on surgeon ability [22]. In [10], we found the da Vinci tool tip to be a satisfactory digitization of the phantom organ surface for rigid model-to-surgical scene registration. Further, we demonstrated ink fiducials placed on the phantom organ surface for periodic re-registration in [21]. Finally, we analyzed surgeon pointing accuracy in phantoms with and without our IGS system [22]. Our IGS system reduced mean pointing error by 67% (from 9.2 mm to 3.0 mm). Phantoms enable a controlled environment in which the results directly inform next iterations of platforms, as demonstrated by our phantom testing results.

However, phantom studies do not fully capture the in vivo surgical setting. As a step toward overcoming this barrier, researchers conducted a study where an ablation probe was inserted in vivo under ultrasound guidance with an IGS system registered in parallel, but not used in the therapeutic process [23]. The distance between the tumor and ablation probe was then measured in both ultrasound images and using the IGS system. While this study provides a useful way to test some IGS systems, it assumes there is already an accurate way to approach the surgical objective (i.e. ultrasound), which is not always the case. It also does not isolate the IGS information from the surgeon's view, so there is no guarantee that IGS information does not influence surgical decision making. In this paper, we provide a more general framework, and isolate surgical decision making from IGS testing.

Note also that the value of manually aligned image guidance has been assessed based on clinical metrics such as complication rates [24], [25], [26], which supports the value of image guidance in soft tissue surgeries, in general. However, since these cases align the IGS display with the anatomy based on human hand-eye coordination, they are subject to an unknown (surgeon and case-specific) amount of registration error, making it challenging to know which types of surgical decisions should rely on IGS information. Thus, a safe method is needed to quantitatively evaluate registration accuracy in vivo for soft tissue image guidance systems.

To enable this, this paper describes an experimental framework we call the “Bystander Protocol” that enables image guidance systems to be safely tested quantitatively, in vivo. To illustrate the bystander protocol in a practical application, we implemented it and used it in a small case series of in vivo robotic partial nephrectomies (rPN's) conducted using the da Vinci Surgical System. Given that the study size is small (6 patients) the objective of this study was not to find statstical significance. Rather it was to illustrate how the new bystander framework we propose can be applied in a practical setting to safely test a new IGS system that is an early stage of development, in human subjects.

II. Materials and Methods

A. The Bystander Protocol

The bystander protocol consists of two surgeons trading places at various points throughout a surgical procedure. One, the “primary surgeon”, performs all aspects of the surgical procedure, but never sees the IGS display. The other, the “bystander”, has access to IGS and collects data using it, but does not perform any surgical actions. The primary surgeon can also make measurements, without IGS, and the results can be compared. The workflow of the bystander protocol is shown in Fig. 1. The type of image guidance system and procedure will determine when and what measurements the surgeons acquire. The advantage of using two surgeons in this way is that, since the primary surgeon never sees the IGS display, any misalignments or other errors in the guidance system cannot – even subconsciously – influence the surgical decisions of the primary surgeon. This enables IGS systems to be tested in vivo much earlier in their development life cycle than would otherwise be possible, revealing the unique features of the in vivo workflow and context earlier.

Fig. 1.

The general bystander protocol workflow where the primary surgeon and bystander surgeon periodically trade control of the system.

B. Image Guidance System for Da Vinci Soft Tissue Surgeries

To provide an example of the bystander protocol, we implement it in the context of an image guidance system for the da Vinci Surgical System (Intuitive Surgical, Inc.). The first version of this system was implemented in Orion [27] on the da Vinci classic system [28]. Optical tracking was used to compensate for uncertainty in robot tool tip positions. The system was later moved to the open source 3D Slicer software and applied to the improved da Vinci Si, and subsequently Xi, systems [22]. This enabled us to dispense with the tracking system because the accuracy of the Xi system is sufficient for the robot itself to be used as a digitizer [29]. In particular, the Xi system dramatically improved the accuracy of the setup joints, enabling the robot to be re-positioned during surgery without having to be re-calibrated or re-registered.

C. Application to Partial Nephrectomy

We applied the bystander protocol with our robotic IGS system in the context of partial nephrectomy. The bystander protocol workflow for this procedure is shown in Fig. 3. We began by segmenting the patient's preoperative CT scan to generate a three-dimensional model including renal parenchyma, the lesion, the renal artery, the inferior vena cava (right kidney), the gonadal vein (left kidney), and the renal vein. CT scan data was collected according to our institution's normal protocol for pre-operative imaging of patients undergoing partial nephrectomy. This involves a slice thickness of 1 mm, with the patient in supine position, and the field of view set to include abdomen and pelvis. Axial, sagittal, and coronal views of the CT scans were used simultaneously for manual segmentation. Each anatomical target was identified and marked across the scan slices and cross-validated between each view to produce a complete three-dimensional anatomical model within 3D Slicer, verified by the bystander surgeons. A subset of our studies utilized automatic segmentation provided by Ceevra, Inc. using their proprietary software. This segmented model was displayed in the da Vinci's surgeon console in 3D Slicer using the TilePro interface.

Fig. 3.

The bystander protocol workflow as applied during robotic partial nephrectomy.

In the operating room, each rPN began with a standard dissection performed by the primary surgeon, uncovering approximately 30% of the kidney's surface, as described in [21], [30]. The tumor and vasculature were not yet exposed. At this point, the primary surgeon stepped immediately outside the room, the bystander took control of the da Vinci, and the IGS system was turned on. The bystander traced the exposed kidney surface with the robotic tool tip, digitizing the anatomy. The resulting point cloud was then used to register the segmented model to patient anatomy, using the iterative closest point (ICP) algorithm, while the bystander maintained control of the da Vinci. Additionally, four fiducials are added to the kidney surface via ink, or, if the surgeon preferred, cautery. These fiducials were used to update the initial rigid registration throughout the remainder of the study by either surgeon as necessary [31]. Note, when a registration update was performed by the primary surgeon, they used only the endoscopic video feed to localize the fiducials. A completed registration is shown in Fig. 2(a). Once the initial registration was complete, the bystander surgeon used the IGS display and endoscopic video feed to point at various anatomical targets, an example of which is depicted in Fig. 2(b).

Fig. 2.

IGS display as seen by the bystander surgeon (a) complete registration including tracing (red dots) and fiducials (purple and yellow dots) (b) lesion centroid localization

The bystander pointed at the following targets: the lesion centroid, the planned arterial clamp location, and the intersection of the renal vein with the inferior vena cava (right kidney) or the gonadal vein (left kidney). For each anatomical target, the bystander pointed at it from several different angles, collecting one measurement from each (these will later be averaged). During the final two human surgical cases, the intersection curve between the exophytic lesion and the kidney parenchyma was also localized by tracing with the instrument.

The image guidance display was then turned off, and the primary surgeon resumed control and collected pointing data following the same procedure as the bystander had previously used, only without viewing the IGS display. Then, the primary surgeon continued with the standard partial nephrectomy dissection, uncovering the anatomical targets in the process. The ink fiducials were used to update the registration periodically. Then, the exposed anatomical targets were touched with the robotic tool tip by the primary surgeon. This touching data is the final measured intraoperative location of the anatomical targets.

Since the centroid of the tumor and vasculature cannot be directly touched, we performed several pointing instances from very close range. To compute the final measured intraoperative location from these pointing instances, we performed an optimization, finding the point that minimized the sum of the perpendicular distances to the set of pointing instance lines. This is depicted in Fig. 4. The black arrows are the pointing instances, and the colored dots are the points that minimize the sum of perpendicular distances.

Fig. 4.

Final intraoperative location pointing measurements collected by the primary surgeon (black arrows) and located point in 3D space for lesion centroid (green) and arterial clamp location (red).

Next, using these final measured locations, our goal is to compute the target localization error (TLE) for each surgeon, based on the several pointing instances each collected before de-fatting. This TLE is defined as the average of the minimum perpendicular distances between the final measured point and the lines defined by each pointing instance.

For the lesion-parenchyma intersection curve, the final measured curve was collected by tracing the da Vinci tool tip along it. For this curve, we defined the error of a single pointing instance as the minimum distance between the curve and the line defined by that pointing instance. The TLE of the curve is then computed as the mean of these errors.

Once the primary surgeon had mobilized the kidney and exposed the parenchymal surface near the lesion, the primary surgeon again stepped immediately out of the room, the IGS display was turned on, and the bystander took control. The bystander then updated the registration using the ink fiducials and traced the planned lesion resection contour. This resection contour indicates where the bystander surgeon would resect the lesion if they were operating. The IGS display was then turned off, the primary surgeon resumed control, and traced their planned resection contour with the robotic tool tip, using only endoscopic video feed. Once the data collection process was complete, the primary surgeon continued the standard procedure.

Written consent was obtained before each procedure in accordance with approval from our Institutional Review Board (Vanderbilt University Medical Center IRB, #191338).

III. Results

Patients radiologically diagnosed with unilateral renal cell carcinoma were prospectively identified within our practice at Vanderbilt for inclusion in this study. Our image-guidance system was deployed via the bystander protocol during six robot-assisted partial nephrectomies between December 2020 and August 2022.

CT scan segmentation for the first five patients was done manually using 3D Slicer [32]. The sixth patient's reconstruction was done in partnership with Ceevra, Inc. using their automated segmentation tools (see e.g. [33]). Patient and lesion characteristics such as age, BMI, American Society of Anesthesiologists (ASA) physical status classification, axial tumor diameter, R.E.N.A.L. nephrometry scores, and tumor location were prospectively recorded (Table I). The overall median age, BMI, and axial lesion diameter were 52 years, 30.65 kg/m, and 2.95 cm. The median nephrometry score was 6.5. Descriptive statistics of the data sets include the median, minimum, maximum, and lower/upper quartile ranges. The statistical differences between the medians of bystander and primary target localization errors were determined via a Kruskal-Wallis test and Chi-square approximation in Matlab.

TABLE I. Patient Characteristics in Our Study.

Patient and Lesion Characteristics
Age	BMI (kg/m)	Axial Lesion Diameter (cm)	Tumor Location	ASA Classification	RENAL Score
53	28.2	3.1	interpolar	3	7
51	33.8	2.4	interpolar	3	6
57	21.9	2.8	upper pole	3	4
50	33.1	1.5	interpolar	2	5
60	39.7	4.6	lower pole	3	7
44	26.6	3.7	interpolar	2	10

Six different surgeons, with an average of 10+ years in practice, participated in these experiments. Each surgery was performed with the Da Vinci Xi surgical robot through transperitoneal access. Our total time added to the surgical procedure was limited to 20 minutes or less in each of our cases, per to our IRB protocol. The initial surface tracing took approximately 75 seconds to collect. Across the six studies, pointing instances varied between 3, 5, and 10 pointings, according to the surgeons' discretion. Across all studies, the median TLE for all targets when comparing the bystander to the primary surgeon was 9.82 mm vs. 8.86 mm. For the lesion centroid, the overall median TLE was 7.44 mm vs. 7.67 mm (bystander vs. primary). For the planned arterial clamp location, the overall TLE was 14.13 mm vs. 18.74 mm (bystander vs. primary). For the intersection of the venous intersection, the overall TLE was 14.45 mm vs. 9.66 mm (bystander vs. primary). For the lesion-parenchyma intersection curve the overall TLE was 3.70 mm vs. 2.18 mm (bystander vs. primary). The TLEs for the individual studies can be found in Table II.

TABLE II. Target Localization Error (TLE) for Each Study and Anatomical Target.

Target Localization Errors in Millimeters
Surgeon	Lesion Centroid	Artery	Vein	Lesion-parenchyma
Primary	6.4	15.1	5.5	NA
Bystander	5.0	7.6	27.4	NA
Primary	4.9	11.0	5.2	NA
Bystander	6.3	7.7	18.7	NA
Primary	4.9	15.5	14.8	NA
Bystander	6.8	14.7	22.2	NA
Primary	6.6	18.0	17.3	NA
Bystander	6.2	21.1	24.8	NA
Primary	7.5	19.7	15.0	2.5
Bystander	6.9	20.4	22.3	4.7
Primary	7.3	18.5	13.3	3.1
Bystander	7.7	18.6	17.6	4.0
Mean Target Localization Errors Across All Cases in Millimeters
Primary	6.3	16.3	11.9	2.8
Bystander	6.5	15.0	22.2	4.35

IV. Discussion

The bystander protocol was successful in enabling us to quantitatively evaluate our robotic IGS system in the context of partial nephrectomy. Thus, the primary objective of this paper was achieved. Our results were inconclusive (no statistically significant differences) with respect to demonstrating that our robotic IGS system can improve accuracy in partial nephrectomy. This may be partially due to small sample size (6 cases). There are also several other challenges that are worth noting, which are opportunities for future research.

First, a significant source of error in our results is associated with the human's ability to point at a desired target using the da Vinci. We have previously studied this and found that attempting to point at the centroid of a sphere that is roughly the size of the tumors in this study introduces error of 5.21 mm [34]. Further, overall registration error contributes to the surgeons' TLEs. Next, our final intraoperative measured location is not truly ground truth. It is subject to tissue deformation during the process of removing fat from and manipulating the kidney, and also involves pointing at subsurface points. This error is largest at the arterial clamp and vein locations, which agrees with our intuition from qualitative observations of deformation during the surgical procedure. We believe that the variation in surgeon TLEs for the vasculature is largely due to this tissue deformation, which can occur at several different time points in the overall process. For example, differences in patient positioning between preoperative scans and surgery have been noted in the past to cause tissue deformation [35]. Furthermore, the de-fatting process that occurs before measurement of the intraoperative final location can cause additional tissue deformation. Indeed, in observing videos of surgery, one can qualitatively see what appears to be significant tissue deformation occurring during de-fatting. The most accurate target in our results was the lesion-parenchyma intersection curve. This too is as expected because it is the one location that can be physically touched by the primary surgeon during the final intraoperative localization.

Thus, in future studies we plan to incorporate ultrasound into our IGS system, which will enable registration to incorporate subsurface features, potentially enhancing accuracy. We also plan to incorporate deformable tissue models to account for deformations induced during the de-fatting and resection process. Such a model would serve to overcome the large vasculature localization error experienced with our current system, as well as account for kidney deformation throughout lesion resection. Note, our current implementation is tested before lesion resection begins. Specific models to account for kidney deformation after cutting will be needed as demonstrated by [18]. However, the overarching purpose of the current paper is the bystander framework, and the quantitative measurements it enables in vivo. Thus, the fact that we were able to make quantitative measurements that yielded the above insights demonstrates the value of the bystander protocol, which is the purpose of this paper.

Note that the bystander protocol itself is agnostic to target anatomy, source of images, and registration method. When applying it to other organs one may wish to, for example, digitize the surface with a tracked pointer or laser range scanner [36]. Similarly, one could use MRI images, segmented using any desired segmentation procedure, or other medical images. Other organs may also provide additional features that can be observed in both medical images and intraoperatively, providing additional data for validation. Furthermore, registration can be from points, surfaces, or even subsurface points or volumes (e.g. collected via intraoperative ultrasound).

It is worth noting that imaging protocols were not a major focus of the current work, but could impact accuracy. Before future clinical deployment, it will be important to examine the impact of factors such as slice thickness, scanner manufacturer, and segmentation algorithm on registration accuracy. None of these factors were a major focus of the current paper, which did not prescribe scanner parameters, and used scans collected by Vanderbilt as part of the normal course of care, which are typically conducted on a Siemens CT scanner with slice thickness of 1 mm.a

V. Conclusion

The bystander protocol enables quantitative in vivo assessment of systems for soft-tissue image guided surgery. By introducing an additional surgeon, the protocol isolates the primary surgeon from IGS system information, preserving the standard of care. Thus, in-development IGS systems can be tested in vivo, without risk of affecting the theraputic process by providing imperfect information to the primary surgeon. We demonstrate the bystander protocol in a series of robotic partial nephrectomies, and were successful in quantitatively evaluating an IGS system with the bystander protocol.

Funding Statement

This work was supported by the National Institutes of Health under Grants R01-EB023717 and T32-EB021937.