Assessing and reducing registration and targeting uncertainty in video-based image-guided renal navigation: An in vitro study featuring virtual targets

Abstract

Minimally invasive image-guided interventions (IGIs) enable better therapy outcomes for patients, but navigation accuracy is highly dependent on the accuracy of the image-/model-to-patient registration. This requires methods to reduce the uncertainty to a level appropriate for the procedure being performed. Since sub-surface tissue landmarks cannot be easily sampled using a tracked stylus and used to perform the patient registration, here we present a method that employs a tracked camera (that mimics a laparoscope) to perform the patient registration or update this registration in case of suspected misalignment within the context of an image-guided renal navigation procedure. We implement and test the method using a simplified patient phantom, which consists of a foam block to which a virtual kidney model featuring both surface and sub-surface landmarks is registered. This setup mimics a situation when a surgeon would navigate a tracked needle to renal landmarks percutaneously, while relying on pre-procedural imaging, optical tracking, and surface video imaging. We conduct several experiments under both optimal phantom registration and purposely altered registration, to not only show the effect of phantom / patient mis-registration on subsequent navigation, but also demonstrate the use of the camera-based registration to restore navigation to an acceptable uncertainty. Our results illustrate that camera-based registration yields a target registration error on the order of 0.4 mm and a subsequent targeting error on the order of 0.6 mm, comparable to the performance achieved following gold-standard landmark-based registration. These results suggest that the proposed method can be used to perform or update the patient registration for image-guided interventions involving sub-surface organs.

1. INTRODUCTION

For image-guided interventions (IGI), surgeons rely on pre-operative or intra-operative images to provide guidance to a region of interest within a patient in a minimally invasive manner. Intra-operative visualization via ultrasound or video imaging may be used to provide real-time guidance, but they may not provide the specific location of the tissue of interest, as it may not be visible in those modalities. This can be mitigated by registering pre-operative images to intra-operative images to augment the view, which implies accurate registration to the patient. Registration has been shown in previous work to be a significant source of error in an image-guided navigation system^1,2.

It is possible to register a patient directly to a tracking system if, for example, rigid fiducial markers are attached to a patient prior to pre-procedural, e.g. computed tomography (CT), imaging. However, rigid fiducials are not as useful for many interventions, particularly for soft tissue interventions, as they are prone to organ shifts. Furthermore, selecting sub-surface landmarks with a tracked stylus to obtain corresponding points is not feasible. One way to perform the registration is to use intra-operative imaging to view the surface of interest, which assumes either corresponding landmarks or a sufficient extent of the surface visible in the video to match the pre-operative model.³ For the purpose of this work, we rely on extracting registration landmarks from video images.

Similar to previous work by Stefansic et al.,⁴ we are interested in understanding the uncertainty of image-/model-to-physical patient (phantom) registration. The study used a direct linear transform applied to hepatic surgery. This differs from our proposed method in that we are using a perspective n-point (PNP) formulation, which assumes the camera has been previously calibrated. The experimental setup mimics a scenario where a laparoscope is used to image the patient’s skin (i.e., the foam block) beneath which the kidney is located. While the video shows the skin (i.e., foam) surface, the kidney is only visible via a pre-procedural model that needs to be registered to the patient (i.e., foam block). Moreover, it is also assumed that there are observable landmarks in the camera images that correspond to pre-operative landmarks located on the patient. This is analogous to viewing the surface of an organ using a laparoscope.

In this work, we demonstrate the use of tracked video images to perform the tracker-to-patient registration using a PNP formulation, and then compare the achieved registration accuracy to the accuracy achieved using a traditional landmark-based registration. As a result of the performance achieved using this technique, we also demonstrate the potential of using a tracked camera to update a previously sub-optimal patient registration, as a means to reduce tracker-to-patient registration uncertainty. Lastly, we also demonstrate the overall targeting accuracy following registration and image-guided navigation.

2. METHODOLOGY

2.1. System Overview

We previously developed a prototype 3D Slicer module to enable image-guided renal navigation via augmented and virtual reality views of the scene.⁵ We used an NDI Polaris Spectra optical tracking system (OTS) to provide tracking for tools. The system also tracks a DRF rigidly attached to a camera which generates tracked video images of the scene. Our camera was a Logitech C910 which acts as a surrogate laparoscope. Digital models from pre-operative images (CT or MRI) are generated, which provide context by augmenting the tracked camera view ith the virtual pre-operative model. The transformations between the various components of the IGI system are shown in Fig. 1. Each $T_a^b$ represents a 4×4 rotation and translation transformation matrix from coordinate space a to b

Figure 1.

Diagram representation for a calibrated IGI system. It shows the relationship between the world or physical object and camera space. Each T represents a transformation matrix between coordinate systems.

2.2. Experimental Overview

We emulated a minimally invasive image-guided renal intervention using a foam block phantom (mimicking the skin), a tracked stylus (mimicking a percutaneous access needle), and a virtual kidney model described in detail below. The phantom was instrumented with four surface landmarks that were subsequently sampled using a tracked stylus to register the phantom to the tracking system. A kidney phantom model was generated from an MRI dataset and twelve target landmarks were defined on the surface and within the volume of the model. The kidney model featuring the associated target landmarks was virtually registered to the foam phantom, mimicking a patient’s kidney (and its associated landmarks) located beneath the skin and not directly visible. The twelve virtual landmarks labeled on the virtual patient-specific kidney model constitute the therapeutic targets to be accessed via image guidance.

We first registered the virtual kidney model to the foam block, such that the virtual kidney is fully embedded within the foam block (i.e., patient) volume. The foam block was then registered to the tracker space using the four defined landmarks. We used the protocol described in⁵ to calibrate the tracked camera. Image-guided navigation trajectories were defined by selecting a target (from the twelve available landmarks) and a entry point on the surface of the foam block. Our image-guided navigation platform has navigation aids consisting of a virtual representation of the foam block surface as a rectangular plane. The stylus is represented by a virtual needle along with an orthogonal plane depicted by a circle. Lastly, the entry point-to-target trajectory is indicated by the corresponding axis and another orthogonal plane depicted by a circle. During navigation, a user aims to align the stylus and trajectory, while their corresponding orthogonal planes are parallel to each other. The user then navigates the tracked stylus to the target of interest and a virtual glyph indicating the location of the tracked stylus tip is “dropped” when the target is reached. This process is repeated for all defined targets.

The next step was to mis-register the foam phantom. We altered the patient registration by “adjusting” the registration transformation between the physical and virtual foam block by a known translation vector (20 mm). Alternatively, mis-registration was also achieved by altering the physical landmarks sampled on the foam block using the tracked stylus (Random Trial #1 & 2). After registration, the steps described above are repeated to obtain a navigation data set.

Following the navigation experiment under the mis-registration scenario, the registration was updated using information available using the tracked camera as described later, then another set of navigation experiments was conducted. Figure 2 shows an example view of the navigation screens from 3D Slicer. This view shows a mis-registered surface to the actual foam surface along with the navigation guidance to one of the target locations.

Figure 2.

Example screenshots of the navigation platform tool showing: A. camera view of the foam block phantom, B. virtual view showing the pre-operative kidney model beneath the surface of the foam block phantom along with the entry point-to-target trajectory and tracked needle displayed along with their navigation aids; C. augmented reality view showing the virtual overlay onto the camera image; and D. image view showing the virtual representation of the tracked stylus and the target to be accessed on an MRI slice.

2.3. Registration Update Procedure

To update the registration between the physical object and tracker, the transformation chain between the patient and the camera is used. A relationship between the patient in the physical world and the image can be written as:

$$X_{\mathit{\text{Image}}}=K{T}_{\mathit{\text{DRF}}}^{\mathit{\text{Camera}}}{\left(T_{\mathit{\text{DRF}}}^{\mathit{\text{Tracker}}}\right)}^{-1}{T}_{\mathit{\text{Physical}}}^{\mathit{\text{Tracker}}}{X}_{\mathit{\text{Physical}}}$$ (1)

where K is the camera intrinsic matrix. Eq. 1 can be seen as a Perspective n-Point (PNP) problem which assumes a pin-hole camera. This can be simplified to

$$X_{\mathit{\text{Image}}}=K{T}_{\mathit{\text{Physical}}}^{\mathit{\text{Camera}}}{X}_{\mathit{\text{Physical}}}.$$ (2)

If the corresponding coordinates of landmarks on the physical object are known in both the physical object and camera image coordinate systems, X_Physical and X_Image respectively, we can solve for $T_{\mathit{\text{Physical}}}^{\mathit{\text{Camera}}}$.

We used the OpenCV SolvePNP method using the iterative solution and using the initial $T_{\mathit{\text{Physical}}}^{\mathit{\text{Camera}}}$ values as an initial guess. Equating the updated transformation to the original chain of transformations, $T_{\mathit{\text{Physical}}}^{\mathit{\text{Camera}}}=T_{\mathit{\text{DRF}}}^{\mathit{\text{Camera}}}{\left(T_{\mathit{\text{DRF}}}^{\mathit{\text{Tracker}}}\right)}^{-1}{T}_{\mathit{\text{Physical}}}^{\mathit{\text{Tracker}}}$, we can solve for $T_{\mathit{\text{Physical}}}^{\mathit{\text{Tracker}}}$ if we know the transforms $T_{\mathit{\text{DRF}}}^{\mathit{\text{Camera}}}$ and ${\left(T_{\mathit{\text{DRF}}}^{\mathit{\text{Tracker}}}\right)}^{-1}$. $T_{\mathit{\text{DRF}}}^{\mathit{\text{Camera}}}$ is constant and while $T_{\mathit{\text{DRF}}}^{\mathit{\text{Tracker}}}$ changes with any movement of the camera, the tracking system reports this transformation and therefore both are known. This is accomplished using Equation 3:

$${\left(T_{\mathit{\text{Physical}}}^{\mathit{\text{Tracker}}}\right)}_{\mathit{\text{Update}}}=\left(T_{\mathit{\text{DRF}}}^{\mathit{\text{Tracker}}}\right){\left(T_{\mathit{\text{DRF}}}^{\mathit{\text{Camera}}}\right)}^{-1}{T}_{\mathit{\text{Physical}}}^{\mathit{\text{Camera}}}.$$ (3)

This procedure gives an updated result of the transformation between the patient and the tracking system.

2.4. Registration and Navigation Uncertainty Metrics

First, to assess registration uncertainty, we calculated the fiducial registration error (FRE) between the virtual foam block surface and the corresponding sampled landmarks. Subsequently, we calculated target registration error (TRE) between the registered landmarks and the ground truth landmarks, under each of three scenarios: optimal registration, intentional mis-registration, and updated registration post PNP-based correction. Lastly, we also computed the targeting error (TE - denoted as such to distinguish it from TRE), defined as the distance between the recorded coordinates of the tracked stylus tip (when the target was reached) and the actual target coordinates. Note that TE is also a TRE, but it refers to the error associated with overall navigation rather than registration alone.²

3. RESULTS

The overall registration and navigation results are shown in Table 1. The baseline error values are associated with registering the foam surface to the tracker directly using the stylus. As expected, the targeting error (TE) slightly exceeds the target registration error (TRE), even under optimal patient registration conditions.

Table 1.

Root-mean Squared (RMS) error associated with registration and navigation (FRE, TRE, and TE) for a baseline case and three mis-registered scenarios

	FRE (mm)	TRE (mm)	TE (mm)
Baseline Registration	0.34	0.35	0.42

	Mis-registration			Camera-based Re-registration
	FRE (mm)	TRE (mm)	TE (mm)	FRE (mm)	TRE (mm)	TE (mm)
Displaced 20 mm	0.34	19.68	19.78	0.36	0.50	0.66
Random Trial 1	6.36	4.40	4.40	0.29	0.49	0.70
Random Trial 2	5.25	13.23	13.27	0.27	0.46	0.88

The mis-registration cases clearly indicate the error associated with the registration landmarks (located at the foam block surface) for the random trials. The small FRE associated with the displaced case not only demonstrates good fit across a set of registration landmarks, despite their overall translation to an incorrect position, but also demonstrates the significant registration and navigation error associated with the target landmarks. The TRE and TE are closely related, indicating that poor target registration leads to poor targeting. However, following camera-based correction and re-registration, both TRE and TE are significantly reduced and are comparable to the baseline performance achieved under optimal registration at less than 1 mm RMS. These performance metrics are comparable to those reported by Stefansic et al.,,⁴ with an overall best result TRE of 0.56 mm.

Figure 3 shows the AR view of the virtual surface and kidney mis-registered and then correctly aligned with the foam phantom. This shows the ability to quickly notice if the system is not correctly registered and make corrections.

Figure 3.

Example augmented reality view showing the alignment before and after updating the registration.

To further investigate the difference between using landmarks sampled with the stylus (baseline) or the PNP solution to calculate the registration transformation, we transformed the fiducial markers by the camera-based registration transformations and calculated the FRE values. This error is the distance between the landmarks transformed by the baseline transformation and the three mis-registration camera-based corrected transformations. The small error (less than 0.5 mm) indicates the similarity between the transformations and shows that the PNP method produces similar results to the landmark based method.

4. CONCLUSION

In this work, we emulated an experimental setup that mimics an image-guided renal navigation intervention, with the overall goal to demonstrate the effect of the patient registration uncertainty on the overall targeting and navigation performance. Moreover, we also proposed a technique that relies on real-time tracked camera imaging, mimicking a laparoscope, to enable a user to update the patient registration in the event of suspected misregistration, and help restore both registration and overall targeting accuracy. Our experimental results showed comparable registration and overall targeting performance following camera-based registration update to the baseline registration and navigation under optimal registration. The proposed method is key when a direct landmark-based registration of an organ of interest located beneath the skin (or another organ surface imaged using video) is not feasible, and when other intra-operative imaging beyond video is not available.

Future work entails the exploration and incorporation of automated methods to identify landmarks in the video images that correspond to the pre-operative landmarks to be used for patient registration, as well as the potential use of the tracked camera to track / localize / and update the position of surgical instruments.