A Mechanics-Based Nonrigid Registration Method for Liver Surgery Using Sparse Intraoperative Data

Abstract

In open abdominal image-guided liver surgery, sparse measurements of the organ surface can be taken intraoperatively via a laser-range scanning device or a tracked stylus with relatively little impact on surgical workflow. We propose a novel nonrigid registration method which uses sparse surface data to reconstruct a mapping between the preoperative CT volume and the intraoperative patient space. The mapping is generated using a tissue mechanics model subject to boundary conditions consistent with surgical supportive packing during liver resection therapy. Our approach iteratively chooses parameters which define these boundary conditions such that the deformed tissue model best fits the intraoperative surface data. Using two liver phantoms, we gathered a total of five deformation datasets with conditions comparable to open surgery. The proposed nonrigid method achieved a mean target registration error (TRE) of 3.3 mm for targets dispersed throughout the phantom volume, using a limited region of surface data to drive the nonrigid registration algorithm, while rigid registration resulted in a mean TRE of 9.5 mm. In addition, we studied the effect of surface data extent, the inclusion of subsurface data, the trade-offs of using a nonlinear tissue model, robustness to rigid misalignments, and the feasibility in five clinical datasets.

I. Introduction

LIVER resection surgery in the open abdomen is a challenging setting for the application of image-guided surgical techniques which have been largely limited to procedures involving the cranium in the past. The difficulty is that surgical liver presentation typically begins with mobilization from the surrounding anatomy, followed by organ stabilization by packing support material underneath and around the organ. Thus, large deformations (on the order of several centimeters) typically occur between the preoperative (when CT imaging was performed) and intraoperative organ states.

While intraoperative imaging has been used to document the extent of deformation [1], and guidance solutions using intraoperative imaging have been proposed [1]–[5], the workflow requirements and the challenges of integrating preoperative imaging data continue to hinder adoption. As a result, a minimally encumbered accurate solution to efficiently align preoperative data to the intraoperative patient state would be of high significance.

A. Related Work

A review by Hawkes et al. [6] recognizes the limitations of the rigid body assumption in image-guided interventions and discusses nonrigid registration problems on various types of data using free-form deformation models, motion models, statistical models, and biomechanical models.

One area that has had considerable investigation is the nonrigid registration of preoperative imaging data using limited intraoperative data and biomechanical models within the context of brain-shift compensation. Typically these approaches involve using intraoperative cortical surface deformation measurements to guide a volumetric biomechanical model [7]–[10] to produce a full 3-D deformation field which best aligns preoperative positions to their intraoperative counterparts. Some have chosen alternative or enhanced approaches using intraoperative ultrasound data [11]–[14].

Guidance for applications in the liver began with studies focused on modeling and accounting for respiratory motion [15]–[18]. The need for accounting for this motion within the context of radiation therapy for abdominal organs is quite compelling [19].

Cross-sectional imaging with CT and/or MRI is critical to preoperative planning, used not only to assess disease extent but also to delineate the relationships between the tumor(s) and the vascular and biliary anatomy. Such information is essential for the safe conduct of complete tumor resections. The use of intraoperative ultrasonography represents a natural extension of these imaging techniques, allowing the acquisition of updated, real-time information for tumor resection [3]. Thus the collection of subsurface data has been well established and integrated into current workflow schemes. However, noise and speckle inherent to ultrasound will always limit contrast and image quality when compared to CT and MR. Thus, a variety of approaches to register preoperative image sets to vascular features using intraoperative 3-D ultrasound has been investigated, such as the combination of iterative closest point (ICP) algorithms and B-splines proposed by [20], and a framework combining landmarks and intensity information in [4].

Similarly, an approach based on a set of assumed deformation modes has been used to align a model to a simulated patch of intraoperative surface data [21]. This is perhaps the most closely related work to the approach in this paper. The main difference arises from the fact that in [21], free-form deformation modes are used which are unrelated to the mechanics of the organ itself (e.g., rigid-body modes, uniform bending, and Gaussian twisting along each axis). In contrast, we propose to use deformation modes based on the response of a patient-specific tissue mechanics model to the types of boundary conditions likely to be experienced in surgery, as detailed in the following section.

In our prior work, Cash et al. [22] studied a combination of rigid and nonrigid registrations to intraoperative surface data that utilized a large extent of coverage and reported encouraging results. Using a similar rigid-followed-by-nonrigid approach Clements et al. [23] developed a deformation atlas approach similar to [10]. Other work by Dumpuri et al. emphasized approaches that reduced the preoperative computational burden and were more accurate [24]. All of these approaches used a linear elastic finite element model generated from the patient’s CT data with each investigating techniques to deploy displacement boundary conditions on the organ surface in order to minimize the remaining partial surface misfit resulting after the rigid alignment. The boundary conditions were determined from signed closest point distances between the data and the model surface, using various methods for extrapolating these conditions across the entire liver surface, e.g., a surface Laplacian, or a radial filter in the case of Dumpuri et al. [25]. While each uses various combinations of preoperative/intraoperative computing, all have the common attractive feature of correcting for deformations without costly, awkward intraoperative imaging equipment.

B. Contributions

In the present work, our aim is to improve accuracy by developing a new nonrigid registration approach which reconstructs the likely physical causes of deformation, and to improve efficiency by not requiring affinite element tissue model to be solved in the intraoperative setting. Our proposed approach aligns a biomechanical liver model (built from preoperative images) to incomplete geometric data that represents sparse liver region locations gathered intraoperatively (e.g., anterior surfaces, tumor centroids). Unlike prior methods that were extrapolative in nature, our proposed approach casts the nonrigid registration as a nonlinear optimization problem. Based on very limited surgical organ presentation assumptions, a set of boundary conditions is parameterized on the deformation-inducing regions of the organ surface. Those parameters are then reconstructed via an iterative algorithm which minimizes the error between the incomplete geometric data and the deformed model counterpart, where precise correspondence is not known or assumed a priori.

We present the details of our proposed method in the following section. In Section IV, we give the results of experiments with liver phantoms undergoing deformations consistent with a typical surgical presentation, and demonstrate that our proposed iterative approach yields more accurate predictions than both rigid registration alone and our previous nonrigid method which uses boundary condition extrapolation [24]. We also explore the effect of incorporating small amounts of additional subsurface data, such as could be gathered by workflow friendly intraoperative imaging modalities like ultrasound. We investigate several aspects of the proposed approach using a representative phantom case, including the effect of perturbations in the initial rigid registration, the effect of using a geometrically nonlinear model, the effect of the extent of available surface data, and the effect of varying the parameters of our algorithm. Finally, five examples using clinical data further illustrate the feasibility of our method.

Some of our work on this subject was presented in preliminary form in [26]. In this paper, we offer several significant contributions beyond our early work including 1) generalization of our basic approach and extensive description of the methods involved in Section II, 2) numerous additional experimental investigations in liver phantoms and extensive experimental analysis of our approach in Section IV, and 3) feasibility studies using clinical case data.

II. Methods

We present our nonrigid registration algorithm as a central component in the context of a patient-specific data pipeline for surgical navigation. Prior to the registration realization, we perform several data acquisition and processing steps. The procedures described below were used in both the phantom experiments and the clinical examples presented in Section IV.

A. Intraoperative Data Collection

Our proposed method is based on intraoperatively acquiring a set of 3-D points corresponding to a portion of the organ surface. For this, we use a custom-built commercial laser range scanner (Pathfinder Technologies, Inc., Nashville, TN, USA). Intraoperatively, once the organ is presented, the laser range scanner sweeps a laser line over the surface of interest and records both shape and color information, i.e., a textured point cloud. Using the color information from the field of view, the organ surface can be rapidly segmented leaving only the sparse liver geometrical data. Alternatively, surface points could be gathered via an optically tracked stylus which can be swabbed over the organ surface. Once intraoperative surface data are acquired, anatomical landmarks are designated from the data (e.g., falciform ligament, inferior ridges, round ligament) and a salient feature ICP method developed in [27] is used to obtain an initial rigid registration.

With respect to additional geometric information, intraoperative ultrasound is routinely used within liver resection surgery. In addition, recently commercial guidance systems have begun to integrate tracking information with the ultrasound to provide references between ultrasound and preoperative images (e.g., Pathfinder Technologies Inc., has a tracked attachment for a t-shaped ultrasound transducer). Given that tumors are often localized with ultrasound, it is conceivable that a tracked probe could be used to locate one or more points of interest inside the organ, e.g., a tumor centroid or large vessel bifurcation. We investigate the effect of including such additional data in our phantom experiments in Section IV.

B. Finite Element Model From Preoperative Image Set

CT image volumes are typically acquired approximately one week prior to performance of the surgical procedure. For clinical cases we use a semiautomatic method developed by Dawant et al. [28] and Pan and Dawant [29], based on the level set method proposed by Sethian [30], to segment the liver from the surrounding anatomical structures in the preoperative tomograms. For the phantom cases studied in Section IV, we manually segmented the liver surface using Analyze (Biomedical Imaging Resource, Mayo Clinic) due to the ease of segmenting phantom data. Isosurfaces are generated from the liver segmentations via the marching cubes algorithm [31] and smoothed via radial basis functions (RBF) (FastRBF toolkit, FarField Technology, Christchurch, New Zealand). A tetrahedral mesh is then generated from this surface using the customized mesh-generation software [32]. Using a nominal tetrahedron edge length of 4 mm results in a triangular surface representation with a resolution such that discrepancies between the RBF and the surface mesh are minimal.

Our linear elastic model entails the use of the standard 3-D Navier–Cauchy equations for the displacement field

$$\frac{E}{2(1+\nu )(1-2\nu )}\nabla (\nabla ·\mathit{u})+\frac{E}{2(1+\nu )}{\nabla }^2\mathit{u}+\mathit{F}=\mathbf{0}$$ (1)

where E is Young’s modulus, ν is Poisson’s ratio, u is the 3-D displacement vector at a point in the body, and F is the applied body force distribution. Using linear basis functions defined on the tetrahedral elements, we perform the standard Galerkin weighted residual method to obtain the standard linear system of equations of the form

$$K\mathit{d}=\mathit{f}$$ (2)

where K is the 3n × 3n global stiffness matrix, d = [u_1x u_1y u_1z ··· u_nx u_ny u_nz]^T is the vector of nodal displacements, and f contains the contributions of the applied body forces and/or surface tractions at each node.

Displacement boundary conditions are applied at a subset of the surface nodes by modifying the corresponding equations in (2), which results in a new system of equations [33]

$$A\mathit{d}=\mathit{b}$$ (3)

which is solved for the nodal displacements that satisfy static equilibrium for the given boundary conditions.

III. Proposed Nonrigid Registration Algorithm

The goal of our proposed approach is to align the volumetric organ model (built from the preoperative image set) with the incomplete geometric patient data gathered intraoperatively. The basic structure of the algorithm is depicted by the flowchart in Fig. 1. A set of parameters described in the following sections is used to define the rigid and nonrigid components of a trial displacement mapping at each iteration. The algorithm initializes these parameters via a rigid registration and calculates the error between the model surface and the data at each iteration, updating the guess for the parameters using a nonlinear optimization routine until the surface fit is sufficiently accurate.

Fig. 1.

Structure of the proposed nonrigid registration approach is depicted in flowchart form. Parameters are updated via a nonlinear optimization routine. In this work, we used the Levenberg–Marquardt algorithm. Required gradients were computed via forward finite differences.

A. Nonrigid Deformation

The nonrigid deformation modes are selected according to an assumed type of surgical presentation. We designate a “support surface” region on the posterior side of the organ where contact often occurs during routine mobilization of the liver from its surrounding anatomy and subsequent “packing” of support material underneath it to stabilize its presentation. We note that the extent and location of the support area can be approximately known in advanced based on a surgical plan, and that reasonably small deviations from this plan only slightly affect the performance of the approach, as demonstrated by the experiments described in Section IV. The geometry of the liver itself usually provides an intuitive way to select the support surface, which is consistent with a typical surgical presentation. As shown in Fig. 9 left, on most of the surface there is a defined edge or corner where the anterior surface (which is red in Fig. 9) transitions into the posterior side (which is blue). This feature is also evident in the clinical organ shapes in Fig. 12. The entire posterior side of the organ can be manually designated as the support surface using this edge as a boundary, and this is the approach we take for all the studies reported in this paper.

Fig. 9.

The “correct” and “incorrect” support surface designations tested for our representative case.

Fig. 12.

Results from two clinical case studies with swab data from a tracked stylus. The blue surfaces are the rigidly registered models. The red surfaces show the nonrigid registration resulting from the proposed algorithm. Compared to the rigid registration, the nonrigid registration improved the average closest point distance between the model surface and data from 4.5 mm to 2.4 mm in swab case 1 and from 7.4 mm to 3.0 mm in swab case 2.

A smoothly varying displacement field for the designated support surface is specified via a bivariate polynomial form as follows:

$${\mathit{u}}_s={\widehat{\mathit{n}}}_s\sum _{1\le i+j\le n}{c}_{ij}{t}_1^i{t}_2^j$$ (4)

where u_s is the displacement vector for a point on the support surface, n̂_s is the average unit normal vector over the designated support region (the area weighted average over the triangular boundary elements), and t₁ and t₂ are tangential coordinates of the point on the support surface (measured from the origin perpendicular to n̂_s in two orthogonal directions). Thus, the constant coefficients c_ij define the nonrigid displacement field over the support region. The sum over 1 ≤ i + j ≤ n avoids redundancy with the subsequent rigid transformation by excluding the constant displacement mode, which is captured within a general rigid-body motion.

For the surface nodes located on the support region, the corresponding displacements given by (4) are assigned as model boundary conditions to obtain the system of equations (3). Thus, solving the model produces a displacement field consistent with the assumed support conditions.

The model response to each of the coefficients c_ij may be precomputed and stored in matrix M, where each column is a displacement vector d_ij obtained by solving the finite element system (3) with the right-hand side vector computed assuming c_ij = 1 with all other coefficients are zero. Because we are employing a linear elastic model, the principle of superposition applies, and M can subsequently be used to rapidly compute the model solution for any combination coefficients as

$$\mathit{d}=\mathit{Mc}$$ (5)

where c = [c₀₁ c₁₀ c₀₂ c₁₁ c₂₀ ··· c_n0]^T is the vector of coefficients.

B. Rigid Transformation

After solving the model, a rigid-body transformation may be applied to the deformed nodal coordinates via a homogeneous transformation matrix consisting a translation vector t = [t_x t_y t_z] ∈ ℝ³ is the translation vector and a rotation matrix R ∈ SO(3) computed as the matrix exponential of a skew-symmetric matrix defined by a rotation vector θ = [θ₁ θ₂ θ₃]^T ∈ ℝ³ as follows:

$$R=exp\left(\left[\begin{array}{ccc}0& -{\theta }_z& {\theta }_y\\ {\theta }_z& 0& -{\theta }_x\\ -{\theta }_y& {\theta }_x& 0\end{array}\right]\right).$$

If after solving the model, the position vector of node i is p_i, then rigid transformation produces a new position vector ${\mathit{p}}_i^{\ast }=R{\mathit{p}}_i+\mathit{t}$. Thus, after nonrigid deformation and rigid transformation, a vector of parameters which defines the total displacement field can be expressed as

$$\mathit{P}={\left[\begin{array}{ccccccc}{\mathit{c}}^T& t_x& t_y& t_z& {\theta }_x& {\theta }_y& {\theta }_z\end{array}\right]}^T.$$

C. Optimization Algorithm

Our registration method is based on a nonlinear optimization framework where the aforementioned parameter set is iteratively chosen to minimize an objective function defined by a metric quantifying the fit between the deformed model and the available data. In this study, we propose the following objective:

$$F=\frac{1}{N}\sum _{i=1}^N{\left({\widehat{\mathit{n}}}_{ci}^T({\mathit{p}}_{di}-{\mathit{p}}_{ci})\right)}^2+\alpha E^2$$ (6)

where p_di is a 3 × 1 vector containing the Cartesian coordinates for the location of the i^th point in the surface data cloud, p_ci is the location of the corresponding point on the model surface (we discuss how correspondence is determined below), n̂_ci is a unit vector normal to the model surface at p_ci. E is proportional to the total strain energy stored in the nonrigid displacement field produced by the model solution (before the rigid transformation is applied), and is calculated as E = d^T Kd. α is a weighting constant, so that the term αE is a regularization term that balances accuracy of shape matching and the distortion of the deformation field.

The nonlinearity in the optimization problem arises from two sources: 1) rigid-body motion is inherently nonlinear due to the rotational component, and 2) we allow the correspondences between the model surface points and the data to update as the optimization progresses. At each iteration, the corresponding model point to each data point is assumed to be the closest-point (using the Euclidean distance) on the displaced model surface (which is defined by the current set of parameters). Thus, the approach implicitly solves the data-to-surface correspondence problem simultaneously with the nonrigid registration problem.

There are many well established optimization methods suitable for updating the parameter set at each iteration of the algorithm. In this study, we found that the Levenberg–Marquardt procedure worked well and we implemented it for the phantom and clinical cases in Section IV. With this method, the parameter update step is computed as follows:

$${\mathit{P}}_{k+1}={\mathit{P}}_{k+1}-{(J^{T}J+\lambda \text{diag}(J^{T}J))}^{-1}{J}^T\mathit{r}$$ (7)

where

$$\mathit{r}={\left[\frac{{\widehat{\mathit{n}}}_{c1}^T}{\sqrt{N}}({\mathit{p}}_{d1}-{\mathit{p}}_{c1}),\dots ,\frac{{\widehat{\mathit{n}}}_{cN}^T}{\sqrt{N}}({\mathit{p}}_{dN}-{\mathit{p}}_{cN}),\sqrt{\alpha }E\right]}^T$$

is the residual vector containing each of the error terms which are subsequently squared and summed in (6), λ > 0 is the damping parameter which can be selected iteratively to achieve optimal performance or set to a constant, and J is the Jacobian matrix of partial derivatives J = ∂r/∂P.

Thus, our nonrigid registration approach proceeds as shown in Fig. 1. First, a set of initial parameters is chosen. We choose the initial nonrigid coefficients c_i,j to be zero, and we use a rigid registration method to obtain an initial guess for the rigid parameters t_x, t_y, t_z, θ_x, θ_y, θ_z. Throughout this paper we have initialized our algorithm using the iterative-closest-point variant studied in [27]. However, any rigid registration method can be used for initialization, and even a nonoptimal approximate alignment may be sufficient. We study the effect of perturbations on the initial registration in Section IV. At each iteration, our algorithm then calculates the deformed nodal locations from using the nonrigid coefficients and (5). The transformation defined by the rigid parameters is then applied to the deformed nodal coordinates to produce the final displacement field for the iteration. Correspondence between the model surface and the data is then via closest point relationships and the residual vector function is evaluated. We compute the Jacobian matrix J via finite differences by evaluating the residual vector for small changes in each parameter [33], and apply the Levenberg–Marquardt update step to calculate the next guess for all parameters. This process is repeated for a fixed number of iterations (we used 10 in our experimental trials) or until the surface fit is sufficiently accurate.

D. Incorporating Model Modifications

We note that in general, it is straightforward to adapt the proposed modeling approach and optimization framework to include deformation effects from a variety of sources other than normal displacements on the support surface, by simply parameterizing them within the elasticity model of the organ. For example, tangential displacements on the support surface can be parameterized in exactly the same way as the normal displacements in (4). This could be useful if the expected surgical presentation involves “unfolding” the liver or stretching it out on the support surface before resection. In addition, distributed tissue forces arising from gravity (due to orientation changes) or fluid perfusion can be modeled if they are expected to play significant role in the organ deformation.

As an example, consider including gravity g as a force distribution vector which is constant over the volume. When building the finite element system in (2), the right-hand side vector f is linear in the components of g, and the matrix K is unaffected. Therefore, the model response to each gravity component can also be precomputed and stored as an additional column of the matrix M used for fast model computation in (5). Then we can simply include the components of g in the parameter vector P so that they are simultaneously selected with the coefficients c_i,j and the rigid parameters within the optimization routine. The decision to include or not include a particular effect can be made based on the anticipated surgical plan and/or the nature of the intraoperatively acquired data, in order to balance trade-offs between concerns for computational speed, model accuracy, and over-fitting.

E. Incorporating Intraoperative Subsurface Data

In the case where additional subsurface data is available intraoperatively, e.g., a tumor location from tracked ultrasound measurements, we can modify our objective function accordingly, as

$$F=\frac{1}{N}\sum _{i=1}^N{\left({\widehat{\mathit{n}}}_{ci}^T({\mathit{p}}_{di}-{\mathit{p}}_{ci})\right)}^2+{\alpha }_1{E}^2+{\alpha }_2{\Vert {\mathit{p}}_d-{\mathit{p}}_c\Vert }^2$$ (8)

where p_d is the location of the point of interest as measured intraoperatively, and p_c is the location of the corresponding model point as predicted by the parameterized displacement field. The point of interest would need to be specified in the preoperative image set before this information could be included in the objective function.

IV. Experimental Validation Studies

In this section, we evaluate our proposed nonrigid registration method in a series of experiments with anthropomorphic liver phantoms. We compare the results of the method to ground-truth fiducial locations throughout the phantoms as measured by CT imaging. First, to verify that the approach is broadly applicable, we analyze its accuracy in four cases where a phantom underwent a set of plausible deformations ranging from small to large. The phantom used in these four cases contained 28 fiducial targets evenly distributed throughout its volume for validation.

Then, to provide a more detailed investigation of the limitations and sensitivities of the method, we describe a number of analyses for a single additional large deformation case where the phantom contained a denser distribution of 58 validation targets. For this representative and data-rich case we evaluate 1) the additional accuracy of the nonrigid registration beyond rigid registration alone and previous nonrigid registration methods, 2) the result of incorporating subsurface data in our framework, 3) the result of employing a nonlinear corotational approach in place of the linear elastic model, 4) sensitivity analysis with respect to choosing of the number of modes and the energy weighting coefficient, 5) robustness analysis with respect to variations in the initial rigid alignment, 6) the result of incorrectly designating the support surface region, and 7) the result of having various extents of surface data coverage, from small to large.

A. Data Collection Procedure

As depicted in Fig. 2(a), a compliant liver phantom was made using a cyrogel of water, Polyvinyl alcohol, and glycerin [34]. This recipe was refined based on our knowledge of organ motion derived from a 75 patient multi-center clinical trial with our industrial collaborator [24] Plastic target beads (visible in CT) were evenly dispersed inside the phantom to serve as ground-truth fiducials in our analysis. A CT scan of the phantom was taken to identify the initial target locations. Our finite element model mesh was generated from this image volume representing the “preoperative,” undeformed organ state. Next, as shown in Fig. 2(b), the phantom was deformed by adding blocks of support material under certain parts of the posterior side of the liver, simulating the intraoperative procedure of organ repositioning and stabilization by packing material underneath. In this deformed state, we captured surface data with a laser range scanner (LRS) to drive our nonrigid registration algorithm, and a second CT scan was taken to identify the post-deformation target locations for validation. An initial rigid registration was acquired via the weighted patch ICP algorithm in [27]. Then our proposed nonrigid algorithm described in Section III was applied to deform the model to match the surface data.

Fig. 2.

(a) Phantom liver tissue in its “preoperative” undeformed state. (b) Thick support material was placed underneath the phantom. (c) Phantom liver in its deformed “intraoperative” state, due to the presence of the extra material underneath.

B. Results Over a Range of Deformations

To evaluate our proposed method over a range of deformations that could be encountered in surgery, we deformed a liver phantom in four different ways as illustrated in Fig. 3 by adding support blocks underneath various parts of the phantom. In case 1, two support blocks were added underneath the middle of the phantom, allowing the side lobes to droop. In case 2, a block was placed underneath the right lobe, and in case 3 a block was placed under the left lobe. In case 4, a block and an extra towel were placed under the right lobe, which gave rise to a significant rotational component in the displacement field as well as a sharp bend in the corner of the left lobe near the falciform ligament. We gathered simulated intraoperative surface data by sampling the post-deformation CT surface, as depicted by the overlaid white points in the third column of Fig. 3. Salient features patches were designated and the weighted iterative closest point algorithm (wICP) studied in [27] was used to obtain the initial rigid registration shown in the middle column of Fig. 3. Then our proposed nonrigid algorithm described in Section III-C was applied to fit the model surface to the LRS data. We used n = 3 for the degree of the bivariate polynomial (resulting in nine nonrigid parameters in addition to the six rigid parameters), and chose the energy weighting coefficient as α₁ = 10⁻⁹ N² with a Young’s modulus of 2100 Pa and Poisson’s ratio of 0.45 in our elastic model. (Note that since the model contains no body forces and only Dirichlet boundary conditions, the value of Young’s modulus does not affect the displacement solution, only the scale of the stored energy). Over the 25 embedded target fiducials dispersed throughout the liver, the mean target registration error (TRE) after rigid registration was 9.5 mm, 13.8 mm, 7.9 mm, and 8.3 mm in cases 1–4, respectively. After applying our nonrigid registration algorithm, the mean target error was reduced to 2.7 mm, 3.0 mm, 3.2 mm, and 3.7 mm, respectively.

Fig. 3.

Top: Post-deformation CT segmented surface is shown in blue, and the model-predicted surface is shown in red for four deformation cases at three stages in the registration process. The surface data used is overlaid in white on the final result. Bottom: A histogram of TRE over the four cases using rigid registration, our previous nonrigid method [24], and the proposed nonrigid method.

C. Comparison of Methods for a Representative Case

The same data collection procedure described above was applied to a liver phantom with 58 embedded target fiducials, which was deformed by placing support blocks under both left and right lobes as well as the front portion of the phantom, as shown in Fig. 2(c). The extra support material caused significant upward displacement of the supported portions of the phantom, while the unsupported portions sagged down to the bottom of the container. We use this large deflection case with a denser distribution of fiducial targets to evaluate various aspects our method in the following subsections.

In Fig. 4(b), we show the deformed model surface resulting from our algorithm in red, with the true deformed surface generated from the second CT scan overlaid in blue. We note that the deformed model visually matches the partial surface data as closely as possible and also displays the same qualitative behavior as the true surface in the posterior region where no data is present. The model predicts that the three supported sides are displaced upward as much as the true surface and the middle and back remain as low as the true surface. Rigid registration using the method of [27] is shown in Fig. 4(a).

Fig. 4.

LRS data cloud is superimposed on the post-deformation CT segmented surface in blue and the registered model surface in red. (a) The initial rigid registration using the method of [27]. (b) The proposed iterative method. (a) Rigid Registration. (b) Proposed Nonrigid Method.

Predicted post-deformation target locations were also generated using the algorithm’s displacement mapping and compared to their CT-measured post-deformation locations to assess ground-truth accuracy, as shown in Fig. 6.

Fig. 6.

Locations of the fiducial targets are shown in the deformed phantom volume, colored, and sized according to their respective registration errors. We note that there is a trend toward greater error as distance from the available surface data increases.

Fig. 5 collects the statistical information for all 58 target locations across the three different registration methods for comparison. The rigid registration method of [27] is shown in red with a mean TRE of 8.0 mm. For completeness, we also show our previous nonrigid method in [24] in yellow is shown in yellow with a mean TRE of 7.2 mm. Our proposed iterative method is shown in light blue with a mean TRE of 4.0 mm. The proposed method’s error distribution has less spread than the other methods indicating better agreement over a wider geometric region is being achieved.

Fig. 5.

Statistical histogram of the 58 target errors resulting from three different methods of registration applied to our representative phantom case. Red: Results of a rigid registration using the weighted patch ICP method of Clements et al. [27]. Yellow: After the rigid registration, the results of a subsequent nonrigid registration using a surface Laplacian to extrapolate boundary conditions as detailed in Dumpuri et al. [24]. Light Blue: After the rigid registration, the results of a subsequent nonrigid registration using the proposed iterative method. Dark Blue: After the rigid registration, the results of a subsequent nonrigid registration using the proposed iterative method with one additional subsurface data point.

D. Improvement Using Subsurface Data

To investigate the effect of incorporating of subsurface data (which could be acquired e.g., by identifying a subsurface feature using tracked ultrasound), a subsurface tumor was simulated by embedding a polyester sphere soaked in barium sulfate within the phantom as shown in the CT slices in Fig. 7. The pre-and post-deformation locations of the simulated tumor centroid were determined from the CT images and used as described in Section II as additional data in the fitting process with α₂ = .01. Including this one additional subsurface data point improved the mean TRE to 3.3 mm, and the resulting TRE distribution is shown in dark blue in Fig. 5

Fig. 7.

(a) Slice from the CT image volume of the phantom liver in its undeformed “preoperative” state, as shown in Fig. 2(a). Embedded tumor phantom is visible in this slice. (b) Corresponding CT slice of the phantom liver in its “intraoperative” deformed state, as shown in Fig. 2(c). Added support material causes a large displacement in the tumor location.

E. Robustness to Initial Registration

We also analyzed the robustness of nonlinear optimization procedure to the initial registration input. In general, we assume that the data input to our algorithm has already been rigidly registered (in this paper we use the wICP algorithm in [27]) to provide a good initial estimate of correspondence between the model and the intraoperative surface data. To assess the role that small initial alignment variations might play during the course of the iterative nonlinear optimization process, a Monte Carlo simulation was performed where the initial alignment between the LRS point cloud and the preoperative liver model was perturbed in 6DOF space by a maximum total angle of 10° and translated by up to 10 mm in each direction. In 100 simulations, all of the final mean TRE values fell between 3.6 mm and 4.4 mm which compares similarly to the 4.0 mm mean TRE obtained with the original wICP alignment.

F. Sensitivity to Weighting and Number of Modes

We investigated the sensitivity of our proposed algorithm to changes in the formulation of the optimization problem, namely the energy weighting coefficient α₁ and the order of the bivariate polynomial which defines the displacement of the support surface. The resulting statistics for the TRE is summarized by the box plots in Fig. 8. We conclude from inspection of the results that n = 3 provides a sufficient number of support surface modes (nine modes) for accurate subsurface predictions, but increasing the number past this yields successively diminishing returns. Similarly, reducing the energy weighting coefficient will yield more accurate results up to a point.

Fig. 8.

Statistical box plots are shown illustrating the sensitivity of the results of the algorithm to changes in the weighting coefficient α₁ and the order n of the bivariate polynomial for the support surface displacements.

The trade-off for decreasing α₁ is that unrealistically large deformation predictions are possible if the chosen mode set cannot easily reproduce the surface data. This could be the case if the designated support surface was wildly incorrect or if there is significant unmodeled behavior such as swelling due to perfusion or transverse stretching of the organ. We note that these types of behavior could also be parameterized and included within our framework, and the choice of modes should be refined over time according to experience and clinical case data. To buffer this uncertainty, α₁ provides a way to control the amount of predicted deformation to a reasonable level. Upon collecting surface data, it can be increased if the deformation looks unreasonable.

G. The Effect of Incorrect Support Surface Designation

One potential limitation to the proposed method is that it requires an a priori assumption about what part of the organ surface is directly contacted by support material. In our experience with image-guided liver surgery, a good estimate of the actual support surface can be easily determined from the preoperative plan performed by the surgeon. The posterior region of the organ as shown in blue in Fig. 9 on the left is usually easy to manually segment by using the high surface curvature at the lobe edges as a boundary. This posterior region serves as an appropriate starting point for the support surface designation which can be modified according to the surgical plan if necessary.

Even with a good estimate of the support surface, accuracy of the nonrigid registration will no doubt be affected by the degree of agreement between this assumed support region and the actual intraoperative support region. To provide an estimate of how much an incorrect support surface designation might affect registration accuracy, we tested our method under a case where the support region was quite incorrectly designated as shown in Fig. 9. The resulting mean target registration error for the incorrectly designated case was 4.6 mm, compared to a mean TRE of 4.0 mm in the correctly designated case. This result indicates that an accurate nonrigid registration may still be obtained even if the support surface designation is somewhat inaccurate.

H. The Effect of Data Coverage Extent

We also investigated the effect of having smaller patches of intraoperative surface data available for registration. As shown in Fig. 10, we tested three sizes of data coverage, the largest of which was the dataset used in our representative case. The resulting mean TRE for the two smaller extents [(b) and (c) in Fig. 10] was 5.0 and 5.2 mm, respectively.

Fig. 10.

(a) Extent of surface data used for our representative case, which produced a mean TRE of 4.0 mm. (b) Smaller extent of surface data which resulted increased the mean TRE to 5.0 mm. (c) Minimal extent of surface data which resulted in a mean TRE of 5.2 mm.

I. The Effect of Including Gravity

As we discuss in Section III-D other forces on the tissue such as gravity are straightforward to include within our model and registration framework. We performed an additional execution of our registration procedure for the representative case of 4, and we included gravity in the model and the parameter set to be optimized as described in Section III-D. The resulting TRE distribution was very similar to that shown in Fig. 5 obtained without including gravity, and the mean TRE was slightly decreased to 3.9 mm.

J. The Effect of Model Linearization

We tested our assumption of linear elasticity by implementing a second model using a nonlinear, corotational finite element framework [35] for comparison. We enforced identical displacement boundary conditions on the support surfaces of both models, taken from the final registration solution for each of the five phantom cases shown in Figs. 4 and 3. After solving both models for each case, we compared the resulting displacement fields from the two solutions. Over this set of cases, the maximum euclidean distance between the location of any point in the linear model solution and the location of that same point in the nonlinear model solution was 3.3 mm, which occurred at the tip of the lobe in the Fig. 4 case. Fig. 11 shows the distribution of the euclidean difference between the two model displacement solutions for this case, with a linearization error of less than 1 mm for the vast majority of the volume. The other four cases exhibited similar patterns with even less linearization error.

Fig. 11.

Deformed surface from the phantom case of Fig. 4 is shown. Color map over the surface illustrates the euclidean difference between the linear and nonlinear model solutions using the same set of boundary conditions. Linearization error is less than 1 mm for the majority of the volume, with a maximum error of 3.3 mm where element rotations are highest.

We consider this amount of linearization error to be relatively small when compared to the displacements exhibited in our phantom experiments (up to 23 mm, with element rotations as large as 30°), suggesting that a linear model is perhaps adequate in this context. However, we do not wish to minimize the importance of nonlinearities, realizing that linearization could potentially have much larger adverse effects depending on the nature and scale of the deformation. Care should always be taken to quantify, as we have done here, the expected amount of linearization error for the types of organ deformation being considered, and future work towards fast, stable nonlinear model implementations is certainly needed.

Currently, the computational trade-offs favor the linear approach for surgical guidance applications. In our phantom experiments, the total computation time for convergence of the proposed algorithm (eight iterations, 149 evaluations of the objective function) was approximately 13 s using a model with 8000 nodes and a surface point cloud containing approximately 4000 3-D data points, using a MATLAB implementation on a standard laptop computer with a 2.67 GHz Intel Core i5 CPU. The total number of model solves required for our registration algorithm to converge is usually on the order of 100, so the increase in computation time required to implement a nonlinear model must be weighed against the potential loss of accuracy in linearization.

The linearization error should also be viewed in the context of the clinical data acquisition and registration problem, and compared to other potential sources of inaccuracy. Aside from the process of data acquisition itself, one source of potential registration inaccuracy comes from driving the registration problem with incomplete surface data having unknown or only approximate correspondence to the model surface. Another error source is the ability of the assumed boundary condition modes to actually reproduce the correct displacement field. We estimate that these two sources combined contribute the majority of the target registration error.

As a case in point, we replaced the fast linear solve portion of our algorithm (which combined deformation modes via superposition as described in Section II) with an iteratively solved corotational finite element framework [35]. The nonrigid registration algorithm with the nonlinear model produced a mean TRE of 4.8 mm, compared to the mean TRE of 4.0 mm obtained with the linear model. When the subsurface tumor data was included, the nonlinear model produced a mean TRE of 3.4 mm, while the linear model produced a mean TRE of 3.3 mm. Thus, even though the nonlinear model represents the physical tissue mechanics more accurately than the linear model, random uncertainty and bias from other sources coincidentally caused the registration algorithm to perform slightly worse with the nonlinear model.

K. Clinical Feasibility Study

We investigated the feasibility of our proposed framework in five clinical cases where anonymized data was gathered by our industrial collaborator Pathfinder Therapeutics Inc. under an Institutional Review Board approved study with informed written patient consent. In three of the cases, a 3-D laser range scan was performed over the visible surface of the liver in the intraoperative setting, providing a dense surface cloud. In two of the cases, swabbed surface data collected with a tracked probe was used, which illustrates the versatility of the proposed method to handle sparse and noisy datasets.

For the three LRS cases, the energy weighting coefficient was chosen as α₁ = 1 × 10⁻⁵. For the two swab cases, some inaccuracy is introduced by intermittent stylus-to-tissue contact and local deformation due to stylus contact, so the energy weighting coefficient was chosen as α₁ = 2 × 10⁻⁴ to avoid fitting the limited data with unreasonably large deformations.

The two swab case results are shown in Fig. 12 with the sparse dataset overlaid on top of the registered model surface. The nonrigid registration is compared to the results from rigid registration using the method of [27]. The three LRS cases are similarly shown in Fig. 13, where instead of overlaying the dense surface dataset, we show the registered surface color coded by signed closest point distance to the data.

Fig. 13.

Results from three clinical case studies with dense laser range scanned surface data. The registered model surface is colored by the signed closest point distance to the surface data. Compared to the rigid registration, the nonrigid registration improved the average normal closest point distance between the model surface and data from 3.1 mm to 1.8 mm in LRS case 1, from 4.3 mm to 3.0 mm in LRS case 2, and from 4.6 mm to 3.5 mm in LRS case 3.

Table I shows the mean surface errors (the closest point distance) for the five clinical cases, showing an improvement in each case over rigid registration alone. We note that the surface errors after nonrigid registration are higher than the same surface metric for our representative phantom case (0.6 mm), which indicates that while much of the deformation has been accounted for, there are still some aspects that our chosen parameterization is unable to reproduce. As discussed in Section III-D, incorporating additional parameters (such as modes which mimic an “unfolding” of the organ) could potentially capture more of the deformation, while increasing the trade-off of potentially over-fitting the data for cases in which this does not occur. Future validation work using intraoperative imaging is needed to fully address these tradeoffs and optimize our proposed approach for clinical deployment.

TABLE I.

Clinical Surface Registration Errors

	Mean Surface Error (mm)
	Rigid Registration	Nonrigid Registration
Swab Case 1	4.5	2.4
Swab Case 2	7.4	3.0
LRS Case 1	3.1	1.8
LRS Case 2	4.3	3.0
LRS Case 3	4.6	3.5

V. Discussion

The distribution of TRE illustrated by the histograms in Figs. 3 and 5 show an average improvement in TRE from 9.5 mm to 3.3 mm over the five phantom experiments when our nonrigid registration approach is employed, and further improvement is shown when a single point of subsurface data was incorporated. In the analysis of our representative large-displacement phantom case with a dense distribution of fiducials, the proposed method demonstrate robustness to variations in initial registration, support surface designation, and the extent of available surface data. The effect of the energy weighting parameter and the number of support surface displacement modes was also investigated for this dataset, and a point of diminishing returns was shown. Our comparison of linear and geometrically nonlinear tissue models demonstrates that linear models are capable of providing good guidance for this surgically realistic large-displacement case. Five cases using clinical data suggest that the method is capable of providing a realistic deformation mapping when driven by both sparse and dense data acquired in the operating room.

With respect to accuracy needs in the clinical setting, our previous work has demonstrated that rigid organ-based registration errors are on the order of 1–2 cm routinely in clinical cases [36]. Adding to this experience, the clinical conventional experience has considered the 1 cm negative margin to be the minimum acceptable resection threshold (as reported by a number of clinical studies [37]–[39]). These would suggest that perhaps a surgeon’s intuitive spatial understanding of the lesion between preoperative and intraoperative experience is compromised by organ deformation. We would suggest that in order to restore an intuitive spatial understanding to the surgeon, it would be beneficial to reduce target errors to below 5 mm on average (an accuracy threshold also suggested by [40]).

While much future work is needed, to our knowledge the results herein are the first to suggest that mean target registration errors less than 5 mm over the volume are possible using a sparse-surface-data driven mechanics-based nonrigid registration method. While [22] and [24] suggest TREs less than 5 mm, the results were achieved with significantly more boundary condition information with very coarse TRE sampling in the former, and involved deformations significantly less than the clinical counterpart in the latter. In this paper, the method we propose needs very little a priori boundary condition information and is validated with a detailed spatial sampling of target error in experiments containing deformations that were carefully generated to mimic the clinical scenario.

We note that the proposed method could also be feasible for laparoscopic procedures if enough surface data can be collected laparoscopically, and research is currently being conducted toward the development a device capable of laparoscopic surface scanning based on conoscopic holography [41]. The limitations on data collection in the laparoscopic case could be somewhat compensated for by the fact that the organ deformation is likely to be less extensive. We leave investigation of nonrigid registration using laparoscopically gathered data to future work.

VI. Conclusion

We conclude that use of the proposed iterative method for nonrigid registration of the preoperative liver to the intraoperative environment is feasible to be incorporated into a surgical workflow with minimal encumbrance, and our experimental analysis shows that the method significantly improves upon the robust rigid registration currently used in commercial systems, as well as previously investigated nonrigid methods. In addition, the method is fully realized for a sparse data acquisition environment thus potentially allowing for wide scale adoption by image-guided surgical platforms for soft-tissue organ guidance. Future work will include further testing on clinical datasets with validation strategies using tracked co-registered ultrasound.