Estimating nonrigid motion from inconsistent intensity with robust shape features

Abstract

Purpose: To develop a nonrigid motion estimation method that is robust to heterogeneous intensity inconsistencies amongst the image pairs or image sequence.

Methods: Intensity and contrast variations, as in dynamic contrast enhanced magnetic resonance imaging, present a considerable challenge to registration methods based on general discrepancy metrics. In this study, the authors propose and validate a novel method that is robust to such variations by utilizing shape features. The geometry of interest (GOI) is represented with a flexible zero level set, segmented via well-behaved regularized optimization. The optimization energy drives the zero level set to high image gradient regions, and regularizes it with area and curvature priors. The resulting shape exhibits high consistency even in the presence of intensity or contrast variations. Subsequently, a multiscale nonrigid registration is performed to seek a regular deformation field that minimizes shape discrepancy in the vicinity of GOIs.

Results: To establish the working principle, realistic 2D and 3D images were subject to simulated nonrigid motion and synthetic intensity variations, so as to enable quantitative evaluation of registration performance. The proposed method was benchmarked against three alternative registration approaches, specifically, optical flow, B-spline based mutual information, and multimodality demons. When intensity consistency was satisfied, all methods had comparable registration accuracy for the GOIs. When intensities among registration pairs were inconsistent, however, the proposed method yielded pronounced improvement in registration accuracy, with an approximate fivefold reduction in mean absolute error (MAE = 2.25 mm, SD = 0.98 mm), compared to optical flow (MAE = 9.23 mm, SD = 5.36 mm), B-spline based mutual information (MAE = 9.57 mm, SD = 8.74 mm) and mutimodality demons (MAE = 10.07 mm, SD = 4.03 mm). Applying the proposed method on a real MR image sequence also provided qualitatively appealing results, demonstrating good feasibility and applicability of the proposed method.

Conclusions: The authors have developed a novel method to estimate the nonrigid motion of GOIs in the presence of spatial intensity and contrast variations, taking advantage of robust shape features. Quantitative analysis and qualitative evaluation demonstrated good promise of the proposed method. Further clinical assessment and validation is being performed.

INTRODUCTION

Image-based motion estimation plays an important role in various biomedical context, such as dynamic imaging and MR based radiation therapy. The problem of image registration, which estimates the deformation based on image information, either between pairs of frames or a sequence, has been intensively studied.^{1, 2, 3, 4, 5} To register images, a certain type of consistency condition needs to be satisfied, typically imposed on the intensity values themselves or the intensity transformations among images, leading to the development of a large body of works on minimal squared error and/or maximum mutual information.^{2, 6, 7, 8, 9} These approaches, unfortunately, are not directly applicable when the heterogeneous contrast variations are present between image pairs or among temporal sequences, as in the imaging processes involving contrast enhancement. In this work, we focus on dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI), where a series of MR images are acquired before and after an intravenous injection of gadolinium.¹⁰ To assess renal function using DCE-MRI, pharmacokinetic models are fitted to the voxelwise enhancement curves to map the renal blood flow and obtain gleomerular filtration rate.¹¹ Kidney motion caused by breathing¹² induces artifacts and inaccuracy in the voxel-based analysis^{13, 14, 15, 16} and need to be corrected via image registration. On the other hand, the kinetics of the contrast agent leads to temporally inconsistent intensity, which makes estimating and subsequently correcting for the physiological and physical motion challenging.

A relatively straightforward way to perform DCE-MR registration is to consider the contrast-variant images as effectively different modalities and apply mutual information as the matching metric.¹⁷ However, the spatially variant intensity changes in DCE-MR does not guarantee such mapping consistency, as maximized in mutual information based methods, and may lead to misalignment.^{7, 18} More recently, efforts have been made to modify generic registration algorithms to address the difficulties caused by intensity and contrast inconsistencies. Merrem et al.¹⁹ modifies a nonrigid registration method that uses the squared cross-correlation as matching metric, by introducing an isotropic Gaussian kernel to smooth the estimated deformation field at each step, to reduce noise sensitivity and alleviate the effect of spatially variant enhancement pattern. This approach is sensitive to the kernel size, and risks misregistration if not selected appropriately. A modified demons algorithm has also been proposed for DCE-MR registration,²⁰ yet it only stabilizes the original demon for low gradient regions by adding a small modifier to the denominator but does not improve sensitivity to contrast change. Haber and Modersitzki²¹ have proposed a normalized gradient method by using the direction of the gradient as registration features. While the gradient direction is robust to contrast changes, it is sensitive to spatially varying noise which is prevalent in practice. It also faces stability issue when certain edges appear in only one of the images under registration. In addition, the efficacy of the above methods were largely assessed visually with only qualitative conclusions.

In this study, we propose and evaluate a novel method to estimate the motion of roughly persistent GOIs in the presence of intensity variations. We propose to use full shape contour as registration features and benefit from their robustness to noise and intensity changes. To quantitatively assess the registration performance, 2D and 3D nonrigid motion were simulated and used as ground-truth for evaluation purposes. We also assessed the performance of the proposed method on real MR series, yielding promising results.

METHOD

Traditional motion estimation/registration methods are often based on the assumption that voxel intensity is temporally invariant or statistically dependent (as in mutual information based method). Those assumptions generally do not hold in the presence of local contrast and intensity changes as in DCE-MRI. Therefore, features that are robust with respect to intensity and contrast variations are preferable. In this study, we propose to utilize full geometric contours as registration features. Depending on gradient in a more regularized fashion, the level set representation of the geometric contour is more robust to local perturbations caused by noise and spatially varying contrast changes.

The proposed method consists of two main steps: (1) extracting GOIs with a level set based segmentation, (2) estimating nonrigid deformation on the full domain by minimizing discrepancy of the corresponding level set functions (LSFs) in the neighborhood of the GOIs subject to regularity conditions.

Level set based segmentation

The level set method Osher and Sethian et al.^{22, 26} is widely used for representing evolving curves and surfaces, especially for segmentation tasks. It has the advantage of being flexible in shape representation and numerically efficient with fixed-grid calculation. In this representation, a curve C is represented implicitly as the zero level set of its corresponding LSF ϕ in a space that is one dimension higher. The LSF is often enforced to have unity gradient magnitude (|∇ϕ| = 1) to represent distance to the curve of interest for numerical stability. We use the level-set based method to segment the GOIs and extract shape contours represented by the LSFs.²³ At each image frame, we perform segmentation by minimizing a variational energy that encodes (i) alignment to high image gradient regions, (ii) curve smoothness, (iii) area/volume preservation between adjacent frames, and (iv) signed distance property of the LSF.

Specifically, for a LSF $\varphi :\Omega \subset {\mathbb{R}}^d\to \mathbb{R}$ (d = 2 or 3), the energy functional is defined as

$$\epsilon \left(\varphi \right)=\lambda {\mathcal{L}}_g\left(\varphi \right)+\frac{\beta }{2}\mathcal{A}\left(\varphi \right)+\mu {\mathcal{R}}_p\left(\varphi \right),$$ (1)

where the weights λ, β, and μ balance the various contributing terms.

Conformation of the boundary to high image gradient area and curve smoothness is encouraged by ${\mathcal{L}}_g\left(\varphi \right)$,²³ defined as

$${\mathcal{L}}_g\left(\varphi \right)={\int }_{\Omega }g\delta \left(\varphi \right)\left|\nabla \varphi \right|d\mathbf{x}.$$ (2)

Here δ is the Dirac Delta function, and g is an edge indicator function that takes small values at locations with large gradient:

$$g=\frac{1}{1+|\nabla G_{\sigma }{*I|}^2},$$ (3)

where I is the input image and G_σ is a Gaussian kernel with standard deviation σ. The kernel parameter σ balances robustness to noise and preservation of image gradient information.

Unlike conventional level set based segmentation methods that often push the contour to either shrink or expand, we design an area regularization component $\mathcal{A}\left(\varphi \right)$ to enforce consistency of area/volume across frames

$$\mathcal{A}\left(\varphi \right)={\left(\left({\int }_{\Omega }H(-\varphi )d\mathbf{x}\right)-\left({\int }_{\Omega }H(-{\varphi }_0)d\mathbf{x}\right)\right)}^2,$$ (4)

where H is the Heavyside function and ϕ₀ is the LSF of the previous image frame. This is motivated by the observation that areas and volumes are expected to vary little between adjacent frames.

Distance regularization is achieved with ${\mathcal{R}}_p\left(\varphi \right)$,²⁴ which enforces the signed distance property of the LSF:

$${\mathcal{R}}_p\left(\varphi \right)={\int }_{\Omega }p\left(\right|\nabla \varphi \left|\right)d\mathbf{x},$$ (5)

where p is a double-well function defined as

$$p\left(s\right)=\left\{\begin{array}{cc}\hfill \frac{1}{{\left(2\pi \right)}^2}(1-\cos \left(2\pi s\right)),& \hfill \text{if}s\le 1\\ \hfill \frac{1}{2}{(s-1)}^2,& \hfill \text{if}s\ge 1\end{array}\right..$$ (6)

The double-well function has two minimums at 0 and 1, respectively, which preserves the signed distance property (|∇ϕ| = 1) near the zero level set and smoothes (|∇ϕ| = 0) the locations that far away from the zero level set. It has been shown that Eq. 5 preserves the signed distance property more efficiently and continuously than the traditional reinitialization methods.^{22, 25, 26}

Numerically, smoothed approximations of H(ϕ) and δ(ϕ) are used²²

$$H_{\epsilon }\left(\varphi \right)=\left\{\begin{array}{cc}\hfill 0,& \hfill \varphi <-\epsilon \\ \hfill \frac{1}{2}+\frac{\varphi }{2\epsilon }+\frac{1}{2\pi }\sin \left(\frac{\pi \varphi }{\epsilon }\right),& \hfill -\epsilon \le \varphi \le \epsilon \\ \hfill 1,& \hfill \varphi >\epsilon \end{array}\right.,$$ (7)

$${\delta }_{\epsilon }\left(\varphi \right)=\left\{\begin{array}{cc}\hfill 0,& \hfill \varphi <-\epsilon \\ \hfill \frac{1}{2\epsilon }+\frac{1}{2\epsilon }\cos \left(\frac{\pi \varphi }{\epsilon }\right),& \hfill -\epsilon \le \varphi \le \epsilon \\ \hfill 1,& \hfill \varphi >\epsilon \end{array}\right..$$ (8)

By taking Gâteaux derivative, energy functional [Eq. 1] can be minimized by the gradient flow in Eq. 9.

$$\begin{array}{ccc}\hfill {\varphi }_t& =& \lambda {\delta }_{\varepsilon }\left(\varphi \right)\nabla ·\left(g\frac{\nabla \varphi }{\left|\nabla \varphi \right|}\right)+\beta {\delta }_{\varepsilon }\left(\varphi \right)\left({\int }_{\Omega }{H}_{\varepsilon }(-\varphi )d\mathbf{x}\right.\hfill \\ & & \left.-{\int }_{\Omega }{H}_{\varepsilon }(-{\varphi }_0)d\mathbf{x}\right)+\mu \nabla ·(d_p\left(\right|\nabla \varphi \left|\right)\nabla \varphi ).\hfill \end{array}$$ (9)

Motion estimation

After segmenting the GOIs from each frame, one has access to ϕ_i—the LSF for frame i, which we use as shape features. To estimate the underlying motion field, we aim for similarity upon motion compensation on the LSFs between adjacent frames. We proceed to perform nonrigid matching to minimize the following functional:

$$\begin{array}{c}\hfill E\left(\omega \right)={\int }_{{\Omega }_{nb}}{({\varphi }_1\left(\mathbf{x}\right)-{\varphi }_2(\mathbf{x}+\omega ))}^{2}d\mathbf{x}+\alpha {\int }_{\Omega }{\Vert \nabla \omega \Vert }_F^{2}d\mathbf{x},\end{array}$$ (10)

where ω is the motion field, ϕ₁ and ϕ₂ are the LSFs of the temporally adjacent frames, and α is the regularization weight. The data fidelity term is defined over a narrow band Ω_nb = {x: |ϕ₁(x)| < η}∪{x: |ϕ₂(x)| < η}, where 2η is the width of the detected organ boundary centered at the zero level set. The smoothness regularization effectively extends the boundary's motion to the rest of the domain. Minimization of the above functional yields the following Euler-Lagrange equations under Neumann boundary conditions:

$$\begin{array}{ccc}& & ({\varphi }_1\left(\mathbf{x}\right)-{\varphi }_2(\mathbf{x}+\omega ){\nabla }_i{\varphi }_2(\mathbf{x}+\omega )+\alpha \nabla ·\left(\nabla {\omega }_i\right)=0,\hfill \\ & & \\ & & \text{for}i=1,2,...,d.\hfill \end{array}$$ (11)

The notation ∇_iϕ₂ represents the partial derivatives of ϕ₂ along the ith direction, and ω_i is the ith vector component of ω. To solve this nonlinear system of equations, we use first-order Taylor expansion to approximate ${\varphi }_2(\mathbf{x}+\omega )$ at each iteration:

$$\begin{array}{ccc}\hfill {\varphi }_2(\mathbf{x}+{\omega }^{n+1})& \approx & {\varphi }_2(\mathbf{x}+{\omega }^n)+\nabla {\varphi }_2(\mathbf{x}+{\omega }^n)\hfill \\ & & ·({\omega }^{n+1}-{\omega }^n)\hfill \end{array}$$ (12)

with ${\omega }^0=0$ as the initialization. At the (n + 1)th iteration, plugging Eq. 12 into Eq. 11 reduces it to the following linear system of equations for ${\omega }^{n+1}$.

$$\begin{array}{ccc}& & \left({\varphi }_1-{\varphi }_2^n-\nabla {\varphi }_2^n·({\omega }^{n+1}-{\omega }^n)\right){\nabla }_i{\varphi }_2^n+\alpha \nabla ·\left(\nabla {\omega }_i^{n+1}\right)=0,\hfill \\ & & \text{for}i=1,2,...,d,\hfill \end{array}$$ (13)

where ${\varphi }_2^n={\varphi }_2(\mathbf{x}+{\omega }^{\mathbf{n}})$,${\nabla }_i{\varphi }_2^n={\nabla }_i{\varphi }_2(\mathbf{x}+{\omega }^{\mathbf{n}})$. This linear system of equations can be solved efficiently with various numerical methods, such as Gauss-Seidel or successive over-relaxation (SOR)²⁷ iterations.

Inspired by Horn and Shunck²⁸ we introduce an inner iteration scheme to solve Eq. 13. At the (m + 1)th inner iteration, the divergence $\nabla ·\left(\nabla {\omega }_i^{n,m+1}\right)$ is approximated with $(\overline{{\omega }_i^{n,m+1}}-{\omega }_i^{n,m+1})$ , where the $\overline{{\omega }_i^{n,m+1}}$ is the local weighted average of (k^d − 1) neighbors for a support of k in d dimensions. This approximation leads to the fixed-point iteration scheme in Eq. 14, where n indexes the outer linear approximation iteration, and m indexes the inner fixed point iteration.

$${\omega }_i^{n+1,m+1}=\frac{{\nabla }_i{\varphi }_2^n\left[{\varphi }_1^n-{\varphi }_2^n+{\nabla }_i{\varphi }_2^n{\omega }_i^n-{\sum }_{j=1,j\ne i}^d{\varphi }_{2,j}\left({\omega }_j^{n,m}-{\omega }_j^n\right)\right]+\alpha \overline{{\omega }_i^{n,m}}}{{\left({\nabla }_i{\varphi }_2^n\right)}^2+\alpha },\text{for}i=1,2...,d$$ (14)

We use multiscale scheme to further improve efficacy and accuracy of the proposed segmentation and motion estimation algorithm. Specifically, we created a coarse to fine image structure. By using this pyramidal structure, we are able to perform the segmentation and motion estimation at coarser image levels and to refine the results at finer levels. Algorithm 1 provides an overview of the proposed algorithm with the multiscale implementation.

Multiscale scheme for proposed algorithm

1: Input: I1(x), I2(x), α, Nscales

2: fors ← Nscales − 1 to 0 do

3:  Downscale images I1(x), I2(x) by η = 2s

4:  Segmentation based on Eq. 1, obtain ${\varphi }_1^s\left(\mathbf{x}\right),{\varphi }_2^s\left(\mathbf{x}\right)$

5:  Motion estimation based on Eq. 10, obtain ${\omega }^s\left(\mathbf{x}\right)$

6:  ifs > 0 then

7:   ${\omega }^{s-1}\left({\mathbf{x}}^{\prime }\right)=\sum \eta {\omega }^s\left(\mathbf{x}\right)B({\mathbf{x}}^{\prime }/\eta -\mathbf{x})$

8:   ${\varphi }_i^{s-1}\left({\mathbf{x}}^{\prime }\right)=\sum \eta {\varphi }_i^s\left(\mathbf{x}\right)B({\mathbf{x}}^{\prime }/\eta -\mathbf{x})$,

9:    where B(t) is the bicubic interpolation kernel.

10:  end if

11: end for

EXPERIMENTS AND RESULTS

To quantitatively assess and validate the performance of the proposed method, we generated artificial but physically reasonable nonrigid motion and applied such to a reference image to simulate the physiological deformation. The baseline performance is first evaluated when intensity is consistent. Inconsistency is then imposed to test the efficacy and robustness of the proposed method. The proposed method was benchmarked against three alternative image registration methods: optical flow,²⁸ B-spline based mutual information^{2, 30} D. Rueckert et al.,³¹ and multimodality demons methods.²⁹ After that, qualitative test was performed using real MR image sequence.

Brief overview of benchmark methods

Optical flow

The optical flow method, first introduced by Horn and Shunck,²⁸ aims to estimate a smooth motion field ω that matches intensity values between reference I₁ and target images I₂, by minimizing the energy $\int {(I_1\left(\mathbf{x}\right)-I_2(\mathbf{x}+\omega ))}^2+{\alpha }_{OF}{\Vert \nabla \omega \Vert }^{2}d\mathbf{x}$, where α_OF is the smoothness coefficient. In this study, we implemented the optical flow method in MATLAB and set α_OF = 0.5.

B-spline based mutual information

The mutual information based method^{2, 30} maximizes the statistical dependency between two images. It is a widely used approach for multimodality registrations. Our benchmark implementation was based on a multiscale B-spline parameterized formulation for nonrigid motion estimation³¹ from Medical Image Registration Toolbox (MIRT). We used 64 histogram bins and 10 × 10 initial control point meshes, which is subsequently refined. In minimizing $-\text{MI}(I_1,I_2\circ T)+{\alpha }_{\text{MI}}\frac{1}{V}\int {\Vert \nabla T\Vert }^{2}d\mathbf{x}$, the regularization weight α_MI was set to 1.

Multimodality demons

The multimodality demons²⁹ is a combination of demons and mutual information methods. It transforms one image representation to another by maximizing mutual information, and then perform demonslike registration while regularizing the smoothness of the motion field. We used the multimodality implementation of Ref. 29 with the default parameters.

For all the implementations, three hierarchal image levels were used. We used a brute force parameter searching scheme to find the optimal parameters near the suggested values.^{28, 29, 31}

Parameter setup and computation complexity

In the segmentation step, one needs to set three parameters λ, β, μ to balance the contribution from edge driving force, area and distance property conservation, respectively, and the time step Δt for curve evolution. Our experiments indicate that the proposed method is insensitive to the choices of λ in the range of (3,30) and μ when μΔt < 0.25 is satisfied, and we used fixed values in all our validation tests. The area weight β, on the other hand, depends on motion amplitude: when motion is small between adjacent frames, β should be relaxed to be zero; for large temporal motions, larger value needs to be used. For all experiments in this study, we fixed the parameters to be λ = 10, β = 0.01, μ = 0.2, Δt = 1.

In the motion estimation step, one needs to set the parameter α to weigh the Tikhonov regularization term, and the width η for the narrowband region. Based on our tests, the model is insensitive to the choice of α ranging from 2 to 5 and η ranging from 11 to 21 voxels. In our implementation, we fixed α to be 2.5 and the width of narrow band η to be 15 voxels for all experiments.

The average running time for segmenting 2D image with size 256 × 256 was about 5 s, and 60 s for segmenting 3D image volume with size 320 × 320 × 32 when the previous frame/volume segmentation was used as initialization. The computation time for estimating motion between 2D images of size 256 × 256 was about 30 s, and 120 s for comparing 3D volumes with size 320 × 320 × 32. The algorithm was implemented with MATLAB 2012b on Mac OS 10.84, running on a macbook pro laptop with quad-core i7 2.3 GHz, 8 GB RAM. No specific code optimization schemes were used.

Experiments on simulated nonrigid 2D motion

We start with a 2D EPI MR image of the kidney with TR = 137 ms, TE = 44 ms, FOV = 390 × 390  mm² and voxel size of 1.52 × 1.52  mm². The kidneys under quiet respiration have predominant motion in the superior-inferior (SI) direction with a mean magnitude of 19 mm.³² To mimic this behavior, we simulated a motion field parameterized by B-splines,³³ with an average motion magnitude of 15 mm in SI direction and 8 mm in left-right (LR) direction around the kidneys in the abdominal MR images, as depicted in Fig. 1. Figure 2 shows the segmentation results for reference and target images, and the corresponding LSF representations. Figure 3 illustrates the registration performance. Figures 3a, 3b compares the preregistration and postregistration agreement of the reference and target images, using a checkerboard presentation which alternatingly displays the intensity squares from either image. Figures 3c, 3d report the absolute intensity difference map to quantify the registration accuracy. The checkerboard in Fig. 3b shows significantly improved continuity on kidney boundaries compared to Fig. 3a, while Fig. 3d demonstrates a large reduction of intensity discrepancies compared to Fig. 3c, demonstrating the efficacy of the proposed registration method. Table 1 reports the error statistics in terms of mean absolute error (MAE = $\frac{1}{\left|\Omega \right|}{\sum }_i\left|e_i\right|$), root means squared error (RMSE = $\sqrt{\frac{1}{\left|\Omega \right|}{\sum }_i{e}_i^2}$), standard deviation (SD), mean, median, and max error in the neighborhood of kidneys, showing voxel-size comparable registration accuracy. The directional error is evaluated by taking the difference between the simulated ground-truth and the estimated deformation field $e_i\left(\mathbf{x}\right)={\omega }_i\left(\mathbf{x}\right)-{\widehat{\omega }}_i\left(\mathbf{x}\right)$ for voxels x in the proper neighborhood of the GOI. The aggregated nD(n = 2or 3) error is measured by $e\left(\mathbf{x}\right)=\sqrt{{\sum }_{i=1}^n{({\omega }_i\left(\mathbf{x}\right)-{\widehat{\omega }}_i\left(\mathbf{x}\right))}^2}$.

Figure 1.

Simulated nonrigid motion field with B-spline parameterization.

Figure 2.

2D level set based segmentation results and extracted GOI features from target and reference images: (a) segmented contour with zero level set in agreement with reference kidney boundaries, (b) segmented contour with zero level set agrees with target kidney boundaries, (c) GOIs of the reference image represented by a narrowband LSF, (d) GOIs of the target image represented by a narrowband LSF. The segmented contours of target and reference images are overlaid in (a) (b).

Figure 3.

2D motion estimation results in checkerboard presentation and absolute intensity difference maps: (a) checkerboard before registration, (b) checkerboard after registration, showing pronounced improvement of boundary continuity, (c) intensity difference map before registration, (d) intensity difference map after registration, showing significantly improved agreement between the target kidney boundary and reference kidney as well as reduced intensity difference.

Table 1.

Error statistics of 2D experiments on simulated motion showing estimation errors comparable to voxel size (1.52 mm × 1.52 mm).

	MAE	RMSE	SD	Mean	Median	Max
LR	1.63	1.92	1.92	0.04	−0.08	4.03
SI	1.26	1.53	1.42	0.57	0.81	3.45
2D	2.25	2.45	0.99	2.25	2.24	5.07

Experiments on 3D dataset

We conducted similar experiment on 3D MR datasets. Figure 4 illustrates the sample slices of T2 weighted MR images, acquired with TR = 700  ms, TE = 71  ms, FOV = 400 × 400  mm², array size of 320 × 320, slice-thickness of 6 mm, and 32 slices in total in the sequence. Figure 5a presents the segmentation results of left kidney and the simulated nonrigid motion. Figures 5b, 5c compares the kidney positions before and after registration, where improved alignment from registration can be appreciated. Error statistics are summarized in Table 2, showing subvoxel accuracy.

Figure 4.

Sample slices of reference 3D MR data: T2 weighted MR images with TR = 700ms, TE = 71  ms, FOV = 400 × 400  mm², array size of 320 × 320 and slice-thickness of 6  mm.

Figure 5.

3D segmentation and motion estimation results: (a) left kidney segmentation result overlaid with simulated nonrigid motion field, (b) overlays for reference kidney and target kidney before registration, (c) overlays of reference kidney and target kidney after registration, showing improved alignment.

Table 2.

Error statistics of 3D experiments of simulated nonrigid motion, showing sub-voxel accuracy (voxel size = 0.8  mm × 0.8  mm × 6  mm).

	MAE	RMSE	SD	Mean	Median	Max
LR	0.38	0.50	0.38	0.33	0.25	1.08
SI	0.41	0.55	0.53	0.14	0.07	2.03
AP	0.75	0.99	0.96	−0.26	−0.17	2.80
3D	1.09	1.24	0.58	1.09	1.04	2.88

Robustness test on images with synthetic intensity and contrast variations

We tested the robustness of the proposed method with the presence of synthetic intensity and contrast changes. Figure 6 shows the simulation flow for contrast and intensity variations. The same deformation field as in Sec. 3C was used. The proposed method was applied to the reference and target images in Fig. 6 and compared against alternative registration methods using optical flow, B-spline based mutual information, and multimodality demons.^{28, 31, 29} The quantitative error comparisons are reported in Fig. 7. The proposed method may perform slightly inferiorly when consistency is present, possibly due to limited information captured by the contour. However, when intensity is inconsistent, it enjoyed the pronounced benefit of being robust. The checkerboard representations of difference methods are presented in Fig. 8, showing significant improvements of boundary continuities from the proposed method compared to the others.

Figure 6.

Simulation workflow for adding contrast and intensity variations and nonrigid deformation to the reference image: Step 1. Changing image contrast by applying the gamma function to the reference image: I₁(x) = I₀(x)^γ with γ = 0.6; Step 2. Introducing spatially varying intensity inhomogeneity to I₁(x) by multiplying it with spatially varying function M(x): $I_2\left(\mathbf{x}\right)=I_1\left(\mathbf{x}\right)*M\left(\mathbf{x}\right)$; Step 3. Introducing nonrigid motion effect to I₂(x) by deforming it with the simulated motion field T(x): I₃(x) = I₂(x)○T(x).

Figure 7.

Performance comparison with benchmark alternatives in terms of mean absolute error (MAE) and standard deviation (SD): when there is no in intensity variations, the proposed method shows slightly inferior result; in the presence of intensity changes, it shows much improved robustness.

Figure 8.

Comparison of registration performance in the presence of intensity and contrast variation between benchmark and proposed method: (a) optical flow, (b) mutual information, (c) multimodality demons, (d) proposed. Arrows in (a)–(c) indicate observable misregistration locations and arrows in (d) indicate the corresponding locations for the proposed method with improved registration performance.

Experiments on real MR image sequences

Finally, we tested the proposed method on real time EPI coronal MR image sequence with TR = 137 ms, TE = 44 ms, and FOV = 390 × 390 mm². Figure 9a presents the estimated motion field using the 1st and 5th image frame in the series. Figures 9b, 9c provide checkerboard illustrations showing registration significantly improves continuity of contours across the alternating squares, demonstrating the validity of proposed registration. Figures 10a, 10b presents the mean boundary motion from applying the proposed method to the first 80 frames (with a duration of approximately 10.2 s) of the MR sequence. It exhibits an approximate two-cycle behavior, which agrees with visual inspection of the image sequence and the general knowledge about typical respiratory with 4–6 s duration per cycle. Figure 10c presents the estimated temporal motion trajectory of the renal boundary from the first 10 frames, showing the algorithm's ability to capture local deformations.

Figure 9.

Motion estimation results using real MR image sequences: (a) segmented contours (reference in magenta and target in cyan) and estimated motion field, showing downward and expansive motion of the kidneys, consistent with visual examination of the sequence; (b) checkerboard visualization preregistration; (c) checkerboard postregistration, where arrows are used to indicate locations with significant improvement, compared to the same locations indicated in (b).

Figure 10.

Temporal evolution of the estimated kidney motion. (a) mean displacement in the superior-inferior direction, (b) mean displacement in the left-right direction, (c) temporal anatomical boundary trajectories reconstructed from estimated motion for the first ten images frames, with corresponding points connected and coded in the same color.

Test of sensitivity and robustness towards optimization parameter selection noise levels

Finally, we tested the robustness of the proposed registration method under various optimization parameter settings and noise conditions. Figure 11 reports the results of performing the registration algorithm with different optimization parameters to balance the terms in Eq. 1. Within a reasonably large range, the segmentation results vary little.

Figure 11.

Test of sensitivity to parameter value selections by varying length λ and area β weighting coefficients: the segmentation results vary little for λ ∈ (3, 20) and β ∈ (0, 0.1), indicating the proposed method's insensitivity of parameter selections. (a) λ = 3, β = 0.01, (b) λ = 10, β = 0.01, (c) λ = 20, β = 0.01, (d) λ = 10, β = 0, (e) λ = 10, β = 0.005, (f) λ = 10, β = 0.1.

To test the sensitivity to image quality, we have simulated Gaussian noise with standard deviation one tenth of the image intensity range, reducing the power SNR to −1.63dB. A spatially heterogeneous intensity modulation pattern was further imposed to simulate the contrast variation. Figure 12 shows that segmentation results only degrade slightly with the addition of this severe noise.

Figure 12.

Test of robustness to intensity variations and noise: the segmented contour only degraded slightly as both intensity variations and significant noise were introduced, when the same fixed optimization parameter values were used. (a) Noise free, (b) with additive gaussian noise, (c) with intensity variations, and (d) with both gaussian noise and intensity variations.

DISCUSSIONS

In this paper, we have proposed and validated a novel motion estimation method based on geometric contour features that are robust to intensity and contrast variations. When image intensity and contrast are consistent, the proposed method performed slightly inferior (MAE = 2.25 mm) to the benchmark methods using optical flow (MAE = 1.23 mm), maximal mutual information (MAE = 2.88 mm), and multimodality demons (MAE = 1.13 mm), but still comparable to voxel size. This is likely caused by the fact that the proposed method only relies on limited amount of information around the extracted contours. When synthetic contrast and intensity variations were introduced, the proposed method showed robust estimation results (MAE = 2.25 mm) and outperformed benchmark registration methods using optical flow (MAE = 9.23 mm), maximal mutual information (MAE = 9.57 mm), and multimodality demons (MAE = 10.07 mm). When applied to clinical image sequence where ground-truth motion is inaccessible, the proposed method generated visually appealing results that are consistent with manual inspection of the sequence. The proposed method is also robust with respect to optimization parameter selection and the specific SNR conditions.

The use of contour shape features endows the proposed method its robustness to contrast variation and noise—it alleviates the gradient magnitude's sensitivity to contrast variations and gradient direction's sensitivity to noise. The combined regularization energy controlling the curve length and enclosing area provides improved robustness for the extracted contour features, compared to isolated high gradient points. The use of this new feature also has its associated limitations. In particular, the registration relies on data fidelity defined on a narrow band around the segmented contours and the motion for voxels away from this band is effectively obtained by “propagating” via the regularization term. Therefore, the method will fail to capture any subtle local motion pattern far from the GOIs. This is not a severe drawback since the major motion estimation interest normally lies within or around the GOI. However, when globally good motion estimation is desired, one can extend the current method to use multiple GOIs, and use multiphase level set presentation,³⁴ for example, to segment and depict a set of contours simultaneously that provide sufficient coverage for the domain of interest.

With the registration relying on distance function representation of the segmented contours, a potential concern is the dependence of segmentation accuracy. There are several aspects to be considered. First, we have utilized good initialization of the LSF with the segmentation results from temporally adjacent frames to encourage more robust and consistent results. Second, we have chosen the level set based segmentation for its flexibility and its natural compatibility into variational optimization framework. In principle, one has the option of using alternative methods to generate contours, e.g., semi-interactive contouring schemes for possibly higher segmentation accuracy, and then deriving distance functions as input to the registration module. Finally, the major goal of the proposed method is motion estimation, not segmentation per se. Therefore, as long as the segmented contours are close enough to the GOIs and the segmentation results are consistent among the frames, the motion estimation results are not affected. In some sense, the segmented contours should be considered as “surrogate” descriptors for the organ or structure boundaries that sharing the same motion.

Note that sometimes GOI may become partially inconspicuous in one frame out of the registration pairs, when contrast kinetic makes it indistinguishable from its surrounding at that particular frame. When such inconspicuousness is severe enough to distort the segmented contour and make it inconsistent with the other contours in the sequence, the proposed method would fail. We are in the process of incorporating prior knowledge on both shape characterization^{35, 36, 37} and deformation fields to improve stability and robustness in these challenging situations.

The proposed method may also be challenged when the underlying presumptions in designing the regularizations are violated. In particular, we used a Tikhonov type regularization for motion field smoothness. When the motion under study is nonsmooth, such as sliding, one may need to either relax such assumption by decreasing the smoothness weight α or adjust the design of the regularization to preserve discontinuity or sliding.^{38, 39, 40}

In this study, we simulated the pairwise intensity and contrast changes to verify the robustness of the proposed method to such variations. We are in the process of incorporating more precise and realistic pharmacokinetic functions⁴¹ for sequence simulation with quantitative clinical verifications.

With the advantage of robustness to contrast and intensity variations, the proposed method is applicable to other problems involving contrast change, such as multimodality registration, dynamic contrast enhanced CT, and subtraction imaging as in angiography.

CONCLUSIONS

In summary, we have developed a novel method to estimate nonrigid motion for geometries of interest. The proposed method is based on robust contour shape features that preserves good consistency in the presence of intensity and contrast variations. Quantitative analyses were made on 2D and 3D datasets showing promising results. Future work will consider incorporating shape and motion priors for better segmentation and registration. Parallel computation is being implemented to speed up the execution. A clinical protocol has been established to collect free-breathing DCE-MRI image sequences of the abdominal region to further validate the proposed method, and collaboration has been established with clinicians to evaluate its performance under different image quality condition and motion patterns.