NONRIGID VOLUME REGISTRATION USING SECOND-ORDER MRF MODEL

Abstract

In this paper, we introduce a nonrigid registration method using a Markov Random Field (MRF) energy model with second-order smoothness priors. The registration determines an optimal labeling of the MRF energy model where the label corresponds to a 3D displacement vector. In the MRF energy model, spatial relationships are constructed between nodes using second-order smoothness priors. This model improves limitations of first-order spatial priors which cannot fully incorporate the natural smoothness of deformations. Specifically, the second-order smoothness priors can generate desired smoother displacement vector fields and do not suffer from fronto-parallel effects commonly occurred in first-order priors. The usage of second-order priors in the energy model enables this method to produce more accurate registration results. In the experiments, we will show comparative results using uni- and multi-modal Brain MRI volumes.

1. INTRODUCTION

Recently, nonrigid registration methods based on a Markov Random Field (MRF) energy model incorporating the discrete energy optimization [1, 2] showed the promising experimental results [3, 4, 5]. These methods model the deformation pattern as a discrete label set where labels correspond to displacements of control points (nodes) placed on a cubic mesh grid. The energy is constructed via the standard pairwise MRF model

$$E(x\mid \theta )\triangleq \sum _{s\in \mathcal{V}}{\theta }_s(x_s)+\sum _{(s,t)\in \mathcal{E}}{\theta }_{st}(x_s,x_t)$$ (1)

where 𝒱 is a set of nodes placed on a cubic grid, ℰ is a set of edges incorporating neighborhood information of nodes, and x_s is the label of s ∈ 𝒱. In this model, the data cost θ_s is computed using similarity measures between reference and input images and the smoothness cost θ_st is computed using displacement differences between s and t. For θ_st, the following first-order smoothness prior (truncated linear potential) is conventionally used

$${\theta }_{st}(x_s,x_t)\triangleq {\lambda }_{st}min({\Vert \mathit{d}(x_s)-\mathit{d}(x_t)\Vert }_1,T_{st})$$ (2)

where λ_st is the regularization constant, d(x_s) represents the 3D displacement vector corresponding to the label x_s, and T_st is a threshold for truncation. However, the energy models based on (2) have limitations. The pairwise potentials penalize the global transformations of the mesh, such as rotation and scaling movements. Mesh nodes are needed to be remained at initial position or translated all together to get low energy. These problems correspond to fronto-parallel effects in stereo matching. In nonrigid registrations, irregular spacing between neighboring nodes in recovered vector fields is commonly appeared by these limitations.

To overcome these limitations, Kwon et al. [6] proposed a registration method using higher-order MRF energy model. This method incorporates second-order spatial interactions to represent natural smoothness of displacement fields. The energy model is formulated using factor graph notations and factor to pairwise graph conversion is used for applying belief propagation (BP) variant optimization methods. In experiments, they showed promising results using 2D images.

Here, we introduce a nonrigid volume registration method using a MRF based energy model. The proposed method extends the second-order MRF model proposed in [6] to work on 3D volumes. As displacements are 3D vectors, graph models in [6] are modified to involve ternary potentials for interlayer interactions in the decomposed scheme. Also, we apply a coarse-to-fine registration scheme for efficient computations and sub-pixel accuracy. In next sections, the proposed MRF energy model for registration is described, and optimization strategies are introduced. In experiments, the proposed method is compared with a first-order MRF registration method [3] using uni- and multi-modal brain MRI data.

2. MRF ENERGY MODEL FOR REGISTRATION

For an input volume, we construct a set 𝒱 which consists of nodes placed on each voxel. Then we generate a factor graph [7] 𝒢_ℱ = (𝒱, ℱ_ℋ) where ℱ_H is a set of factors for second-order potentials. For each s ∈ 𝒱, let x_s be a label taking values in some discrete set ℒ. A function d: ℒ → ℝ³ maps a label to a corresponding 3D displacement where each label x_s corresponds to a displacement vector d(x_s) = (d_x(x_s), d_y(x_s), d_z(x_s)). In 𝒢_ℱ, an unary potential θ_s(x_s) is defined for each node s ∈ 𝒱 and a second-order potential θ_stu(x_s, x_t, x_u) is defined for each factor (s, t, u) ∈ ℱ_ℋ.¹

2.1. Second-Order Smoothness Prior

In [8], deformation energy of the mesh is usually defined as a sum of squared second derivatives of its nodes. This deformation energy describes the natural representation of inherent deformedness of the mesh which depends only on the relative locations of mesh nodes. Following the analysis on [6], we incorporate second-order smoothness priors into the energy model. To apply second-order smoothness priors, the ternary potential θ_stu(x_s, x_t, x_u) is constructed on every collinear three nodes s, t and u as follows

$${\theta }_{\mathit{stu}}(x_s,x_t,x_u)\triangleq {\lambda }_{\mathit{stu}}min({\Vert \mathit{d}(x_s)-2\mathit{d}(x_t)+\mathit{d}(x_u)\Vert }_1,T_{\mathit{stu}}).$$ (3)

By applying second-order smoothness priors, the final MRF energy model is

$$E(x\mid \theta )\triangleq \sum _{s\in \mathcal{V}}{\theta }_s(x_s)+\sum _{(s,t,u)\in {\mathcal{F}}_{\mathcal{H}}}{\theta }_{\mathit{stu}}(x_s,x_t,x_u)$$ (4)

where ℱ_ℋ is defined on a set of collinear three nodes² on a cubic grid.

2.2. Data Cost Computation

The unary term θ_s(x_s) of (4) which measures the cost when a node s has a label x_s is defined as

$${\theta }_s(x_s)\triangleq f(s_x,s_y,s_z,s_x+d_x(x_s),s_y+d_y(x_s),s_z+d_z(z_s)).$$ (5)

This cost is calculated by the dissimilarity measure f using two volume patches centered on (s_x, s_y, s_z) in a source volume and (s_x+d_x(x_s), s_y+d_y(x_s), s_z+d_z(z_s)) in a target volume, respectively. As this representation is flexible, we can incorporate various dissimilarity measures to f. In this paper, we use the normalized cross correlation (NCC) for uni-modal registrations and the normalized mutual information (NMI) for multi-modal registrations. For computing NCC, we used cubic B-spline weighting scheme used in [3] with patch width 4. For NMI, we compute local NMI [9] for each nodes with patch width 8.

3. OPTIMIZATION STRATEGY

Optimizing the proposed energy model (4) directly is time-consuming when a large search space ℒ should be applied to cover large displacements. To reduce computational burden, we apply a decomposed scheme with a coarse-to-fine scheme.

3.1. Decomposed Scheme

Based on the scheme introduced in [10], we generate a new graph 𝒢^D by making three layers of nodes 𝒱^x, 𝒱^y and 𝒱^z corresponding to x, y and z displacements from the original nodes 𝒱. This is only applicable when potentials for spatial priors are summation of a potential for each dimensional displacement. Using the upper bound approximation, we decouple smoothness priors (3) as follows

$$\begin{array}{l}{\theta }_{\mathit{stu}}(x_s,x_t,x_u)\le {\theta }_{\mathit{stu}}^x(x_s^x,x_t^x,x_u^x)+{\theta }_{\mathit{stu}}^y(x_s^y,x_t^y,x_u^y)+{\theta }_{\mathit{stu}}^z(x_s^z,x_t^z,x_u^z),\\ {\theta }_{\mathit{stu}}^i(x_s^i,x_t^i,x_u^i)\triangleq {\lambda }_{\mathit{stu}}min(\mid d_i(x_s)-2d_i(x_t)+d_i(x_u)\mid ,T_{\mathit{stu}})\end{array}$$ (6)

where i ∈ {x, y, z}. In the graph 𝒢^D, factor sets include ${\mathcal{F}}_{\mathcal{H}}^x,{\mathcal{F}}_{\mathcal{H}}^y$ and ${\mathcal{F}}_{\mathcal{H}}^z$ for intra-layer interaction potentials and ${\mathcal{F}}_{\mathcal{H}}^{\mathit{xyz}}$ for inter-layer interaction potentials defined as

$${\theta }_{\mathit{stu}}^{\mathit{xyz}}(x_s^x,x_t^y,x_u^z)\triangleq f(s_x,t_y,u_z,s_x+d_x(x_s),t_y+d_y(x_t),u_z+d_z(z_t)).$$ (7)

Then, the MRF energy on 𝒢^D is defined as follows

$$E(x\mid \theta )\triangleq \sum _{(s,t,u)\in {\mathcal{F}}_{\mathcal{H}}^D}{\theta }_{\mathit{stu}}(x_s,x_t,x_u)$$ (8)

where ${\mathcal{F}}_{\mathcal{H}}^D={\mathcal{F}}_{\mathcal{H}}^{\mathit{xyz}}\cup {\mathcal{F}}_{\mathcal{H}}^x\cup {\mathcal{F}}_{\mathcal{H}}^y\cup {\mathcal{F}}_{\mathcal{H}}^z$. The unary term (5) is changed to ternary term (7) and its set is included in ${\mathcal{F}}_{\mathcal{H}}^D$. The energy model applying decomposed scheme consists of ternary potentials only.

3.2. Coarse-to-Fine Scheme

To cover large displacements efficiently, we compute multilevel displacement in a coarse-to-fine manner [9]. For a pyramidal dense descriptor representation, input volumes of finer level are smoothed and down-sampled for generating volumes of the coarser level. The displacements are computed on each level with propagated displacement offset which is scaled from the coarser levels. To achieve sub-pixel accuracy, we perform the sub-pixel refinement on the finest level.

3.3. Discrete Optimization Method

To optimize the proposed energy model (4) or (8), tree reweighted message passing (TRW) [11, 1] is applied. The TRW message passing provides the lower bound guaranteed not to decrease. A recent comparative study shows the TRW gives the state-of-the-art performances among the various discrete optimization methods [12]. As the TRW theory is built on the pairwise MRF, we need to convert factor graphs representations to pairwise interactions. The detailed conversion procedure is referred to [6]. As the proposed energy model uses higher-order smoothness priors, converting to the hierarchical gradient node graph [13] is possible. Although optimization strategy we introduced decrease computational burden significantly, running TRW or BP on 3D volumes still require high computational power. However, as BP variant optimization algorithms have highly parallel structures, we implement TRW message passing algorithms for 3D registrations on GPU using NVIDIA CUDA. Using GPU, the time required for 1 cycle of whole message passing in a volume takes few seconds. The total time for registration of the proposed method takes about 20 ~ 40 minutes in the experiments.

4. EXPERIMENTS

In this section, we show registration results using uni- and multi-modal brain MRI data. The results are compared to those of a method using the first-order MRF energy model [3]. In experiments, λ_st = 6, T_st = 20, λ_stu = 6 and T_stu = 20 are used. For label widths, we use d_i ∈ {−4, …, 4} for the coarsest level, d_i ∈ {−(2 + (n − 1)/2), …, (2 + (n − 1)/2)} for n^th level (n = 1 for the finest level) and d_i ∈ {−1.0,−0.75, …, 0.75, 1.0} for the sub-pixel refinement where i ∈ {x, y, z}. We use 4 levels for coarse-to-fine schemes and 2mm voxel spacing for finest level. All parameters are empirically chosen.

4.1. Registration Using Synthetically Deformed Data

We generate synthetically deformed data sets from 10 aligned T1 and T2 weighted brain MRI pairs. The deformed volume is generated by TPS [8] warping with control points (placed on grid with 30mm spacing) perturbed with random variation [−6mm,6mm] in each axis. More detailed procedure for generating synthetic images are described in [14]. The registration is performed between T1 and warped T1 volumes (uni-modal) and T1 and warped T2 volumes (multi-modal), respectively. As intensity characteristics between T1 and T2 images are not consistent, registrations between T1 and T2 are more challenging problem. In Table 1, we show averaged root mean square (RMSE), mean (MEAN) and standard deviation (STD) for the magnitude of differences (mm) between the ground truth and the recovered displacement fields. One can see the performances of the proposed method are better than the first order model. In Fig. 1, we show registration results for a selected case. In the figure, the proposed method produces smoother displacements with higher accuracy than the other method.

Table 1.

Registration Errors for Synthetically Deformed Data

Data	Method	RMSE	MEAN	STD
uni-modal	Initial	3.42	3.18	1.27
	First-Order[3]	1.01	0.84	0.56
	Proposed	0.49	0.38	0.31
multi-modal	Initial	3.42	3.18	1.27
	First-Order[3]	1.44	1.29	0.64
	Proposed	1.10	1.01	0.41

Fig. 1.

Selected results for Synthetically Deformed Data. Slice images from original T1, synthetically deformed T1 and T2 volumes shown in (a), (b) and (c), respectively. The displacement vectors are shown using spherical coordinate (ρ, θ, ϕ). The color coded displacements are shown in (d) ρ-θ and (e) ρ-ϕ. The ground truth displacement used for generating (b) and (c) is shown in (d). The recovered displacements using First-Order [3] and Proposed shown in (g) and (h), respectively. In (f–h), left and right figure correspond to ρ-θ and ρ-ϕ parts of the displacements.

4.2. Registration Using Inter-Subject Data

For inter-subject registrations, we use IBSR V2.0 data set [15] which consists 18 T1 weighted brain MRI volumes with manual segmentations. The volumes are scaled to have 1mm voxel spacing in each axis. We select first data as a template and remaining data sets are registered to the template. The recovered displacement fields are used to deform the corresponding segmentations, and then deformed segmentations are compared to the manual ones. We show DICE score (DICE), the sensitivity (SENS) and the specificity (SPEC) for gray matters, white matters and ventricle in Table 2. For limited spaces, only two selected cases are shown. One can see the proposed method have better scores in terms of DICE, SENS and SPEC measures. In Fig. 2, we show Case 2 results. In the figure, the proposed method produces smoother displacements and better overlapping regions than the other method. The proposed method especially works better in central regions on brain than the other method.

Table 2.

Registration Errors for Inter-Subject Data

Case	Matter	Method	DICE	SENS	SPEC
Case 1	Gray	Initial	0.622	0.679	0.980
		First-Order[3]	0.843	0.846	0.993
		Proposed	0.847	0.850	0.993
	White	Initial	0.589	0.654	0.988
		First-Order[3]	0.797	0.798	0.995
		Proposed	0.804	0.804	0.995
	Ventricle	Initial	0.398	0.721	0.999
		First-Order[3]	0.763	0.872	1.000
		Proposed	0.775	0.897	1.000
Case 2	Gray	Initial	0.616	0.721	0.979
		First-Order[3]	0.836	0.865	0.991
		Proposed	0.840	0.881	0.991
	White	Initial	0.643	0.646	0.981
		First-Order[3]	0.798	0.762	0.996
		Proposed	0.813	0.775	0.996
	Ventricle	Initial	0.608	0.723	1.000
		First-Order[3]	0.818	0.780	1.000
		Proposed	0.843	0.816	1.000

Fig. 2.

Case 2 results for Inter-Subject Data. Slice images from source T1 and target T1 volumes are shown in (a) and (b). The recovered displacements using First-Order [3] and Proposed shown in (f) and (j), respectively. We use color coded displacements along with Fig 1. Overlapped segmentations of source and deformed target from initial, (f) and (j) are shown in (c–e), (g–i) and (k–m), respectively. In these figures, overlapped regions are colored white and non-overlapped regions are colored darker gray for source and light gray for deformed target.

5. CONCLUSION

In this paper, we proposed a nonrigid registration method using the MRF model with second-order smoothness priors. The second-order prior enables us to generate smoother displacement fields with better accuracy. In experiments, we showed the proposed method outperformed a MRF energy model with first-order smoothness priors on uni- and multimodal brain MRI registrations using synthetically deformed data and inter-subject brain MRI data.