Fast Image Registration by Hierarchical Soft Correspondence Detection

Abstract

A new approach, based on the hierarchical soft correspondence detection, has been presented for significantly improving the speed of our previous HAMMER image registration algorithm. Currently, HAMMER takes a relative long time, e.g., up to 80 minutes, to register two regular sized images using Linux machine (with 2.40GHz CPU and 2-Gbyte memory). This is because the results of correspondence detection, used to guide the image warping, can be ambiguous in complex structures and thus the image warping has to be conservative and accordingly takes long time to complete. In this paper, a hierarchical soft correspondence detection technique has been employed to detect correspondences more robustly, thereby allowing the image warping to be completed straightforwardly and fast. By incorporating this hierarchical soft correspondence detection technique into the HAMMER registration framework, the robustness and the accuracy of registration (in terms of low average registration error) can be both achieved. Experimental results on real and simulated data show that the new registration algorithm, based the hierarchical soft correspondence detection, can run nine times faster than HAMMER while keeping the similar registration accuracy.

1. Introduction

Deformable registration is important for comparing medical images across different subjects. So far, various image registration algorithms [1–19] have been proposed in the literature. Most registration methods are based on matching simple image information such as intensities, while other methods use features for image registration. Features can be local features [4, 20] such as image boundary or curvature, or regional features such as geometric moment invariants (GMIs) [21, 22], wavelet coefficients [23], multiple-scale histogram features [24], local descriptor features [25], or combinations of various local features [25]. In the image registration, besides maximizing the matching of image features, the estimated dense deformation field is required to be smooth as well. This can be achieved by using a Laplacian smoothness constraint [21, 22], an elastic energy [8], or even the statistical deformation constraint learned from the training samples [26].

We have previously developed a hierarchical image registration algorithm, called HAMMER [21, 22]. There are two novel ideas in this algorithm. First, an attribute vector is defined for each point in the image, in order to characterize and better distinguish this point from its neighboring points; the image registration can be accordingly formulated as a problem of attribute vector matching or correspondence detection. Second, a strategy for hierarchical approximation of energy function is used to reduce the chance of local minima in the registration. In particular, the registration algorithm is designed to focus initially on the registration of only a small set of active points (with distinctive attribute vectors). This process implicitly selects only a small number of major energy terms in the whole energy function for optimization, thus potentially achieving global optimization for the new energy function. With the progress of registration, more and more points with less distinctive attribute vectors are added into the set of active points for actively driving the image registration, once these less distinctive points have been moved close to their targets and become stable for registration. In the final registration stage, all points in the images are considered as active points to drive image registration, which means that all energy terms in the whole energy function will be finally considered for optimization. By using this hierarchical registration strategy, we can improve the performance as well as the speed of registration. Also, since not all points are initially considered as active points to drive image registration, thus effectively reducing the scale of image registration problem and becoming fast and robust to complete.

HAMMER has been successfully applied to many large clinical research studies and clinical trials involving over 5,000 MR brain images. These applications include one of the largest longitudinal studies of aging in the world to date - which involves an 18-year annual follow-up of 150 elderly individuals [27], a relatively large schizophrenia imaging study involving 148 participants [28], a study of cocaine-exposed children [29], a morphometric study of children with genetic abnormalities [30], a large study of the effects of organolead-exposure on the brain [31], and a study of effect of sustained, heavy drinking on the brain [32, 33]. Many of these studies are the leading studies in their respective fields, and have received ample international attention.

To further improve the performance of HAMMER, various research works have been performed. In order to improve the performance of correspondence detection using attribute vectors, the best-scale geometric moment invariants (GMIs) have been learned for registration [34]. Also, the separate sets of best features have been specially learned for different parts of brain, since it is believed that different types of features have different functions in characterizing image points, and different parts of brain need different sets of best features to distinguish themselves. Moreover, by learning the normal brain variations from a set of brain image samples, we can use this learned information to constrain the image registration, thus improving the robustness of registration [25, 26].

It is worth noting that all previous improvements were focusing on improving the robustness and accuracy of registration in HAMMER, e.g., using the average registration error. Less attention has been paid to increasing the registration speed. Currently, HAMMER takes up to 80 minutes for registering two MR brain images with the size of 256×256×124, which might be too long for some applications. Therefore, it is important to design methods to improve the speed of registration algorithm.

One reason that HAMMER needs a relatively long time for registration is that the correspondence detection results, used to guide image registration, can be ambiguous in some complex structures, and image warping has to be performed conservatively and thus slow. Many advanced correspondence detection methods have been proposed in the literature [35, 36]. Traditionally, one to one correspondence is established for each point in the image, which we can call as hard correspondence detection. Some advanced methods such as in [35, 36] allow to identify correspondences in a soft way, e.g., each point can be have multiple points as candidate correspondences, each assigned with a possibility. The total possibility is equal to 1. This is called soft correspondence detection. In the initial correspondence detection procedure, candidate correspondences can have similar amount of probability, indicating the uncertainty is high. With the progress of correspondence detection process, it will allow only a single correspondence or several candidate correspondences having the high probability and others close to zero, which increases the specificity of correspondence detection.

In this paper, we propose to detect soft correspondences [35, 36] for active points selected during the registration procedure. Thus, correspondence detection can be robust and allows the image warping to perform fast. Moreover, soft correspondence detection can be implemented hierarchically to render the registration both robust and accurate. For example, in the initial registration procedure, the distant points with high attribute similarity can also contribute significantly to the soft correspondence detection, thus potentially increasing the robustness of correspondence detection. With the progress of registration, the distant points even with high attribute similarity decrease gradually their contribution to the soft correspondence detection, thus effectively increasing the specificity of final correspondence detection results. By mathematically formulating this soft correspondence detection idea into the HAMMER registration framework, the speed of the final registration algorithm is reduced to only the one-ninth of original computational cost required by HAMMER, without affecting much for registration performance. In particular, the proposed registration algorithm can complete the alignment of two 256×256×124 images in about 9 minutes using Linux machine (with 2.40GHz CPU and 2-Gbyte memory).

The structure of this paper is organized as follows. In Section 2, the definitions for attribute vector and similarity, a new energy function for registration, and details for implementation are all discussed one by one. In particular, the ideas of how to mathematically allow each active point to have multiple correspondences in the other image, and how to make soft correspondence detection strategy robust as well as accurate are also explained in detail. In Section 3, various experiments on simulated and real data are conducted to confirm that the performance of the proposed registration method is not decreased much even with the significant deduction of computation cost. This paper concludes in Section 4.

2. Method

Image registration problem is formulated here as an iterative process of detecting soft correspondences for the active points. The active points are selected according to the distinctiveness of their attribute vectors, which include image intensity, edge type, and geometric features from the neighborhood. The active points are also selected hierarchically during the registration procedure, e.g., initially a small set of active points with the most distinctive attribute vectors and gradually a larger and larger set of active points by including less distinctive points. The dense correspondences in the entire image domain are interpolated by the mean correspondences estimated in the active points.

The correspondence for each active point is softly determined. Instead of one-to-one correspondence detection, each active point is allowed to have multiple correspondences detected in the other image; each detected correspondence is assigned with a probability ranging from 0 to 1, and all the detected correspondences have a total probability of 1. The location of soft correspondence is determined as a statistical mean location of all detected correspondences. Note that, if a point v is considered as a correspondence of an active point x, the respective probability is determined by evaluating not only the similarity of attribute vectors of points v and x, but also the similarity of attribute vectors of all points in their neighborhoods. To increase the speed of detecting candidate correspondences and measuring their probabilities, the evaluation of the similarity of attribute vectors of the two points can be performed first, in order to select only a small set of candidate points {v} to perform the computation-consuming comparison of attribute vectors in the whole neighborhoods.

It is worth noting that the soft correspondence detection is more robust than the one-to-one correspondence detection, thus the robustness of registration using the soft correspondence detection can be improved. In the initial registration procedure, both local and distant points with similar attribute vectors can contribute to the correspondence detection. Gradually, in the later registration procedure, only the most similar points are allowed to contribute to the correspondence detection. By using these strategies, we can achieve both the robustness and the accuracy for the registration results.

In the following, we first briefly summarize the definitions for attribute vectors and their similarity, which are the same as in HAMMER [21, 22]. Then, we give a definition of new energy function for the proposed registration algorithm. Finally, we provide the detailed descriptions of the soft correspondence matching within the original HAMMER registration framework.

2.1 Attribute Vector and Similarity

Definition

An attribute vector is defined for each point x in the image, to better distinguish one point from its neighboring points and also to identify its corresponding points in the other image by matching attribute similarity. The attribute vector a(x) includes three parts of information, i.e., edge type a₁(x), image intensity a₂(x), and Geometric Moment Invariants (GMIs) a₃(x) that are calculated from the structure in the neighborhood. Hence, a(x) = [a₁ (x) a₂ (x) a₃ (x)]; each of these three sub-vectors is explained one by one in the next three paragraphs.

For the hard tissue segmentation case [37] as employed in this paper, a₁(x) is a 1×1 vector, which takes one of 7 discrete values corresponding to 7 edge types such as non-edge and six combinations of edges between grey matter (GM), white matter (GM), and CSF. In the fuzzy segmentation case [38], a 3×1 fuzzy vector [C_WM, C_GM, C_CSF]^T is typically used to represent the probability of WM, GM, and CSF on an image point x, where C_WM, C_GM, and C_CSF all take a value between 0 and 1 and C_WM + C_GM + C_CSF = 1.0. Thus, in the fuzzy segmentation case, a₁(x) is designed as a 3×1 vector to measure the maximal direct difference (3×1 vector) of the tissue segmentation results between the image point x and any neighboring points.

a₂(x) represents direct information on the image point x, which can be image intensity, tissue segmentation result, or both. In this paper, a₂(x) is selected as a 1×1 vector to represent the intensity on the image point x, which has been normalized between 0 and 1.

For the brain images, the vector a₃(x) comprises the GMIs of the respective GM, WM, and CSF, calculated from the multiple-scale spherical neighborhoods around the image point x. For a particular scale and tissue type, thirteen rotation invariants can be calculated from the zero-order, second-order, and third-order 3D regular moments [39]. Therefore, a₃(x) can be a vector with K=3×13×L, where L is the number of scales used. Four rotation invariants that are formulated from the zero-order and the second order moments are listed next, while other nine rotation invariants, formulated from the third-order moments and both the second-order and the third-order moments, can be found in [39].

$$\begin{array}{l}{I}_1=M_{0,0,0}\\ I_2=M_{2,0,0}+M_{0,2,0}+M_{0,0,2}\\ I_3=M_{2,0,0}{M}_{0,2,0}+M_{2,0,0}{M}_{0,0,2}+M_{0,2,0}{M}_{0,0,2}-M_{1,0,1}^2-M_{1,1,0}^2-M_{0,1,1}^2\\ I_4=M_{2,0,0}{M}_{0,2,0}{M}_{0,0,2}-M_{0,0,2}{M}_{1,1,0}^2+2M_{1,1,0}{M}_{1,0,1}{M}_{0,1,1}-M_{0,2,0}{M}_{1,0,1}^2-M_{2,0,0}{M}_{0,1,1}^2\end{array}$$

Note that the first GMI corresponds to the volume of a certain tissue.

Similarity

As mentioned above, there are three parts of attributes included in each attribute vector, with different parts of attributes having different contributions to the overall similarity of attribute vectors. For example, a₁(x) represents the edge type, which provides important information for matching the same type of boundary points in the two individual brain images. a₃(x) provides the geometric features, to allow the detailed comparison of geometric similarity between two images. Moreover, the sizes of these three different parts of attributes are different. Thus, we can first compare the similarity of short vectors of important attributes, such as a₁(x), in the two attribute vectors, in order to avoid comparing the similarity of long vectors of attributes, such as a₃(x), if the similarity on short vectors is low. In this way, we can significantly reduce the overall computation time for similarity measurement. In particular, the dissimilarity of two attribute vectors, a(x) and a(y), of two points x and y can be defined as:

$$d(a(\mathbf{x}),a(\mathbf{y}))=\{\begin{array}{ll}1,\hfill & if\mid a_1(\mathbf{x})-a_1(\mathbf{y})\mid >\epsilon \hfill \\ 1-c([a_2(\mathbf{x})a_3(\mathbf{x})],[a_2(\mathbf{y})a_3(\mathbf{y})]),\hfill & \mathbf{otherwise}\hfill \end{array}$$

If |a₁ (x) − a₁ (y)| is larger than ε, the overall dissimilarity d (a(x), a(y)) is one. In the hard tissue segmentation case, ε is equal to zero; in the fuzzy segmentation case, ε is a certain value that should be determined in the experiments. c([a₂ (x) a₃ (x)],[a₂ (y) a₃ (y)]) is the similarity of the second and the third parts of the two attribute vectors in a(x) and a(y). To be strict on the similarity measure, each element in a₂ and a₃ is first normalized into a range from 0 and 1, and then compared and further integrated as a way described next:

$$c([a_2(\mathbf{x})a_3(\mathbf{x})],[a_2(\mathbf{y})a_3(\mathbf{y})])=(1-\mid a_2(\mathbf{x})-a_2(\mathbf{y})\mid )·\prod _{i=1}^K(1-\mid a_3^i(\mathbf{x})-a_3^i(\mathbf{y})\mid ),$$

where $a_3^i(\mathbf{x})$ is the i-th element of a₃ (x) which has total K elements.

2.2 Energy Function

The goal of image registration is to best align a template image T (x) with a subject image S(y) by maximizing the similarity of attribute vectors on the corresponding points. If a template point x is corresponding to a subject point y, their attribute vectors a_T (x) and a_S (y) should be as similar as possible, i.e., d (a_T (x), a_S (y)) should be as small as possible. On the other hand, the estimated deformation field {h(x), x ∈ Ω_T}, where Ω_T denotes the template image domain, should be able to warp the template point x very close to its corresponding subject point y, i.e., h(x) = y.

It is important to allow each active point to have multiple correspondences detected in the other image, in order to increase the robustness of overall registration algorithm. This is significant particularly when the one-to-one correspondence detection is difficult. Accordingly, in this paper, each active point x in the template is allow to have multiple points {v} detected in the subject image as correspondences, with {p_x,v} representing the probability of a template active point x and a subject point v as a pair of corresponding points. The deformation field h(x) in the template point x can be thus estimated as a statistical mean location of all correspondences detected in the subject, i.e., $h(\mathbf{x})={\displaystyle \sum _{\mathbf{v}}}{p}_{\mathbf{x},\mathbf{v}}·\mathbf{v}$. Note that the soft or fuzzy correspondence will be progressively forced to the one-to-one correspondence during the registration procedure, thus achieving not only the robustness but also the accuracy of registration. The robustness is achieved by initially allowing all points with similar attribute vectors to contribute to the correspondence detection, while the accuracy or the specificity of registration is achieved by finally allowing only the most similar points to contribute to the correspondence detection.

Our registration algorithm can be accordingly formulated as an iterative optimization process involving three steps. The first step is to select active points from both template and subject images, with the number of active points progressively increased during the registration. The second step is to detect multiple candidate correspondences and respective probabilities for each active point; candidate points can be selected initially from a large neighborhood and later from a small or even immediate neighborhood. The third step is to calculate the mean deformation for each template active point based on the detected correspondences, and also to interpolate the mean deformations in the other locations of the template image domain and make them smooth spatially. By iteratively performing these three steps, the registration algorithm can register images robustly and accurately.

The energy function is defined next to facilitate the iterative performance of these three steps in the registration procedure.

Energy terms

The first term in the whole energy function describes the overall cost of matching a template image with a subject image (see Fig. 1 for reference). For a template point x, its mean deformation is defined as h(x). For a subject point v in the neighborhood of h(x), v ∈ n₁ (h(x)), its probability of being selected as a correspondence for the template point x is p_x,v. n₁ denotes a searching neighborhood around the mean deformation h(x); the size of n₁ is large initially, and decreases gradually with the progress of registration, in order to achieve the robustness as well as the accuracy of registration. Three types of costs need to be minimized once a subject point v is selected as a correspondence of the template point x. The first type of cost is the dissimilarity of attribute vectors of the template point x and the subject point v, i.e., d (a_T (x), a_S (v)). The second type of cost is the overall dissimilarity in the neighborhoods around the template point x and the subject point v, i.e., $\frac{1}{{\displaystyle \sum _{\mathbf{u}\in n_2(\mathbf{x})}}1}{\displaystyle \sum _{\mathbf{u}\in n_2(\mathbf{x})}}d(a_T(\mathbf{u}),a_S(h(\mathbf{u})+(\mathbf{v}-h(\mathbf{x}))))$ where all neighboring points {u} of the template point x, u ∈ n₂ (x), are compared with the neighboring points h(u) + (v − h(x)) of the subject point v, by using attribute vector dissimilarity. n₂ represents a matching neighborhood, whose size changes from an initial large size to a final small size with the progress of registration. The third type of cost is the distance of the subject point v away from the mean deformation h(x), i.e., ||v − h(x)||². By integrating all of these three types of cost from all template points, the total cost for aligning the template with the subject images can be obtained as the first energy term in whole energy function.

Fig. 1.

Schematic explanation for correspondence detection, probability estimation, and neighborhood matching in our registration algorithm. A template point x is tentatively warped to a location h(x) in the subject. In the subject space, a neighboring point v of h(x) is considered as a candidate correspondence for the template point x, with the probability p_x,z calculated from the similarity of attribute vectors on points x and v, as well as the similarity of attribute vectors in the respective neighborhoods (e.g., a dotted grey circle n₂ in the template and one deformed dotted grey circle around point v in the subject). The deformed dotted circle around point v is shifted by v−h(x) from the one around point h(x), which is a warped version of an original circle in the template. In the searching neighborhood n₁ (red circle), only several points (green cross points) are detected as candidate correspondences for template point x.

For increasing the consistency of registration [5, 21, 22, 40, 41], we also define a cost function for matching the subject image with the template image, as described by the second energy term in the whole energy function. h⁻¹(y) is the backward transformation from the subject to the template, which is the inverse of the forward transformation h(x). Cost definitions in the second energy term are similar to those in the first energy term.

As mentioned, by allowing each point to have multiple correspondences, the robustness of registration algorithm can be increased. Also, in order to increase the specificity or the accuracy of registration results, it is necessary to finally determine only a single correspondence for each point, or require all the detected correspondences very close to the mean deformation h(x). The latter requirement can be implemented by increasing the parameter σ (in the energy function) with the progress of registration, to constrain the distance between each correspondence (v) and the mean location of all correspondences (h(x)). Also, the progressive decrease of the size of searching neighborhood n₁ ensures the selection of candidate correspondences from the nearby points. The former requirement can be implemented by adding an entropy measurement on the probabilities {p_x,v} and {p_v,y}, which is included as the third energy term in the whole energy function. The parameter τ controls the significance of this energy term [42, 43]. By adaptively adjusting parameter τ with the progress of registration, i.e., changing from extremely large to extremely small value, we can actively allow the registration algorithm to detect multiple correspondences from an initial large neighborhood to later a small neighborhood, and also from initially even the low similarity points to latter only the high similarity points.

The last energy term evaluates the smoothness of the mean deformation field {h(x)} estimated in the entire image domain, using a Laplacian operator. Based on the multiple correspondences and the respective probabilities detected for each active point, the mean deformations {h(x)} in all active points {x} can be statistically determined and then used to refine the previous estimations in the same active points. The mean deformations in the other locations of the image can be interpolated with an interpolation kernel, e.g., a Gaussian kernel used in this paper. Finally, this updated deformation field can be made as regular as possible in the entire image domain by using a Laplacian smoothness constraint. A parameter β is used to control the importance of the Laplacian smoothness constraint in the whole energy function.

The following is the whole energy function that our image registration algorithm will minimize.

$$\begin{array}{l}E=\sum _{\mathbf{x}\in {\mathrm{\Omega }}_T}{\omega }_T(\mathbf{x})\sum _{\mathbf{v}\in n_1(h(\mathbf{x}))}{p}_{\mathbf{x},\mathbf{v}}\left\{d(a_T(\mathbf{x}),a_S(\mathbf{v}))+\frac{1}{{\displaystyle \sum _{\mathbf{u}\in n_2(\mathbf{x})}}1}\sum _{\mathbf{u}\in n_2(\mathbf{x})}d(a_T(\mathbf{u}),a_S(h(\mathbf{u})+(\mathbf{v}-h(\mathbf{x}))))+\sigma ·{\Vert \mathbf{v}-h(\mathbf{x})\Vert }^2\right\}+\\ \sum _{\mathbf{y}\epsilon {\mathrm{\Omega }}_S}{\omega }_S(\mathbf{y})\sum _{\mathbf{v}\in n_1(h^{-1}(\mathbf{y}))}{p}_{\mathbf{v},\mathbf{y}}\left\{d(a_T(\mathbf{v}),a_S(\mathbf{y}))+\frac{1}{{\displaystyle \sum _{\mathbf{u}\in n_2(\mathbf{y})}}1}\sum _{\mathbf{u}\in n_2(\mathbf{y})}d(a_T(h^{-1}(\mathbf{u})+(\mathbf{v}-h^{-1}(\mathbf{y}))),a_S(\mathbf{u}))+\sigma ·{\Vert \mathbf{v}-h^{-1}(\mathbf{y})\Vert }^2\right\}+\\ \tau \left\{\sum _{\mathbf{x}\in {\mathrm{\Omega }}_T}\sum _{\mathbf{v}\in n_1(h(\mathbf{x}))}{p}_{\mathbf{x},\mathbf{v}}log{p}_{\mathbf{x},\mathbf{v}}+\sum _{\mathbf{y}\in {\mathrm{\Omega }}_S}\sum _{\mathbf{v}\in n_1(h^{-1}(\mathbf{y}))}{p}_{\mathbf{v},\mathbf{y}}log{p}_{\mathbf{v},\mathbf{y}}\right\}+\beta \sum _{\mathbf{x}}\Vert {\nabla }^{2}h(\mathbf{x})\Vert \end{array}$$

Subject to the constraints:

$$\begin{array}{l}0\le p_{\mathbf{x},\mathbf{v}}\le 1,\sum _{\mathbf{v}\in n_1(h(\mathbf{x}))}{p}_{\mathbf{x},\mathbf{v}}=1\text{for}\text{each}\text{template}\text{point}\mathbf{x};\\ 0\le p_{\mathbf{v},\mathbf{y}}\le 1,\sum _{\mathbf{v}\in n_1(h^{-1}(\mathbf{y}))}{p}_{\mathbf{v},\mathbf{y}}=1\text{for}\text{each}\text{subject}\text{point}\mathbf{y}.\end{array}$$

Hierarchical optimization

As mentioned, our registration algorithm will hierarchically select a set of active points with distinctive attribute vectors to drive the image registration. The set of active points will be progressively increased with the progress of registration by adding less distinctive points when they are deformed close to their targets and become reliable to drive the image registration. This strategy can be implemented by hierarchically assigning different values to the parameters ω_T(x) and ω_S (y) in the energy function, thus effectively selecting the respective template and subject points (x and y) to be actively involved in the image registration.

We can initially assign large values to the points with distinctive attribute vectors, and zero to other points in the images. In this case, the whole energy function above describes a problem of determining correspondences for those selected active points and also interpolating a smooth dense deformation field according to the mean correspondence locations determined in those active points. After solving this relative small scale problem of correspondence detection and deformation field estimation, we can increase the set of active points by including other less distinctive points, and then start to solve a relative large scale problem of correspondence detection on more active points. Since the sizes of the searching neighborhood n₁ and the matching neighborhood n₂ will be decreased with the progress of registration, the computational cost for detecting correspondences for more active points will not be necessarily increased as confirmed by our experiments.

In the final, by considering all image points into the energy minimization problem, we solve the exact energy function that our registration algorithm should originally minimize. Importantly, the advance of using this hierarchical optimization strategy is that we can have high chances to avoid local minima. The subsection next describes the details for implementing this hierarchical optimization strategy.

2.3 Implementation

The whole energy function above can be hierarchically minimized by iteratively estimating (1) the correspondence probabilities {p_x,v} and {p_v,y} for the current selected active points, (2) the mean deformations in the current selected active points, and (3) the dense mean deformation field in the entire image domain. The descriptions for each of these three iterative steps are provided next.

Estimating correspondence probabilities

The probability for each pair of correspondences in {p_x,v} can be determined by solving an equation of ∂E/∂P_x,v = 0, with the result given next. Note that the probabilities in {p_v,y} can be similarly determined.

$$p_{\mathbf{x},\mathbf{v}}=c·m(\mathbf{x},\mathbf{v})·m_{\mathbf{Sub}}(\mathbf{x},\mathbf{v})·m_{\mathbf{Dist}}(\mathbf{x},\mathbf{v})$$

where c is a constant, m(x, v) measures the similarity of attribute vectors between a template point x and a subject point v, and m_Sub (x, v) represents the similarity of whole neighborhoods around the template point x and the subject point v, and m_Dist(x, v) measures how close two points x and v are. Expressions of these measures are given next:

$$\begin{array}{l}m(\mathbf{x},\mathbf{v})=exp\left(\frac{-{\omega }_T(\mathbf{x})·d(a_T(\mathbf{x}),a_S(\mathbf{v}))}{\tau }\right)\\ m_{\mathbf{sub}}(\mathbf{x},\mathbf{v})=exp\left(-\frac{{\omega }_T(\mathbf{x})·\frac{1}{{\displaystyle \sum _{\mathbf{u}\in n_2(\mathbf{x})}}1}{\displaystyle \sum _{\mathbf{u}\in n_2(\mathbf{x})}}d(a_T(\mathbf{u}),a_S(h(\mathbf{u})+(\mathbf{v}-h(\mathbf{x}))))}{\tau }\right)\\ m_{\mathbf{Dist}}(\mathbf{x},\mathbf{v})=exp\left(-\frac{{\omega }_T(\mathbf{x})·\sigma {\Vert \mathbf{v}-h(\mathbf{x})\Vert }^2}{\tau }\right)\end{array}$$

The estimated probabilities {p_x,v} are normalized separately for each template point x as follows, in order to ensure that the total probability for detecting correspondences for the point x is 1.0.

$$p_{\mathbf{x},\mathbf{v}}=\frac{p_{\mathbf{x},\mathbf{v}}}{{\displaystyle \sum _{\mathbf{v}\in n_1(h(\mathbf{x}))}}{p}_{\mathbf{x},\mathbf{v}}},\mathit{for}\mathit{every}\mathit{template}\mathit{point}\mathbf{x}.$$

It is worth noting that, the similarity of neighborhoods m_Sub (x, v) is generally expensive to compute, if the number of points {v} in the neighborhood n₂ is large. To avoid this computation problem, we propose to first calculate the measurement m(x, v) for all neighboring points {v}. Then, we check whether any neighboring points have the measurement m(x, v) over a certain threshold. (Note that this threshold can be gradually adjusted with the progress of registration, e.g., from initially a large value (such as 0.8 in our paper) to later a small value.) Only for the points with the expected measurement m (x, v), their neighborhood similarity m_Sub (x, v) will be consequently computed. By using this strategy, we can potentially assign zero probability for many neighboring points v, thus significantly reducing the overall computational time.

Estimating the mean deformations in active points

Once the probabilities {p_x,v} and {p_v,y} have been determined, the mean deformation, or the location of soft correspondence, for each active point can be immediately determined. By taking active points in the template image as an example, we can determine the mean deformation for each template active point x by optimizing the sub-energy term, ${\displaystyle \sum _{\mathbf{x}\in {\mathrm{\Omega }}_T}}{\omega }_T(\mathbf{x}){\displaystyle \sum _{\mathbf{v}\in n_1(h(\mathbf{x}))}}{p}_{\mathbf{x},\mathbf{v}}\left\{\sigma ·{\Vert \mathbf{v}-h(\mathbf{x})\Vert }^2\right\}$. The solution is $h(\mathbf{x})={\displaystyle \sum _{\mathbf{v}\in n_1(h(\mathbf{x}))}}(p_{\mathbf{x},\mathbf{v}}·\mathbf{v})$. This means that the new location of soft correspondence for the active point x is the statistical mean location of all correspondences detected in the subject image.

By determining the new location for each of these active points in the template and the subject images, we can deform all points in the neighborhoods of these active points by using a Gaussian kernel as employed in HAMMER [21, 22]. By selecting the standard deviation of the Gaussian kernel as small as the one-third of the size of searching neighborhood n₁, we can smoothly update the deformation field without introducing possible self-intersection. Note that the entire deformation field might be not sufficiently smooth due to this independent, local updating of deformations. This necessitates the use of Laplacian smoothness constraint to regularize the entire deformation field, as briefly summarized next.

Regularizing the dense deformation field

Once deformations have been updated in active points and their neighborhoods, the last energy term, e.g., Laplacian smoothness constraint, can be used to ensure both global and local smoothness of the entire deformation field. In the discrete case, the Laplacian smoothness constraint actually requires the deformation of each point to be close to the average deformation in its immediate neighborhood. Therefore, in the implementation, we can gradually push the deformation of each point to the average deformation in the immediate neighborhood, e.g., a 3×3×3 neighborhood. By repeating this step, we can finally obtain a smooth dense deformation field {h(x)}.

3. Experimental Results

The performance of the proposed registration algorithm has been evaluated using both simulated and real data. The results on simulated data show that the proposed registration method can achieve a registration accuracy comparable to the one produced by HAMMER, while the results on the real longitudinal data indicate that the proposed registration algorithm can achieve better longitudinal consistency than HAMMER. Importantly, the speed of the proposed registration algorithm has been significantly improved, e.g., taking only about the one-ninth of computation time required by HAMMER. For example, it takes about 9 minutes for registration of two 256×256×124 brain images, using Linux machine (with 2.40GHz CPU and 2-Gbyte memory).

Three subsections next compare the performances of our proposed registration algorithm and HAMMER in registering simulated data, constructing average brains from 18 individual brains, and aligning longitudinal data of real patients, respectively. The registration is performed in the isotropic image space. Note that, all of these experimental results are provided to confirm that the performance of the proposed registration algorithm is comparable to that of HAMMER, even with the significant improvement of registration speed.

2.1 Simulated Data

The proposed registration method has been applied to the simulated data for quantitatively measuring its registration performance. The same simulated data used to validate HAMMER in [21] is used here. The simulated data was created by first randomly selecting a brain, and then using the STAR elastic warping method [8] to warp this selected brain to five individual human brains by respectively placing forces on the ventricles and the cortex. Finally, five simulated brains were obtained, which are actually the different warped versions of the same selected brain. Note that the overall shapes of these five simulated brains are similar to those five individual brains used to guide the warping.

It is worth noting that the ground-truth deformations from the selected brain to each of these simulated brains are known. Thus, by applying the proposed registration method to estimating the deformations between the selected brain and each of these five simulated brains, we can quantitatively measure the accuracy of the proposed registration algorithm by comparing our estimated deformations with the ground-truth ones. The average registration error by the proposed algorithm is 1.13mm, which is comparable to 0.97mm error produced by HAMMER. Note that, as mentioned above, the proposed algorithm took only the one-ninth of the computational cost required by HAMMER.

2.2 Average Brain

The proposed registration algorithm has been applied to 18 real brain images for constructing an average brain from these 18 normalized brain images in the template space. The same 18 brain images used in HAMMER [21] are used here for direct comparison of our constructed average brain with that by HAMMER.

Fig. 2 show the average images constructed from 18 individual brain images using the proposed registration algorithm and HAMMER, respectively. The average brain image constructed by the proposed method (the 2^nd from left in the bottom row) is relatively fuzzy compared to the one constructed by HAMMER (left). Considering that the proposed registration method takes only the one-ninth of computation time to align images, the quality of registration is still promising. If we spend a relatively long time for registration, i.e., about 1/5 or 1/3 of computation time used by HAMMER, we can obtain average brain images closer and closer to the one obtained by HAMMER.

Fig. 2.

Average brain images constructed from 18 normalized individual brain images by HAMMER, and by the proposed registration algorithm that took about 1/9, 1/5, or 1/3 computation time of HAMMER. The difference image between the result by HAMMER and each result by the proposed method is provided in the top row. These different images have been enhanced for better visual inspection.

2.3 Longitudinal Data

The proposed registration method has been applied to the longitudinal data obtained from our Baltimore Longitudinal Study of Aging (BLSA) [27], to check the longitudinal stability of our registration algorithm in aligning images of the same subject acquired at different times. In the BLSA project, SPGR MR brain images have been acquired annually by a GE 1.5T scanner for 150 elderly subjects over 13 years. We applied our registration algorithm to the first five images of the same patients (acquired in the first five years), and the performance is compared with that achieved by HAMMER.

Fig. 3 shows the results of five longitudinally acquired images collected once per year for five years of the same subject. These images were spatially normalized by HAMMER (1^st row) and the proposed algorithm (2^nd row), respectively. Since the normalization for each time image was performed independently, the normalization results might be not longitudinally consistent. It can be observed that the normalization results by HAMMER are not consistent in the fourth time image as indicated by the arrow. In contrast, the normalization results achieved by the proposed algorithm are more stable across different times, indicating the better performance of the proposed algorithm than HAMMER in aligning longitudinal data. On the other hand, it is worth noting that the normalization results from these two methods are somewhat different, particularly in the light hemisphere sulcal patterns (right in the image). More evaluations need to be done in the future for complete comparison of the two methods.

Fig. 3.

Comparison on the longitudinal consistency of the registration on five different images of the same subject, using (a) HAMMER and (b) the proposed registration algorithm.

4. Conclusion

We have presented a new approach to significantly improve the speed of our previous HAMMER registration algorithm. This is achieved by using a hierarchical soft correspondence detection technique to replace the one-to-one correspondence detection strategy in HAMMER, thus the correspondence detection can be performed more robustly and the image warping can be conducted straightforwardly and fast. In particular, for each active point selected during the registration procedure, it is allowed initially to have multiple correspondences detected in the other image, and gradually forced to have few (or even one) correspondences detected in the other image. By using this hierarchical soft correspondence detection technique, we can achieve not only the robustness but also the accuracy of registration.

In summary, the image registration is here formulated as a process of hierarchically detecting a soft correspondence for each active point selected during the registration procedure, and also a process of interpolating and regularizing the dense smooth deformation field in the entire image domain. The experimental results indicate that, compared to HAMMER, first the speed of the proposed registration algorithm has been significantly increased without sacrificing much for the registration performance, and second the longitudinal consistency has been considerably improved.

In the future work, we will extensively evaluate and optimize all the parameters in our proposed registration algorithm, by using more real and simulated datasets with ground-truth information provided by manual raters or computer simulations. The final registration algorithm will be released publicly, as we did for the HAMMER registration algorithm, http://www.nitrc.org/projects/hammer/.