A Generative Model for Image Segmentation Based on Label Fusion

Abstract

We propose a nonparametric, probabilistic model for the automatic segmentation of medical images, given a training set of images and corresponding label maps. The resulting inference algorithms rely on pairwise registrations between the test image and individual training images. The training labels are then transferred to the test image and fused to compute the final segmentation of the test subject. Such label fusion methods have been shown to yield accurate segmentation, since the use of multiple registrations captures greater inter-subject anatomical variability and improves robustness against occasional registration failures. To the best of our knowledge, this manuscript presents the first comprehensive probabilistic framework that rigorously motivates label fusion as a segmentation approach. The proposed framework allows us to compare different label fusion algorithms theoretically and practically. In particular, recent label fusion or multiatlas segmentation algorithms are interpreted as special cases of our framework. We conduct two sets of experiments to validate the proposed methods. In the first set of experiments, we use 39 brain MRI scans—with manually segmented white matter, cerebral cortex, ventricles and subcortical structures—to compare different label fusion algorithms and the widely-used FreeSurfer whole-brain segmentation tool. Our results indicate that the proposed framework yields more accurate segmentation than FreeSurfer and previous label fusion algorithms. In a second experiment, we use brain MRI scans of 282 subjects to demonstrate that the proposed segmentation tool is sufficiently sensitive to robustly detect hippocampal volume changes in a study of aging and Alzheimer’s Disease.

I. Introduction

This paper investigates a probabilistic modeling framework to develop automatic segmentation tools that delineate anatomical regions of interest in a novel medical image scan. The objective is to learn a segmentation protocol from a collection of training images that have been manually labeled by an expert. This protocol is then employed by the algorithm to automatically segment a new (test) image. Such supervised segmentation tools are commonly used in many medical imaging applications, including surgical planning [27] and the study of disease progression, aging or healthy development [23], [50], [74]. As an application domain, this paper focuses on magnetic resonance (MR) imaging of the brain. However, most of the ideas we discuss here can be easily extended to other modalities and applications, particularly with the recent development of fast algorithms for pairwise registration in other imaging domains [71], [73]. We will thus consider the problem of segmenting the MRI volume scan of a novel subject, based on other subjects’ MRI scans that have been delineated by an expert.

Early MR segmentation algorithms mainly dealt with the problem of tissue classification, where local image intensity profiles contain a significant amount of the relevant information [10], [15], [41], [68]. A detailed parcellation of the brain anatomy into structurally or functionally defined regions of interest (ROI) typically requires supervision, commonly in the form of labeled training data, since the local appearance of most such structures is not easily distinguishable [23], [30], [40], [51], [72]. The training data is commonly obtained via a time-consuming and/or expensive procedure such as manual delineation, histology or functional localization experiments [22], [48], [60]. Automating the painstaking procedure of labeling improves the reliability and repeatability of the study, while allowing for the analysis of large pools of subjects.

One of the simplest ways to automatically segment an image using a single training dataset is to perform a nonrigid, dense registration between the labeled image and test image. The resulting warp can then be used to map the training labels onto the coordinates of the test image [16], [26], [31], [43]. The quality of such a registration-based approach is limited by the accuracy of the pairwise registration procedure and the anatomical similarity between the labeled and test subjects.

To reduce the bias due to the labeled subject and to model anatomical variability, multiple subjects can be employed in the training phase. A common method is to use a parametric model to summarize the training data in a common coordinate system [8], [24], [29], [36], [48], [62]–[64], [72]. In this approach the training data are co-registered to compute probability maps that encode the prior probability of observing a particular label at each point in the common (atlas) coordinates. The test subject is then normalized to the atlas coordinates through a pairwise registration with a template image that represents the average subject. This registration can be completed as a preprocessing step, or can be unified with the segmentation procedure, as in [8], [18], [48], [72]. Once the test subject is spatially normalized, one can use a variety of models of shape and appearance to devise a segmentation algorithm. Traditionally, generative models have been popular, where simple conditionally independent Gaussian models are used for appearance [24], [48]. More sophisticated shape models that encourage certain topological properties have also been proposed to improve segmentation quality [47], [62], [70].

Rather than relying on a single atlas coordinate system, an alternative strategy is to register each training subject to the test subject separately. Similar to the registration-based segmentation approach, these pairwise warps are then used to transfer the training labels into the coordinate system of the test subject. Given these transferred labels, segmentation has been commonly formulated as a label fusion problem [30], [42], [51]–[53], [58]. Intriguingly, Majority Voting, which is probably the simplest label fusion method, has been demonstrated to yield powerful segmentation tools [1], [30], [51], [52]. Empirical results in these studies suggest that errors in the manual labeling and registration procedures are reduced during label fusion, resulting in accurate segmentation. Recent work has shown that weighted averaging can be used to improve segmentation quality [6], [32], [53], [58]. The underlying intuition is that training subjects more similar to the test subject should carry more weight during label fusion. The practical advantages of various strategies based on this idea have recently been demonstrated [6]. These strategies include using the whole image to determine a single, global weight for each training subject [6], employing local image intensities for locally adapting the weights [6], [32] and using estimates of classifier performance in determining a weight for each label and training subject [53], [67]. To the best of our knowledge, however, none of the label fusion segmentation methods are derived within a probabilistic framework that explicitly models the relationship between the training data and test subject.

Label fusion methods offer two main advantages: 1) across-subject anatomical variability is better captured than in a single atlas, which can be viewed as a parametric model that typically uses single mode distributions (e.g., Gaussian) to encode anatomical appearance, and 2) multiple registrations improve robustness against occasional registration failures. The main drawback is the computational burden introduced by the multiple registrations and information fusion from the entire training data. To alleviate this problem, various heuristics for selecting only a subset of the training subjects for use in registration and label fusion have been proposed [1]. In the atlas construction literature, there has also been recent work on estimating multiple templates from a set of images [2], [11], [54], [66]. We believe that this approach can also be used to reduce the computational cost of label fusion by summarizing the training data. We leave this issue for future work.

The central contribution of this paper is to propose and investigate a generative model that leads to label fusion style image segmentation algorithms. Within the proposed framework, we derive several methods that combine transferred training labels into a single segmentation estimate. Using a dataset of 39 brain MRI scans and corresponding label maps obtained from an expert, we experimentally compare these segmentation algorithms. Additionally, we compare against other benchmarks including FreeSurfer’s whole brain segmentation tool, which has been widely used in a large number of studies [69], and STAPLE [67], a method that combines multiple segmentation estimates based on a probabilistic performance model. Our results suggest that the proposed framework yields accurate and robust segmentation tools that can be employed on large multisubject datasets. In a second experiment, we used one of the proposed segmentation algorithms to compute hippocampal volumes in MRI scans of 282 subjects. A comparison of these measurements across clinical groups indicate that the proposed algorithm is sufficiently sensitive to robustly detect hippocampal volume differences associated with aging and early Alzheimer’s Disease.

The generative model described in this paper is an extension of the preliminary ideas we presented in recent conference papers [55], [57]. The present paper offers detailed derivations, discussions and experiments that were not contained in those papers. The remainder of the paper is organized as follows. Sections II and III present the generative model and its instantiation, respectively. In Section IV, we develop several label fusion style segmentation algorithms based on the proposed generative model. Section V presents empirical results. In Section VI, we discuss the contributions of the paper along with the drawbacks of the proposed algorithms, while pointing to future research directions. Section VII concludes with a summary.

II. Generative Model

In this section, we present the probabilistic model that forms the core of this paper. We use {Ĩ_n} to denote training images with corresponding label maps {L̃_n}, n = 1, …, N. We assume the label maps take discrete values from 1 to (including a “background” or “unknown” label) at each spatial location. While the training images are defined on a discrete grid, we treat them as spatially continuous functions on ℝ³ by assuming a suitable interpolator. Let Ω ⊂ ℝ³ be a finite grid where the test subject is defined. We denote Φ_n: Ω ↦ ℝ³ to be the spatial mapping (warp) from the test subject coordinates to the coordinates of the nth training subject. For simplicity, we assume that {Φ_n} have been precomputed using a pairwise registration procedure, such as the one described in Appendix A. This assumption allows us to shorthand {Ĩ_n, Φ_n} and {L̃_n, Φ_n} with I_n and L_n, respectively, where we drop ~ to indicate that we know the transformation Φ_n that maps the training data into the coordinates of the test subject.

The goal of segmentation is to estimate the label map L associated with the test image I. This can be achieved via maximum-a-posteriori (MAP) estimation

$$\begin{array}{l}\widehat{L}=\underset{L}{\text{argmax}}p(L\mid I;\{{\stackrel{\sim }{L}}_n,{\stackrel{\sim }{I}}_n,{\mathrm{\Phi }}_n\})\\ =\underset{L}{\text{argmax}}p(L,I;\{L_n,I_n\})\end{array}$$ (1)

where p(L, I; {L_n, I_n}) denotes the joint probability of the label map and image given the training data.

Instead of using a parametric model for p(L, I; {L_n, I_n}), we employ a nonparametric estimator, which is an explicit function of the entire training data, not a statistical summary of it, as shown in Fig. 1. The model assumes that the test subject is generated from one or more training subjects, the index or indices of which are unknown. This modeling strategy is parallel to Parzen window density estimators, where the density estimate can be viewed as a mixture distribution over the entire training data, and each new sample is associated with a single training sample, the index of which is unknown and thus is marginalized over. In dealing with images, we may want to allow for this membership index to vary spatially. Therefore we introduce M: Ω ↦ {1, … N} to denote the latent random field that specifies for each voxel in the test image I, the (membership) index of the training image I_n it was generated from.

Fig. 1.

Graphical model that depicts the relationship between the variables. Squares indicate nonrandom parameters, circles indicate random variables. Replications are illustrated with plates (bounding L(x) and I(x)). The |Ω| in the corner of the plate indicates the variables inside are replicated that many times (i.e., once for each voxel), and thus are conditionally independent. Shaded variables are observed.

In the following, we make the assumption that the image intensity values I(x) and labels L(x) at each voxel are conditionally independent (as illustrated with a plate around these variables in Fig. 1), given the random field M, and the training data {L_n, I_n}. Furthermore, we assume that each voxel is generated from a single training subject indexed with M(x), i.e., p(L(x)|M; {L_n}) = p(L(x)|M(x); L_{M (x)}) and p(I(x)|M; {I_n}) = p{I(x)|M(x); I_M(x)), which we will shorthand with p_M(x)(L(x); L_M(x)) and p_M(x)(I(x); I_M(x)), respectively. We can thus construct the conditional probability of generating the test image and label map

$$p(L,I\mid M;\{L_n,I_n\})=\prod _{x\in \mathrm{\Omega }}p(L(x),I(x)\mid M(x);\{L_n,I_n\})$$ (2)

$$=\prod _{x\in \mathrm{\Omega }}{p}_{M(x)}\left(L(x);L_{M(x)}\right)p_{M(x)}\left(I(x);I_{M(x)}\right).$$ (3)

Given a prior on M, we can view the image I and label map L as generated from a mixture model

$$p(L,I;\{L_n,I_n\})=\sum _{M}p(M)p(L,I\mid M;\{L_n,I_n\})$$ (4)

where Σ_M denotes the marginalization over the unknown random field M. Substituting (3) into (4) yields the final cost function

$$\widehat{L}=\underset{L}{\text{argmax}}\sum _{M}p(M)\prod _{x\in \mathrm{\Omega }}{p}_{M(x)}(L(x);L_{M(x)})\times p_{M(x)}(I(x);I_{M(x)}).$$ (5)

The conditional independence assumption between the label map and image may seem simplistic at first. Yet conditional independence does not imply independence and the relationship between L and I is given by marginalizing over the unknown M as in (4). Therefore, our model implicitly includes complex dependencies between labels and intensity values. For instance p(I|L), a term commonly modeled explicitly in the segmentation literature can be expressed as

$$\begin{array}{l}p(I\mid L;\{L_n,I_n\})=\sum _{M}p(M\mid L;\{L_n,I_n\})p(I\mid L,M;\{L_n,I_n\})\\ =\sum _{M}p(M\mid L;\{L_n\})p(I\mid M;\{I_n\}).\end{array}$$

Thus, given a model instantiation, the conditional intensity distribution of a particular label at a location of interest can be estimated by examining the training subjects that exhibit that label in the proximity of the location of interest. This is exactly what atlas-based segmentation algorithms do, which underscores the similarity between the proposed probabilistic model and parametric models used in the literature. But unlike atlas-based methods that use a parametric model for p(I|L), the proposed framework explicitly employs the entire training data set.

III. Model Instantiation

This section presents the specific instantiations of the individual terms in (5) that we use in this work to derive segmentation algorithms.

A. Image Likelihood

We adopt a Gaussian distribution with a stationary variance σ² as the image likelihood term

$$p_n(I(x);I_n)=\frac{1}{\sqrt{2\pi {\sigma }^2}}exp\left[-\frac{1}{2{\sigma }^2}{\left(I(x)-{\stackrel{\sim }{I}}_n({\mathrm{\Phi }}_n(x))\right)}^2\right].$$ (6)

For σ → ∞, (6) reduces to an improper distribution p_n(I(x); I_n) ∝ C, where C is a constant. As we discuss in Section IV-B, this simple model leads to the Majority Voting strategy in label fusion, whereas for a finite σ², (6) yields a weighted averaging strategy.

B. Label Prior

In this work, we investigate two representations to define the label prior term p_n(L(x); L_n). One representation uses the logarithm of odds (LogOdds) model based on the signed distance transform [49]. Let ${\stackrel{\sim }{D}}_n^l$ denote the signed distance transform of label l in training subject n (in the native coordinates), which is assumed to be positive inside the structure of interest. We define the label prior as

$$p_n(L(x)=l;L_n)=\frac{1}{Z_{n,\rho }(x)}exp\left(\rho {\stackrel{\sim }{D}}_n^l({\mathrm{\Phi }}_n(x))\right)$$ (7)

where ρ > 0 is the slope constant, $Z_{n,\rho }(x)={\sum }_{l=1}^{\mathcal{L}}exp(\rho {\stackrel{\sim }{D}}_n^l({\mathrm{\Phi }}_n(x)))$ is the partition function, and is the total number of labels including the background label. The prior p_n(L(x) = l; L_n) encodes the conditional probability of observing label l at voxel x ∈ Ω of the test image, given that it was generated from the nth training image.

The second representation, commonly used in the literature, employs the probability vector image of L̃_n(x): each voxel is a length- binary vector with the lth entry equal to 1 if L̃_n(x) = 1 and 0 otherwise. To define the label prior p_n(·; L_n), the transformation Φ_n is applied to the probability vector image of L̃_n(x). In this method, non-grid locations need to be interpolated using a suitable method (e.g., trilinear or nearest neighbor) that ensures positive and normalized probability values. In general, it is well known that trilinear interpolation yields better segmentation results than nearest neighbor interpolation [51], [55]. The LogOdds model of (7) has the advantage of yielding nonzero probabilities everywhere, which makes the use of the logarithm of the probability numerically more stable. As discussed in our experiments presented in Section V-A, we find that the LogOdds model produces more accurate results.

C. Membership Prior

The latent random field M : Ω ↦ {1, …, N} encodes the local association between the test image and training data. We employ a Markov random field (MRF) prior on M

$$p(M)=\frac{1}{Z_{\beta }}\prod _{x\in \mathrm{\Omega }}exp\left(\beta \sum _{y\in {\mathcal{N}}_x}\delta (M(x),M(y))\right)$$ (8)

where β ≥ 0 is a scalar parameter, is a spatial neighborhood of voxel x, Z_β, is the partition function that only depends on β, and δ is the Kronecker delta. This particular type of MRF is often referred to as the Potts model. In our implementation, includes the immediate six neighbors of each voxel. Similar models have been used in the segmentation literature [64], [72], mainly as priors on label maps to encourage the spatial relationships of labels observed in the training data. In contrast, we use the MRF prior to (indirectly) pool local intensity information from within a neighborhood in determining the association between the test subject and the training data. Here we adopt a simple form of the MRF that does not include singleton and/or spatially varying terms. This is unlike the common usage of MRFs in the segmentation literature where the label prior typically varies spatially.

The parameter β influences the average size of the local patches of the test subject that are generated from a particular training subject. In this work, we consider three settings of the parameter β. With β = 0, the model assumes that each test image voxel is generated from the training subjects with equal probability and that the membership is voxel-wise independent. β → +∞ forces the membership of all voxels to be the same and corresponds to assuming that the whole test subject is generated from a single unknown training subject, drawn from a uniform prior. A positive, finite β encourages local patches of voxels to have the same membership.

IV. Label Fusion Based Image Segmentation

In this section, we derive several label fusion style image segmentation algorithms based on the model and MAP formulation described above. These algorithms correspond to variations in the image likelihood, label prior and membership prior models described in Section III.

A. Local Weighted Voting

Let us assume β = 0, which, thanks to the adopted simple MRF form, implies that M(x) is independent and identically distributed according to a uniform distribution over all labels for all x ∈ Ω

$$p(M)=\frac{1}{N^{\mid \mathrm{\Omega }\mid }}$$ (9)

where |Ω| is the cardinality of the image domain (the number of voxels). Using the image likelihood term of (6), the segmentation problem in (5) reduces to

$$\widehat{L}(x)=\underset{l\in \{1,\dots ,\mathcal{L}\}}{\text{argmax}}\sum _{n=1}^N{p}_n(L(x)=l;L_n)p_n(I(x);I_n).$$ (10)

This optimization problem can be solved by simply comparing numbers at each voxel: the fused label of each voxel is computed via a local weighted (fuzzy) voting strategy. The local image likelihood terms serve as weights and the label prior values serve as votes. Therefore, at each voxel, training images that are more similar to the test image at the voxel after registration are weighted more. Interestingly, a similar approach was recently proposed in the context of CT cardiac segmentation by Isgum et al. [32] where the transferred training labels are fused in a weighted fashion. The heuristic weights proposed in that paper have a different form however and are spatially smoothed with a Gaussian filter to pool local neighborhood information. In Section IV-D, we discuss a more principled approach to aggregate statistical information from neighboring voxels into the weighted label fusion procedure.

B. Majority Voting

Majority voting, which has been widely used as a label fusion method [30], [51] can be derived as a special case of Local Weighted Voting. The key modeling assumption is to set σ → ∞ in the image likelihood term, effectively using an improper distribution p_n(I(x); I_n) ∝ C and assigning equal weight to all training subjects, which reduces (10) to

$$\widehat{L}(x)=\underset{l\in \{1,\dots ,\mathcal{L}\}}{\text{argmax}}\sum _{n=1}^N{p}_n(L(x)=l;L_n).$$ (11)

If we use the probability vector image of L̃_n(x) to define the label prior, we arrive at the traditional Majority Voting algorithm where each training image casts a single, unit vote, with no regards to the similarity between the training image and the test image. If one uses nearest neighbor interpolation, each vote corresponds to one particular label [30], whereas tri-linear interpolation yields a fuzzy voting strategy with each vote potentially spread over multiple labels [51].

C. Global Weighted Fusion

Here, we consider β → +∞. As we now show, this results in an algorithm where, at each voxel, training images that are globally more similar to the test image after registration are weighted more. With β → +∞, the membership prior defined in (8) only takes nonzero values if membership values at all voxels are equal, i.e.,

$$p(M)=\{\begin{array}{ll}\frac{1}{N},\hfill & \text{if}M(x)=j\forall x\in \mathrm{\Omega },\exists j\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}$$ (12)

Thus, (4) is equivalent to a mixture model where the test subject is assumed to be generated from a single, unknown training subject

$$p(L,I;\{L_n,I_n\})=\frac{1}{N}\sum _{n=1}^N{p}_n(L;L_n)p_n(I;I_n).$$ (13)

The segmentation problem in (5) reduces to

$$\widehat{L}(x)=\underset{L}{\text{argmax}}\sum _{n=1}^N\prod _{x\in \mathrm{\Omega }}{p}_n(L(x);L_n)p_n(I(x);I_n).$$ (14)

Equation (14) cannot be solved in closed form. However, an efficient solution to this MAP formulation can be obtained via expectation maximization (EM) [20]. Appendix B contains the derivations of the algorithm. Here, we present the summary.

1. E-Step: Let L̂⁽ⁱ⁻¹⁾(x) denote the segmentation estimate in the (i−1)th iteration. The E-step updates the posterior of the membership associated with each training image

   $$m_n^{(i)}\propto \prod _{x\in \mathrm{\Omega }}{p}_n(I(x);I_n)p_n({\widehat{L}}^{(i-1)}(x);L_n)$$ (15)

   where $m_n^{(i)}$ is the current estimate of the posterior probability that the test image was generated from the nth training image. Therefore ${\sum }_n{m}_n^{(i)}=1$. The E-step in (15) determines a single, global membership for each training image, based on all voxels.

2. M-Step: The M-step updates the segmentation estimate (16)

   $${\widehat{L}}^{(i)}(x)=\underset{l\in \{1,\dots ,\mathcal{L}\}}{\text{argmax}}\sum _{n=1}^N{m}_n^{(i)}log{p}_n(L(x)=l;L_n)$$ (16)

   $$=\underset{l\in \{1,\dots ,\mathcal{L}\}}{\text{argmax}}\sum _{n=1}^N{m}_n^{(i)}{\stackrel{\sim }{D}}_n^l({\mathrm{\Phi }}_n(x))$$ (17)

   where ${\stackrel{\sim }{D}}_n^l$ denotes the signed distance transform of label l in training subject n. We note that (17) uses the LogOdds model of (7). The M-step in (16) performs an independent optimization at each voxel x ∈ Ω; it determines the mode of a length- vector, where is the number of labels. This vector is computed as a weighted average of log label priors (i.e., signed distance transforms, ${\stackrel{\sim }{D}}_n^l$). The global memberships computed in the previous E-step serve as weights and depend on the current label map estimate. The algorithm iterates between the weighted averaging of the M-step in (16) and the global weight computation of the E-step in (15). We initialize the EM algorithm with $m_n^{(1)}\propto {\prod }_{x\in \mathrm{\Omega }}{p}_n(I(x);I_n)$ and terminate the algorithm when the average change in the membership weights is less than a small threshold, e.g., 0.01. We found that our EM algorithm typically converges in fewer than 10 iterations.

D. Semi-Local Weighted Fusion

Finally, we consider a finite, positive β that results in an algorithm, where at each voxel, training images that are more similar to the test image in a local neighborhood of the voxel after registration are weighted more. Because the MRF prior couples neighboring voxels, the exact marginalization in (5) becomes computationally intractable.

An efficient approximate solution can be obtained via variational EM (also referred to as variational mean field) [33]. Variational EM uses a fully factorized distribution q over the random field M

$$q(M)=\prod _{x\in \mathrm{\Omega }}{q}_x(M(x))$$ (18)

to approximate the posterior distribution p(M|L̂, I; {L_n, I_n}) of the random field M given the segmentation estimate L̂, test image and the training data. q_x(n) can thus be viewed as the approximate posterior probability that voxel x was generated from the n-th training image. The algorithm alternates between updating the approximate posterior q (the E-step) and the segmentation L̂ (M-step). Appendix C includes the details of the derivation. Here, we present the summary.

1. E-Step: Let L̂⁽ⁱ⁻¹⁾(x) denote the segmentation estimate in the (i − 1)th iteration. The approximate posterior q is the solution of the following fixed-point equation:

   $$q_x^{(i)}(M(x))\propto p_{M(x)}\left(I(x);I_{M(x)}\right)\times p_{M(x)}\left({\widehat{L}}^{(i-1)}(x);L_{M(x)}\right)exp\left(\beta \sum _{y\in {\mathcal{N}}_x}{q}_y^{(i)}(M(x))\right)$$ (19)

   where ${\sum }_n{q}_x^{(i)}(n)=1$. We compute (19) repeatedly, to iteratively solve for q⁽ⁱ⁾.

2. M-Step: The M-step updates the segmentation estimate L̂ as follows:

   $${\widehat{L}}^{(i)}(x)=\underset{l\in \{1,\dots \mathcal{L}\}}{\text{argmax}}\sum _{n=1}^N{q}_x^{(i)}(n)log{p}_n(L(x)=l;L_n)$$ (20)

   $$=\underset{l\in \{1,\dots \mathcal{L}\}}{\text{argmax}}\sum _{n=1}^N{q}_x^{(i)}(n){\stackrel{\sim }{D}}_n^l({\mathrm{\Phi }}_n(x)).$$ (21)

   Equation (21) uses the LogOdds model of (7) and is computed independently for each voxel. Optimizing (21) entails determining the mode of a length- vector, which is a weighted average of the log label priors corresponding to each training subject. The current approximate posterior q⁽ⁱ⁾ for each training subject serve as weights.

The variational EM algorithm consists of two levels of iterations: the inner loop that repeatedly computes (19) in the E-step and the outer loop that alternates between the E- and M-steps, until convergence. In the inner loop, at each iteration all q_x’s are updated using (19) and the neighboring values {q_y : y ∈ } from the previous iteration. Once this inner loop converges, the algorithm updates the segmentation using (21). To determine the convergence of the outer loop, one can monitor the change in the segmentation estimate. In practice, we terminate the algorithm when less than a predetermined fraction, e.g., 0.01% of voxels change their segmentation estimate from one iteration to the next. Typically convergence is achieved in fewer than 10 iterations.

V. Experiments

In this section, we present two sets of experiments. In the first experiment we compare automatic segmentation results against manual delineations to objectively quantify the accuracy of segmentation algorithms. The second experiment employs a separate collection of brain MRI scans from 282 subjects to demonstrate that hippocampal volume measurements obtained using the proposed label fusion framework can detect subtle volume changes associated with aging and Alzheimer’s disease.

A. Experiment I: Comparison Against Manual Segmentation

The first set of experiments employs 39 brain MRI scans and corresponding manual delineations of nine anatomical regions of interest (ROI) in two hemispheres. The images were selected from a large data set, including an Alzheimer’s cohort, the recruitment of which is described elsewhere [19], [35], [38]. The 39 subjects were selected to span a wide age range and reflect a substantial anatomical variation due to dementia pathology. We note that these are the same subjects used to construct FreeSurfer’s released probabilistic segmentation atlas. Out of the 39 subjects, 28 were healthy and 11 were patients with questionable (N = 5, Clinical Dementia Rating 0.5) or probable Alzheimer’s (N = 6, CDR 1). Ten of the healthy subjects were young (less than 30 years), nine middle-aged (between 30 and 60 years), and nine old (older than 60 years). The MRI images are of dimensions 256 × 256 × 256, 1 mm isotropic voxels and were computed by averaging three or four scans. Each scan was a T1-weighted MP-RAGE, acquired on a 1.5 T Siemens Vision scanner. All scans were obtained in a single session. Acquisition details are as follows: TR 9.7 ms, TE 4.0 ms, TI 20 ms, Flip angle 10°. These high quality images were then gain-field corrected and skull-stripped. All the preprocessing steps were carried out using FreeSurfer tools [69]. The anatomical ROIs we used¹ are white matter (WM), cerebral cortex (CT), lateral ventricle (LV), hippocampus (HP), thalamus (TH), caudate (CA), putamen (PU), pallidum (PA), and amygdala (AM). The labeling protocol we employed was developed by the Center for Morphometric Analysis and has been published and validated elsewhere [13], [28], [37], [59]. An example segmentation obtained via the local weighted voting method of Section IV-A is visualized in Fig. 2.

Fig. 2.

A typical segmentation obtained with the local mixture model. 2D slices are shown for visualization only. All computations are done in 3D.

We use a volume overlap measure known as the Dice score [21] to quantify the quality of automatic segmentations. Given an automatic segmentation L̂ and the corresponding manual segmentation L, the Dice score of label l is defined as

$$\text{Dice}(l;\widehat{L},L)=2\frac{\mid \{x\in \mathrm{\Omega }\mid L(x)=l\&\widehat{L}(x)=l\}\mid }{\mid \{x\in \mathrm{\Omega }\mid L(x)=l\}\mid +\mid \{x\in \mathrm{\Omega }\mid \widehat{L}(x)=l\}\mid }$$ (22)

where |·| denotes set cardinality. The Dice score varies between 0 and 1, with 1 indicating a perfect agreement between the two segmentations.

1. Setting the Free Parameters Through Training: The proposed label fusion algorithms have two stages, registration and label fusion, each with several input parameters. To set these parameters properly, we initially performed training on nine out of the 39 subjects. These nine subjects were then only used as training subjects during the testing phase. Therefore, all results reported in this paper are on the remaining 30 subjects and reflect generalization performance.

   The registration stage has two independent parameters (as described in Appendix A): γ controls the step size in the Gauss–Newton optimization and α determines the smoothness of the final warp. We registered 20 random pairs of the nine training subjects for a range of values of γ and α. For each pair of subjects, we measured pairwise overlap by computing the Dice score between the warped manual labels of the “moving” subject and the manual labels of the “fixed” subject. We then selected (γ^*, α^*) that resulted in the best registration quality as measured by the average pairwise label overlap.

   The label fusion stage also has several independent parameters, depending on the method used. These include the standard deviation σ of the image likelihood in (6), the slope ρ of the distance transform used to compute the label prior in (7), and the Markov weight β which is nonzero for the semi-local method in Section IV-D and controls the average size of the image patches associated with the same training subject.

   To determine ρ, we performed nine leave-one-out segmentations on the training subjects using the Majority Voting method of Section IV-B and label prior model of (7) for a range of ρ values. The value that achieved the best segmentation accuracy was ρ^* = 1. We employed Local Weighted Voting (Section IV-A) and Global Weighted Fusion (Section IV-C) to determine a local and global optimal value for σ^* (10 and 30), respectively. The optimal standard deviation for σ^* the local model was then used to determine the optimal value for β^* (0.75) for the semi-local model.

   We performed leave-one-out cross-validation on the 30 test subjects using these optimal parameters. For each test subject, all remaining 38 subjects were treated as training subjects.

2. Comparison of Label Prior Models: Using the Majority Voting method (Section IV-B), we compare three different label prior models (Section III-B): the LogOdds (based on the signed distance transform) model of (7) and two instantiations of the common approach that interpolates the vector image L̃_n(x) of indicator probability vectors, based on nearest neighbor interpolation (e.g., [30]) or tri-linear interpolation (e.g., [51]). Fig. 3 shows a box-plot of Dice scores for these three different models and all the ROIs. These results indicate that the LogOdds representation provides a significantly better label prior for the label fusion framework. This finding is in agreement with the main point of [52], where the authors propose to employ signed distance transforms when “fusing” labels, essentially arguing that this representation is more suitable for averaging complex shapes such as the cortex. In the remainder of the paper, we use the LogOdds model based on the signed distance transform to compute the label prior. Note that the corresponding majority voting procedure is different from simply averaging the signed distance transforms as proposed in [52] since the signed distance transforms are exponentiated [see (7) and (11)] and converted into probabilities before averaging. Interestingly, however, in Global and Semi-local Weighted Fusion, the algorithms apply a weighted averaging to the signed distance transforms directly at each iteration [see (17) and (21)].

3. Comparison of Label Fusion Methods and Benchmarks: In this section we provide a comparison between the three weighted label fusion algorithms we derived in our framework and four benchmarks.

   The first benchmark is the whole-brain segmentation tool available in the FreeSurfer software package [69]. The FreeSurfer segmentation tool employs a unified registration-segmentation procedure that models across-scanner intensity variation [24], [29]. We consider this a state-of-the-art whole-brain segmentation tool since numerous imaging studies across multiple centers have shown FreeSurfer’s robustness and accuracy as a segmentation tool. Furthermore, this is our only benchmark that does not rely on the preprocessing step of pairwise registering each training image with the test image. FreeSurfer’s segmentation tool uses a probabilistic atlas constructed from the training data. In our experiments, we constructed a separate leave-one-out atlas for each test subject based on the remaining 38 subjects.

   The second benchmark is the Majority Voting scheme based on the LogOdds prior, which is similar to the shape averaging method proposed in [52] and other voting based algorithms, e.g., [30], [51].

   Our third benchmark uses the STAPLE algorithm [67] to fuse the propagated labels. STAPLE was originally developed to combine manual tracings of the same subject from multiple raters and estimate the underlying “ground truth” labeling. In doing so, it ignores MRI intensity information and utilizes a probabilistic model that encodes and estimates the rater performance for each label. STAPLE employs Expectation Maximization to efficiently solve the estimation problem. One can also use STAPLE to combine the transferred labels from each training subject, as suggested in [53]. In our experiments we implemented a multilabel version of STAPLE to combine the transferred training labels.

   Our fourth benchmark is a modified version of majority voting that efficiently selects a subset of training subjects that are “closest” to the test subject to vote on the labels [1]. Various strategies to define the similarity between images have been proposed. In our experiments, we found that the following strategy gave us the best results. First, all training subjects were co-registered using an affine transformation model. The test subject was then normalized to this space by an affine registration with the group mean image. Next, we computed the sum of squared intensity differences (SSD) between each training subject and test subject within a predefined mask. Similar to [1], the mask was computed as the intersection of the foreground labels in all training subjects. Finally, the training subjects that had the smallest SSD were used for majority voting. In the results we present here, we fix the number of training subjects that were used to 10 and call the algorithm “Majority10.” Later in this section, we investigate the effects of varying the number of training subjects.

   Fig. 4 reports segmentation accuracy for benchmarks and the three weighted label fusion methods: Local Weighted Voting, Global Weighted Fusion, and Semi-Local Weighted Fusion. Table I provides the mean Dice scores averaged over all subjects and both hemispheres. Fig. 5 provides an overall comparison between the average Dice scores achieved by the algorithms.

   Semi-local Weighted Fusion yields the most accurate segmentations in all ROIs but the Cortex (CT). The difference with the benchmarks is statistically significant (p < 0.05, Bonferroni corrected) for all ROIs, except the CT for which FreeSurfer yields the best accuracy. Similarly, Local Weighted Voting yields statistically better segmentations than the benchmarks.

   On average, Local Weighted Voting and Semi-local Weighted Fusion yield better segmentations than Global Weighted Fusion, mainly due to the large improvement in the white matter, cerebral cortex and lateral ventricles, the segmentation of which clearly benefits from the additional use of local intensity information. A paired permutation test between the Local and Semi-local models reveals that in all ROIs, a statistically significant improvement is achieved with the MRF model that pools local intensity information (p < 0.05, Bonferroni corrected). Yet, as can be seen from Fig. 6, this improvement is overall quite modest: less than 0.14% per ROI.

   Majority Voting which has gained recent popularity [1], [30], [42], [51], performs significantly worse than the weighted label fusion methods. This result highlights the importance of incorporating image similarity information into the label fusion framework. We note, however, that the results we report for our Majority Voting implementation are lower than the ones reported in [30]. This might be due to differences in the data and/or registration algorithm. Specifically, normalized mutual information (NMI) was used as the registration cost function in [30]. Entropy-based measures such as NMI are known to yield more robust alignment results. We leave a careful analysis of this issue to future work.

   Majority10 performs slightly better than Majority Voting. The improvement is particularly significant in subcortical ROIs such as the caudate. STAPLE, an alternative weighted fusion strategy, also yields slightly better average segmentation accuracy than Majority Voting. STAPLE’s performance, however, is significantly worse than the three weighted label fusion algorithms derived based on the proposed probabilistic framework. Once again, this difference underscores the importance of employing the MRI intensity information in determining the weights for label fusion. FreeSurfer, which we consider to represent the state-of-the art atlas based segmentation, on average, yields better segmentation accuracy than our remaining benchmarks. Yet we stress that FreeSurfer integrates registration and segmentation, while the performance of the remaining benchmarks were limited by our choice of the pairwise registration preprocessing step.

   Fig. 7 illustrates the most common mistakes made by Global Weighted and Semi-local Weighted Fusion. Overall, we observe that Global Weighted Fusion tends to over-segment convoluted shapes such as the cortex. This is probably due to the difficulty of aligning such shapes. Majority voting has no way of recovering from such errors since it does not utilize image intensity information. Local Weighted Voting and Semi-local Weighted Fusion do not employ information on neighborhood structure, whereas atlas-based methods, such as FreeSurfer, do. For example, FreeSurfer uses a non-stationary, anisotropic Markov Random Field model to encode that the pallidum is more medial to the putamen and there is no white matter in between the two, i.e., they border each other. Our label fusion framework does not model such high-level topological information and thus may yield segmentations where anatomical relationships are violated.

4. The Effect of the Number of Training Subjects: In the previous section, for all the algorithms we employed a leave-one-out validation strategy, where for each test subject all remaining 38 subjects were treated as the training data. In this section, we explore how the accuracy results vary as one varies the number of training subjects. This point has received considerable attention in prior work, e.g., [1], [30]. We investigate two strategies: 1) randomly selecting a set of training subjects, 2) selecting the best training subjects that are globally most similar to the test subject. As described in the previous section, based on [1], [42], we implemented strategy 2 by computing the sum of squared differences between the intensity values of the test subject and each training subject in a predefined mask after affine registration. This strategy does not add a substantial computational burden to the algorithm, since the training subjects are co-registered offline and the affine normalization of the test subject takes negligible time compared with the pairwise registrations required for the label fusion stage. We compare the performance of two segmentation algorithms using these two strategies: Majority Voting and Local Weighted Voting. Fig. 8 reports the mean Dice score as a function of the number of training subjects. The mean Dice score is an average over all ROIs, all 30 test subjects, and at least 20 random trials (in strategy 1). The individual results for each ROI are qualitatively similar to this average plot and thus are not shown. For strategy 1, i.e., the random selection of the training subjects, average segmentation performance monotonically improves as a function of the number of training subjects for both Majority Voting and Local Weighted Voting. With Majority Voting, this monotonic behavior was also observed and modeled in [30]. The difference between the two label fusion algorithms is approximately constant. The rate of improvement seems to be flattening around mid-thirties, suggesting that we should not expect a significant improvement in performance with the addition of more subjects within the observed anatomical variation. For a fixed number of training subjects, selecting the globally most similar training subjects, i.e., strategy 2 improves the performance of both algorithms. With strategy 2, Majority Voting’s performance however is not monotonic and starts to decrease beyond 10. In agreement with the empirical observations in [1], as the number of training subjects approaches the total number of available training subjects in the database, the performance of the strategies for training set selection converges to the same level, the accuracy obtained by employing the whole dataset for training. This level of accuracy is higher for Local Weighted Voting, the performance of which monotonically increases with the number of training subjects for both strategies. Interestingly, with strategy 2, the performance of Local Weighted Voting flattens around 15 training subjects. Based on this observation, we conclude that we can use strategy 2 to speed up Local Weighted Voting substantially with minimal effect on segmentation accuracy.

5. The Effect of the MRF Prior: To investigate the effect of the MRF membership prior we applied the Semi-local Weighted Fusion method with four different values of β = {0.5, 0.75, 1.0, 1.25} to the 30 test subjects. Note that during the training phase, we established β = 0.75 as optimal. Fig. 9 reports the average Dice scores for Semi-local Weighted Fusion with these β values, Global Weighted Fusion, which corresponds to β → ∞, and Local Weighted Voting, which corresponds to β = 0.

   In four out of the nine ROIs (WM, CT, LV, TH) Semi-local Weighted Fusion with β = 0.075 achieves the best results. In the remaining ROIs, β = 0.75 yields one of the best. This underscores the importance of pooling local intensity information in the label fusion procedure and the success of our training phase. The improvement over Global Weighted Fusion is particularly pronounced for the white matter, cerebral cortex and lateral ventricle, which are essentially defined by tissue intensity. For putamen (PU) and pallidum (PA), Global Weighted Fusion achieves one of the best results. This suggests that the segmentation of these two ROIs has little to gain by employing local intensity information. The gap between the Semi-local (β = 1.25) and Global Weighted Fusion in these two ROIs is probably because variational EM fails to find good optima for these regions whereas Expectation Maximization is better at exploring the space of solutions in the simpler case of global memberships. Furthermore, this difference can be due to the fact that the image intensities in the putamen and pallidum are similar to neighboring structures and thus local intensity information is less useful in segmenting these ROIs.

6. Runtime: Table II lists the average run-times for the seven algorithms compared above. Majority10 and FreeSurfer are the fastest algorithms with less than 10 h of CPU time required for each test subject. Majority10 uses only 10 training subjects, which are globally most similar to the test subject as measured by the sum of squared differences after affine-normalization. The initial training subject selection stage takes about an hour. The second stage, i.e., Majority Voting with 10 training subjects takes about a quarter of what Majority Voting takes with all 38 training subjects. FreeSurfer, on the other hand, employs a parametric atlas and needs to compute only a single registration. The remaining algorithms take more than 20 h of CPU time on a modern machine (Intel Xeon 3 GHz with a 32 GB RAM), most of which is dedicated to the many registrations of the test image with the training data. Local Weighted Voting and Majority Voting require minimal computation time once the registrations are complete, since they simply perform voxelwise averaging. The remaining three algorithms (STAPLE, Global and Semi-local Weighted Fusion) employ iterative optimization methods (EM, EM and variational EM, respectively) and require longer run-times. It is important to note that these run times can be reduced substantially using the same preselection strategy as Majority 10. In particular, our experiments with Local Weighted Voting suggest that we can lower the run time of this method by at least a half with almost no reduction in accuracy.

Fig. 3.

Dice scores obtained using Majority Voting and various label prior models: Nearest Neighbor (red), Tri-linear (green), and LogOdds (blue). Dice scores for the two hemispheres were averaged. On each box, the central mark is the median, the edges of the box are the 25th and 75th percentiles. The whiskers extend to 2.7 standard deviations around the mean, and outliers are marked individually as a “*.”

Fig. 4.

Dice scores for all methods (top: left hemisphere, bottom: right hemisphere): FreeSurfer (red), Majority Voting (black), STAPLE (light green), Majority10 (dark green), Global Weighted Fusion (light blue), Local Weighted Voting (dark blue), Semi-local Weighted Fusion (purple). On each box, the central mark is the median, the edges of the box are the 25th and 75th percentiles. The whiskers extend to 2.7 standard deviations around the mean, and outliers are marked individually as a “*.”

TABLE I.

Comparison of Average Dice Scores. Boldface Font Indicates Best Scores for Each ROI That Have Achieved Statistical Significance (Paired Permutation Test: p < 0.05, Bonferroni Corrected). ITALIC FONT Indicates Best Scores That are Statistically Equivalent. As a Reference, Last Row Lists Approximate Average Volumes

	WM	CT	LV	HP	TH	CA	PU	PA	AM
FreeSurfer	0.923	0.941	0.878	0.851	0.879	0.849	0.840	0.786	0.796
Majority Vote	0.873	0.760	0.853	0.821	0.891	0.819	0.876	0.822	0.799
STAPLE	0.884	0.774	0.857	0.807	0.889	0.827	0.881	0.821	0.792
Majority 10	0.861	0.750	0.869	0.827	0.897	0.848	0.886	0.829	0.806
Global Weighted Fusion	0.890	0.790	0.880	0.836	0.907	0.849	0.892	0.835	0.814
Local Weighted Fusion	0.943	0.871	0.912	0.864	0.907	0.865	0.891	0.833	0.818
Semi-local Weighted Fusion	0.949	0.884	0.914	0.868	0.908	0.871	0.894	0.836	0.822
Volumes (×103mm3)	450	448	25	7	14	7	10	3	3

Fig. 5.

Average Dice scores for each algorithm (FS: FreeSurfer, Majority: Majority Voting, STAPLE, Majority10, Global: Global Weighted Fusion, Local: Local Weighted Voting, and Semi-Local: Semi-local Weighted Fusion). Error bars show standard error. Each subject and ROI was treated as an independent sample with equal weight.

Fig. 6.

Average Dice differences: Semi-Local Weighted Fusion minus Local Weighted Voting. Overall, Semi-Local Weighted Fusion achieves better segmentation. Error bars show standard error.

Fig. 7.

The segmentations of the subject that Semi-local Weighted Fusion performed the worst on. Left to right: FreeSurfer, Global and Semi-local Weighted Fusion. Common mistakes (indicated by arrows): (A) Global Weighted Fusion tends to over-segment complex shapes like the cortex. (B) Semi-local Weighted Fusion does not encode topological information, as FreeSurfer does. Hence it may assign an “unknown” or “background” label (white) in between the pallidum (blue), putamen (pink), and white matter (green).

Fig. 8.

The average Dice score for Majority Voting (Majority) and Local Weighted Voting (Local) as a function of the number of training subjects. We consider two strategies to select the training subjects: (1) randomly selecting a set of training subjects (Rand), (2) selecting the best training subjects that are globally most similar to the test subject (Best). The average Dice score reaches 83.9% for Majority Voting and 87.8% for Local Weighted Voting, when all 38 subjects are used.

Fig. 9.

Average Dice scores for different β values in the MRF membership prior of (8). Error bars show standard error.

TABLE II.

Average Run-Time (in CPU Hours on a Modern Machine -Intel Xeon 3 GHz With a 32 GB RAM-) to Segment One Test Subject. The Time Taken up by Registration is Listed in Parentheses. Global: Global Weighted Fusion, Local: Local Weighted Voting, Semi-Local: Semi-Local Weighted Fusion

FreeSurfer	Majority	STAPLE	Majority 10	Global	Local	Semi-local
10(9)	24 (23)	28 (23)	8(7)	32 (23)	24 (23)	40 (23)

B. Experiment II: Hippocampal Volumetry

In a second set of experiments, we aim to demonstrate that the proposed label fusion framework yields accurate volumetric measurements of the hippocampus. Hippocampal volume has been shown to correlate with aging and predict the onset of probable Alzheimer’s Disease [25], [34].

We use a FreeSurfer tool (mri_label_volume) [69] that computes volumetric measurements from a given label map to obtain hippocampal volume measurements. This tool uses a partial volume model for boundary voxels to accurately estimate the volume of a particular ROI. As an initial experiment, we compare hippocampal volume measurements obtained via automatic segmentations (FreeSurfer, Global Weighted Fusion, Local Weighted Voting, and Semi-local Weighted Fusion) against the ones computed from manual delineations in the 39 subjects used in the first experiment. Fig. 10(a) reports volume differences between the automatic and manual segmentations;

Fig. 10.

Hippocampal volume differences on the data from Experiment 1. On each box, the central mark is the median, the edges of the box are the 25th and 75th percentiles. The whiskers extend to 2.7 standard deviations around the mean. (a) Automatic minus Manual volumes. (b) Relative volume differences [(23)].

Fig. 10(b) shows relative volume difference values, defined as in [16]

$$\text{Volume}\text{Difference}(l;\widehat{L},L)=2\frac{\mid V(l,\widehat{L})-V(l,L)\mid }{V(l,\widehat{L})+V(l,L)}$$ (23)

where V(l, L) denotes the computed volume of label in label l map L. These results indicate that both Local Weighted Voting and Semi-local Weighted Fusion provide more accurate hippocampal volume measurements than Global Weighted Fusion and Majority Voting.

Since Local Weighted Fusion is much less computationally expensive than Semi-local Weighted Fusion, we choose to employ the former algorithm in the second part of the experiment, where we automatically segment brain MRI scans of 282 individuals. This data set excludes the 39 subjects used in training. The MRI images are of dimensions 256 × 256 × 256 with 1 mm isotropic voxels and were obtained using the same processing and acquisition protocol as the 39 images of the first experiment. The population included young (less than 30 years, N = 105), middle-aged (between 30 and 60 years, N = 30) and old subjects (older than 60 years N = 78) (see Fig. 11), in addition to patients suffering from very mild to mild Alzheimer’s Disease (AD) (N = 69). Among the patients, 48 subjects met the Clinical Dementia Rating (CDR) [44] criteria of questionable AD (CDR 0.5) and 21 subjects had probable AD (CDR 1). The CDR 0.5 subpopulation contained 32 women, whereas the probable AD group included 21 women. The groups did not differ in years of education. Detailed descriptions of the recruitment procedures and criteria for subject recruitment have been published elsewhere [19], [35], [38].

Fig. 11.

Age histogram of 282 subjects in Experiment 2.

Based on age and clinical data, we subdivided the 282 subjects into five groups. The first three groups contained healthy individuals of different age groups: young, middle-aged, and old. The fourth group included the questionable AD patients (CDR 0.5) and the fifth group included the probable AD (CDR 1.0) subpopulation.

Fig. 12 shows the average hippocampal volume measurements for these five groups. Volumetric reduction due to aging and AD can be seen from this figure. These findings are in agreement with known hippocampal volume changes in AD and aging [23] and demonstrate the use of the proposed label fusion method on a large pool of subjects, for which manual segmentation may not be practical.

Fig. 12.

Hippocampal volumes for five different groups in Experiment 2. Error bars indicate standard error across subjects. Stars indicate that the volume measurements in the present group are statistically significantly smaller than the measurements in the neighboring group to the left. (Unpaired, single-sided t-test. * p < 0.05, ** p < 0.01). (a) Left hippocampus. (b) Right hippocampus.

VI. Discussion

Our experiments demonstrate the accuracy and usefulness of the label fusion framework as a segmentation tool. The proposed framework yields better accuracy than current state-of-the-art atlas-based segmentation algorithms, such as FreeSurfer.

The proposed framework should be viewed as an initial attempt to generalize segmentation algorithms based on label fusion, or a multi-atlas approach, which have recently shown promise and gained popularity with hardware advancements and developments of fast registration algorithms. In this paper, we investigated several modeling assumptions and derived four different instantiations of label fusion, one of which is the popular Majority Voting.

Majority Voting simply determines the most frequent label at each voxel, where each training subject gets an equal vote. Yet, recent work suggests that incorporating the similarity between the test image and training subjects can improve segmentation quality. For example, [1] employs a subset of training subjects that are close in age to the test subject. Alternative strategies include using an image-based measure to quantify anatomical similarity, either at a local or global level. This similarity can then weigh the label votes during fusion, where more similar training subjects are given a larger weight.

Our theoretical development based on the proposed nonparametric probabilistic model yields three such algorithms, which solve the same problem for different settings of a single model parameter β. This parameter controls the interactions between neighboring voxels in the Markov prior we construct on the latent membership random field encoding the (unknown) association between the test subject and training data. Smaller β values allow for this association to vary more locally. Specifically, β = 0 treats each voxel independently, whereas β → ∞ corresponds to assuming a single association for the whole image. A finite, nonzero β encourages local patches of voxels to have the same membership.

These three cases are solved with different inference algorithms. The most efficient case corresponds to β = 0, where the global optimum can be computed via simple voxelwise counting. The other two cases are solved with more expensive iterative optimization methods, such as Expectation Maximization and Variational EM. Exact inference for the finite, nonzero β case is intractable, yet our experiments suggest that approximate numerical solutions yield good segmentation accuracy.

The development of the proposed framework makes several simplifying assumptions. In the following, we discuss a number of directions that can be explored to relax these assumptions. We consider these as important avenues for future research, which promise to improve the performance of label fusion style segmentation.

1. In the graphical model of Fig. 1, we made the convenient assumption that the transformations {Φ_n} are known and solved for these in a preprocessing pairwise registration step (see Appendix A). Ideally, however, one would like to integrate over all possible transformations, which has a prohibitively high computational cost. Recent work attempted to approximate this integration for a single registration [3], [40]. A more practical approach is to compute the registrations jointly with the segmentations, cf. [8], [72]. Here, we avoided this particular route, since the multiple registrations performed between the test subject and training data were already computationally challenging.

2. The simple additive Gaussian noise model presented in Section III-A has two crucial consequences: 1) the registration cost function is a sum of squared intensity differences, and 2) in weighted label fusion, the weights are a function of sum of squared intensity differences, i.e., anatomical similarity is measured based on squared differences of intensity values. This model makes the algorithm sensitive to intensity variations due to imaging artifacts. Thus, the presented algorithms are only suitable for intensity-normalized images. An alternative strategy is to employ a more sophisticated image likelihood model that would motivate information theoretic similarity measures, such as mutual information.

3. The main drawback of label fusion style algorithms is the computational complexity introduced by the multiple pairwise registrations and the manipulation of the entire training data. Traditional atlas-based segmentation approaches avoid this problem by using parametric models of anatomical variation in a single coordinate system. In recent work, we used a mixture modeling strategy, called iCluster, to model anatomical heterogeneity with multiple atlases [54]. We believe a combination of the label fusion framework presented in this paper and iCluster can be employed to reduce the computational burden by summarizing the training data with a small number of templates [66].

4. An alternative strategy to reduce the computational demand of label fusion is to employ a nonparametric model in a single coordinate system, to which the test subject is normalized with a single registration procedure. This approach, which entails the co-registration of the training subjects akin to atlas-based segmentation, was recently shown to produce accurate segmentation [5], [17]. The application of this strategy within the proposed label fusion framework is a direction to be explored.

5. Another strategy to reduce computational burden is to preselect the most useful training subjects and apply label fusion on these, as recently proposed by Aljabar et al. [1]. We explored one particular instantiation of this approach, where the subset of training subjects was selected to include the training subjects globally most similar to the test subject after affine normalization. It is clear that this criterion to preselect the most relevant training subjects is related to our definition of the image likelihood term p(I; I_n). Yet, a crucial difference is that the image likelihood term is computed by nonlinearly registering the training and test images, while the preselection is done based on an affine normalization. Alternative preselection strategies should also be investigated.

VII. Conclusion

In this paper, we investigated a generative model that leads to label fusion style image segmentation methods. Within the proposed framework, we derived several algorithms that combine transferred training labels into a single segmentation estimate. With a dataset of 39 brain MRI scans and corresponding label maps obtained from an expert, we empirically compared these segmentation algorithms with FreeSurfer’s widely-used atlas-based segmentation tool [69]. Our results demonstrate that the proposed framework yields accurate and robust segmentation tools that can be employed on large multi-subject datasets. In a second experiment, we employed one of the developed segmentation algorithms to compute hippocampal volumes in MRI scans of 282 subjects. A comparison of these measurements across clinical and age groups indicate that the proposed algorithms are sufficiently sensitive to detect hippocampal volume differences associated with early Alzheimer’s Disease and aging.

Appendix A. Pairwise Registration

In deriving the segmentation algorithms in this paper, we make a crucial simplifying assumption that the warps {Φ_n} that map individual training images to the test image coordinates are observed. In practice, we compute these warps via a pairwise registration algorithm. Over the past decade, a wide variety of registration algorithms have been proposed in the literature. The optimal choice of a registration algorithm remains an open question that can be partially guided by a recent study that compares a broad set of pairwise registration algorithms [39]. Here, we make our choice based on the following three criteria.

1. Speed and computational efficiency. Since the test subject must be registered with each training subject, we need to perform registration many times. Recent algorithms based on Thirion’s Demons algorithm [61] yield fast and efficient inter-subject registration.

2. Rich deformation model. The quality of the segmentation results depends on the accuracy of the alignment. The Klein et al. study [39] found that a rich, dense deformation model, in general, yields better alignment accuracy. In this work, we rely on a particular parametrization of diffeomorphic—smooth and invertible—transformations.

3. SSD similarity measure. To be consistent with our choice of the image likelihood model in (6), we use the sum of squared differences (SSD) as the similarity measure. As we discuss in Section VI, this particular choice is probably not optimal and can be improved by employing more sophisticated similarity measures, e.g., mutual information.

Based on these criteria, we choose the asymmetric bidirectional image-template registration algorithm, details of which are presented in [56]. This method is based on an efficient Demons-style algorithm [65] that uses a one-parameter subgroup of diffeomorphism. The warp Φ is parameterized with a smooth, stationary velocity field v: ℝ³ ↦ ℝ³ via an Ordinary Differential Equation (ODE) [4]

$$\frac{\partial \mathrm{\Phi }(x,t)}{\partial t}=v(\mathrm{\Phi }(x,t))$$

and the initial condition Φ(x, 0) = x. The warp Φ(x) = exp (v)(x) can be computed efficiently using scaling and squaring and inverted by using the negative of the velocity field: Φ⁻¹ = exp(−v) [4].

We impose an elastic-like regularization on the stationary velocity field

$$p(\mathrm{\Phi }=exp(v))=\frac{1}{Z_{\lambda }}exp\left[-\lambda \sum _{y\in \mathrm{\Omega }}\sum _{j,k=1,2,3}{\left({\frac{{\partial }^2}{\partial x_j^2}{v}_k(x)|}_{x=y}\right)}^2\right]$$ (24)

where λ > 0 is the warp stiffness parameter, Z_λ is a partition function that depends only on λ, and x_j and v_k denote the jth and kth component (dimension) of position x and velocity v, respectively. Higher values of the warp stiffness parameter λ yield more rigid warps.

To construct the registration objective function, we assume a simple additive Gaussian noise model that leads to the following optimization problem for registering the nth training image to the test subject

$${\widehat{v}}^n=\underset{v}{\text{argmin}}\sum _{y\in \mathrm{\Omega }}\left[{(I(y)-{\stackrel{\sim }{I}}_n(exp(v)(y)))}^2+2\lambda {\sigma }^2\sum _{j,k=1,2,3}{\left({\frac{{\partial }^2}{\partial x_j^2}{v}_k(x)|}_{x=y}\right)}^2\right]$$ (25)

where σ² is the stationary noise variance, and Φ_n ≜ exp(v̂ⁿ).

Note that (25) warps the training image (i.e., template) which makes the model truly probabilistic (for a discussion see [2], [40]). Bi-directional or symmetric approaches that apply warps to both images seem to yield more accurate alignment [7], [9], [14], [65]. Recently, we proposed an objective function that reconciles the practical advantages of symmetric registration with the asymmetric nature of image-template registration [56]

$${\widehat{v}}^n=\underset{v}{\text{argmin}}\sum _{y\in \mathrm{\Omega }}{\left[I(y)-{\stackrel{\sim }{I}}_n(exp(v)(y))\right]}^2+{\left[I(exp(-v)(y))-{\stackrel{\sim }{I}}_n(y)\right]}^{2}det(\nabla exp(-v)(y))+4\lambda {\sigma }^2{\sum _{j,k=1,2,3}\left({\frac{{\partial }^2}{\partial x_j^2}{v}_k(x)|}_{x=y}\right)}^2$$ (26)

where det(∇ exp(−v)(y)) denotes the determinant of the Jacobian of the inverse transformation exp(−v)(·) with respect to the spatial coordinates and arises from the change of coordinates y = Φ(x) and discretization of the continuous similarity measure ∫_ℝ³ [I(x) − Ĩ_n (Φ(x))]² dx. The Jacobian term is larger than 1 in regions where an area in image I is mapped to a smaller area in Ĩ_n and less than 1 elsewhere.

We solve (26) using the log-domain Demons framework [65], which decouples the optimization of the first and second terms by introducing an auxiliary transformation. The update warp is first computed using the Gauss–Newton method. The regularization is achieved by smoothing the updated warp field. It can be shown that the smoothing kernel corresponding to (24) can be approximated with a Gaussian; $K(x)\propto exp(-\alpha {\sum }_{n=1,2,3}{x}_i^2)$, where α = γ/8λσ² and γ > 0 controls the size of the Gauss-Newton step [12], [46].

Appendix B. Derivation of the EM Algorithm

Here, we present the derivation of the EM algorithm for the Global Weighted Fusion method presented in Section IV-C. We first rewrite (1) using (4)

$$\begin{array}{l}\widehat{L}=\underset{L}{\text{argmax}}log\sum _{M}p(M)p(L,I\mid M;\{L_n,I_n\})\\ =\underset{L}{\text{argmax}}logp(L,I;\{L_n,I_n\}).\end{array}$$ (27)

Let q be any distribution on M. The objective function of (27) is bounded from below by

$$logp(L,I;\{L_n,I_n\})-\text{KL}(q(M)||p(M\mid L,I;\{L_n,I_n\}))$$ (28)

where KL(·||·) denotes the nonnegative Kullback–Leibler (KL) divergence defined as

$$\text{KL}(q(M)||p(M\mid L,I;\{L_n,I_n\}))=\sum _{M}q(M)log\frac{q(M)}{p(M\mid L,I;\{L_n,I_n\})}.$$ (29)

Equation (28) is the negative of what is usually called the free energy [33], which we will denote (q, L) since it is a function of two unknown variables: the label map (segmentation) L and the distribution q. The KL divergence achieves zero only at q^*(M) = p(M|L, I; {L_n, I_n}), in which case the negative free energy (q^*, L) is merely a function of L and is equal to the objective function of (27).

By simple algebraic manipulations, the free energy can be equivalently expressed as

$$\mathcal{F}(q,L)=-H(q)-\sum _{M}q(M)logp(L,I,M;\{L_n,I_n\})$$ (30)

where H(q) denotes the entropy of q.

One interpretation of the Expectation Maximization algorithm is based on a minimization of the free energy over the two variables q and L [45]. The free energy provides a bound on the true objective function, which is optimized via coordinate-descent by alternating between the optimization over L for fixed q and vice versa. Crucially, we can see from (29) that for a fixed L̂, q^* (M) = p(M|L̂, I; {L_n, I_n}) minimizes (·, L̂). The computation of q^*(M) is called the E-step. In the M-step, the algorithm minimizes (q^*(M), L) over L. Since (q^*, L̂) is equal to the objective function evaluated at L̂, the M-step is guaranteed to improve the objective function, effectively guaranteeing convergence to a local optimum.

We can now derive the update equations for the two steps.

E-Step

For a fixed segmentation estimate L̂⁽ⁱ⁻¹⁾, we obtain

$$\begin{array}{l}{q}^{(i)}(M)=p(M\mid {\widehat{L}}^{(i-1)},I;\{L_n,I_n\})\\ \propto p({\widehat{L}}^{(i-1)},I\mid M;\{L_n{I}_n\})p(M)\\ =p_M({\widehat{L}}^{(i-1)};\{L_n\})p_M(I;\{I_n\})p(M)\end{array}$$ (31)

where (31) follows from the assumptions of the generative model presented in Section II. Since, in the global model we assume β → ∞, p(M) (and therefore q^*(M)) is zero unless M(x) is equal to a constant index n ∈ {1, …, N} for all x ∈ Ω. We denote these nonzero elements with m_n. Thus, the E-step can be simplified to

$$m_n^{(i)}\propto p_n(I;I_n)p_n({\widehat{L}}^{(i-1)};L_n).$$ (32)

M-Step

For a given q⁽ⁱ⁾ (or, equivalently $m_n^{(i)}$), the label maps are updated using (30)

$$\begin{array}{l}{\widehat{L}}^{(i)}=\underset{L}{\text{argmax}}\sum _M{q}^{(i)}(M)logp(L,I,M;\{L_n,I_n\})\\ =\underset{L}{\text{argmax}}\sum _n\sum _{x\in \mathrm{\Omega }}{m}_n^{(i)}log{p}_n(L(x);L_n)\end{array}$$ (33)

where (33) uses (32) and the assumptions of the generative model presented in Section II, and omits terms that are constant with respect to L. Note that in (33), each voxel can be optimized independently.

Appendix C. Derivation of the Variational EM Algorithm

Here, we derive the variational EM algorithm for the Semi-local Weighted Fusion method presented in Section IV-D. The key difference between the EM algorithm derived in the previous section and the variational EM algorithm is due to the MRF prior p(M) (Section III-C) with a finite, nonzero β that allows M(x) to vary spatially, while encouraging neighboring voxels to take the same value. The coupling between neighboring voxels introduced by the MRF prior makes the M-step of the algorithm computationally intractable. One approach to compute an approximate solution is to impose a structure on q[33]. This loosens the lower bound of (28), since q^*(M) = p(M|L, I; {L_n, I_n}) may not have the structure we impose on q. In other words, variational EM relaxes the optimization problem by replacing the objective function with an easier-to-optimize loose lower bound.

In our model, we use a (standard) fully factorized structure for q

$$q(M)=\prod _{x\in \mathrm{\Omega }}{q}_x(M(x)).$$ (34)

E-Step

First, we derive the update rule for q for fixed L̂⁽ⁱ⁻¹⁾. Substituting (3) into (28) and omitting terms not containing M, we obtain

$${\widehat{q}}^{(i)}=\underset{q}{\text{argmin}}\text{KL}(q(M)||p(M\mid {\widehat{L}}^{(i-1)},I;\{L_n,I_n\}))$$ (35)

$$=\underset{q}{\text{argmin}}\sum _{M}q(M)log\frac{q(M)}{p({\widehat{L}}^{(i-1)},I\mid M;\{L_n,I_n\})p(M)}$$ (36)

$$=\underset{q}{\text{argmin}}{\mathbb{E}}_q\left(-logp(M)\right)+\sum _{x\in \mathrm{\Omega }}{\mathbb{E}}_{q_x}\left(log{q}_x(M(x))-log{p}_{M(x)}({\widehat{L}}^{(i-1)}(x);L_{M(x)})p_{M(x)}(I(x);I_{M(x)})\right)$$ (37)

where denotes expectation with respect to q. Using (8), we rewrite the first term of (37) as

$${\mathbb{E}}_q\left(-\beta \sum _{x\in \mathrm{\Omega }}\sum _{y\in {\mathcal{N}}_X}\delta (M(x),M(y))\right)+log{Z}_{\beta }=-\beta \sum _{x\in \mathrm{\Omega }}\sum _{y\in {\mathcal{N}}_X}\langle q_x,q_y\rangle +log{Z}_{\beta },$$ (38)

$$=-\beta \sum _{x\in \mathrm{\Omega }}{\mathbb{E}}_{q_x}\left(\sum _{y\in {\mathcal{N}}_X}{q}_y(M(x))\right)+log{Z}_{\beta }$$ (39)

where we used the linearity of expectation and 〈q_x, q_y〉 denotes the inner product between the probability vectors q_x and q_y. Inserting (39) into (37) we obtain

$${\widehat{q}}^{(i)}=\underset{q}{\text{argmin}}\sum _{x\in \mathrm{\Omega }}{\mathbb{E}}_{q_x}\left(\beta \sum _{y\in {\mathcal{N}}_X}{q}_y(M(x))+log\frac{q_x(M(x))}{p_{M(x)}({\widehat{L}}^{(i-1)}(x);L_{M(x)})p_{M(x)}(I(x);I_{M(x)})}\right).$$ (40)

To solve (40), we can differentiate the objective function with respect to each q_x and equate to zero. With the probability constraint, i.e., ${\sum }_n{q}_x^{(i)}(M(x)=n)=1$, ∀x ∈ Ω, this implies (19).

M-Step

With the structure of (34), the segmentation update can be computed using (30)

$${\widehat{L}}^{(i)}=\underset{L}{\text{argmax}}\sum _{x\in \mathrm{\Omega }}\sum _{n=1}^N{q}_x^{(i)}(M(x)=n)\times logp(L(x),I(x)\mid M(x)=n;L_n,I_n)$$ (41)

$$=\underset{L}{\text{argmax}}\sum _{x\in \mathrm{\Omega }}\sum _{n=1}^N{q}_x^{(i)}(M(x)=n)log{p}_n(L(x);L_n).$$ (42)

Since each voxel can be considered independently, we obtain (20).