Segmenting magnetic resonance images via hierarchical mixture modelling

Abstract

We present a statistically innovative as well as scientifically and practically relevant method for automatically segmenting magnetic resonance images using hierarchical mixture models. Our method is a general tool for automated cortical analysis which promises to contribute substantially to the science of neuropsychiatry. We demonstrate that our method has advantages over competing approaches on a magnetic resonance brain imagery segmentation task.

1. Introduction

Gaussian mixture modelling is ubiquitous throughout laboratories in the world for image analysis and segmentation, and has played a core role in medical image analysis, and in particular Human Brain Mapping for the past several decades. A fundamental aspect of Human Brain Mapping research today is cortical reconstruction of brain volumes into their macroscopic components, including gray matter (G), white matter (W), or cerebrospinal fluid (C). Throughout the field of brain mapping, almost without exception the construction of cortical manifolds requires some form of segmentation via probabilistic model fitting. Reconstruction of volumetric submanifolds such as the thalamus, hippocampus, deep nuclei, and other structures are being performed via automatic segmentation methodologies by many investigators (Fischl et al., 2001; Grabowski et al., 2000; Holmes et al., 1998; Kapur et al., 1996; Joshi et al., 1999; Miller et al., 2000; Ratnanather et al., 2001; Shattuck et al., 2001; Robb, 1999; Teo et al., 1997; Wells-III et al., 1996; Xu et al., 1999). Such approaches boil down to some form of classical Bayesian segmentation and Neyman–Pearson likelihood-ratio testing on Gaussian models and Gaussian mixture models. Even segmentation methods which apply segmentation implicitly through mapping without direct Bayesian hypothesis testing, such as those approaches based on deformable template matching (Christensen et al., 1997) or active surface methods (Yezzi et al., 2002; Westin et al., 2000; Tagare, 1997; Shen et al., 2001; Sclaroff and Liu, 2001; Schultz and Conradsen, 1998; Montagnat et al., 1999, 2001; Montagnat and Delingette, 1997; Mignotte and Meunier, 2001; Kervrann and Heitz, 1999; Jain et al., 1996; Garrido and de la Blanca, 1998, 2000; Chen and Metaxas, 2000; Caunce and Taylor, 2001; Xu et al., 2000; Xu and Prince, 2000; Pham et al., 2000)are usually based on some form of Gaussian mixture modelling. Usually, the data models are implicitly conditionally Gaussian, with the resulting matching functions of the quadratic-norm type.

At the heart of such brain mapping efforts is mixture modelling in which each tissue class is modelled as a single Gaussian component. Sometimes additional “partial volume” components are included to account for the inadequacy of the single Gaussian model. Of course, selecting the dimension of the model, i.e. the number of partial volume fits, reduces to the model selection problem; as well there is no reason to believe that a single model order fits all tissue and or imaging modality types. The focus of this paper is to propose a method which allows for the flexibility of a Gaussian mixture model—with model complexity selected adaptively from the data—for each tissue class. Our procedure involves modelling each class as a semiparametric mixture of Gaussians. Thus, for each subject j, the subject-specific marginal f_j is taken to be a hierarchical mixture model—a mixture of Gaussian mixtures. For each of the three classes c ∈ {C, G, W,} the density f_j has one mixture component f_jc, and each of these components is itself a mixture of Gaussians. The major difficulty solved in this paper associated with employing such semi-parametric methods is to solve dynamically the model selection problem. The crucial step of determining class-conditional mixture complexities for (unlabeled) test data is hopeless in the unsupervised case—without labeled voxel observations known to belong to the three classes, the hierarchical mixture model is hopelessly unidentifiable. We accomplish this by matching models to a predefined data base of hand labelled experimental tissue samples. Thus, the use of a registry of expertly generated data becomes a fundamental part of our solution. This approach of course builds on the entire philosophy now emerging through the Biomedical Informatics Research Network (http://www.nbirn.net) initiatives in which data bases are federated and available for laboratory comparison and analysis. As we show, the performance improvement obtained is significant, implying that the emergence of hand labelled information from laboratories across the country which are combined with automated methods for analysis promise tremendous improvements in future automated brain analysis methods.

Of course, the importance of such improvements is in that a fundamental aspect of human brain mapping today is the subtle examination of small changes in brain morphometry. For instance, to investigate whether schizophrenia or Alzheimer's disease alters the structure of gray matter in the cingulate gyrus of affected subjects, accurate segmentations into tissue types of the cingulate gyrus for both affected and control subjects are required. Such segmentations are currently obtained via labor- and cost-intensive hand-segmentation of all images in a sample. For the purpose of investigating subtle differences between the affected and control populations, larger sample sizes afford greater statistical power and are, of course, desirable. This leads to a conflict between the desire for a large sample size and the desire to keep the number of hand-segmentations small. This manuscript presents a method for taking a hand-segmented training subset of the images to be segmented and producing accurate automated segmentations for the remaining images in the sample. This will allow for a larger sample of segmented images, thereby increasing the power of subsequent statistical analyses.

Our method proceeds according to the following outline.

• (1) For each subject/class pair in the available training data set, estimate the marginal subject-specific class-conditional probability densities. Notice that for this supervised step (since these are training images, individual voxel class labels are available) semi-parametric mixture complexity estimation is appropriate.

• (2) Find the “closest” training model to the (unlabeled) test data.

• (3) Fit a mixture to the test data, using the training model obtained in step (2) to determine the class-conditional mixture complexities and starting locations.

• (4) Classify voxels from the test data using the plug-in Bayes rule, where the mixture component class labels are inherited from the selected training model (but the mixture itself is estimated from the test data in step (3)).

This manuscript is structured as follows. In Section 2 we describe a data set of MR images to be investigated. Section 3 gives the mixture modelling methodology in detail. Section 4 describes the experimental protocol employed, and Section 5 reports the results of our investigations. Finally, in Section 6, we conclude with a discussion of the implications of our results and our methodology.

2. Data

Dysfunction of specific subregions of the cerebral cortex has been implicated in the pathophysiology of several neuropsychiatric disorders, including schizophrenia. Although widespread thinning of the gray matter cortical mantle has been observed in subjects with schizophrenia (Zipursky et al., 1992), specific abnormalities of neuronal architecture may exist within portions of the cortical mantle and form the neuroanatomical basis for specific cognitive deficits or symptom groupings. Shown in Fig. 1 are examples of cingulate, occipital and medial prefrontal in the human brain, which are studied here. Such brain areas consist of different mixtures of gray matter, white matter, and cerebrospinal fluid.

Fig. 1.

Panel 1 shows the cingulate gyrus, panel 2 is the occipital gyrus, and panel 3 is the medial prefrontal gyrus. (Adapted from Duvernoy, 1999).

The cingulate gyrus, a large C-shaped gyrus that courses around the corpus callosum and cerebral ventricles, has attracted considerable interest because of its role in attention and other cognitive functions that appear to be critical for understanding schizophrenia and other neuropsychiatric disorders (Csernansky and Bardgett, 1998). The medial prefrontal cortex has been implicated in a variety of recent functional and structural neuroimaging studies of both normal affect regulation and differences associated with affective disorders such as major depressive disorder or bipolar affective disorder (Swerdlow and Koob, 1987; Drevets et al., 1992; Drevets and Todd, 1997; Botteron and Figiel, 1997; Drevets, 1999). The occipital cortex has been the focus of several major functional neuroimaging studies of visual perception; see, e.g. (Yantis, 2001; Tootell et al., 1998).

High-resolution MR imaging now affords an unprecedented opportunity to acquire detailed images of the neuroanatomical configurations and tissue characteristics of the living human brain. Substantial advances in our understanding of neuropsychiatric disorders are anticipated as this technology is used more widely in populations of individuals with such illnesses. However, most neuropsychiatric researchers still apply manual methods to assess the boundaries and internal tissue segmentations of cortical subregions in MR scans. These manual methods are limited by their dependency on the constant availability of human experts and by between rater error (e.g., Bartzokis et al., 1993; Haller et al., 1997).

We investigate first a set of cingulate gyrus images provided by John Csernansky of the Washington University School of Medicine, St. Louis. High-resolution MR scans were acquired using a turbo-FLASH sequence (TR = 20 ms, TE = 5.4 ms, flip angle = 30°, ACQ-1, Matrix 256 × 256, Scanning Time = 13.5 min) with 1 mm × 1 mm × 1 mm isotropic resolution across the entire cranium. (See, for instance, Cho et al. (1993) for MR imaging background.) MR image data were reformatted via intensity scale adjustment to maximize tissue contrast, using the voxel intensities in the corpus callosum and the lateral ventricle as limiting values. MR scans were then interpolated into 0.5 mm × 0.5 mm × 0.5 mm isotropic resolution using trilinear interpolation.

We consider J = 10 subjects; five with schizophrenia (labeled “CG 1” through “CG 5”) and five control subjects (labeled “CG 6” through “CG 10”). For each subject, the voxels in the cingulate gyrus of the right hemisphere, have been hand-segmented by expert neuroanatomists into one of three tissue classes; 𝓒 ={C, G, W}. (See Figs. 2 and 3.) Hand segmented files were created using Analyze (Robb, 1999) and a region of interest (ROI) defined by expert raters to isolate the cingulate gyrus. The ROI included white matter, gray matter and cerebrospinal fluid of one hemisphere that encompassed the cingulate gyrus and immediate surrounding region. A three-dimensional ROI encompassing the entire cingulate gyrus in the right hemisphere was outlined. The ROI was constructed in coronal sections. In each MR section, an enclosure which consisted of the gray matter of the right cingulate gyrus and its neighboring gray matter, white matter, and cerebrospinal fluid was drawn by hand. This was a fast procedure as it did not require manual segmentation of exact tissue types or precise anatomical structures. An example of such an ROI section is shown in Fig. 2; note that some voxels were not labeled and thus not included in the ROI. The right cingulate gyrus was manually delineated, according to anatomical rules. For each scan, a set of cingulate manual segmentations by an expert rater was obtained in the ROI. The manual segmentations consisted of three tissue classes. Voxel classifications by the rater were based on visual inspection of morphological cues such as gyri curvature and connectivity, as well as visual intensity cues between three tissue classes.

Fig. 2.

Panel 1 shows the original MRI, and Panel 2 is the hand-segmentation of (one slice of) CG 9 into C (dark gray), G (light gray), W (white); black is unsegmented region.

Fig. 3.

Frequency histograms for CG 9: entire cingulate gyrus, right hemisphere (top), and C (bottom left), G (bottom center), and W (bottom right).

There are between 390,000 and 700,000 class-labeled voxels per subject. The smallest subject-specific class-conditional sample size is 45,709 (C for subject CG 6). See Table 1 for details.

Table 1.

Sample sizes for 10 cingulate gyri

	Number of voxels
	CG 1	CG 2	CG 3	CG 4	CG 5	CG 6	CG 7	CG 8	CG 9	CG 10
C	150,128	90,035	91,452	85,063	121,138	45,709	108,617	84,402	76,886	125,409
G	210,636	186,529	136,856	124,227	130,717	154,993	181,538	169,583	161,138	131,230
W	333,341	359,746	263,401	243,651	274,943	193,633	320,293	276,141	264,607	339,410
Total	694,105	636,310	491,709	452,941	526,798	394,335	610,448	530,126	502,631	596,049

3. Semiparametric mixture modelling

Model selection in Gaussian mixtures—the estimation of mixture complexity—is of fundamental importance in many applications of finite mixture models. An enormous body of literature exists regarding the application, computational issues, and theoretical aspects of mixture models when the number of components is known (see, e.g., McLachlan and Peel, 2000; Everitt and Hand, 1981; McLachlan and Basford, 1988; Titterington et al., 1990), but estimating the unknown number of components remains an area of intense research effort. See, for instance, Chen and Kalbfleisch (1996); Dacunha-Castelle and Gassiat (1997, 1999); Escobar and West (1995); Henna (1985); James et al. (2001); Keribin (2000); McLachlan (1987); Priebe and Marchette (2000); Roeder and Wasserman (1997) for recent progress in this area.

We model the class-conditional probability density functions via the “alternating kernel and mixture” (AKM) method (Priebe and Marchette, 2000; James et al., 2001). This methodology results in an automatic estimate of mixture complexity. We do not claim AKM is necessarily the best approach; but it is reasonable and illustrative, and can have advantages over other techniques. In Priebe and Marchette (2000) AKM is compared to the Bayesian methodology of Roeder and Wasserman (1997), while James et al. (2001) presents a comparison with numerous competing approaches.

3.1. Modelling

For each subject j ∈ 𝒥: ={1, …, J }, consider magnetic resonance voxel observations 𝒳_j : ={X_j1, …, X_jnj}; the subject-specific sample sizes are denoted by n_j. Let the marginal probability density function for subject j voxel observations be given by

$$f_i=\underset{c\in 𝒞}{\Sigma }{\pi }_{\mathit{jc}}{f}_{\mathit{jc}}.$$ (1)

That is, f_j=π_jCf_jC+π_jGf_jG+π_jWf_jW. The subject-specific class-conditional marginals are denoted f_jc, and the subject-specific class-conditional mixing coefficients π_jc are nonnegative and sum to unity. We assume that the X_ji are identically distributed according to f_j (but not independent).

Each subject-specific class-conditional marginal f_jc is itself modelled as a mixture of normals

$$f_{\mathit{jc}}=\underset{t=1}{\overset{k_{\mathit{jc}}}{\Sigma }}{\pi }_{\mathit{jct}}{\phi }_{\mathit{jct}},$$ (2)

where the subject-specific class-specific mixing coefficients satisfy π_jct ≥0 for each j, c, t, and ${\Sigma }_t^{k_{\mathit{jc}}}{\pi }_{\mathit{jct}}=1$ for each j, c, and ${\phi }_{\mathit{jct}}=\phi (\cdot ;{\mu }_{\mathit{jct}},{\sigma }_{\mathit{jct}}^2)$ denotes the Gaussian probability density function with mean μ_jct variance ${\sigma }_{\mathit{jct}}^2$. Thus the subject-specific marginals f_j are modelled as hierarchical mixtures—mixtures of Gaussian mixtures;

$$f_j=\underset{c\in 𝒞}{\Sigma }{\pi }_{\mathit{jc}}\underset{t=1}{\overset{k_{\mathit{jc}}}{\Sigma }}{\pi }_{\mathit{jct}}{\phi }_{\mathit{jct}}.$$ (3)

This model is precisely analogous to the model used in “mixture discriminant analysis” (Hastie and Tibshirani, 1996). The subject-specific class-conditional mixture complexities k_jc are to be estimated from the data via AKM.

3.2. Estimation

Given class-labelled training data, we have available subject-specific class-conditional sample sizes n_jc such that n_j = Σ_cn_jc. For the subject-specific mixing coefficients π_jc we use the empirical estimate

$${\widehat{\pi }}_{\mathit{jc}}=n_{\mathit{jc}}∕n_j,$$ (4)

the ratio of the subject-specific class-conditional sample size to the total subject-specific sample size. Let 𝒳_jc denote that subset of ᵋ_j for which the class label is c.

We estimate the subject-specific class-conditional mixture complexities k_jc, mixing coefficients π_jct, and mixture components φ_jct, via AKM. This estimation is semiparametric; the mixture complexities k̂_jc are estimated from the data. AKM employs an iterative estimation scheme, comparing the k-component mixture estimate against the k + 1-component mixture estimate. When the improvement obtained by adding a k + 1st component is negligible (less than some penalty term) the iteration is terminated and the resulting k-component mixture is used as the estimate. This general version of model selection—looking for the “elbow” or “knee” in a complexity vs. penalized estimation accuracy curve—is quite common. A distinguishing feature of AKM is the process of using successive kernel estimates to guide the successive mixture estimates.

The filtered kernel estimator (Marchette et al., 1996) extends the basic kernel estimator by allowing multiple bandwidths driven by a pilot mixture model. Given a Gaussian mixture

$$f=\underset{t=1}{\overset{k}{\Sigma }}{\pi }_t{\phi }_t$$ (5)

and a bandwidth h, define the filtered kernel estimator f̃(·; 𝒳, f, h) based on the mixture f and using the data 𝒳 = {X₁, …, X_n} to be

$$\tilde{f}(x;𝒳)=\frac{1}{n}\underset{i=1}{\overset{n}{\Sigma }}\underset{t=1}{\overset{k}{\Sigma }}\frac{{\pi }_t{\phi }_t\left(X_i\right)}{f\left(X_i\right)h{\sigma }_t}{\phi }_0\left(\frac{x-X_i}{h{\sigma }_t}\right),$$ (6)

where ${\sigma }_t^2$ is the variance of the tth component of the mixture f and φ₀ is the standard zero mean, unit variance normal. This allows for different bandwidths, guided by the mixture model f.

Let f̂¹ be the single normal component with mean and variance determined by the sample moments of 𝒳_jc—that is, f̂¹ is a trivial one-component mixture. Let f̃¹ be the filtered kernel estimate based on f̂¹—that is, f̃¹ is the standard kernel estimate using the normal reference rule to determine the bandwidth. For k = 2, 3, …, define in turn f̂^k first to be the k-component mixture best matched to the nonparametric estimate f̃^k−1

$${\widehat{f}}^k:=\arg \underset{f\in {ℱ}^k}{\min }{\parallel f-{\tilde{f}}^{k-1}\parallel }_2^2,$$ (7)

where ${\parallel f-g\parallel }_2^2:={\int }_{-\infty }^{\infty }{(f\left(x\right)-g\left(x\right))}^2\mathrm{d}x$ is the integrated squared error and 𝓕^k is the class of k-component Gaussian mixtures. Subsequently define f̃^k to be the filtered kernel estimate based on the mixture f̂^k.

Let

$$\ell (f,𝒳):=\log \prod _{x\in 𝒳}f\left(x\right)=\underset{x\in 𝒳}{\Sigma }\log f\left(x\right).$$ (8)

(This would be the log-likelihood if the observations were independent; alas, this is not the case for the brain voxels. We address the issue of spatial dependence in the next subsection.) The estimate of mixture complexity used here (analogous to the estimates proposed in Priebe and Marchette (2000) and James et al. (2001))is given by

$${\widehat{k}}_{\mathit{jc}}=\arg \min \{k\in \{1,2,\dots \}:\ell ({\widehat{f}}^{k+1},{𝒳}_{\mathit{jc}})-({\widehat{f}}^k,{𝒳}_{\mathit{jc}})<a(n_{\mathit{jc}},k+1)\}.$$ (9)

The penalty term a(n_jc, k + 1) in the above equation—a function of sample size and model complexity—is the key practical issue in this version of model selection. A simple choice for a(n_jc, k + 1) is 3 log(n_jc). Asymptotic considerations require a(n_jc, k + 1)/n_jc → 0 as n_jc → ∞ for fixed k, but not too fast. In practice, the choice of a(n_jc, k + 1) drives the parsimoniousness of the resultant model. Akaike (1974), Rissanen (1978), and many others since (see, notably, George and Foster, 2000) have weighed in on the choice of this penalty term; nevertheless, for finite sample sizes the choice quite resembles an artistic balancing of parsimony and fit—the conventional bias-variance trade-off.

3.3. Spatial dependence

Because there is correlation amongst the voxel observations, and thus ℓ(·, ·) defined above is not the log-likelihood, simple interpretation of the penalty term a(n_jc, k+1) as a function of sample size calls for conditional covariance modelling of the three-dimensional spatial process. Covariograms (Cressie, 1993) for the subject-specific class-conditional random fields indicate that significant spatial correlation is evident at a distance of up to 10 voxels. (Stationarity and isotropy are indicated as well.) This suggests that the effective sample sizes are significantly smaller than the number of voxels n_jc.

One way to look at the effect of dependent observations on our estimate of mixture complexity is to note that, if the observations in the subject-specific class-conditional sample 𝒳_jc are positively correlated, then the terms ℓ(f̂^k+1, 𝒳_jc) and ℓ(f̂^k, 𝒳_jc) in Eq. (9) should be scaled accordingly. That is, if the “effective number of independent observations” is n′_jc:=n_jc/δ for 1≤δ≤n_jc, then substituting δ⁻¹ ℓ(·, ·) into Eq. (9) will account for the correlation. The case δ = 1 represents independence and, as (positive) spatial correlation increases, δ increases as well. This can be presented as

$${\widehat{k}}_{\mathit{jc}}=\arg \min \{k\in \{1,2,\dots \}:\ell ({\widehat{f}}^{k+1},{𝒳}_{\mathit{jc}})-\ell ({\widehat{f}}^k,{𝒳}_{\mathit{jc}})<\delta a(n_{\mathit{jc}}∕\delta ,k+1)\}.$$ (10)

The idea of accounting for spatial correlation via an “effective number of independent observations” (Cressie, 1993) is analogous to the “resolution elements” found in Worsley et al. (1992), e.g. Solid mathematical justification for adopting this approach is lacking except in the most restricted cases (Leadbetter et al., 1983). Direct modelling of the dependency structure, via methods such as Markov random field models, is the subject of intense ongoing research. Our ad hoc approach is meant as a simple and straight-forward method enabling semiparametric hierarchical mixture modelling.

Estimating δ is a sticky wicket. Nevertheless, as will be seen in the experimental results presented below, a significant performance improvement can be obtained by employing even a crude covariogram-based estimate δ̂ in the complexity selection methodology. Furthermore, Occam's Razor may suggest a preference to err on the side of parsimony. Our rule of thumb is to choose δ̂ to be the number of voxels in a ball whose radius is given by the distance at which the omnidirectional empirical covariogram drops below some threshold, rounded to the nearest integer.

Using a(n, k) = 3 log(n), we arrive at the estimate of mixture complexity

$${\widehat{k}}_{\mathit{jc}}=\arg \min \{k\in \{1,2,\dots \}:\ell ({\widehat{f}}^{k+1},{𝒳}_{\mathit{jc}})-\ell ({\widehat{f}}^k,{𝒳}_{\mathit{jc}})<\widehat{\delta }3\log (n_{\mathit{jc}}∕\widehat{\delta })\}.$$ (11)

Given k̂_jc selected in this manner, the mixture model estimate of the marginal probability density for subject j, tissue class c, is then given by

$${\widehat{f}}_{\mathit{jc}}=\arg \underset{f\in {ℱ}^{{\widehat{k}}_{\mathit{jc}}}}{\min }{\parallel f-{\tilde{f}}^{{\widehat{k}}_{\mathit{jc}}-1}\parallel }_2^2,$$ (12)

the k̂_jc-component mixture identified in the AKM procedure. Results will be reported below for three choices of δ̂: 1, 25, and 99. The case δ̂ = 1 corresponds to ignoring the dependence; the choices δ̂ = 25 and δ̂ = 99 are obtained with threshold values 2/3 and 1/3,respectively.

4. Segmentation method

We consider segmenting the cingulate gyrus for subject γ ∈ 𝓙; in this case, we have J − 1 training subjects corresponding to indices j ∈ 𝓙_γ := 𝓙\{γ}.

4.1. Training

For j ∈ 𝓙_γ, we obtain

$${\widehat{f}}_j=\underset{c\in 𝒞}{\Sigma }\frac{n_{\mathit{jc}}}{n_j}{\widehat{f}}_{\mathit{jc}}=\underset{c\in 𝒞}{\Sigma }\frac{n_{\mathit{jc}}}{n_j}\underset{t=1}{\overset{{\widehat{k}}_{\mathit{jc}}}{\Sigma }}{\widehat{\pi }}_{\mathit{jct}}{\widehat{\phi }}_{\mathit{jct}}$$ (13)

via AKM. For γ we have the test data set 𝒳_γ.

Write

$$ℒ(\gamma ,f)≔\prod _{x\in {𝒳}_{\gamma }}f\left(x\right).$$ (14)

(This would be the likelihood if the subject-specific voxel observations were independent.) We proceed by choosing the index γ* specifying the training subject “closest” to the test data set. However, even though we proceed under the assumption that the subject-specific class-conditional tissue mixture models f̂_jc for one training subject can be usefully transferred to the test sample, there is good reason to treat the subject-specific class-conditional mixing coefficients as nuisance parameters; while the tissue class probability distributions may be transferrable, we wish to assume that the percentage of gray matter, say, may vary from training subject to test subject. Toward this end we define

$${ℱ}_j≔\left\{\underset{c\in 𝒞}{\Sigma }{p}_{\mathit{jc}}{\widehat{f}}_{\mathit{jc}}:p_{\mathit{jc}}\in (0,1){\forall }_c\text{and}\underset{c\in 𝒞}{\Sigma }{p}_{\mathit{jc}}=1\right\},$$ (15)

the collection of densities which agree with f̂_j up to subject-specific class-conditional mixing coefficients. Then

$${\gamma }^{\ast }≔\arg \underset{j\in {𝒥}_{\gamma }}{\max }\underset{f\in {ℱ}_j}{\max }ℒ(\gamma ,f).$$ (16)

The selection of training subject γ* provides the test subject γ with class-conditional complexity estimates k̂_γ*c, as well as initial conditions obtained from the f̂_γ*c and n_γ*c/n_γ*, for use in modelling 𝒳_γ. (There are, of course, choices other than the likelihood-based option (Eq. (14)) for determining the “closest” training subject γ*.)

4.2. Testing

4.2.1. Mixture modelling

Given γ* selected as above, we estimate f_γ by estimating the parameters

$$\theta =[{\pi }_{\gamma C},{\pi }_{\gamma G},{\pi }_{\gamma C1},\dots ,{\pi }_{\gamma C({\widehat{k}}_{\gamma C}-1)},{\mu }_{\gamma C1},{\sigma }_{\gamma C1}^2,\dots ,{\mu }_{\gamma C{\widehat{k}}_{\gamma C}},{\sigma }_{\gamma C{\widehat{k}}_{\gamma C}}^2,{\pi }_{\gamma G1},\dots ,{\pi }_{\gamma G({\widehat{k}}_{\gamma G}-1)},{\mu }_{\gamma G1},{\sigma }_{\gamma G1}^2,\dots ,{\mu }_{\gamma G{\widehat{k}}_{\gamma G}},{\sigma }_{\gamma G{\widehat{k}}_{\gamma G}}^2,{{\pi }_{\gamma W1},\dots ,{\pi }_{\gamma W({\widehat{k}}_{\gamma W}-1)},{\mu }_{\gamma W1},{\sigma }_{\gamma W1}^2,\dots ,{\mu }_{\gamma W{\widehat{k}}_{\gamma W}},{\sigma }_{\gamma W{\widehat{k}}_{\gamma W}}^2]}^{\prime }$$ (17)

via

$${\widehat{f}}_{\gamma }≔\arg \underset{\theta \in ϴ}{\max }\prod _{x\in 𝒳}\underset{c\in 𝒞}{\Sigma }{\pi }_{\gamma c}\underset{t=1}{\overset{{\widehat{k}}_{\gamma c}}{\Sigma }}{\pi }_{\gamma \mathit{ct}}{\phi }_{\gamma \mathit{ct}}\left(x\right).$$ (18)

Notice that this involves estimation of $3{\Sigma }_{c\in 𝒞}{\widehat{k}}_{{\gamma }^{\ast }c}-1$ parameters. Algorithmically, this M-estimate is obtained using the EM algorithm (see, for instance, McLachlan and Krishnan, 1997) with the training model f̂_γ* as the starting point;

$${\widehat{f}}_{\gamma }≔\mathit{EM}({𝒳}_{\gamma };{\widehat{f}}_{{\gamma }^{\ast }}).$$ (19)

The estimate obtained in this manner can be written as

$${\widehat{f}}_{\gamma }=\underset{c\in 𝒞}{\Sigma }{\widehat{\pi }}_{\gamma c}{\widehat{f}}_{\gamma c}=\underset{c\in 𝒞}{\Sigma }{\widehat{\pi }}_{\gamma c}\underset{t=1}{\overset{{\widehat{k}}_{\gamma c}}{\Sigma }}{\widehat{\pi }}_{\gamma \mathit{ct}}{\widehat{\phi }}_{\gamma \mathit{ct}}.$$ (20)

Class complexities and component labels are inherited from f̂_γ*, so that (for c = G, for instance) the mixing coefficient π̂_γG indicates the amount (proportion) of gray matter in the test image while ${\widehat{f}}_{\gamma G}={\Sigma }_{t=1}^{{\widehat{k}}_{\gamma G}}{\widehat{\pi }}_{\gamma \mathit{Gt}}{\widehat{\phi }}_{\gamma \mathit{Gt}}$ provides a model for that gray matter.

This unsupervised modelling of the test subject mixture allows estimation of the proportion and character of the different tissue types as they appear in the test subject, with a goal toward improving the segmentation for this subject.

4.2.2. Classification

We next consider the Bayes plug-in classifier

$$g\left(x\right)=\arg \underset{c\in 𝒞}{\max }{\widehat{\pi }}_{\gamma c}{\widehat{f}}_{\gamma c}\left(x\right).$$ (21)

The voxel x is to be labelled as belonging to the class which maximizes posterior probability of class membership. See Devroye et al. (1996) for a thorough discussion of the Bayes plug-in classifier.

This is (operationally) equivalent to the likelihood ratio test procedure (our estimates are indeed marginal density estimates) given by considering

$${\mathit{LRT}}_{\mathrm{C}∕\mathrm{G}}\left(x\right)=\frac{{\widehat{\pi }}_{\gamma C}{\widehat{f}}_{\gamma C}\left(x\right)}{{\widehat{\pi }}_{\gamma G}{\widehat{f}}_{\gamma G}\left(x\right)}≕r_1\left(x\right)$$ (22)

and

$${\mathit{LRT}}_{\mathrm{G}∕\mathrm{W}}\left(x\right)=\frac{{\widehat{\pi }}_{\gamma G}{\widehat{f}}_{\gamma G}\left(x\right)}{{\widehat{\pi }}_{\gamma W}{\widehat{f}}_{\gamma W}\left(x\right)}≕r_2\left(x\right).$$ (23)

Our automatic segmentation is then given by the following rules:

• r₁(x) > 1 implies voxel x is to be labelled as tissue class C.

• r₂(x) < 1 implies voxel x is to be labelled as tissue class W.

• r₁(x) < 1 & r₂(x) > 1 implies voxel x is to be labelled as tissue class G.

• r₁(x) > 1 & r₂(x) < 1 should not occur.

  • ○ (Fig. 3 suggests stochastic ordering, C<^st G<^stW; this event is labelled “unknown” in the event that it does occur.)

• (Ties are to be broken arbitrarily.)

5. Results

An example of the subject-specific, class-conditional marginal density estimates obtained via the AKM procedure presented above is presented in Fig. 4. Table 2 presents the complexity estimation results k̂_jc for the 10 cingulate gyri under consideration for three choices of δ̂. From Fig. 4 we see that these (representative) fits are quite accurate. From Table 2 we see that δ̂ = 1 leads to a very high complexity, while δ̂ = 25 and δ̂ = 99 provide a reasonable framework to investigate the parsimony-fit trade-off.

Fig. 4.

Depicted are mixtures (δ̂ = 25) and histograms for CG 9. From left to right: C, G, W. Estimated model complexities are k̂_CG9C = 2, k̂_CG9G = 3, k̂_CG9W = 3.

Table 2.

Mixture complexity estimation results for 10 cingulate gyri

	Estimated number of components
	CG 1	CG 2	CG 3	CG 4	CG 5	CG 6	CG 7	CG 8	CG 9	CG 10
δ̂ = 1
C	3	1	4	3	3	2	26	3	2	2
G	2	16	3	1	3	1	1	1	3	3
W	3	13	5	16	17	13	6	16	12	11
δ̂ = 25
C	3	1	2	3	3	1	2	2	2	2
G	1	2	3	1	2	1	1	1	3	3
W	3	2	3	4	3	2	3	4	3	5
δ̂ = 99
C	1	1	1	1	1	1	2	1	1	1
G	1	1	2	1	2	1	1	1	1	2
W	1	2	3	3	3	2	2	2	3	3

There is no “gold standard” available for this investigation; that is, we do not know the true voxel class labels. The expert's hand-segmentation is the “lead standard” against which we compare. Competing segmentation methodologies include:

• An overly-simplistic model in which each tissue class is modelled with but a single normal.

• Partial volume approaches (PV1 and PV2 in Table 3).

These two partial volume methodologies are sketched here.

Table 3.

Segmentation results for 10 cingulate gyri

	Probability of misclassification
	CG 1	CG 2	CG 3	CG 4	CG 5	CG 6	CG 7	CG 8	CG 9	CG 10
PV1	0.15	0.09	0.23	0.20	0.28	0.15	0.15	0.16	0.20	0.24
PV2	0.09	0.06	0.13	0.11	0.18	0.09	0.09	0.11	0.12	0.15
AKM (δ̂ = 25)	0.13	0.07	0.12	0.09	0.11	0.10	0.07	0.10	0.09	0.11
AKM (δ̂ = 99)	0.10	0.07	0.13	0.10	0.15	0.09	0.07	0.10	0.08	0.10

The partial volume approaches PV1 (Ratnanather et al., 2001) and PV2 (Ratnanather et al., 2004) involve a five-component Gaussian mixture fit via the EM algorithm; initial conditions are hand-selected based on the frequency histogram for the entire data set (e.g., the top panel in Fig. 3). One mixture component is initialized to account for each of the three tissue classes, and a “partial volume” mixture component is initialized between C & G and another between G & W. The difference between the two approaches is in the manner in which these partial volumes are allocated to the various tissue classes. These partial volume approaches are competitive with the most successful semi-automated MR brain image segmentation methodologies available in the literature, and are in use in current brain segmentation projects (Ratnanather et al., 2001, 2004).

In method PV1, τ_C/G = 1/2 of the C/G partial volume component is allocated to C and the other 1 − τ_C/G to G. That is, a voxel observation x for which the C/G partial volume component maximizes (over the five mixture components) posterior probability of class membership is assigned to tissue class C (resp. G) if it takes a value smaller (resp. larger) than the mean of the C/G partial volume component. The G/W partial volume component is allocated analogously.

Method PV2, on the other hand, optimizes probability of misclassification over τ_C/G ∈ [0, 1] and τ_G/W ∈ [0, 1]. For the cingulate gyrus data set under consideration herein, this optimization results in ${\tau }_{C∕G}^{\ast }=0.85$ and ${\tau }_{G∕W}^{\ast }=0.92$.

Table 3 presents the probability of misclassification results for the 10 cingulate gyri under consideration for the AKM hierarchical mixture model with two choices of δ̂, as well as analogous results for the two competing partial volume approaches described above. Despite the small sample size of just 10 subjects, Table 3 suggests that the hierarchical mixture model methodology may provide superior automatic segmentation results to the competing approaches. For example, a paired one-sided t-test of AKM (δ̂ = 99) vs. PV2 yields a p-value of p = 0.03.

6. Discussion and conclusions

Voxel segmentation is one step of a sequence of procedures used in cortical analysis. Knowledge of volumes of gray matter, the coordinates of gray matter voxels and vertices of the gray/white surface is essential in a careful scientific analysis of cortical regions in normal and diseased subjects.

A real constraint on the ability to detect differences in two populations of MR brain imagery, and hence to investigate important neuropsychiatric hypotheses, is the cost of obtaining hand-segmentated brain volumes. This cost restricts the available sample sizes. Automated segmentation promises to allow for significantly larger sample sizes, and thus to advance the state of the science.

A hierarchical mixture modelling approach is demonstrated to provide automatic segmentation of magnetic resonance images superior to that of competing partial volume approaches. Our modelling methodology provides a key aspect missing in the methods currently in use: automatic subject-specific, class-conditional model complexity selection. The statistically innovative aspect of our work is the use of classified training data to obtain a model complexity for the unclassified test data, and hence allowing for richer test data models (and, subsequently, superior segmentation performance).

While specific details of the model fitting employed herein may be beneficially altered for particular applications, the general approach of (1) semiparametric estimation of subject-specific class-conditional marginal densities for a set of training volumes, (2) nearest neighbor matching of the test data to the training models providing for automated class-conditional mixture complexities, (3) parameter fitting of the selected training model to the test data, and (4) plug-in Bayes classification of unlabelled voxels, provides an advance in the state-of-the-art for automated MR brain image segmentation.

It is instructive to know which voxels are misclassified. In fact, for AKM the misclassified voxels are likely to be near class boundaries. This knowledge should be exploited to improve performance. However, such improvements may need to take into account that the “lead-standard” against which we test is based on expert hand-segmentation, and must not be considered a perfect “gold-standard.”

A useful generalization of our methodology involves “combining classifiers”; that is, rather than considering just the closest training model, one can perform the classification as above for each training model and then combine the results by weighting the classification obtained via the jth training model inversely proportional to max _{f ∈ 𝓕j} 𝓛(γ,f). For our cingulate gyrus investigation this alteration improves the performance for seven of the 10 subjects. Note, however, that computational requirements increase proportional to the number of models used in the combination.

One aspect of our methodology for which further investigation is particularly warranted is our ad hoc accounting for spatial dependence (our δ̂). It will be beneficial to develop and use a more sophisticated spatially varying model to replace this crude covariogram estimate of the “effective number of independent observations” used herein.

The application of our methodology is not limited to just the cingulate gyrus. Experiments have been performed on medial prefrontal gyrus and occipital gyrus data sets; results indicate, again, the potential advantage of AKM. Indeed, MR brain image segmentation via AKM is a general tool for automated cortical analysis.