Joint inference on structural and diffusion MRI for sequence-adaptive Bayesian segmentation of thalamic nuclei with probabilistic atlases

Abstract

Segmentation of structural and diffusion MRI (sMRI/dMRI) is usually performed independently in neuroimaging pipelines. However, some brain structures (e.g., globus pallidus, thalamus and its nuclei) can be extracted more accurately by fusing the two modalities. Following the framework of Bayesian segmentation with probabilistic atlases and unsupervised appearance modeling, we present here a novel algorithm to jointly segment multi-modal sMRI/dMRI data. We propose a hierarchical likelihood term for the dMRI defined on the unit ball, which combines the Beta and Dimroth-Scheidegger-Watson distributions to model the data at each voxel. This term is integrated with a mixture of Gaussians for the sMRI data, such that the resulting joint unsupervised likelihood enables the analysis of multi-modal scans acquired with any type of MRI contrast, b-values, or number of directions, which enables wide applicability. We also propose an inference algorithm to estimate the maximuma-posteriori model parameters from input images, and to compute the most likely segmentation. Using a recently published atlas derived from histology, we apply our method to thalamic nuclei segmentation on two datasets: HCP (state of the art) and ADNI (legacy) – producing lower sample sizes than Bayesian segmentation with sMRI alone.

1. Introduction

Automated segmentation of MRI scans is a prerequisite for most human neuroimaging studies. Most of the algorithms commonly used for this task rely solely on structural MRI (sMRI) scans, and belong to one of three categories: Bayesian segmentation with a probabilistic atlas (e.g., [1,2]); multi-atlas segmentation [3]; and, more recently, convolutional neural networks (e.g., [4]). Typically, these techniques segment the brain into tissue types (i.e., gray matter, white matter, and cerebrospinal fluid), or into finer anatomical structures (e.g., hippocampus, ventricle). Bayesian methods drive the primary segmentation modules of the most widespread neuroimaging packages, like FreeSurfer [2], FSL [5], or SPM [1].

The aforementioned approaches rely mostly on T1 contrast to distinguish between gray and white matter. However, some boundaries between structures are nearly invisible in T1 (and other structural MR contrasts) due to insufficient difference in proton density and relaxation times. This is exacerbated by lower contrast-to-noise ratio in deep-brain structures, due to greater distance from the head coil. Two examples from the state-of-the-art Human Connectome Project (HCP) dataset [6] are shown in Fig. 1. In the first example, the lateral boundary of the thalamus appears very faint (Fig. 1a). In the second, the lateral boundary of the globus pallidus is visible thanks to the contrast with the neighboring putamen, but the medial boundary is not (Fig. 1c).

Fig. 1.

(a) Coronal plane across the thalami of a T1 scan from the HCP. (b) Corresponding map of principal diffusion directions, and manual delineation of the left thalamus. (c) Axial plane of the T1 scan across the basal ganglia, and manual delineation of the boundary between the putamen and globus pallidus. (d) Corresponding principal diffusion directions (weighted by FA), with the joint boundary of the putamen and pallidum (in yellow, visible in this map) and the contour from the T1 (in red).

These issues create a need for fusing data from several MR modalities to better delineate structure boundaries. A natural complement to sMRI is diffusion MRI (dMRI), which may help discriminate between certain tissue types, despite its lower resolution. For example, in Fig. 1b, the lateral boundary of the thalamus is clearly discernible in the principal diffusion direction map obtained from dMRI. The diffusion data also complement the T1 scan in the pallidum, which can be delineated by combining contours obtained from the two modalities (medial from dMRI, lateral from sMRI, see Fig. 1d).

Most prior work on segmentation of dMRI focuses on delineating white matter structures, using tractography [7,8,9] or volumetric segmentation [10,11]. Tractography has also been used to subdivide subcortical structures (e.g., thalamus [12], amygdala [13]) based on long-range connections. Surprisingly, the literature on joint modeling of multimodal sMRI/dMRI is sparse. When sMRI and dMRI are used by the same tool, this is most often done serially, e.g., a segmentation derived from sMRI is used to analyze the dMRI (e.g., to derive priors for Bayesian tractography [9]). To the best of our knowledge, the only works analyzing sMRI and dMRI simultaneously have been on thalamic nuclei segmentation with random forests [14,15]. The main concern with such discriminative techniques is their generalization ability to other datasets, which is limited by differences in MRI acquisition. For sMRI segmentation, this problem can be ameliorated with data augmentation and pretraining [4]. However, this is harder to do in dMRI, where acquisition protocols are much less standardized.

The ability to generalize across datasets is critical when software is released publicly and few assumptions can be made on the acquisition. In such scenarios, Bayesian segmentation methods that automatically estimate appearance models from input images remain very popular, as they are agnostic to the MRI contrast of the input scan, and thus robust to acquisition differences. These methods are used for tissue segmentation by major neuroimaging packages (e.g., Unified Segmentation [1] in SPM, and FAST [16] in FSL). However, they can be inaccurate when segmenting structures with poor sMRI contrast (see Fig. 1).

Here we propose a sequence-adaptive Bayesian algorithm that uses a probabilistic atlas to segment sMRI and dMRI data simultaneously⁵. This is achieved via a novel dMRI likelihood term, which relies on a hierarchical model for the fractional anisogropy (FA) and principal diffusion orientation. Combined with a Gaussian likelihood for sMRI, this model of image intensities is flexible enough to produce accurate segmentations, while keeping dimensionality low. We also propose a novel inference algorithm to automatically segment scans by fitting the model to multi-modal sMRI/dMRI data. Thanks to unsupervised intensity modeling, applicability across a wide range of acquisition protocols is achieved, which is demonstrated experimentally on two considerably different datasets.

2. Methods

2.1. Forward probabilistic model

The graphical model of our framework is shown in Fig. 2a. The observed variables are a bias field corrected (possibly multispectral) sMRI scan S = [s₁, . . . , s_V ] defined on V voxels, a dMRI scan D = [d₁, . . . , d_V ] defined at the same voxel coordinates (which might require resampling), and a probabilistic atlas A, which provides the probabilities of observing one of C neuroanatomical classes at every location across a reference spatial coordinate system. The model is governed by three sets of deterministic hyperparameters specified by the user: γ^a, γ^s and γ^d.

Fig. 2.

(a) Graphical model of proposed framework. (b) Hierarchical model of dMRI likelihood. Circles represent random variables (open if hidden, shaded if observed). Smaller solid circles are deterministic parameters. Plates indicate replication.

At the top of the generative model we find the atlas A, along with a set of related parameters θ^a that deform this atlas into the space of the MRI data. These parameters are a sample of a distribution that regularizes the deformation field by penalizing, e.g., its bending energy. The strength of the regularization is controlled by the set of hyperparameterst γ^a.

Given the deformed atlas, a labeling (segmentation) L = [l₁, . . . , l_V], with l_υ ∈ {1, . . . , C}, is obtained by independently sampling the categorical distribution defined by the deformed atlas at each voxel location. Given L, the observed sMRI and dMRI data are assumed to be conditionally independent from each other and across voxels. The sMRI data s_υ at υ follows a distribution (typically a Gaussian) whose parameters ${\mathit{\theta }}_c^s$ depend on the corresponding label c = l_υ. Any prior knowledge on these parameters is encoded in their priors, which are governed by hyperparameter vectors $\left\{{\mathit{\gamma }}_c^s\right\}$. Similarly, d_υ is also assumed to be a mixture conditioned on the segmentation, described by parameters $\left\{{\mathit{\theta }}_c^d\right\}$ and hyperparameters $\left\{{\mathit{\gamma }}_c^d\right\}$, which yields a symmetric likelihood model (Fig. 2a). The joint probability density function (PDF) of the model is therefore:

$$\begin{gathered} p\left(\mathit{S},\mathit{D},\mathit{L},{\mathit{\theta }}^a,{\mathit{\theta }}^s,{\mathit{\theta }}^d|\mathit{A},{\mathit{\gamma }}^a,{\mathit{\gamma }}^s,{\mathit{\gamma }}^d\right) \\ =p\left(\mathit{S}|\mathit{L},{\mathit{\theta }}^s\right)p\left(\mathit{D}|\mathit{L},{\mathit{\theta }}^d\right)p\left(\mathit{L}|\mathit{A},{\mathit{\theta }}^a\right)p\left({\mathit{\theta }}^s|{\mathit{\gamma }}^s\right)p\left({\mathit{\theta }}^d|{\mathit{\gamma }}^d\right)p\left({\mathit{\theta }}^a|{\mathit{\gamma }}^a\right) \\ =\left(\prod _{v=1}^{V}p({\mathit{s}}_v|{\mathit{\theta }}_{l_v}^s)p({\mathit{d}}_v|{\mathit{\theta }}_{l_v}^d)p_v(l_v|\mathit{A},{\mathit{\theta }}^a)\right)\left(\prod _{c=1}^{C}p\left({\mathit{\theta }}_c^s|{\mathit{\gamma }}_c^s\right)p\left({\mathit{\theta }}_c^d|{\mathit{\gamma }}_c^d\right)\right)p\left({\mathit{\theta }}^a|{\mathit{\gamma }}^a\right), \end{gathered}$$ (1)

where ${\mathit{\theta }}^s={\left\{{\mathit{\theta }}_c^s\right\}}_{c=1,\dots ,C}$, ${\mathit{\gamma }}^s={\left\{{\mathit{\gamma }}_c^s\right\}}_{c=1,\dots ,C}$, ${\mathit{\theta }}^d={\left\{{\mathit{\theta }}_c^d\right\}}_{c=1,\dots ,C}$, ${\mathit{\gamma }}^d={\left\{{\mathit{\gamma }}_c^d\right\}}_{c=1,\dots ,C}$.

2.2. Model instantiation

Probabilistic atlas

We follow the framework of the thalamic atlas [17] that we use in the experiments in Section 3, in which the atlas is encoded as a tetrahedral mesh. Deforming the mesh is penalized by a regularizer R, weighted by the mesh stiffness λ. The prior is given by (see further details in [18]):

$$p\left({\mathit{\theta }}^a|{\mathit{\gamma }}^a\right)\propto \exp \left[-\lambda R\left({\mathit{\theta }}^a\right)\right],\text{and}{l}_v\sim \mathrm{Cat}\left[{\mathit{A}}_v\left({\mathit{\theta }}^a\right)\right],$$ (2)

where A_υ(θ^a) = [A_υ1, . . . , A_υc]^t is simply the vector of C label probabilities provided by the atlas at voxel υ when deformed with parameters θ^a; Cat[·] is the categorical distribution; and hyperparameters γ^a comprise just λ, i.e., γ^a = {λ}.

Likelihood of sMRI

In order to model the sMRI intensities given the segmentation L, we follow the Bayesian brain MR segmentation literature (e.g., [1,16,17]) and use Gaussian intensity distributions, such that ${\mathit{\theta }}_c^s=\left\{{\mathit{\mu }}_c,{\mathit{\Sigma }}_c\right\}$, i.e., the mean and covariance of the intensities of class c. We place a Normal Inverse Wishart (NIW) distribution on these parameters (i.e., the conjugate prior), with zero degrees of freedom for the covariance, as we found it difficult in practice to inform such parameter a priori. Therefore, we have: ${\mathit{\gamma }}_c^s=\left\{{\mathit{M}}_c,n_c\right\}$, where M_c is the hypermean and n_c is the scale. The sMRI likelihood is thus:

$$\begin{gathered} {\mathit{\mu }}_c,{\mathit{\Sigma }}_c|{\mathit{M}}_c,n_c\sim \mathrm{NIW}\left({\mathit{M}}_c,n_c,0,0\mathit{I}\right), \\ {\mathit{s}}_v|l_v,\left\{{\mathit{\mu }}_c\right\},\left\{{\mathit{\Sigma }}_c\right\}\sim \mathcal{N}\left({\mathit{\mu }}_{l_v},{\mathit{\Sigma }}_{l_v}\right), \end{gathered}$$ (3)

where $\mathcal{N}(\cdot ,\cdot )$ is the Gaussian distribution and I is the identity matrix.

Likelihood of dMRI

We have two requirements for the likelihood function of the dMRI: low demands on gradient directions and b-values to accommodate legacy data; and low number of parameters to facilitate unsupervised clustering (yet sufficient to separate the classes). To satisfy the first requirement, we adopt the diffusion tensor imaging (DTI) model, which can be fit from virtually all available dMRI data. Rather than modeling the tensors directly (e.g., with a Wishart distribution, which we found in pilot experiments to fade too quickly from its mode), we use a hierarchical model (Fig. 2b) that only considers the FA f_υ and the principal eigenvector ϕ_υ at each voxel, i.e., d_υ = {f_υ, ϕ_υ}.

At the first level, we model the FA conditioned on the class, with Beta distributions parameterized by {α_c, β_c}. We chose the Beta because it can model location and dispersion of signals defined on the [0,1] interval with two parameters. At the second level, we model the principal eigenvector with the Dimroth-Scheidegger-Watson (DSW) distribution, which is axial (i.e., antipodally symmetric), accommodating the directional invariance of dMRI [19]. This distribution is also rotationally symmetric around a mean direction ψ and its opposite −ψ (∥ψ∥ = 1), with a dispersion around the mean parameterized by a concentration κ. It has fewer parameters than other axial distributions, such as the (non rotationally symmetric) Bingham. Its PDF is given by [20]:

$$f(\mathit{\varphi };\mathit{\psi },\kappa )={[Z(\kappa )]}^{-1}\exp \left[\kappa {\left({\mathit{\psi }}^t\mathit{\varphi }\right)}^2\right],$$ (4)

with domain ∥ϕ∥ = 1, and where the partition function is the Kummer function in 3D [20]: $Z\kappa ={\int }_0^1\exp \left(\kappa t^2\right)dt$. We further assume that the concentration is modulated (multiplied) by the FA. This is a simple yet effective way of modeling the higher directional dispersion in voxels with low FA (e.g., in areas of unrestricted diffusion or fiber crossings), without having to resort to mixtures or additional parameters. The overall model for the dMRI likelihood is thus:

$$\begin{gathered} f_v|l_v,\left\{{\alpha }_c\right\},\left\{{\beta }_c\right\}\sim \mathrm{Beta}\left({\alpha }_{l_v},{\beta }_{l_v}\right), \\ {\mathit{\varphi }}_v|l_v,f_v,\left\{{\mathit{\psi }}_c\right\},\left\{{\kappa }_c\right\}\sim \mathrm{DSW}\left({\mathit{\psi }}_{l_v},f_v{\kappa }_{l_v}\right), \end{gathered}$$ (5)

and the set of parameters is thus: ${\mathit{\theta }}_c^d=\left\{{\alpha }_c,{\beta }_c,{\mathit{\psi }}_c,{\kappa }_c\right\}$, with ∥ψ_c∥ = 1,∀c. We decided not to inform these parameters, such that $p\left({\mathit{\theta }}_c^d\right)\propto 1$, and ${\mathit{\gamma }}_{\mathit{c}}^{\mathit{d}}=\varnothing$. We note that this likelihood model defines a PDF on the unit ball for vector f_υϕ_υ.

2.3. Segmentation as Bayesian inference

Within our joint generative model of sMRI and dMRI, we pose segmentation as an optimization problem, seeking to maximize the posterior probability of the labeling, given the known hyperparameters and observed input data:

$$\begin{gathered} \underset{\mathit{L}}{\mathrm{argmax}}\int \int \int p\left(\mathit{L}|{\mathit{\theta }}^a,{\mathit{\theta }}^s,{\mathit{\theta }}^d,\mathit{S},\mathit{D},\mathit{A}\right)p\left({\mathit{\theta }}^a,{\mathit{\theta }}^s,{\mathit{\theta }}^d|\mathit{S},\mathit{D},\mathit{A},{\mathit{\gamma }}^a,{\mathit{\gamma }}^s\right)d{\mathit{\theta }}^{a}d{\mathit{\theta }}^{s}d{\mathit{\theta }}^d, \\ \approx \underset{\mathit{L}}{\mathrm{argmax}}p\left(\mathit{L}|{\widehat{\mathit{\theta }}}^a,{\widehat{\mathit{\theta }}}^s,{\widehat{\mathit{\theta }}}^d,\mathit{S},\mathit{D},\mathit{A}\right), \end{gathered}$$ (6)

where we have made the standard approximation that the posterior distribution of the parameters is heavily peaked around point estimates ${\widehat{\mathit{\theta }}}^a$, ${\widehat{\mathit{\theta }}}^s$, ${\widehat{\mathit{\theta }}}^d$ given by:

$$\{{\widehat{\mathit{\theta }}}^a,{\widehat{\mathit{\theta }}}^s,{\widehat{\mathit{\theta }}}^d\}=\underset{\{{\mathit{\theta }}^a,{\mathit{\theta }}^s,{\mathit{\theta }}^d\}}{\mathrm{argmax}}p\left({\mathit{\theta }}^a,{\mathit{\theta }}^s,{\mathit{\theta }}^d|\mathit{S},\mathit{D},\mathit{A},{\mathit{\gamma }}^a,{\mathit{\gamma }}^s\right).$$ (7)

Therefore, we segment a scan by first estimating the parameters with Eq. 7, and then obtaining the (approximate) most likely labeling with Eq. 6.

Applying Bayes rule to Eq. 7, marginalizing over the hidden segmentation L, and considering the structure of the model and our design choices, we obtain:

$$\{{\widehat{\mathit{\theta }}}^a,{\widehat{\mathit{\theta }}}^s,{\widehat{\mathit{\theta }}}^d\}=\underset{\{{\mathit{\theta }}^a,{\mathit{\theta }}^s,{\mathit{\theta }}^d\}}{\mathrm{argmax}}p\left({\mathit{\theta }}^a|{\mathit{\gamma }}^a\right)p\left({\mathit{\theta }}^s|{\mathit{\gamma }}^s\right)\sum _{\mathit{L}}p\left(\mathit{S}|\mathit{L},{\mathit{\theta }}^s\right)p\left(\mathit{D}|\mathit{L},{\mathit{\theta }}^d\right)p\left(\mathit{L}|{\mathit{\theta }}^a,\mathit{A}\right).$$

Expanding and taking logarithm, we obtain the following objective function:

$$\begin{gathered} O\left({\mathit{\theta }}^a,\left\{{\mathit{\mu }}_c,{\mathit{\Sigma }}_c,{\alpha }_c,{\beta }_c,{\mathit{\psi }}_c,{\kappa }_c\right\}\right)=\log p\left({\mathit{\theta }}^a|{\mathit{\gamma }}^a\right)+\sum _{c=1}^C\log p\left({\mathit{\mu }}_c,{\mathit{\Sigma }}_c|{\mathit{M}}_c,n_c\right) \\ +\sum _{v=1}^V\log \left[\sum _{c=1}^{C}p\left({\mathit{s}}_v|{\mathit{\mu }}_c,{\mathit{\Sigma }}_c\right)p\left(f_v|{\alpha }_c,{\beta }_c\right)p\left({\mathit{\varphi }}_v|f_v,{\mathit{\psi }}_c,{\kappa }_c\right)p\left(l_v=c|\mathit{A},{\mathit{\theta }}^a\right)\right]. \end{gathered}$$ (8)

We maximize Eq. 8 with a Generalized Expectation Maximization (GEM) algorithm [21], iterating between expectation (E) and maximization (M) steps:

E step: In the E step, we use Jensen’s inequality to build a lower bound for the objective function, which touches it at the current value of the parameters:

$$\begin{gathered} O\ge Q=\sum _{v=1}^V\sum _{c=1}^C{w}_{vc}\log \left[p\left({\mathit{s}}_v|{\mathit{\mu }}_c,{\mathit{\Sigma }}_c\right)p\left(f_v|{\alpha }_c,{\beta }_c\right)p\left({\mathit{\varphi }}_v|f_v,{\mathit{\psi }}_c,{\kappa }_c\right)p\left(l_v=c|\mathit{A},{\mathit{\theta }}^a\right)\right] \\ +\log p\left({\mathit{\theta }}^a|{\mathit{\gamma }}^a\right)+\sum _{c=1}^C\log p\left({\mathit{\mu }}_c,{\mathit{\Sigma }}_c|{\mathit{M}}_c,n_c\right)-\sum _{v=1}^V\sum _{c=1}^C{w}_{vc}\log w_{vc}, \end{gathered}$$ (9)

where {ω_υc} a soft segmentation according to the current parameter estimates:

$$\begin{gathered} w_{vc}^{\prime }=p\left({\mathit{s}}_v|{\mathit{\mu }}_c,{\mathit{\Sigma }}_c\right)p\left(f_v|{\alpha }_c,{\beta }_c\right)p\left({\mathit{\varphi }}_v|f_v,{\mathit{\psi }}_c,{\kappa }_c\right)p\left(l_v=c|\mathit{A},{\mathit{\theta }}^a\right) \\ ={\left|{\mathit{\Sigma }}_c\right|}^{-1/2}\exp \left[-\frac{1}{2}{\left({\mathit{s}}_v-{\mathit{\mu }}_c\right)}^t{\mathit{\Sigma }}_c^{-1}\left({\mathit{s}}_v-{\mathit{\mu }}_c\right)\right]f_v^{{\alpha }_c-1}{\left(1-f_v\right)}^{{\beta }_c-1}{\left[B\left({\alpha }_c,{\beta }_c\right)\right]}^{-1} \\ \times {\left[Z\left(f_v{\kappa }_c\right)\right]}^{-1}\exp \left[f_v{\kappa }_c{\left({\mathit{\psi }}_c^t{\mathit{\varphi }}_v\right)}^2\right]A_{vc}\left({\mathit{\theta }}^a\right),\text{and}{w}_{vc}=w_{vc}^{\prime }/\sum _{c^{\prime }=1}^C{w}_{v{c}^{\prime }}^{\prime }, \end{gathered}$$ (10)

where B is the Beta function.

M step: In the generalized M step, we seek to improve the lower bound Q in Eq. 9. While optimizing the bound with respect to all parameters simultaneously is difficult, optimizing different subsets each time (coordinate ascent) is feasible.

Optimizing θ^a: Fixing all other parameters and switching signs, we obtain:

$$\underset{{\mathit{\theta }}^a}{\mathrm{argmin}}\sum _{v=1}^V\sum _{c=1}^C{w}_{vc}\log \left[w_{vc}/A_{vc}\left({\mathit{\theta }}^a\right)\right]+\lambda R\left({\mathit{\theta }}^a\right).$$ (11)

This is a registration problem combining the regularizer R with a data term: the Kullback–Leibler (KL) divergence between the deformed atlas and the current soft segmentation. We solve it numerically with the conjugate gradient method.

Optimizing {μ_c, Σ_c}: Setting derivatives to zero yields a closed-form solution:

$${\mathit{\mu }}_c=\frac{n_c{\mathit{M}}_c+{\sum }_{v=1}^V{w}_{vc}{\mathit{s}}_v}{n_c+{\sum }_{v=1}^V{w}_{vc}},$$ (12)

$${\mathit{\Sigma }}_c=\frac{n_c\left({\mathit{\mu }}_c-{\mathit{M}}_c\right){\left({\mathit{\mu }}_c-{\mathit{M}}_c\right)}^t+{\sum }_{v=1}^V{w}_{vc}\left({\mathit{s}}_v-{\mathit{\mu }}_c\right){\left({\mathit{s}}_v-{\mathit{\mu }}_c\right)}^t}{1+{\sum }_{v=1}^V{w}_{vc}}.$$ (13)

Optimizing {α_c, β_c}: Substituting the expression of the Beta distribution into Eq. 9, the problem decouples across classes:

$$\underset{{\alpha }_c,{\beta }_c}{\mathrm{argmax}}\left({\alpha }_c-1\right)\sum _{v=1}^V{w}_{vc}\log f_v+\left({\beta }_c-1\right)\sum _{v=1}^V{w}_{vc}\log \left(1-f_v\right)-\log B\left({\alpha }_c,{\beta }_c\right)\sum _{v=1}^V{w}_{vc}.$$ (14)

This is a simple 2D optimization problem, which we solve with conjugate gradient. In the first iteration, we use the method of moments for initialization.

Optimizing {ψ_c}: This optimization can also be carried out one c at the time:

$$\underset{{\mathit{\psi }}_c:\Vert {\mathit{\psi }}_c\Vert =1}{\mathrm{argmax}}\sum _v{w}_{vc}{f}_v{\left({\mathit{\psi }}_c^t{\mathit{\varphi }}_v\right)}^2=\underset{{\mathit{\psi }}_c:\Vert {\mathit{\psi }}_c\Vert =1}{\mathrm{argmax}}{\mathit{\psi }}_c^t\left[{\sum }_v{w}_{vc}{f}_v{\mathit{\varphi }}_v{\mathit{\varphi }}_v^t\right]{\mathit{\psi }}_c,$$

with closed-form solution given by the leading eigenvector of: ${\sum }_v{w}_{vc}{f}_v{\mathit{\varphi }}_v{\mathit{\varphi }}_v^t$.

Optimizing {κ_c}: This optimization problem also decouples across classes:

$$\underset{{\kappa }_c}{\mathrm{argmax}}{\kappa }_c\sum _{v=1}^V{w}_{vc}{f}_v{\left({\mathit{\psi }}_c^t{\mathit{\varphi }}_v\right)}^2-\sum _{v=1}^V{w}_{vc}\log Z\left(f_v{\kappa }_c\right),$$ (15)

which we solve with conjugate gradient, initializing κ_c = 10 in the first iteration.

Final Segmentation

It is straightforward to show that the approximate posterior probability of the segmentation from Eq. 6 factorizes across voxels and is given by $p(\mathit{L}|{\widehat{\mathit{\theta }}}^a,{\widehat{\mathit{\theta }}}^s,{\widehat{\mathit{\theta }}}^d,\mathit{S},\mathit{D},\mathit{A},{\mathit{\gamma }}^a,{\mathit{\gamma }}^s)={\prod }_{v=1}^V{\widehat{w}}_{v,l_j}$, where ${\widehat{w}}_{v,l_j}$ is obtained by evaluating Eq. 10 at the optimal parameter values ${\widehat{\mathit{\theta }}}^a$, ${\widehat{\mathit{\theta }}}^s$, ${\widehat{\mathit{\theta }}}^d$. Therefore, the optimal segmentation can be computed independently at each location υ as:

$${\widehat{l}}_v=\underset{c}{\mathrm{argmax}}{\widehat{w}}_{vc},$$ (16)

and the expected value of the volume of class c is given by: ${\sum }_{v=1}^V{\widehat{w}}_{vc}$ (in voxels).

Algorithm 1.

Bayesian segmentation with sMRI and dMRI

Require: A, S, D, γa, γs, {Mc, nc}

Ensure: ${\widehat{\mathit{\theta }}}^a$, $\{{\widehat{\mathit{\mu }}}_c,{\widehat{\mathit{\Sigma }}}_c,{\widehat{\alpha }}_c,{\widehat{\beta }}_c,{\widehat{\mathit{\psi }}}_c,{\widehat{\kappa }}_c\}$

Initialize θa, with affine registration and mutual information

Initialize ωυc ← p(lυ = c|A, θa), ∀υ, c

Initialize κc = 10,∀c

Initialize αc, βc with method of moments, ∀c

it ← 0

while θa, {μc, Σc, αc, βc, ψc, κc} change AND it ≤ itmax do

it ← it + 1

for c = 1, . . . , C do

Update μc, Σc with Eqns. 12 and 13

Update αc, βc by numerically optimizing Eq. 14 with conjugate gradient

Update ψc ← leading eigenvector of: ${\sum }_v{w}_{vc}{f}_v{\varphi }_v{\varphi }_v^t$

Update κc by numerically optimizing Eq. 15 with conjugate gradient

end for

if mod(its,5)=0 then

Update θa by numerically optimizing Eq. 11 with conjugate gradient

end if

Update ωυc with Eq. 10, ∀υ, c

end while

Compute final segmentation with Eq. 16.

Implementation details

Since GEM only requires improving the bound at each iteration, we follow a schedule in which all the model parameters except for θ^a are updated once in the M step. Since updating θ^a requires solving a more computationally expensive registration problem, we only update θ^a in the M step every five GEM iterations. The method is summarized in Algorithm 1.

In practice, we also force some parameters $\left\{{\mathit{\theta }}_c^s\right\}$ and $\left\{{\mathit{\theta }}_c^d\right\}$ to be shared across classes, for increased robustness of the algorithm. For the sMRI parameters $\left(\left\{{\mu }_c,{\sigma }_c^2\right\}\right)$, we follow [17] and force parameter sharing across: cortex, hippocampus and amygdala; reticular nucleus and white matter; mediodorsal and pulvinar nuclei; rest of thalamic nuclei; and contralateral structures. For the FA, parameters {α_c, β_c} are shared across each of the six groups of thalamic nuclei in Table 2 of [17], and across contralateral structures. The same grouping – but without contralateral constraints – is used for the directional parameters {ψ_c, κ_c}.

3. Experiments and results

3.1. Data

We evaluate our method with a recent probabilistic atlas of 25 thalamic nuclei and surrounding regions derived from histology [17]. The thalamus is an excellent target region, due to its faint lateral boundaries in sMRI (as explained in Section 1), and its set of nuclei with different connectivity. We use two considerably different datasets in evaluation: HCP (state of the art) and ADNI (legacy).

HCP: Isotropic T1 and dMRI scans from 100 healthy subjects (age 29.1±3.3, 44 males), at 0.7 mm (T1) and 1.25 mm resolution (dMRI). We fit the DTI model to the b=1000 s/mm² shell (180 directions) and 12 scans with b=0 (details in [22]).

ADNI: T1 and dMRI scans from 77 subjects from ADNI2: 39 Alzheimer’s disease (AD) and 38 age-matched controls (74.1±8.1 years; 40 females total). T1 resolution: 1.2×1×1 mm (sagittal); dMRI resolution:1.35×1.35×2.7 mm (axial); 5 volumes with b=0, 41 directions (b=1000 s/mm²; details at adni-info.org).

3.2. Experimental setup

We evaluate three competing methods: (i) Segmentation of the whole thalamus with FreeSurfer [2]; (ii) Segmentation of thalamic nuclei using Bayesian segmentation on T1 only [17]; and (iii) Segmentation of thalamic nuclei with the full model, including dMRI. We compare these approaches in three experiments: (i) Qualitative assessment of segmentation and tractography in HCP; (ii) Correlation between thalamic and total intracranial volume (ICV) in HCP; and (iii) Ability to discriminate AD and control subjects based on volumes in ADNI. The sMRI and dMRI data are resampled to 0.5 mm isotropic in a bounding box around the thalami (DTI is interpolated in a log-euclidean framework [23]). We set λ = 0.05 as in [17], M_c to the median T1 intensity in class c according to the main FreeSurfer segmentation, and n_c to the volume of the class in mm³.

3.3. Results

Figure 3 shows qualitative results on an HCP subject. FreeSurfer almost completely misses the left pallidum (yellow arrow in the figure) and oversegments the thalami. We test the effects of the latter on tractography by reconstructing the full dMRI data with generalized q-sampling [24], performing whole-brain tractography, and isolating the tracts that intersect the whole thalami, as automatically segmented by the three competing methods. The FreeSurfer thalamus yields many false positive tracts, mostly due to overlap with the internal capsule (red arrow). Aggregating the nuclei produced by Bayesian segmentation on the T1 produces a more accurate boundary, but still oversegments the anterior thalamus (white arrow), and undersegments the pulvinar nucleus (black arrow). Our multi-modal method yields less false positive tracts, and segments thalamic nuclei that are more homogeneous in terms of diffusion orientation and FA.

Fig. 3.

Top two rows: axial slice of a T1 scan and principal diffusion directions of an HCP subject, with segmentations from FreeSurfer, Bayesian segmentation (T1 only), and the proposed method (T1+dMRI). Bottom row (left to right): Whole-brain tractography (25,000 tracts); subset of tracts going through the thalami (in yellow) as segmented by: FreeSurfer (2,602 tracts); Bayesian segmentation of T1 (2,193 tracts); and proposed method (1,676 tracts). See Section 3.3 for a description of the arrows.

We also evaluate segmentation performance quantitatively on HCP, in an indirect fashion, by computing the correlation of total thalamic volume obtained by each method (left-right averaged) with the ICV estimated by FreeSurfer; noisy thalamic segmentations are expected to degrade this correlation. Scatter plots and regression lines are shown in Fig. 4. The FreeSurfer volumes are quite large on average, and their correlation with ICV is ρ=0.71. Bayesian segmentation with T1 yields ρ=0.68 (not significantly different, with p=0.37 on a two-tailed Steiger test). The proposed algorithm produces fewer outliers than the other two, and yields the highest correlation (ρ=0.81), significantly higher than those of FreeSurfer (p=0.006) and T1-only segmentation (p=0.001).

Fig. 4.

Scatter plot for intracranial vs. whole thalamic volumes (left-right averaged), and regression lines (black) with 95% confidence intervals (in red).

Finally, we evaluate the ability of the segmented volumes to classify the ADNI subjects into AD and controls. We use a simple linear discriminant analysis, whose performance is mostly determined by the quality of the volumes. We project the volumes (left-right averaged, corrected for ICV and age) onto the normal to the discriminant hyperplane in a leave-one-out fashion. We use the projections to compute the area under the ROC curve (AUC), accuracy at its elbow, and sample sizes (α = 0.05, β = 0.2). Results are shown in Table 1. Our method yields a fair improvement compared with T1-only Bayesian segmentation (increase of 7 points in accuracy and AUC, and reduction of 6 samples). Compared with FreeSurfer, our algorithm reduces the sample size by 60%.

Table 1.

AUC, accuracy at elbow, and sample size for the AD classification experiment.

Method	FreeSurfer (whole)	Bayes. Seg. T1	Proposed
AUC	60.3%	66.5%	73.6%
Accuracy at elbow	61.0%	67.5%	74.0%
Sample size	50	26	20

4. Conclusion

We have presented a Bayesian method for joint segmentation of sMRI and dMRI, which is robust to changes in acquisition platform and protocol – as shown with two substantially different datasets. Compared with Bayesian segmentation using sMRI alone, our method produces more accurate boundaries for subcortical structures, and yields smaller sample sizes in an AD classification task.

Future work at the methodological level will follow five main directions: (i) Modeling partial voluming in the dMRI, which may be important for smaller structures; (ii) Exploring other axial PDFs, as well as mixtures; (iii) Placing a prior on the dMRI likelihood parameters, e.g., to utilize prior knowledge on the FA; (iv) Modeling the bias field in the sMRI data, e.g., as in [1]; and (v) Adding connectivity derived from tractography to the dMRI likelihood, which may be challenging because tractography results depend largely on the MR acquisition.

We also plan to manually trace structures some of the HCP and AD data, with three purposes. First, to include white matter bundles in the atlas, as modeling the whole cerebral white matter with a single Beta-DSW is not realistic (not even within a bounding box). Second, to enable direct evaluation of our segmentations. And third, to help us explain discrepancies in AD classification accuracy between our results and those presented in [17], which may be due to the their larger dataset, their different ADNI sample, or some other factor.

As high-resolution dMRI becomes more common in neuroimaging, we believe that segmentation techniques that jointly model gray and white matter with sMRI and dMRI – like the one in this paper – will become increasingly important.