Two-stage multishape segmentation of brain structures using image intensity, tissue type, and location information

Abstract

Purpose: The authors propose a fast, robust, nonparametric, entropy-based, coupled, multishape approach to segment subcortical brain structures from magnetic resonance images (MRIs).

Methods: The proposed method uses three types of information: Image intensity, tissue types, and locations of structures. The image intensity information is captured by estimating the probability density function (pdf) of the image intensities in each structure. The tissue type information is captured by applying an unsupervised tissue segmentation method to the image and estimating a probability mass function (pmf) for the tissue type of each structure. The location information is captured by estimating pdf of the location of each structure from the training datasets. The resulting pmf’s and pdf’s are used to define an entropy function whose minimum corresponds to a desirable segmentation of the structures. The authors propose a three-step optimization strategy for the segmentation method. In the first step, a powerful automatic initialization method is developed based on tissue type and location information of the structures. In the second step, a quasi-Newton method is used to optimize the parameters of the energy function. To speed up the iterations, derivatives of the energy function with respect to its parameters are analytically derived and used in the optimization process. In the last step, the limitations related to the prior shape model are removed and a level-set method is applied for the fine tuning of the segmentation results.

Results: The proposed method is applied to two different datasets and the results are compared to those of previous methods in literature. Experimental results are presented for lateral ventricles, caudate, thalamus, putamen, pallidum, hippocampus, and amygdala.

Conclusions: The results illustrate superior performance of the proposed segmentation method compared to other methods in literature. The execution time of the algorithm is a few minutes, suitable for a variety of applications.

INTRODUCTION

A major category of the methods proposed for the segmentation of brain structures from magnetic resonance images (MRIs) optimizes an energy function with several parameters that represent the underlying shapes. In the definition of the energy function, earlier methods use the boundary information for the structures of interest.^{1, 2, 3} Later methods use regional information such as intensity histogram (parametric and nonparametric, offline or online) or the intensity variance in an area.^{4, 5, 6} Others combine boundary and regional information.⁷ Recent methods benefit from a priori knowledge about the structures of interest.^{8, 9, 10, 11} This makes the segmentation process robust to the imperfect image conditions. For the methods developed based on the a priori information, a registration process is essential to integrate the prior model into the segmentation process. Many of these methods use probabilistic techniques to represent this information. For example, Chupin et al.¹² introduced a method driven by hybrid constraints. They defined an energy function based on global and local data attachments and nonstationary anisotropic Markovian regularization terms. In addition, they applied two additional terms for volume and surface control. Their method has many parameters and is well adapted for their specific datasets.

The anatomical structures in the brain are related to the neighboring structures through their location, size, orientation, and shape. An integration of these relations into the segmentation process improves accuracy and robustness.^{13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23} Litvin et al.¹⁸ introduced shape distribution as a new concept for the segmentation of coupled objects. Their model is constructed from distributions of features related to the shape. Their method is 2D and its extension to 3D is not reported yet. Addition of new terms in their energy function leads to challenges in the calculation of the derivatives required for the curve evolution. The deformable models introduced in Refs. ^{13, 21, 23} define coupled shape models of multiple structures. Akselrod-Ballin et al.¹⁴ proposed a knowledge-based multiscale segmentation method that applied a graph representation in different levels. They used the probabilistic information derived from an atlas and a likelihood function estimated from the training datasets.

Tu and Toga²² developed a hybrid method that applied a multiclass classifier for the learning∕computing of the multiclass discriminative models and a learned edge field to constrain the region boundaries. Bazin and Pham¹⁵ introduced a segmentation method that used a statistical and topological atlas generated from the training data along with some pre-existing general atlases. Development of the topological atlas requires manual editing and is thus semiautomatic. The authors applied the energy function used in the FANTASM method.²⁴ Corso et al.¹⁶ proposed a graph shifts algorithm using a dynamical hierarchical representation of the image. The terms in the energy function were learned from the training data. Wu and Chung²⁵ introduced a method based on Markov dependence tree. In this method, a framework for segmentation of multiple brain structures is introduced that uses edge, region, and partial Hausdorff distance. The partial Hausdorff distance is used to define constraints that minimize the distance between the boundaries of objects and the image edge maps.

In this paper, we propose a 3D segmentation method, which considers coupling information of the shapes of the related structures in a nonparametric entropy-based energy function. We use principal component analysis (PCA) to extract principal shapes of different structures.^{13, 21, 23} The proposed method is robust, fast, and accurate with a small number of parameters to set. It integrates information obtained from different sources in the energy function. In addition, while using the prior knowledge for the extraction of the shape model, it allows flexibility of fine-tuning by relaxing constraints related to the shape model in the final stage of the algorithm. There is another category of segmentation methods in literature that has used prior knowledge-based terms in the energy function to limit shape variation and gain flexibility.^{9, 10} However, these methods are sensitive to the weights used for different energy terms. If the weights of the shape constraints, extracted from the training datasets to limit the shape variation around a mean shape, are large, there will be almost no flexibility. In other words, the shape of the segmented structure is limited to the mean shapes extracted from the training datasets. On the other hand, if these weights are small, the segmentation loses robustness because the information extracted from the training datasets is not used. Moreover, these types of methods are sensitive to the initialization.

We consider tissue type information in the energy function by constructing of a probability mass function (pmf) using a fuzzy membership function. In addition, we consider position and shape of the structures to define a probability density function (pdf) for the location of each structure. The energy function is built based on this information and nonparametric pdf’s for the intensity of the structures. In our method for shape representation, we use the Euclidean sign distance function as a powerful shape representation method.¹⁰ The energy function has two types of parameters, which are the weights of the principal shapes and the transformation parameters of each shape for the local alignment. To find the minimum of the energy function, first we use a powerful initialization strategy using tissue type and location information. Then, we apply a quasi-Newton based algorithm to optimize the parameters. In the third step, we utilize the segmentation result of the previous step as the initialization of a level-set to remove the limitations of the shape deformation due to the principal shapes. Figure 1 shows the general steps of the proposed method.

Figure 1.

Steps of the proposed method in the training and test phases.

The main contributions of this work are threefold: (1) An integration of the location and tissue type information with a prior knowledge of the structures in a new energy function, (2) a multistage optimization process, and (3) a powerful initialization approach. To see the effectiveness of each part of the algorithm, we apply the proposed method in four steps. In the first step, we use the training datasets for the pdf estimation and use the coupling of the shape and pose. We name this method “PCA-Only.” This is a modified version of our method in Ref. 13 that improves the registration, the selection of the number of the principal components, and the optimization procedures. In the second step, we use a modified initialization, update the intensity pdf’s iteratively, and also used location and tissue type information by adding the pdf for the location and the pmf for the tissue type information. Because of the modification in the probability function relative to the PCA-Only, we name it “Modified-Probability.” In the third step, we change the coupling method in the Modified-Probability to consider only the shape coupling and thus name it “Shape-Coupled.” Finally, in the last step, we add the fine-tuning algorithm to the Shape-Coupled and name it “Fine-Tuned.” For convenience, these names are used throughout the paper. Table 1 compares these four methods based on their features. The rest of the paper is organized as follows. Section 2 explains the proposed shape model and its properties. In Sec. 3, we explore the energy function and the proposed method for its optimization. Experimental data, procedures, and results are presented in Sec. 4. Discussion and conclusions are presented in Secs. 5, 6, respectively.

Table 1.

Brief comparison of four different implemented methods based on the utilized features.

	Shape coupling	Initialization	Online pdf estimation	Location and tissue type	Fine-tuning	Pose coupling
Fine-Tuned	√	√	√	√	√	×
Shape-Coupled	√	√	√	√	×	×
Modified-Probability	√	√	√	√	×	√
PCA-Only	√	×	×	×	×	√

SHAPE MODEL

In many segmentation methods, a shape model is used where richer models generate more accurate results. A shape model can be more sophisticated than a shape representation, but for the sake of simplicity, we use them as synonyms here. Several shape representation methods are used in literature.^{24, 25, 26, 27, 28, 29} Cootes et al.²⁷ introduced a point based method. Hong et al.²⁸ used kernel integrals for shape representation and Pizer et al.²⁹ used medial shape representation. Powerful methods for shape representation are based on distance function, implicit representation, and relationships among different shapes including pose, orientation, and other geometrical relations.^{30, 31}

Colliot et al.³⁰ used fuzzy forces to describe spatial relation of different structures and Bahman-Bijari et al.³¹ used symmetry properties of structures for segmentation. Scherrer et al.³² used a unified Markov random field framework for the segmentation of the tissue types as well as specific structures. They use location constraints provided by fuzzy descriptor as a priori anatomical knowledge to improve the segmentation results. Ciofolo and Barillot³³ introduced a segmentation method using level set and a fuzzy decision system. In their method, the initial contours of multiple structures evolve using the fuzzy decision system that combines a priori knowledge extracted from an anatomical atlas with the intensity pdf of the image and relative positions of the contours.

In this paper, shape relation and individual transformations are used to develop shape models of the brain structures. To this end, PCA is applied on the 3D training datasets. Each training dataset has a label map that shows manual segmentation of the structures by the expert. In this image, each structure has a specific label (fixed value∕intensity). Before applying PCA, the label maps are aligned to an atlas. The atlas has two aligned images: Intensity image and the label map that assigns to each voxel a label of a specific structure. There are two ways to align the label maps of the training datasets to the label map of the atlas. The label map of a specific structure can be aligned individually or the labels of all of the structures can be aligned collectively. When the label map that includes all of the structures is used for alignment, in effect, both of the shape and pose information are used for the coupling. This alignment is used in the PCA-Only and Modified-Probability. On the other hand, when the label maps of the structures are aligned individually, then in effect, only the shape information is used. This alignment is used in the Shape-Coupled and Fine-Tuned. The alignment method is described next.

Alignment of training datasets for shape variability extraction

In knowledge-based segmentation methods that use shape variation, alignment is a critical step. It reduces shape variability due to misalignment of different subjects. A good review of the basic registration methods can be found in Ref. 34. For 3D objects, several transformations such as similarity, affine, and Euler can be used. The Euler transformation is applicable when there is no scale difference between the moving and the target images. The affine transformation is applicable when there is a small misalignment. In addition, it has 12 parameters, making the optimization process complex. Therefore, we first apply a similarity transformation with seven parameters (one scaling, three rotations, and three translations). Then, we apply an affine transformation for the fine-tuning of the registration process.

For the metric, several options are available in literature. They vary based on the types of the images used as the moving and target (reference) images. Since our goal is to align the labels, the metric in our implementation is a cardinality metric that computes the number of pixels that are not matched between the moving and target images. This metric is equal to zero when all of the pixels are matched. It can be used for both of the cases described above.

For interpolation of binary images, we use a nearest neighbor interpolator. For the optimization, since the proposed metric is nondifferentiable, we use an optimizer that does not need derivatives. This is the Nelder–Mead (amoeba) method, which is a popular direct search method for minimizing unconstrained real functions with our desired property.³⁵

Now, suppose that there are n 3D labeled images available for the training, each with m structures labeled as {l₁,l₂,…,l_m}. Based on the methodology used for the coupling, we align the label maps. For each structure, we define a 3D binary image that represents the desired structure with one (foreground) and other pixels with zero (background). Therefore, for each structure, there are n 3D binary images that are aligned. The results are used to extract the variability of the shape of each structure in the training datasets.

To achieve an accurate and robust registration, a good initialization is needed. We use the difference between the centers of mass of the moving and reference images as the initial translation parameters, the angle between the main principal axes of the two images as the initial rotation parameter, and 1 for the initial scale parameter. Using these initializations, the method does not have any problems with the small sizes of the structures. We do not make any assumptions about the images and do not use any regions of interest (ROIs) at this point.

Using the above methods, we extract shape variability of the desired structures for model construction as explained in Sec. 2B. It should be noticed that all of the other parts of the segmentation algorithm are the same for both of the coupling methods described above.

Implicit parametric shape representation

As stated in Ref. 10, implicit parametric shape representation has advantages such as computational efficiency, accuracy, capturing wide range of shape variability, and handling topological changes. We use a distance map for shape representation that is zero on the boundary of a structure and in other points is the Euclidian distance from the boundary (negative inside, positive outside). For simultaneously segmenting m structures that are coupled, we embed their boundaries as the zero level sets of the corresponding signed distance functions {ψ₁,ψ₂,…,ψ_m}. We have n aligned binary images for each one of the m desired structures.

After extracting n distance maps for each one of the m desired structures (${\psi }_k^i$ shows the distance map of the kth structure of the ith dataset), we subtract the mean distance map of each structure, computed by averaging of the training datasets $\left({\overline{\Psi }}_k\right)$, from each of the n signed distance maps to remove similar parts in different shapes and show them with ${\tilde{\psi }}_k^i$,

$${\overline{\Psi }}_k=\frac{1}{n}\sum _{i=1}^n{\psi }_k^i,$$$${\tilde{\psi }}_k^i={\psi }_k^i-{\overline{\Psi }}_k.$$ 1

We use these n×m maps to show the variability of different structures in the training dataset. We collect n column vectors in matrix $S=\left\{{\tilde{\psi }}^1,{\tilde{\psi }}^2,\dots ,{\tilde{\psi }}^n\right\}$ with size mN_xN_yN_z×n, where N_x and N_y are the number of pixels in the x and y directions in each slice and N_z is the number of slices. Each column is built from a concatenation of the vectors that are the lexicographical order of N_x×N_y×N_z matrices $\left\{{\tilde{\psi }}_1^i,{\tilde{\psi }}_2^i,\dots ,{\tilde{\psi }}_m^i\right\}$. We need the eigenvectors of SS^T to define the eigenshapes. However, SS^T is a very large matrix. To minimize computational complexity, we find the eigenvalues of S^TS and call its eigenvectors dⁱ. Then, we find the eigenvectors of SS^T using the transformation ϕⁱ=Sdⁱ. This generates up to n different eigenshapes for each of the m structures, denoted by ${\varphi }_k^i$. Each one of these eigenshapes includes the relationships between different structures and takes the coupling into account. This is because the data from all structures are used in the definition of the eigenshapes. When the label maps of a training dataset are aligned individually, the shape relation of the structures is encoded in the model (Shape-Coupled and Fine-Tuned). On the other hand, when they are aligned collectively, both of their shape and pose relations are encoded (PCA-Only and Modified-Probability).

To allow limited, robust shape variability, we use q≤n eigenvectors to represent each shape. The number of the principal components is calculated using

$$q=\text{min}\left\{k|1\le k\le n,\sum _{i=1}^k{\lambda }_i\ge f\sum _{i=1}^n{\lambda }_i\right\},$$ (2)

where λ_i’s are the eigenvalues and f determines the threshold. The eigenvalues show variations in the model in the directions of the eigenshapes and are ordered decreasingly.

In addition, to consider pose differences, we add 12 local alignment parameters to the shape parameters of each structure. This is used in Shape-Coupled and Fine-Tuned. Our experiments show that using similarity transforms with seven parameters or a single global transformation is insufficient; different shape models need independent local alignments with more flexibility. For PCA-Only and Modified-Probability, we use single global transformation with 12 parameters. Finally, for each structure, we may write

$${\varphi }_k\left(\mathbf{x}\right)={\varphi }_k\left[\mathbf{w},{\mathbf{p}}_k\right]\left(\mathbf{x}\right)={\varphi }_k\left[\mathbf{w},{\mathbf{p}}_k\right]\left(x,y,z\right)={\overline{\Psi }}_k\left({\tilde{x}}_k,{\tilde{y}}_k,{\tilde{z}}_k\right)+\sum _{i=1}^q{w}_i{\varphi }_k^i\left({\tilde{x}}_k,{\tilde{y}}_k,{\tilde{z}}_k\right),$$ (3)

where w is the vector of eigenvectors multipliers and p_k is the vector containing 12 transformation parameters for the alignment of the kth structure according to the following equations:

$$\left[\begin{array}{c}{\tilde{x}}_k\\ {\tilde{y}}_k\\ {\tilde{z}}_k\\ 1\end{array}\right]=T\left({\mathbf{c}}_k\right)A\left({\mathbf{p}}_k\right)T\left(-{\mathbf{c}}_k\right)\left[\begin{array}{c}{x}_k\\ y_k\\ z_k\\ 1\end{array}\right]=M_k\left[\begin{array}{c}{x}_k\\ y_k\\ z_k\\ 1\end{array}\right],$$ (4)

where T and A are the translation and affine transformation matrices, respectively, and c is the fixed center of rotation, computed using the midposition of the ROI for each structure. This choice of the center of rotation improves the local alignment of the model to the data. In this manner, each structure’s pose may change, while shape class covariations are used for the coupling. For the PCA-Only and Modified-Probability, p_k and c_k are the same for all of the structures and the center of mass is computed using the mid position of the ROI of the segmentation. Also, we define P as the vector that contains all of the parameters. The number of the parameters equals m×12+q for the Shape-Coupled and Fine-Tuned (each of the local alignments has 12 parameters) and equals 12+q for the PCA-Only and Modified-Probability (they have only one global transformation). In Sec. 3, we present the proposed entropy-based segmentation method using the shape model described above.

SEGMENTATION PROCESS

After construction of the shape models and their covariations, we register the image that we want to segment to the intensity image of the atlas. To this end, the image is registered using normalized mutual information and linear interpolation. To improve the results, the skull is stripped from the images using the BET method.³⁶ The other parts of the registration protocol are the same as before. An energy function is defined for the segmentation process as explained below.

Energy model

An energy function integrates a variety of information sources for segmentation. To this end, multiple sources of information should be used without adding unwanted complexity to the method. In addition, initialization is critical for optimization of the energy function. In our method, three types of information are used: Image intensity, tissue types, and locations of structures. The image intensity information is captured by estimating the pdf of the image intensities in each structure. This information is used in all of the methods. The tissue type information is captured by applying an unsupervised tissue segmentation method to the image and estimating a pmf for the tissue type of each structure. The location information is captured by estimating pdf of the location of each structure from the training datasets. The tissue type and location information are used in the Modified-Probability, Shape-Coupled, and Fine-Tuned. As explained next, the resulting pmf’s and pdf’s are used to define an entropy function whose minimum corresponds to a desirable segmentation of the structures.

To segment m coupled structures with closed boundaries, there are m regions for these structures. We set the area outside of the m structures as m+1 and use this notation throughout the paper. Based on the estimated entropy of these regions, EN(Ω_k) {Ω_k, k=1,…,m+1} an energy function is defined as $J\left({\Omega }_1,\dots ,{\Omega }_{m+1}\right)=\left|\Omega \right|{\sum }_{k=1}^{m+1}{P}_{{\Omega }_k}EN\left({\Omega }_k\right)$, where |Ω| is the cardinality of a set Ω (number of pixels) and P_Ωk=|Ω_k|∕|Ω|. When all of the regions are as uniform as possible, are in the correct tissue as much as possible, and also are in the correct location, the energy function is at its minimum. The entropy of the kth structure is estimated using EN(Ω_k)=(−1∕|Ω_k|)∫_Ωkln(p_k(x)×p_fk(x)×p_sk(x))dx. Here, p_k(x) is an estimate of the pdf of image intensities in the kth region of the 3D image I, p_fk(x) is the pmf of the tissue type of the kth region, and p_sk(x) is the pdf of the location of the kth region. In the PCA-Only, we utilize the pdf of image intensities (p_k(x)) and we do not consider the other two prob-ability functions. Thus, we can consider p_fk(x) and p_sk(x) equal to those of the PCA-Only. In the other three methods (Modified-Probability, Shape-Coupled, and Fine-Tuned), we also make use of the other two probabilities. Some researchers estimated the pdf’s of image intensities offline, i.e., they used the pixel intensities in the training datasets to construct the pdf’s of image intensities and used the results for the segmentation of the testing datasets without adapting them to the datasets being segmented.^{19, 21} In the PCA-Only approach, we estimate the pdf of the image intensities offline. However, we observed dissimilar dynamic ranges of image intensities in different datasets and concluded that an offline pdf is suboptimal. Therefore, we use online estimation, i.e., we estimate the intensity pdf’s in each iteration of the algorithm. To estimate the intensity pdf’s in each iteration, we use a powerful and robust method (the Parzen window method³⁷) with a Gaussian kernel,

$$p_j\left(\mathbf{x}\right)=p_j\left[\mathbf{w},{\mathbf{p}}_j\right]\left(\mathbf{x}\right)=\frac{{\int }_{\Omega }H\left(-{\varphi }_j\left[\mathbf{w},{\mathbf{p}}_j\right]\left(\widehat{\mathbf{x}}\right)\right)K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}}{{\int }_{\Omega }H\left(-{\varphi }_j\left[\mathbf{w},{\mathbf{p}}_j\right]\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}},$$$$p_{m+1}\left(\mathbf{x}\right)=p_{m+1}\left[\mathbf{P}\right]\left(\mathbf{x}\right)=\frac{{\int }_{\Omega }{\prod }_{k=1}^{m}H\left({\varphi }_k\left[\mathbf{w},{\mathbf{p}}_k\right]\left(\widehat{\mathbf{x}}\right)\right)K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}}{{\int }_{\Omega }{\prod }_{k=1}^{m}H\left({\varphi }_k\left[\mathbf{w},{\mathbf{p}}_k\right]\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}},$$ 5

where H is the Heaviside (unit step) function. This is the approach that we use in the other three methods. To translate the integrals over the changing regions into an integral over the entire image, the Heaviside functions

$$H\left(s\right)=\{\begin{array}{cc}0& ifs<0\\ 1& ifs\ge 0\end{array}\phantom{\}}$$

are used. In these equations, K is a Gaussian kernel with the standard deviation (sigma) that sets the resolution of the estimated pdf. Care must be taken in selecting appropriate values for sigma. Small sigma values make pdf estimation sensitive to noise and large values remove useful details from the estimated pdf. In the literature, values between 1 and 3 are used; we apply the method of Silverman³⁸ to choose it adaptively,

$$\sigma =0.9A{N}^{1∕5},whereA=\text{min}\left\{s,\frac{R}{1.34}\right\},$$ (6)

where s is the empirical standard deviation of the samples in each region, R is the interquartile range of the intensities, and N is the number of samples.

To estimate the pmf’s of the tissue types of the structures, we apply a fuzzy clustering method named FANTASM (Ref. ²⁴) to segment the brain MRI into white matter (WM), gray matter (GM), and cerebrospinal fluid regions. This method is an extension of the standard fuzzy clustering, which is robust to the image intensity nonuniformities and noise. It also considers the information of the neighboring voxels in the clustering process. In its application to the T1-weighted images, the resulting clusters are ordered such that the center of the first cluster has the lowest value (representing darkest regions of the image) and the last cluster center has the highest value (representing the brightest region of the image). Our experiments show that each structure may partially belong to different tissue types and also the ranges of the intensities may be different in different structures. For example, amygdala and hippocampus are in the gray matter but they have different intensity ranges. To capture this information, we use a new strategy where we apply the clustering twice with 3 and 10 clusters. The 3-class segmentation results capture the global intensity information of the normal tissues in the brain while the 10-class segmentation captures their local or fine intensity information. Our experiments have shown that the 10-class clustering captures details of the structures’ intensity information and is sufficiently robust to the intensity inhomogeneities and artifacts.

To use the results, we model the pmf of the tissue type of each structure, i.e., the probabilities of the 10 clusters in the structure as

$$p_{fk}\left(\mathbf{x}\right)=\{\begin{array}{ll}\epsilon ,\hfill & l\left(\mathbf{x}\right)\le L1-3,l\left(\mathbf{x}\right)\ge L2+3\hfill \\ \frac{1}{4},\hfill & l\left(\mathbf{x}\right)=L1-2,l\left(\mathbf{x}\right)=L2+2\hfill \\ \frac{3}{4},\hfill & l\left(\mathbf{x}\right)=L1-1,l\left(\mathbf{x}\right)=L2+1\hfill \\ 1,\hfill & L1\le l\left(\mathbf{x}\right)\le L2,\hfill \end{array}\phantom{\}}$$ (7)

where l(x) shows the cluster number in 10-class segmentation. It should be noted that the probabilities in each voxel only depend on the cluster numbers. In addition, L1 and L2 are computed based on the following equation:

$$\{\begin{array}{c}L1=\text{arg}\underset{1\le i\le 10}{\text{min}}abs\left(C_3\left(2\right)-C_{10}\left(i\right)\right)+l_1\\ L2=\text{arg}\underset{1\le i\le 10}{\text{min}}abs\left(C_3\left(3\right)-C_{10}\left(i\right)\right)+l_2.\end{array}\phantom{\}}$$ (8)

In this equation, C_n(k) shows the intensity of the center of the kth class (cluster) in the n-class clustering. L1 and L2 are not fix numbers; they are calculated for each dataset by Eq. 8. In other words, the closest cluster number in the 10-class segmentation to the WM or GM centers are not fixed; they vary in different datasets. In addition, the model is designed for the T1-weighted images used to study anatomy of the structures; for other MR images that have different contrasts, different models may be needed.

Based on the brain anatomy, each structure is located in a specific tissue type and based on the MRI intensities, it has a specific intensity range. This information is used to estimate the parameters l₁ and l₂ that are listed in Table 2 for different structures. Because of specific features of ventricles, L1=1 and L2=arg min_1≤i≤10abs(C₃(2)−C₁₀(i))−2 are considered for the ventricles. For the region outside all of the m structures (region m+1), if all of the other structures have the same pdf, we use 1−p, but if there are different pdf’s, we use p=0.5, as described in Eq. 9,

$$p_{fm+1}\left(\mathbf{x}\right)=\{\begin{array}{lll}1-p_{f1}\left(\mathbf{x}\right)\hfill & if\forall k,k^{\prime }\ne k∊\left\{1,\dots ,m\right\}\hfill & p_{fk}\left(\mathbf{x}\right)=p_{f{k}^{\prime }}\left(\mathbf{x}\right)\hfill \\ 0.5\hfill & otherwise.\hfill & \hfill \end{array}\phantom{\}}$$ (9)

Table 2.

Shift values used for the construction of tissue type pmf’s for different structures.

Structure	l1	l2
Thalamus	+1	−1
Caudate	+1	−2
Hippocampus	0	−2
Amygdala	−1	−2
Putamen	+1	−1
Pallidum	+3	0

To capture the a priori information about the locations of the structures, we construct a pdf for the location of each structure. We previously registered the image of interest (a test dataset) to the intensity image of the atlas. Thus, alignment of the intensity images of the training datasets can be used to find location information of the structures. To this end, the intensity images of the training datasets are registered using the same algorithm described above to the intensity image of the atlas. Again, the skull is stripped from the images using the BET method.³⁶ The same registration protocol is applied to all of the training datasets. Then, the resulting transformations are applied to the binary images (labels) of the structures. After registration of all of the training datasets, the strategy explained for shape model extraction is applied to find the mean shape of each structure ${\overline{\Phi }}_k$ [this notation is used to show the difference between this mean shape, extracted from the registered intensity images, and the one extracted from the registered label images explained before $\left({\overline{\Psi }}_k\right)$].

Next, the ${\overline{\Phi }}_k\left(\mathbf{x}\right)$ are used to construct the pdf’s for the locations of the structures. Clearly, ${\overline{\Phi }}_k\left(\mathbf{x}\right)$ is negative for the voxels inside the mean structure, zero on the boundary, and positive outside. If for a point, this value is more negative than another one, it means that it is more inside the structure and∕or is inside the structure in a larger number of the train-ing datasets than the other one. In addition, the points on the boundary are neither inside nor outside. Thus, the following decreasing function of ${\overline{\Phi }}_k\left(\mathbf{x}\right)$ which is 1 for the minimum of ${\overline{\Phi }}_k\left(\mathbf{x}\right)$ and is 0.5 for ${\overline{\Phi }}_k\left(\mathbf{x}\right)=0$ (boundary voxels) is defined:

$$p_{sk}\left(\mathbf{x}\right)=\{\begin{array}{l}\text{ln}\left(\left(e-e^{0.5}\right)\left(\frac{{\overline{\Phi }}_k\left(\mathbf{x}\right)}{{\text{min}}_{\mathbf{x}}\left({\overline{\Phi }}_k\left(\mathbf{x}\right)\right)}\right)+e^{0.5}\right)if\text{ln}\left(\left(e-e^{0.5}\right)\left(\frac{{\overline{\Phi }}_k\left(\mathbf{x}\right)}{{\text{min}}_{\mathbf{x}}\left({\overline{\Phi }}_k\left(\mathbf{x}\right)\right)}\right)+e^{0.5}\right)>\epsilon \hfill \\ \epsilon ={10}^{-4}otherwise,\hfill \end{array}\phantom{\}}$$$$p_{sm+1}\left(\mathbf{x}\right)=\underset{1\le k\le m}{\text{min}}\left(1-p_{sk}\left(\mathbf{x}\right)\right).$$ 10

Here, for the region outside all of the structures, we find 1−p for all of the structures and use their minimum for constructing the pdf of this region. ε is used to avoid ln(0) in Eq. 11. The resulting functions can be considered as scaled pdf’s.

Figure 2 shows the probability map of the image intensity (p_k(x)), the tissue type (p_fk(x)), and the location (p_sk(x)) of the thalamus in a sample slice along with their multiplication (p_k(x)×p_fk(x)×p_sk(x)) and the segmentation result generated by the Fine-Tuned (gray line) and the expert segmentation (white). Brighter points show higher probabilities. It should be noted that none of the probabilities predicts the corresponding structure accurately and thus cannot be used for the segmentation by itself. However, multiplication of the probabilities localizes the structure of interest accurately. In other words, each term captures a specific attribute of a structure only while their multiplication captures all of the attributes. To summarize, the pdf’s and pmf’s estimated by the above methods are used in the following energy function (also introduced earlier in this section):

$$J\left({\Omega }_1,\dots ,{\Omega }_{m+1}\right)=J\left(P\right)=\sum _{k=1}^{m+1}-{\int }_{{\Omega }_k}\text{ln}\left(p_k\left(\mathbf{x}\right)\times p_{fk}\left(\mathbf{x}\right)\times p_{sk}\left(\mathbf{x}\right)\right)d\mathbf{x}.$$ (11)

The following equation demonstrates the detailed dependency of the energy function to the different parameters:

$$J=\sum _{j=1}^m{\int }_{\Omega }H\left(-{\varphi }_j\left[\mathbf{w},{\mathbf{p}}_j\right]\left(\mathbf{x}\right)\right)F\left(p_j\left[\mathbf{w},{\mathbf{p}}_j\right]\left(\mathbf{x}\right)\times p_{fj}\left(\mathbf{x}\right)\times p_{sj}\left(\mathbf{x}\right)\right)d\mathbf{x}+{\int }_{\Omega }\prod _{k=1}^{m}H\left({\varphi }_k\left[\mathbf{w},{\mathbf{p}}_k\right]\left(\mathbf{x}\right)\right)F\left(p_{m+1}\left[\mathbf{P}\right]\left(\mathbf{x}\right)\times p_{fm+1}\left(\mathbf{x}\right)\times p_{sm+1}\left(\mathbf{x}\right)\right)d\mathbf{x},$$ (12)

where p_sk(x) and p_sk(x) do not depend on the parameters in the vector P. In this equation, F(p)=−ln(p). However, in the Modified-Probability, Shape-Coupled, and Fine-Tuned, p_k(x) depends on the parameters according to Eq. 5; for the PCA-Only, it is fixed.

Figure 2.

The probability map of the image intensity (a), the tissue type (b), and the location (c) of the thalamus in a sample slice along with their multiplication (d) and segmentation result generated by the Fine-Tuned (gray line) and the expert segmentation (white) (e) are shown. Brighter points indicate higher probabilities. It should be noted that each term captures a specific attribute of the structure, while their combination (multiplication) captures all attributes of the structure.

Energy optimization

To optimize the energy function, we have three steps. An initialization, a quasi-Newton algorithm to optimize the parameters of the shapes, and a level set algorithm for fine tuning. First, we present the initialization.

As explained in Sec. 1, initialization is very important for a segmentation algorithm. We use the center of mass of the negative portions of the ${\overline{\Phi }}_k$ to place the center of mass of the kth shape model of the brain structures, ${\overline{\Psi }}_k$. Then, we fine-tune the initialization using the location and tissue type information (their estimated pdf and pmf). This is done by optimizing the following energy function using its derivative with respect to d:

$$E_j\left(\mathbf{d}\right)=-{\int }_{{\Omega }_j}{p}_{fj}\left(\widehat{\mathbf{x}}+\mathbf{d}\right)\times p_{sj}\left(\widehat{\mathbf{x}}+\mathbf{d}\right)d\widehat{\mathbf{x}},$$$$\frac{\partial E_j}{\partial {\mathbf{d}}_i}={\int }_{{\Gamma }_j}\frac{\partial {\varphi }_j\left(\widehat{\mathbf{x}}+\mathbf{d}\right)}{\partial {\mathbf{d}}_i}\times p_{fj}\left(\widehat{\mathbf{x}}+\mathbf{d}\right)\times p_{sj}\left(\widehat{\mathbf{x}}+\mathbf{d}\right)d\widehat{\mathbf{x}},$$ 13

where d is the translation vector rearranged as a translation matrix in Eq. 4 and ∫_Γj is used to denote an integration over the boundary of the region Ω_j. The method to compute the derivatives will be described after the second step. We use this step in the Modified-Probability, Shape-Coupled, and Fine-Tuned.

In the next step, to minimize the energy function, we use a quasi-Newton algorithm along with the Broyden–Fletcher–Goldfarb–Shanno method for Hessian matrix estimation.³⁹ Although methods such as steepest descent are also popular in literature,²¹ we obtained superior results using the quasi-Newton algorithm. This method needs more memory and is more complex than the steepest descent algorithm but finds the optimal point faster. It requires the gradient of the objective function with respect to the parameters. The gradients can be estimated using numerical methods but analytical computation is faster and generates more accurate results. As such, we computed them analytically resulting in the following equations for the derivatives of the energy function with respect to the parameters w and p_k (Modified-Probability, Shape-Coupled, and Fine-Tuned):

$$\frac{\partial J}{\partial {\mathbf{w}}_i}=\sum _{j=1}^m{\int }_{{\Gamma }_j}\frac{\partial {\varphi }_j\left(\widehat{\mathbf{x}}\right)}{\partial {\mathbf{w}}_i}\times \left\{\text{ln}\left({\widehat{p}}_j\left(\widehat{\mathbf{x}}\right)\times p_{fj}\left(\widehat{\mathbf{x}}\right)\times p_{sj}\left(\widehat{\mathbf{x}}\right)\right)-\text{ln}\left({\widehat{p}}_{m+1}\left(\widehat{\mathbf{x}}\right)\times p_{fm+1}\left(\widehat{\mathbf{x}}\right)\times p_{sm+1}\left(\widehat{\mathbf{x}}\right)\right)+\frac{1}{\left|{\Omega }_j\right|}\times {\int }_{{\Omega }_j}\left(K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)∕{\widehat{p}}_j\left(\mathbf{x}\right)\right)d\mathbf{x}-\frac{1}{\left|{\Omega }_{m+1}\right|}\times {\int }_{{\Omega }_{m+1}}\left(K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)∕{\widehat{p}}_{m+1}\left(\mathbf{x}\right)\right)d\mathbf{x}\right\}d\widehat{\mathbf{x}},$$ (14)

$$\frac{\partial J}{\partial {\mathbf{p}}_j^i}={\int }_{{\Gamma }_j}\frac{\partial {\varphi }_j\left(\widehat{\mathbf{x}}\right)}{\partial {\mathbf{p}}_j^i}\times \left\{\text{ln}\left({\widehat{p}}_j\left(\widehat{\mathbf{x}}\right)\times p_{fj}\left(\widehat{\mathbf{x}}\right)\times p_{sj}\left(\widehat{\mathbf{x}}\right)\right)-\text{ln}\left({\widehat{p}}_{m+1}\left(\widehat{\mathbf{x}}\right)\times p_{fm+1}\left(\widehat{\mathbf{x}}\right)\times p_{sm+1}\left(\widehat{\mathbf{x}}\right)\right)+\frac{1}{\left|{\Omega }_j\right|}\times {\int }_{{\Omega }_j}\left(K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)∕{\widehat{p}}_j\left(\mathbf{x}\right)\right)dx-\frac{1}{\left|{\Omega }_{m+1}\right|}\times {\int }_{{\Omega }_{m+1}}\left(K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)∕{\widehat{p}}_{m+1}\left(\mathbf{x}\right)\right)d\mathbf{x}\right\}d\widehat{\mathbf{x}}.$$ (15)

Note that if for a point on the boundary, the probability of being in the outside region is larger than being in the inside region, then the boundary should expand. This happens because in this scenario, the terms inside the braces in the first lines of Eqs. 14, 15 become negative, and assuming that the first terms on the first lines are positive, result in an increase in the parameters, as expected. The same holds true if the first terms are negative.

If we use the training datasets for the intensity pdf estimation of each structure and fix it during the optimization, we have simpler equations in the following form [which is used for the PCA-Only with p_fk(x) and p_sk(x) equal to one for all of the voxels]:

$$\frac{\partial J}{\partial {\mathbf{w}}_i}=\sum _{j=1}^m{\int }_{{\Gamma }_j}\frac{\partial {\varphi }_j\left(\widehat{\mathbf{x}}\right)}{\partial {\mathbf{w}}_i}\times \left\{\text{ln}\left({\widehat{p}}_j\left(\widehat{\mathbf{x}}\right)\times p_{fj}\left(\widehat{\mathbf{x}}\right)\times p_{sj}\left(\widehat{\mathbf{x}}\right)\right)-\text{ln}\left({\widehat{p}}_{m+1}\left(\widehat{\mathbf{x}}\right)\times p_{fm+1}\left(\widehat{\mathbf{x}}\right)\times p_{sm+1}\left(\widehat{\mathbf{x}}\right)\right)\right\}d\widehat{\mathbf{x}},$$ (16)

$$\frac{\partial J}{\partial {\mathbf{p}}_j^i}={\int }_{{\Gamma }_j}\frac{\partial {\varphi }_j\left(\widehat{\mathbf{x}}\right)}{\partial {\mathbf{p}}_j^i}\times \left\{\text{ln}\left({\widehat{p}}_j\left(\widehat{\mathbf{x}}\right)\times p_{fj}\left(\widehat{\mathbf{x}}\right)\times p_{sj}\left(\widehat{\mathbf{x}}\right)\right)-\text{ln}\left({\widehat{p}}_{m+1}\left(\widehat{\mathbf{x}}\right)\times p_{fm+1}\left(\widehat{\mathbf{x}}\right)\times p_{sm+1}\left(\widehat{\mathbf{x}}\right)\right)\right\}d\widehat{\mathbf{x}}.$$ (17)

Thus, we do not need to update pdf’s in each iteration. In this case, we lose quality due to the intensity variations between the training and testing datasets. Also, $\partial {\varphi }_j\left(\mathbf{x}\right)∕\partial {\mathbf{p}}_j^i$ and ∂ϕ_j(x)∕∂w_i are calculated as follows:

$$\frac{\partial {\varphi }_k\left(\mathbf{x}\right)}{\partial {\mathbf{w}}_i}={\varphi }_k^i\left(\mathbf{x}\right),$$ (18)

$$\frac{\partial {\varphi }_k\left(\mathbf{x}\right)}{\partial {\mathbf{p}}_k^i}{\left[\begin{array}{c}\frac{\partial {\varphi }_k\left({\tilde{x}}_k,{\tilde{y}}_k,{\tilde{z}}_k\right)}{\partial {\tilde{x}}_k}\\ \frac{\partial {\varphi }_k\left({\tilde{x}}_k,{\tilde{y}}_k,{\tilde{z}}_k\right)}{\partial {\tilde{y}}_k}\\ \frac{\partial {\varphi }_k\left({\tilde{x}}_k,{\tilde{y}}_k,{\tilde{z}}_k\right)}{\partial {\tilde{z}}_k}\\ 0\end{array}\right]}^T\times \frac{\partial M_k}{\partial {\mathbf{p}}_k^i}\times \left[\begin{array}{c}x\\ y\\ z\\ 1\end{array}\right],$$ (19)

where $\partial M_k∕\partial {\mathbf{p}}_k^i$ is computed using derivatives of each matrix element.

Note that we do not use the intensity information (estimated pdf) in this step because the initial location is such that the estimated pdf may be inaccurate. Usually, parts of the initial model are outside the correct region of the structure, and thus the estimated pdf is inaccurate and may cause large segmentation errors. However, after fine-tuning the placement of the centers of mass of the models, the variances of the Gaussian kernels of the Parzen window for estimation of the intensity pdf’s are estimated once using Eq. 6. Then, the iterative procedure explained in the previous subsection to optimize the energy function in Eq. 11 is executed.

This approach is very effective for the segmentation but it is limited to the shape model. We propose a strategy to overcome this limitation by adding an extra step, where the quasi-Newton based algorithm introduced above is followed by a fine-tuning step using a level-set method. To this end, the same energy function defined in Eq. 11 is minimized using a level-set formulation. This is the approach that we utilize only in the Fine-Tuned. With some mathematical operations, it can be shown that the level-set evolution of the region j can be described with the following partial differential equation:

$$\frac{\partial {\varphi }_j\left(\mathbf{x}\right)}{\partial t}=\left|\nabla {\varphi }_j\left(\mathbf{x}\right)\right|\left(-\text{ln}\left(p_j\left(\mathbf{x}\right)\times p_{fj}\left(\mathbf{x}\right)\times p_{sj}\left(\mathbf{x}\right)\right)+\text{ln}\left(p_{m+1}\left(\mathbf{x}\right)\times p_{fm+1}\left(\mathbf{x}\right)\times p_{sm+1}\left(\mathbf{x}\right)\right)-\frac{1}{\left|{\Omega }_j\right|}\times {\int }_{{\Omega }_j}\left(K\left(I\left(\widehat{\mathbf{x}}\right)-I\left(\mathbf{x}\right)\right)∕p_j\left(\widehat{\mathbf{x}}\right)\right)d\widehat{x}+\frac{1}{\left|{\Omega }_{m+1}\right|}\times {\int }_{{\Omega }_{m+1}}\left(K\left(I\left(\widehat{\mathbf{x}}\right)-I\left(\mathbf{x}\right)\right)∕p_{m+1}\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}\right).$$ (20)

The initial sign distance functions of this step are the final shapes of the previous step (segmentation based on quasi-Newton algorithm). The proposed method overcomes the two problems explained in Sec. 1 (initialization and sensitivity to the weights) at the cost of the time used for this step, which is not significant because of the good initialization used as explained above.

EXPERIMENTS AND RESULTS

Data

To test and evaluate the proposed method, we have applied it on the MRI data from two different sources. We have used datasets from the Internet Brain Segmentation Repository⁴⁰ as the primary data. These datasets are T1-weighted volumetric images with slightly different voxel sizes (0.938×0.938×1.5, 1.0×1.0×1.5, and 0.837×0.837×1.5 mm³). There are 18 datasets on which expert physicians have segmented 43 brain structures. These datasets consist of 256×256×128 noncubical voxels. The smallest physical size that covers all of the datasets is 256×256×192 mm³. As a preprocessing step, all datasets are resampled into 1.0×1.0×1.0 mm³ cubical voxels after the registration phase. The resampling is a prerequisite for using the sign distance functions defined on isotropic spaces. The second datasets are the images and segmentation results of the hippocampi of 21 temporal lobe epilepsy (TLE) patients acquired using a General Electric 1.5 T Signa System (GE Medical Systems, Milwaukee, WI) and processed using the EIGENTOOL software.⁴¹ The TLE may cause changes in the size and shape of the hippocampus. Coronal T1-weighted MRI, acquired using a spoiled gradient-echo sequence with a pixel size of 0.781×0.781 mm² and a slice thickness of 2.0 mm is used.

Procedure

Based on the structures of interest, a subvolume (ROI) is chosen to improve speed and robustness of the algorithm. This improves the execution speed and accuracy of the method since the pdf’s of all regions inside and outside of each desired structure need to be estimated. We define the ROI as 1.05 times of the smallest cube that covers all of the desired structures in all of the training datasets. The additional 5% confidence interval has been sufficient for the test datasets (image segmentation phase) in our experiments.

To evaluate the results, we use the Dice coefficient (k) as a similarity index. In addition, as an alternative evaluation measure, we use the Hausdorff distance (H), which is the maximum of the minimum distances between the corresponding surface points of the two volumes. Due to the outliers, the 95 percentile of the Hausdorff distance (H₉₅) is used in our study as well as many previous studies. Moreover, the mean distance (M) is used as another evaluation measure. For 3D visualization of the structures, we use 3D SLICER (Ref. ⁴²) and for registration of the training and testing datasets, we use the Insight Segmentation and Registration Toolkit (ITK) library.⁴³ For optimization and extraction of the principal shapes, we use MATLAB.⁴⁴ All programs are run on a 3.2 GHz Windows XP workstation with 4 Gbyte of RAM.

For the IBSR dataset, we segment the following seven structures on the two sides of the brain for a total of 14 structures: (1) Lateral ventricles, (2) caudate, (3) thalamus, (4) putamen, (5) pallidum, (6) hippocampus, and (7) amygdala. These structures are either used in many applications or hard to segment because of their unclear boundaries. They include a variety of shapes with different signal intensities, sizes, and other geometric features. We use the leave-one-out strategy to test the proposed segmentation algorithm. We first segment each of the left and right structures individually using the principal shapes extracted from the training datasets. We also segment the left and right structures or multiple structures together by the proposed coupling method. For the TLE datasets, we use the leave-one-out strategy and apply the same procedure used for the IBSR datasets.

Results

The quantitative measures (k,M,H₉₅) used to evaluate the segmentation results for the selected structures of the 18 IBSR datasets are compared in Tables 3, 4, 5, respectively. It should be noted that the measurements are obtained after the final sign distance functions are thresholded at zero. The results of the four implementations of our method are compared to eight other methods from the literature that have used the IBSR datasets and reported their results when using all of the datasets. Note that the comparison does not include the methods that although used the IBSR datasets but did not report the above measures, e.g., Ref. 45. In the following, we first compare the performance measures of our proposed methods with those of the other methods in literature. Then, we compare the four implementations of our approach with each other. First, we compare the methods based on the k values in Table 3.

Table 3.

Dice coefficients (k) for the segmentation results of the proposed methods for seven brain structures in test datasets and the results of the other methods that reported their segmentation results for the IBSR datasets (Refs. ¹⁴, ¹⁵, ²⁵, ³³, ⁴⁶).

	Thalamus	Putamen	Caudate	Pallidum	Hippocampus	Amygdala	Ventricle
Fine-Tuned	0.86±0.04	0.82±0.05	0.81±0.04	0.76±0.04	0.73±0.05	0.70±0.08	0.80±0.06
Shape-Coupled	0.85±0.04	0.80±0.06	0.78±0.04	0.75±0.05	0.71±0.06	0.68±0.09	0.73±0.10
Modified-Probability	0.86±0.02	0.80±0.07	0.78±0.03	0.72±0.10	0.68±0.08	0.67±0.10	0.75±0.07
PCA-Only	0.85±0.04	0.75±0.14	0.72±0.12	0.72±0.10	0.66±0.07	0.64±0.10	0.67±0.11
Akselrod et al.	0.84	0.79	0.80	0.74	0.69	0.63	–
Wu et al.	0.84	0.81	0.80	–	–	–	–
ISCA	0.80	0.78	0.74	0.70	0.64	0.58	–
Naive Prior	0.83	0.77	0.65	0.72	0.62	0.65	–
Gouttard et al.	–	0.78	0.76	0.71	0.67	0.64	0.85
Coifolo et al.	0.77	0.70	0.65	0.62	–	–	–
Bazin et al.	0.78±0.05	0.82±0.05	0.78±0.04	–	–	–	0.85±0.04
Hammer	0.77±0.05	0.60±0.07	0.74±0.04	–	–	–	0.66±0.08

Table 4.

Mean (M) distances for the segmentation results of the proposed methods for seven brain structures in test datasets and the results of the other methods that reported their segmentation results for the IBSR datasets (Refs. ¹⁴, ¹⁵, ²⁵, ³³, ⁴⁶).

	Thalamus	Putamen	Caudate	Pallidum	Hippocampus	Amygdala	Ventricle
Fine-Tuned	0.65±0.18	0.58±0.19	0.44±0.13	0.70±0.16	0.80±0.17	0.76±0.27	0.68±0.23
Shape-Coupled	0.75±0.22	0.72±0.23	0.61±0.14	0.74±0.19	0.84±0.18	0.78±0.29	0.70±0.28
Modified-Probability	0.74±0.13	0.77±0.34	0.64±0.14	0.83±0.35	0.98±0.32	0.83±0.32	0.66±0.24
PCA-Only	0.78±0.24	1.00±0.70	0.87±0.51	0.82±0.32	1.01±0.23	0.98±0.27	0.87±0.41
Akselrod et al.	1.44	1.6	1.44	1.43	1.88	1.67	–
ISCA	1.55	1.72	1.84	1.55	1.91	1.78	–
Coifolo et al.	1.70	1.46	1.71	1.51	–	–	–
Bazin et al.	1.45±0.22	0.97±0.26	0.94±0.19	–	–	–	0.78±0.17

Table 5.

Hausdorff (H₉₅) distances for the segmentation results of the proposed methods for seven brain structures in test datasets and the results of the other methods that reported their segmentation results for the IBSR datasets (Refs. ¹⁴, ¹⁵, ²⁵, ³³, ⁴⁶).

	Thalamus	Putamen	Caudate	Pallidum	Hippocampus	Amygdala	Ventricle
Fine-Tuned	2.31±0.58	2.11±0.60	2.17±0.45	2.32±0.61	2.51±0.64	2.88±0.85	3.55±3.35
Shape-Coupled	2.38±0.65	2.32±0.63	2.17±0.51	2.38±0.69	2.62±0.63	2.99±0.94	4.90±4.34
Modified-Probability	2.28±0.42	2.29±0.87	2.28±0.66	2.58±0.89	3.34±1.09	3.06±1.04	4.55±3.59
PCA-Only	2.32±0.62	2.88±1.82	2.56±1.30	2.53±0.76	3.30±0.91	3.60±1.11	5.72±3.76
Akselrod et al.	2.9	3.36	3.07	2.75	4.57	3.38	–
ISCA	3.2	3.89	4.46	3.2	4.44	3.89	–

A comparison of the PCA-Only (simplest case) and the eight methods from the literature shows that our method is generally superior to the methods of ISCA,¹⁴ Naive Prior,¹⁴ Coifolo et al.,³³ and Hammer.¹⁵ It is, however, inferior to the methods of Refs. ^{14, 46, 15}. The standard deviation values suggest that the PCA-Only is not as robust (stable) as some of the other methods. For example, for the putamen, it has a standard deviation of 0.14, which is relatively large. Many of the methods did not report the standard deviation but the method of Bazin and Pham¹⁵ reported a smaller standard deviation (0.05).

Next, we compare the Modified-Probability, Shape-Coupled, and Fine-Tuned with the other methods. The Modified-Probability is generally superior to all of the methods except for the methods of.^{14, 15, 46} The Shape-Coupled is superior to the other methods and similar to the method of.¹⁵ The Fine-Tuned is similar to the method of Ref. 15 for the putamen while being superior to the rest. It is also superior to the other methods for the amygdala and hippocampus. However, it is inferior to the methods of Refs. ^{15, 46} but superior to the rest for the ventricles. In general, the Fine-Tuned is more robust than the other methods.

A comparison based on the mean distance values (M) in Table 4 shows that our methods are superior to the other methods for all structures. The only exception is the ventricles where the PCA-Only is inferior to that of Ref. 15. However, the standard deviations of the Fine-Tuned are about 15% lower than those of the method of Ref. 15, on average. Moreover, a comparison based on the H₉₅ values in Table 5 shows that our method is superior to the others for all structures. The only exception is the amygdala where the PCA-Only is inferior to that of Ref. 14.

Overall, the method with most similar values of the three quantitative measures to those of our methods is the method of Ref. 14. A comparison of the Fine-Tuned with the method of Ref. 14 shows that our method produces superior results based on M and H₉₅. For example, for the caudate, it results in the M value of 0.44 mm, which is about 30% of that of the method of Ref. 14. In addition, our method is superior to theirs for the amygdala based on k.

Next, we compare the Fine-Tuned with our other methods (PCA-Only, Modified-Probability, and Shape-Coupled). A comparison of the Fine-Tuned with the Shape-Coupled shows superiority of the segmentations of the lateral ventricles, hippocampus, and caudate based on the k (p<0.05). In addition, it shows significant improvement for the segmentation of the caudate and putamen based on M (p<0.05). The overall superiority (considering all structures) of the Fine-Tuned relative to the Shape-Coupled is about 3% based on the k and about 8% (p<0.05) based on the M. Moreover, the standard deviations of the k and M values of the Fine-Tuned are 18% and 13% less than those of the Shape-Coupled, on average.

A comparison of the Fined-Tuned and Modified-Probability illustrates that the first method is superior to the latter for the segmentation of the caudate and hippocampus based on the k (p<0.05). In addition, the segmentation results of the Fine-Tuned are superior to those of the Modified-Probability for the putamen and caudate based on the M (p<0.05). A comparison of the Fine-Tuned and PCA-Only shows that the Fine-Tuned generates significantly superior results for the caudate, pallidum, hippocampus, amygdala, and ventricles based on k (p<0.05). Also, it is superior for the segmentation of caudate, putamen, and ventricles based on M (p<0.05). In overall, the superiority of the Fine-Tuned relative to the PCA-Only is 9% and 37% based on k and M, respectively (p<0.05). In other cases, the superiority of the Fine-Tuned is not statistically significant.

Now, we compare the three methods with the coupling information and without fine-tuning (PCA-Only, Modified-Probability, and Shape-Coupled). A comparison of PCA-Only and Shape-Coupled illustrates that in overall (mean of the performance measures for all structures) the k and M of the Shape-Coupled are about 7% and 19% superior to the PCA-Only (p<0.05). The difference in the performance may be attributed to the approach by which the two methods use the coupling information. In the Shape-Coupled, the coupling information is just based on the shape variation among the structures but in the PCA-Only, there is additional variation due to the pose of the structures. This additional variation might have made their eigenshapes different from those of the brain structures.

A comparison of Modified-Probability and Shape-Coupled shows similar findings. The superiority of the Shape-Coupled based on k is about 1% but based on M and H₉₅ is about 6% and 3%, respectively. Additionally, the superiority of the Shape-Coupled is more pronounced for small structures with large pose variations (pallidum, hippocampus, and amygdala). In Fig. 3, a sample segmentation of the selected structures by the PCA-Only, Modified-Probability, Shape-Coupled, and Fine-Tuned are shown along with the expert segmentation. An important observation here is the marginal quality of the manual segmentation (with unsmooth surfaces) which is partly due to the slice-by-slice approach used in the manual segmentation. This leads to imperfect (nonrepresentative) values for the performance measures especially for H₉₅. In addition, Fig. 4 shows a combination of the pdf of the image intensities (p_k(x)), the pmf for the tissue type (p_fk(x)), and the pdf of the location (p_sk(x)) of the lateral ventricles, caudate, putamen, pallidum, hippocampus, and amygdala in a sample slice along with the segmentation results generated by the Fine-Tuned (gray line) and the expert segmentations (white).

Figure 3.

Comparison of 3D views of the structures segmented manually (a), by PCA-Only (b), by Modified-Probability (c), by Shape-Coupled (d), and by Fine-Tuned (e).

Figure 4.

Combination of the probability maps of the image intensities, the tissue type, and the location are shown along with the segmentation results generated by the Fine-Tuned (gray line) and the expert segmentations (white) for the lateral ventricles (a), caudate (b), putamen (c), pallidum (d), hippocampus (e), and amygdala (f) in sample slices. Brighter points indicate higher probabilities.

For further comparison, we use the TLE datasets. Table 6 compares the results of the methods and illustrates the superiority of the Fined-Tuned to the rest. Note that the fine-tuning step has improved the segmentation results of the patients’ images more than the IBSR datasets (normal brains). This can be attributed to the fact that the hippocampi of the TLE patients are not symmetric (one hippocampus is usually smaller than the other). In this case, the superiority of the Fine-Tuned relative to the Shape-Coupled is 7% and 15% based on k and M, respectively, where the improvement based on the k is statistically significant (p<0.05). Consequently, alleviating the constraint imposed by the principal shapes allows accurate segmentation of the abnormal parts of the structures that are not covered by the principal shapes. In addition, because of the complex relation between shape and pose of the hippocampi in the TLE patients, the Modified-Probability cannot extract the relation between the pose and shape appropriately. The superiority of the Shape-Coupled relative to the Modified-Probability based on k and M are 11% and 12%, respectively. Again, the improvement based on k is statistically significant (p<0.05). Figure 5 compares the automatic and expert segmentation of the hippocampus on 6 coronal slices.

Table 6.

Dice coefficients (k) and mean (M) and Hausdorff (H₉₅) distances for the segmentation results of the proposed methods for the hippocampus in the test datasets for the temporal lobe epilepsy datasets.

	k	M	H95
Fine-Tuned	0.73±0.06	0.90±0.29	3.34±1.12
Shape-Coupled	0.68±0.08	1.06±0.33	3.74±1.21
Modified-Probability	0.61±0.11	1.21±0.40	5.29±1.75
PCA-Only	0.43±0.15	1.93±0.48	7.45±2.56

Figure 5.

Comparison of the segmentation results for the hippocampus generated by the Fine-Tuned (gray line) and the expert segmentation (white) in a series of coronal images from a temporal lobe epilepsy dataset.

Finally, a critical issue for an automated segmentation method in clinical applications is its execution time. The mean execution time of our method is about 9 min for the algorithm with coupling information. This time is for the registrations (2 steps), the FANTASM (3- and 10-class clustering), the initialization, the quasi-Newton step, and the level-set step.

DISCUSSION

Tsai et al.²¹ introduced a 2D method using PCA for coupling using the nonparametric pdf in a mutual information framework. They used training datasets to estimate the pdf’s and fixed them during the segmentation process. Because of the intensity variations among different structures, the use of the training datasets to estimate the pdf’s of the image intensities in different structures fails in many cases.^{17, 21} Thus, we estimate these pdf’s in each iteration of the algorithm and have introduced an extended method in Ref. 13 to update the estimated pdf’s during the iterations of the optimization algorithm.

A common problem of the aforementioned methods is that they are not stable in some cases and do not use any specific prior knowledge in the segmentation process. In addition, they have initialization problems and heavily depend on the training datasets. Akhondi-Asl and Soltanian-Zadeh²⁶ introduced a constrained optimization strategy to develop a robust segmentation method. Albeit its superiority in performance, it takes considerable time to execute and is somewhat sensitive to the parameters of the constraints. Another problem in many of the segmentation methods is the sensitivity of the results to the initialization of the algorithm.^{13, 21, 23}

An important observation is that the fine tuning step improves the segmentation results by overcoming the limitations of the prior shape model (principal shapes extracted from the training datasets). In other words, each dataset has parts similar to the training datasets that are represented by the prior knowledge. In addition, there are dissimilarities between the training datasets and the testing datasets that are not represented by the prior knowledge. The fine-tuning step removes the constraints imposed by the prior shape model and thus improves the segmentation results. Based on this analysis, the improvements are expected to be more pronounced when segmenting the patient data (abnormal structures). In addition, the fine-tuning step allows closer adjustment of the segmentation results to the image edges and thus the improvements generated by this approach are quantified the best by the Hausdorff distance.

Note that performances of different algorithms on different structures are different; none of the methods generate perfect segmentation results for all of the structures. Moreover, due to the small size of the amygdala relative to the other brain structures, the performance measure k shows a lower performance for the automatic segmentation of this structure. However, it is interesting to note that the improvements obtained by our proposed methods are more pronounced in this case.

A critical issue in comparing different segmentation methods is that unless the same datasets are used, the performance measures are incomparable. This is based on the fact that depending on the quality and variation of the datasets, an algorithm can produce significantly different results. For example, Powell et al.²⁰ reported highly accurate segmentation results for the caudate, hippocampus, and putamen for a specific high-resolution dataset that is not publicly available. Yes, it is not clear how good that method will perform on the publicly available datasets like the IBSR dataset.

CONCLUSIONS

We have presented a new method for the segmentation of the brain subcortical structures using their shape relationships and tissue type and location information. In the model development phase, the proposed method registers each structure individually before estimating their shape variations. The registration is done in two steps using similarity and affine transformations. In the segmentation phase, the proposed method applies an independent transformation with 12 parameters (rotation, scaling, translation, and shearing) to each structure, improving flexibility of the segmentation model. The energy function used for the segmentation is based on the entropy of different structures. Tissue type information is used to improve robustness and accuracy of the segmentation process. In addition, location information is used to further improve the segmentation process. With a powerful automatic initialization of the structures which considers tissue type and location information and the use of the quasi-Newton algorithm, a local minimum of the energy function is found. In the final step, a level-set based optimization is applied. Throughout the algorithm, the tissue type and location information are used to generate robust and accurate segmentations. To achieve accurate results, the intensity pdf’s are calculated in each iteration of the algorithm and gradients are computed analytically. Experimental results have illustrated superiority of the proposed framework to the other methods in the literature. The execution time of our algorithm is a few minutes, which is appropriate for a variety of applications.

APPENDIX: ANALYTIC DERIVATION OF GRADIENTS

In this section, mathematical derivation of the derivatives of the energy function with respect to the optimization parameters is presented. The energy function is based on the functions defined on m+1 regions. However, the regionm+1 may be constructed from the other m regions. These regions have no intersections in the beginning due to the proposed initialization method and it is assumed that they do not overlap during the segmentation process either. Also, a common notation, ω, is used to represent the parameters of the energy function that are called w_i’s and $p_k^i$’s in the body of the paper. Then, the following equation is derived for the derivative of the function for the region m+1 with respect to the parameters:

$$\frac{\partial \left({\prod }_{k=1}^{m}H\left({\varphi }^k\left(\mathbf{x}\right)\right)\right)}{\partial \omega }=\sum _{k=1}^m\delta \left({\varphi }^k\left(\mathbf{x}\right)\right)\frac{\partial {\varphi }^k\left(\mathbf{x}\right)}{\partial \omega }.$$ (A1)

Note that, as expected, the derivatives for the region m+1 depend on those of the other m regions.

Next, the following equations are derived for the derivatives of the estimated intensity pdf’s with respect to the parameters. Note that the pdf of the region m+1 is defined based on the pdf’s of the other regions. As such, their analytical expressions are different and their derivatives are presented separately in the following equations:

$$\frac{\partial p_k\left(\mathbf{x}\right)}{\partial \omega }=\frac{\partial }{\partial \omega }\left(\frac{{\int }_{\Omega }H\left(-{\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}}{{\int }_{\Omega }H\left(-{\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}}\right)=\frac{-\left({\int }_{\Omega }\delta \left({\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)\frac{\partial {\varphi }_k\left(\widehat{\mathbf{x}}\right)}{\partial \omega }K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}\right)\times \left({\int }_{\Omega }H\left(-{\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}\right)}{{\left({\int }_{\Omega }H\left(-{\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}\right)}^2}+\frac{\left({\int }_{\Omega }\delta \left({\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)\frac{\partial {\varphi }_k\left(\widehat{\mathbf{x}}\right)}{\partial \omega }d\widehat{\mathbf{x}}\right)\times \left({\int }_{\Omega }H\left(-{\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}\right)}{{\left({\int }_{\Omega }H\left(-{\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}\right)}^2}=\frac{p_k\left(\mathbf{x}\right)\times \left({\int }_{\Omega }\delta \left({\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)\frac{\partial {\varphi }_k\left(\widehat{\mathbf{x}}\right)}{\partial \omega }d\widehat{\mathbf{x}}\right)-\left({\int }_{\Omega }\delta \left({\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)\frac{\partial {\varphi }_k\left(\widehat{\mathbf{x}}\right)}{\partial \omega }K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}\right)}{\left|{\Omega }_k\right|},$$ (A2)

$$\frac{\partial p_{m+1}\left(\mathbf{x}\right)}{\partial \omega }=\frac{\partial }{\partial \omega }\left(\frac{{\int }_{\Omega }{\prod }_{k=1}^{m}H\left({\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}}{{\int }_{\Omega }{\prod }_{k=1}^{m}H\left({\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}}\right)=\frac{\left({\int }_{\Omega }{\sum }_{k=1}^m\left(\delta \left({\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)\frac{\partial {\varphi }_k\left(\widehat{\mathbf{x}}\right)}{\partial \omega }\right)K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}\right)\times \left({\int }_{\Omega }{\prod }_{k=1}^{m}H\left({\varphi }_k\left(\mathbf{x}\right)\right)d\widehat{\mathbf{x}}\right)}{{\left({\int }_{\Omega }{\prod }_{k=1}^{m}H\left({\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}\right)}^2}-\frac{\left({\int }_{\Omega }{\sum }_{k=1}^m\left(\delta \left({\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)\frac{\partial {\varphi }_k\left(\widehat{\mathbf{x}}\right)}{\partial \omega }\right)d\widehat{\mathbf{x}}\right)\times \left({\int }_{\Omega }{\prod }_{k=1}^{m}H\left({\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}\right)}{{\left({\int }_{\Omega }{\prod }_{k=1}^{m}H\left({\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}\right)}^2}=-\frac{p_{m+1}\left(\mathbf{x}\right)\times \left({\int }_{\Omega }{\sum }_{k=1}^m\left(\delta \left({\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)\frac{\partial {\varphi }_k\left(\widehat{\mathbf{x}}\right)}{\partial \omega }\right)d\widehat{\mathbf{x}}\right)-\left({\int }_{\Omega }{\sum }_{k=1}^m\left(\delta \left({\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)\frac{\partial {\varphi }_k\left(\widehat{\mathbf{x}}\right)}{\partial \omega }\right)K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)d\widehat{\mathbf{x}}\right)}{\left|{\Omega }_{m+1}\right|}.$$ (A3)

Now, we can move on to the derivation of the derivatives of the energy function with respect to the parameters. To simplify the equations, p_ti(x)=p_fi(x)×p_si(x) is used since neither of p_fi(x) and p_si(x) depends on the parameters; the only parameter dependent part is p_j(x). In addition, to visualize all of the terms, the −ln function in the energy function is replaced by a general function F, resulting the following general energy function: $J={\sum }_{j=1}^{m+1}{\int }_{{\Omega }_j}F\left(p_j\left(\mathbf{x}\right)\times p_{tj}\left(\mathbf{x}\right)\right)d\mathbf{x}$:

$$\frac{\partial J}{\partial \omega }=\sum _{j=1}^{m+1}\frac{\partial {\int }_{{\Omega }_j}F\left(p_j\left(\mathbf{x}\right)\times p_{tj}\left(\mathbf{x}\right)\right)d\mathbf{x}}{\partial \omega }=\sum _{j=1}^m\frac{\partial {\int }_{\Omega }H\left(-{\varphi }_j\left(\mathbf{x}\right)\right)F\left(p_j\left(\mathbf{x}\right)\times p_{tj}\left(\mathbf{x}\right)\right)d\mathbf{x}}{\partial \omega }+\frac{\partial {\int }_{\Omega }\prod _{k=1}^{m}H\left({\varphi }_k\left(\mathbf{x}\right)\right)F\left(p_{m+1}\left(\mathbf{x}\right)\times p_{tm+1}\left(\mathbf{x}\right)\right)d\mathbf{x}}{\partial \omega }=-\sum _{j=1}^m{\int }_{\Omega }\delta \left({\varphi }_j\left(\mathbf{x}\right)\right)\frac{\partial {\varphi }_j\left(\mathbf{x}\right)}{\partial \omega }F\left(p_j\left(\mathbf{x}\right)\times p_{tj}\left(\mathbf{x}\right)\right)d\mathbf{x}+\sum _{j=1}^m{\int }_{\Omega }H\left(-{\varphi }_j\left(\mathbf{x}\right)\right)F^{\prime }\left(p_j\left(\mathbf{x}\right)\times p_{tj}\left(\mathbf{x}\right)\right)\times p_{tj}\left(\mathbf{x}\right)\frac{\partial p_j\left(\mathbf{x}\right)}{\partial \omega }d\mathbf{x}+{\int }_{\Omega }\sum _{k=1}^m\left(\delta \left({\varphi }_k\left(\mathbf{x}\right)\right)\frac{\partial {\varphi }_k\left(\mathbf{x}\right)}{\partial \omega }\right)F\left(p_{m+1}\left(\mathbf{x}\right)\times p_{tm+1}\left(\mathbf{x}\right)\right)d\mathbf{x}+{\int }_{\Omega }\prod _{k=1}^{m}H\left({\varphi }_k\left(\mathbf{x}\right)\right)F^{\prime }\left(p_{m+1}\left(\mathbf{x}\right)\times p_{tm+1}\left(\mathbf{x}\right)\right)\times p_{tm+1}\left(\mathbf{x}\right)\frac{\partial p_{m+1}\left(\mathbf{x}\right)}{\partial \omega }d\mathbf{x}.$$ (A4)

By substituting ∂p_k(x)∕∂ω and ∂p_m+1(x)∕∂ω from Eqs. A2, A3 into Eq. A4, we get

$$\frac{\partial J}{\partial \omega }={\int }_{\Omega }\sum _{j=1}^m\left(\delta \left({\varphi }_j\left(\mathbf{x}\right)\right)\frac{\partial {\varphi }_j\left(\mathbf{x}\right)}{\partial \omega }\right)\times \left(F\left(p_{m+1}\left(\mathbf{x}\right)\times p_{tm+1}\left(\mathbf{x}\right)\right)-F\left(p_j\left(\mathbf{x}\right)\times p_{tj}\left(\mathbf{x}\right)\right)\right)d\mathbf{x}+\sum _{j=1}^m{\int }_{\Omega }H\left(-{\varphi }_j\left(\mathbf{x}\right)\right)F^{\prime }\left(p_j\left(\mathbf{x}\right)\times p_{tj}\left(\mathbf{x}\right)\right)\times p_{ti}\left(\mathbf{x}\right)\frac{\partial p_j\left(\mathbf{x}\right)}{\partial \omega }d\mathbf{x}+{\int }_{\Omega }\prod _{k=1}^{m}H\left({\varphi }_k\left(\mathbf{x}\right)\right)F^{\prime }\left(p_{m+1}\left(\mathbf{x}\right)\times p_{tm+1}\left(\mathbf{x}\right)\right)\times p_{tm+1}\left(\mathbf{x}\right)\frac{\partial p_{m+1}\left(\mathbf{x}\right)}{\partial \omega }d\mathbf{x}={\int }_{\Omega }\sum _{j=1}^m\left(\delta \left({\varphi }_j\left(\mathbf{x}\right)\right)\frac{\partial {\varphi }_j\left(\mathbf{x}\right)}{\partial \omega }\right)\times \left(F\left(p_{m+1}\left(\mathbf{x}\right)\times p_{tm+1}\left(\mathbf{x}\right)\right)-F\left(p_j\left(\mathbf{x}\right)\times p_{tj}\left(\mathbf{x}\right)\right)\right)d\mathbf{x}+\sum _{j=1}^m{\int }_{\Omega }H\left(-{\varphi }_j\left(\mathbf{x}\right)\right)F^{\prime }\left(p_j\left(\mathbf{x}\right)\times p_{tj}\left(\mathbf{x}\right)\right){\times }_{tj}\left(\mathbf{x}\right)\times \left(\frac{{\int }_{\Omega }\delta \left({\varphi }_j\left(\widehat{\mathbf{x}}\right)\right)\frac{\partial {\varphi }_j\left(\widehat{\mathbf{x}}\right)}{\partial \omega }\left(p_j\left(\mathbf{x}\right)-K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)\right)d\widehat{\mathbf{x}}}{\left|{\Omega }_j\right|}\right)d\mathbf{x}-{\int }_{\Omega }\prod _{k=1}^{m}H\left({\varphi }_k\left(\mathbf{x}\right)\right)F^{\prime }\left(p_{m+1}\left(\mathbf{x}\right)\times p_{tm+1}\left(\mathbf{x}\right)\right)\times p_{tm+1}\left(\mathbf{x}\right)\times \left(\frac{\left({\int }_{\Omega }{\sum }_{k=1}^m\left(\delta \left({\varphi }_k\left(\widehat{\mathbf{x}}\right)\right)\frac{\partial {\varphi }_k\left(\widehat{\mathbf{x}}\right)}{\partial \omega }\right)\left(p_{m+1}\left(\mathbf{x}\right)-K\left(I\left(\mathbf{x}\right)-I\left(\widehat{\mathbf{x}}\right)\right)\right)d\widehat{\mathbf{x}}\right)}{\left|{\Omega }_{m+1}\right|}\right)d\mathbf{x}.$$ (A5)

Now, F(p) is replaced by −ln(p) to get the derivatives of the proposed energy function. With further simplification, we obtain the final results in Eqs. 11, 12.