Combining Generative Models for Multifocal Glioma Segmentation and Registration

Abstract

In this paper, we propose a new method for simultaneously segmenting brain scans of glioma patients and registering these scans to a normal atlas. Performing joint segmentation and registration for brain tumors is very challenging when tumors include multifocal masses and have complex shapes with heterogeneous textures. Our approach grows tumors for each mass from multiple seed points using a tumor growth model and modifies a normal atlas into one with tumors and edema using the combined results of grown tumors. We also generate a tumor shape prior via the random walk with restart, utilizing multiple tumor seeds as initial foreground information. We then incorporate this shape prior into an EM framework which estimates the mapping between the modified atlas and the scans, posteriors for each tissue labels, and the tumor growth model parameters. We apply our method to the BRATS 2013 leaderboard dataset to evaluate segmentation performance. Our method shows the best performance among all participants.

1 Introduction

Gliomas are the most common primary brain tumors that arise within the brain parenchyma. They are commonly categorized according to their malignancies, from low-grade to high-grade, but nearly all low-grade gliomas eventually progress to high-grade malignancy [8]. Glioblastoma is the most malignant form of gliomas and has median survival rates of 12-18 months. The standard treatment includes partial or complete resection, chemotherapy, and radiation therapy [14]. Accurate delineation of glioma and edematous parenchyma is helpful for treatment planning and progression monitoring. However, the segmentation of brain gliomas is a challenging task of critical importance in medical image analysis, due to the complex shape and heterogeneous textures of such tumors. Moreover, multifocal gliomas, having 8-10% incidence among gliomas [1], are even more difficult to segment especially for methods assuming a single-focal mass.

To perform this challenging task, many techniques have been proposed. They can be predominantly classified as either discriminative or generative models [10]. Discriminative models extract image features for each voxel and train classifiers using these features guided by annotated training data [15, 16]. As these models directly learn classifiers from image features, they do not require domain-specific knowledge and can concentrate on the specific features relevant to the segmentation. However, their segmentation is restricted to images from the same protocol as the training data, since these models are often carefully fitted to the training data. Generative models incorporate prior information about the appearance and spatial distribution for each tissue type [2, 12]. For the prior information, the appearance of tumor and edema are modeled as outliers to the healthy tissue, or tumor growth models are used for localizing tumor structures. However, designing effective prior models requires significant efforts and the performance is limited by the range of domain-specific knowledge employed.

In this paper, we propose a new method for joint segmentation and registration (JSR) of brain gliomas. In order to generate a patient-specific atlas, our method grows tumors on the atlas, with parameters estimated at the same time, and registers the scans to this atlas to infer the segmentation. Differently from the previous JSR framework of [2], we also allow multiple tumor seed points to segment multifocal gliomas. For our method, tumors are grown on each seed using a tumor growth model and combined into the single tumor probability map. Also, we incorporate a tumor shape prior into the framework by introducing an empirical Bayes model [11]. The tumor shape prior is estimated by the random walk with restart [5] using tumor seeds as initial foreground information, which helps the framework to find accurate tumor shapes for di cult cases. Since this shape prior can be considered as another generative model, in principle, our method systematically combines two kinds of generative models. In the rest of this paper, we describe the atlas generation method for multifocal tumors in Sec. 2 and our JSR framework with shape prior using this atlas in Sec. 3. In Sec. 4, we present our quantitative and qualitative evaluations and conclude the paper in Sec. 5.

2 Atlas Generation for Multifocal Gliomas

In this section, we use a tumor growth model to embed multifocal masses in an atlas of a healthy population. We denote by $\mathcal{T}$ the label map and by t the possible tissue type. Then we denote by ‘${\mathcal{T}}_t\mid \mathbf{x}$’ the tissue type being t at voxel x, namely ‘$\mathcal{T}=t\mid \mathbf{x}$’. The atlas p_A is defined as a set of probability maps $p_A({\mathcal{T}}_t\mid \mathbf{x})$ for white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF), i.e. t ∈ {WM,GM,CSF}, obtained by averaging aligned segmentations of healthy brains. We adapt the atlas to the subject space by simulating the tumor growth via the diffusion-reaction-advection model of [3].

Unlike most previous methods assuming only a single mass, such as [2], we allow multiple tumor seeds and grow tumors for each given seed location. We then merge each result into the single spatial probability map of tumor (TU). If we assume the subject shows M tumors, then each tumor i ∈ {1, … , M} is characterized by seed location o_i, its size r_i, and other shared tumor parameters including the diffusion coefficient for white matter D_WM and the proliferation coefficient p₁. Therefore, each tumor growth is completely defined by the parameters q_i ≜ {o_i, r_i, D_WM, p₁}, and the tumor probability d_i and its associated deformation (mass effect) u_i are obtained. We also define q as the set of all tumor parameters q = {q₁, … , q_M}. The merged tumor probability $p({\mathcal{T}}_{\mathit{TU}}\mid \mathbf{q},\mathbf{x})$ and mass effect u(x) at voxel x are simply defined as the sum of each estimation, that is, $p({\mathcal{T}}_{\mathit{TU}}\mid \mathbf{q},\mathbf{x})\triangleq \min \left\{{\sum }_{i=1}^M{\mathbf{d}}_i\right(\mathbf{x}),1\}$ and $\mathbf{u}\left(\mathbf{x}\right)\triangleq {\sum }_{i=1}^M{\mathbf{u}}_i\left(\mathbf{x}\right)$. We then construct spatial probability maps for GM and CSF (t ∈ {GM, CSF}) by deforming the atlas via the mass effect u and weighting them with the complement of $p({\mathcal{T}}_{\mathit{TU}}\mid \mathbf{q},\mathbf{x})$:

$$p({\mathcal{T}}_t\mid \mathbf{q},\mathbf{x})\triangleq p_A({\mathcal{T}}_t\mid \mathbf{u}(\mathbf{x}\left)\right)\cdot (1-p({\mathcal{T}}_{\mathit{TU}}\mid \mathbf{q},\mathbf{x}\left)\right).$$ (1)

For the spatial probability map for edema (ED), we model the close proximity of edema to tumor via the Heaviside function H(·) (H(a) = 0 for a ≤ 0 and H(a) = 1 for a > 0) resulting in

$$p({\mathcal{T}}_{\mathit{ED}}\mid \mathbf{q},\mathbf{x})\triangleq 0.5\cdot p_A({\mathcal{T}}_{\mathit{WM}}\mid \mathbf{u}(\mathbf{x}\left)\right)\cdot (1-p({\mathcal{T}}_{\mathit{TU}}\mid \mathbf{q},\mathbf{x}\left)\right)\cdot H\left(p\right({\mathcal{T}}_{\mathit{TU}}\mid \mathbf{q},\mathbf{x}\left)\right),$$ (2)

where we multiply 0.5 to avoid preference of edema over WM. The spatial probability for WM is defined by the complement of spatial probability maps of the other tissue types:

$$p({\mathcal{T}}_{\mathit{WM}}\mid \mathbf{q},\mathbf{x})\triangleq 1-p({\mathcal{T}}_{\mathit{TU}}\mid \mathbf{q},\mathbf{x})-p({\mathcal{T}}_{\mathit{ED}}\mid \mathbf{q},\mathbf{x})-p({\mathcal{T}}_{\mathit{GM}}\mid \mathbf{q},\mathbf{x})-p({\mathcal{T}}_{\mathit{CSF}}\mid \mathbf{q},\mathbf{x}).$$ (3)

After growing tumors on the atlas, a set of tissue type Θ is now defined as {TU, ED, WM, GM, CSF}. A sample set of the spatial probabilities is shown in Fig. 1 (c)-(f). These spatial probabilities are modified from the atlas shown in Fig. 1 (h)-(i) using the tumor growth model.

Fig. 1.

An example of spatial probabilities for multifocal glioma. We show subject scans in (a)-(b), normal atlas in (h)-(i), spatial probabilities $p({\mathcal{T}}_t\mid \mathbf{q},\mathbf{x})$ obtained by growing tumors on normal atlas in (c)-(f), tumor shape prior in (g), spatial probabilities $p({\mathcal{T}}_t\mid \mathbf{q},\mathit{h},\mathbf{x})$ aligned to the scan in (j)-(m), and tumor posterior $p({\mathcal{T}}_{\mathit{TU}}\mid Y,\mathbf{x})$ in (n).

3 Joint Segmentation-Registration with Shape Prior

Having defined the spatial probabilities $p({\mathcal{T}}_t\mid \mathbf{q},\mathbf{x})$, we now describe our JSR framework using the atlas we constructed in the previous section as well as shape priors for tumor. The parameters of our JSR framework include the tumor parameters q, the mapping h from the subject space Ω to the atlas space, and the tissue specific means and covariances Φ. We then find optimal parameters by solving the following problem:

$$\{{\mathbf{q}}^{\ast },h^{\ast },{\Phi }^{\ast }\}\triangleq \arg \underset{\mathbf{q},\mathit{h},\Phi }{\max }\left\{\prod _{\mathbf{x}\in \Omega }\sum _{t\in \Theta }p({\mathcal{T}}_t\mid \mathbf{q},\mathit{h},\mathbf{x})\cdot p(Y\mid {\mathcal{T}}_t,\Phi ,\mathbf{x})\cdot p(\mathbf{q},\mathit{h}\mid {\eta }^{\ast },\mathbf{x})\right\},$$ (4)

where Y consists of the subject images, $p({\mathcal{T}}_t\mid \mathbf{q},\mathit{h},\mathbf{x})\triangleq p({\mathcal{T}}_t\mid \mathbf{q}\mathit{h}(\mathbf{x}\left)\right)$ is the aligned atlas obtained by warping the tumor grown atlas $p({\mathcal{T}}_t\mid \mathbf{q},\mathbf{x})$ via h, and $p(Y\mid {\mathcal{T}}_t,\Phi ,\mathbf{x})$ is the image likelihood defined as the multivariate Gaussian for Φ. p(q, h∣η*, x) is the prior function for {q, h} to be used as part of an empirical Bayes approach [11], and we assume the probability distribution of hyperparameters η given Y , i.e. p(η∣Y), is sharply peaked at. For this prior function, we assume deformed atlas for tumor $p({\mathcal{T}}_{\mathit{TU}}\mid \mathbf{q},\mathit{h}(\mathbf{x}\left)\right)$ tends to match the tumor shape prior $p_E({\mathcal{T}}_{\mathit{TU}}\mid {\eta }^{\ast },\mathbf{x})$ as follows

$$p(\mathbf{q},\mathit{h}\mid {\eta }^{\ast },\mathbf{x})\triangleq \frac{1}{Z}\exp (-\lambda \cdot {\Vert p({\mathcal{T}}_{\mathit{TU}}\mid \mathbf{q},\mathit{h}(\mathbf{x}\left)\right)-p_E({\mathcal{T}}_{\mathit{TU}}\mid {\eta }^{\ast },\mathbf{x})\Vert }^2),$$ (5)

where Z is a normalization constant and λ is a parameter controlling the weight of the shape prior on this function. If λ = 0, our problem turns into the JSR problem of [2] when a single tumor seed is used. The tumor shape prior $p_E({\mathcal{T}}_{\mathit{TU}}\mid {\eta }^{\ast },\mathbf{x})$ is inferred directly from images utilizing tumor seed locations using the random walk with restart (RWR) [5]. This method is suitable for estimating heterogeneous tumor regions as it showed strong performance in finding weak boundaries and separating textures in cluttered scenes. Now, we briefly describe the RWR method.

Given subject images Y, let us construct an undirected graph $G=(\mathcal{V},\mathcal{E})$ with nodes $v\in \mathcal{V}$ and edges $e\in \mathcal{E}$. We assume G is defined on 26-connected neighborhoods. For each edge e_ij, we assign the weight w_ij which measures the likelihood for having the same label between node v_i and v_j:

$$w_{\mathit{ij}}\triangleq \exp (-{\Vert {\mathbf{y}}_i-{\mathbf{y}}_j\Vert }^2∕\sigma ),$$ (6)

where y_i is a multi-modal intensity vector for node v_i and σ is a constant for normalizing intensity differences. Then we define the adjacency matrix W = [w_ij]_N×N, the degree matrix D = diag(D₁, … ,D_N) with $D_j={\sum }_{i=1}^N{w}_{\mathit{ij}}$, and the transition matrix P = D⁻¹ × W. We also define $\tilde{\mathbf{b}}={\left[{\tilde{b}}_i\right]}_{N\times 1}$ as starting locations of the random walker where ${\tilde{b}}_i=1$ if v_i is within distance s from tumor seed points and ${\tilde{b}}_i=0$ otherwise. The random walker iteratively steps to a neighboring location with the probability proportional to the edge weight. Also, it has a restarting probability c to return to the seed points. Our shape prior for tumors is calculated as the steady-state probability of the random walker for the transition matrix P:

$${\left[p_E\right({\mathcal{T}}_{\mathit{TU}}\mid {\eta }^{\ast },{\mathbf{x}}_i\left)\right]}_{N\times 1}\propto c{(\mathbf{I}-(1-c\left)\mathbf{P}\right)}^{-1}\tilde{\mathbf{b}},$$ (7)

where x_i is a location of a node v_i. In Fig. 1 (g), we show the tumor shape prior obtained using (7) on the subject scan shown in (a)-(b). This shape prior shows high probability values in the tumor region.

Returning to the JSR problem (4), we obtain q*, h*, and Φ* via an implementation of the Expectation Maximization (EM) algorithm [13]. The EM algorithm iteratively determines the solution by computing the posterior

$$p({\mathcal{T}}_t\mid Y,\mathbf{x})\triangleq p({\mathcal{T}}_t\mid Y,{\mathbf{q}}^{\prime },{\mathit{h}}^{\prime },{\Phi }^{\prime },\mathbf{x})\propto p({\mathcal{T}}_t\mid {\mathbf{q}}^{\prime },{\mathit{h}}^{\prime }(\mathbf{x}\left)\right)\cdot p(Y\mid {\mathcal{T}}_t,{\Phi }^{\prime },\mathbf{x})$$ (8)

in the E-Step and updating the parameters in the M-Step by sequentially maximizing the following cost function

$$\begin{array}{cc}\hfill Q\{& (\mathbf{q},\mathit{h},\Phi );({\mathbf{q}}^{\prime },{\mathit{h}}^{\prime },{\Phi }^{\prime })\}\hfill \\ \hfill & \triangleq \sum _{\mathbf{x}\in \Omega }\sum _{t\in \Theta }p({\mathcal{T}}_t\mid Y,\mathbf{x})\cdot \log \left(p({\mathcal{T}}_t\mid \mathbf{q},\mathit{h},\mathbf{x})\cdot p(Y\mid {\mathcal{T}}_t,\Phi ,\mathbf{x})\cdot p(\mathbf{q},\mathit{h}\mid {\eta }^{\ast },\mathbf{x})\right).\hfill \end{array}$$ (9)

In detail, Φ’ is obtained from arg max_Φ Q{(q’, h’, Φ’)} using a closed form of [7] and h’ is obtained from arg max_h Q{(q’, h, Φ’); (q’, h’, Φ’)} which iteratively can be solved as in [2]. For updating q, we maximize Q(q, h’, Φ’); (q’, h’, Φ’)} using a derivative-free pattern search library [4] as there {exists no analytical expression for the derivatives of this function with respect to q. We iterate E-Step and M-Step until convergence is achieved. After convergence, we assign {q’, h’, Φ’} to {q*, h*, Φ*}, respectively. We show aligned spatial probabilities in Fig. 1 (j)-(m) optimized via the EM algorithm. These spatial probabilities and the tumor posterior (computed using (8)) shown in (n) now fit well to healthy tissue and pathological regions shown in (a)-(b).

4 Experiments

Our method is semi-automatic and requires minimal user inputs including seed point and radius for each tumor to initialize q and one sample point for each tissue class to initialize means of Φ. For preprocessing, we co-registered all four modalities (T1, T1-CE, T2, and FLAIR), corrected MR field inhomogeneity, and scaled intensities to fit [0, 255]. We solved the tumor growth model on a lattice of 64×64×64 nodes for efficiency reasons. To differentiate enhancing tumor (ET) and others (NT) including necrosis and non-enhancing core within tumor regions, we assume all sub-structures of tumor are equally probable, i.e. $p({\mathcal{T}}_{\mathit{ET}}\mid \mathbf{q},\mathbf{x})=p({\mathcal{T}}_{\mathit{NT}}\mid \mathbf{q},\mathbf{x})=p({\mathcal{T}}_{\mathit{TU}}\mid \mathbf{q},\mathbf{x})∕2$. In experiments, we substitute TU in Θ as {ET, NT}. To find best λ in (5), we measured Dice scores for the tumor core region over λ ∈ [0, 60] using the training data consisting of 10 subjects having manual segmentations. As shown in Fig. 2, Dice scores were not significantly different on λ ∈ [10, 60] (p > 0.1 using the Wilcoxon signed-rank test) and we chose λ = 40 for all experiments. For the shape prior in (7), we used c = 0.0004 as suggested by [5], s = 5mm, and σ = max_ij{∥y_i − y_j∥²}/60.

Fig. 2.

Box plots of Dice score (core) for varying λ ∈ [0, 60]

We tested our method to the BRATS 2013 leaderboard data via the BRATS online tools [9]. The leaderboard data set is the main data set used for comparing results of participants of BRATS [10] and it consists of 21 high-grade and 4 low-grade glioma subjects. When segmentation results are uploaded to the online tools the performance is measured using manual segmentation labels which are not available for download. The performance measures include Dice scores, positive predictive value (PPV), and sensitivity for three interest regions: whole (complete abnormal regions including tumor and edema), core (tumor regions), and active (enhancing regions of tumor). The top 4 results among 16 participants excerpted from [9] are shown in Table 1. Zhao and Guo used generative models based on a learned MRF model on supervoxel clusters and active contours with manual initializations, respectively. Tustison used a discriminative model based on the decision forest [16]. Note that Tustison and Zhao are fully-automatic methods and might therefore be at a disadvantage in this comparisons. The details of performance measures and participants’ methods are described in [10]. Our method (Kwon) performed best among all participants and showed highest average ranks for all regions. The score gap with respect to other participants for the core region is bigger than those of the other regions, which means our method performed especially well for segmenting core regions. The average running time of our method was 85 min on an Intel Core i7 3.4 GHz machine with Windows operating system. Considering it usually takes under 10 min for the user input, the total running time is comparable to Tustison [10].

Table 1.

BRATS 2013 Leaderboard Results (Top 4).

	Dice			PPV			Sensitivity			Rank
Method	whole	core	active	whole	core	active	whole	core	active	whole	core	total
Kwon	0.86	0.79	0.59	0.88	0.84	0.60	0.86	0.81	0.63	2.00	1.33	1.89
Zhao	0.83	0.73	0.55	0.77	0.67	0.46	0.94	0.89	0.78	2.67	3.00	3.67
Tustison	0.79	0.65	0.53	0.83	0.70	0.51	0.81	0.73	0.66	4.33	4.00	4.78
Guo	0.79	0.71	0.50	0.73	0.77	0.50	0.88	0.76	0.60	5.33	2.67	5.56

In Fig. 3, segmentation and registration results from our method are displayed. The spatial probabilities aligned to the scan in (d)-(g) show that they fit well to the scan. The tumor shape priors in (h) help to align spatial probabilities for tumor as they initially estimate tumor regions reasonably well. However, our method also showed robust estimation of tumor regions when the shape prior leaked into nearby regions as indicated by arrow in the fourth row of (h). The spatial probabilities for tumor were not expanded to this leaked region as the image likelihood for tumor kept lower values than those of healthy tissues on this region during the EM iterations. As a result, segmentations in (c) show visually reasonable tissue estimation especially for ET, NT, and ED regions.

Fig. 3.

Segmentation and registration results for 6 subjects selected from the BRATS 2013 leaderboard data set. The top 4 rows show a single-focal glioma and bottom 2 rows show multi-focal gliomas. We show subject images in (a)-(b), segmentation results in (c) (indicating ET, NT, ED, WM, GM, and CSF in blue, red, cyan, white, gray, and yellow colors, respectively), spatial probabilities $p({\mathcal{T}}_t\mid \mathbf{q},\mathit{h},\mathbf{x})$ in (d)-(g), and tumor shape priors in (h).

5 Conclusion

In this paper, we proposed a new joint segmentation and registration method for multifocal glioma images. This method allows multiple tumor seed points to grow each focal mass and combines them to single tumor density for modifying a normal atlas into one with tumors and edema. To perform robustly on tumors having complex shapes and heterogeneous textures, we incorporate a tumor shape prior directly estimated from images into our framework. We then find the optimal solution via the EM algorithm. Our method was evaluated on the BRATS 2013 leaderboard data set and showed the best performance among all participants. Although we only quantitatively validated our segmentation results, our method also produces mappings between a normal atlas and subject scan and simultaneously estimates tumor parameters. These additional results of our method could be used for understanding brain tumor development and for the development of location-based biomarkers. We also expect that our method could improve the registration of pre-operative and post-recurrence brain scans for multifocal gliomas [6].