Integrated 3D Anatomical Model for Automatic Myocardial Segmentation in Cardiac CT Imagery

Abstract

Segmentation of epicardial and endocardial boundaries is a critical step in diagnosing cardiovascular function in heart patients. The manual tracing of organ contours in Computed Tomography Angiography (CTA) slices is subjective, time-consuming and impractical in clinical setting. We propose a novel multi-dimensional automatic edge detection algorithm based on shape priors and principal component analysis (PCA). We have developed a highly customized parametric model for implicit representations of segmenting curves (3D) for Left Ventricle (LV), Right Ventricle (RV), and Epicardium (Epi) used simultaneously to achieve myocardial segmentation. We have combined these representations in a region-based image modeling framework with high level constraints enabling the modeling of complex cardiac anatomical structures to automatically guide the segmentation of endo/epicardial boundaries. Test results on 30 short-axis CTA datasets show robust segmentation with error (mean ± std mm) of (1.46 ± 0.41), (2.06 ± 0.65), (2.88 ± 0.59) for LV, RV and Epi respectively.

1. Introduction

In 2015, the American Heart Association (AHA) reported (CDC 2015) that heart disease is the No. 1 cause of death in the United States, killing nearly 787,000 people in 2011 alone. Among various types of heart diseases, coronary artery disease is the most common, killing nearly 380,000 people annually. Further, the direct and indirect costs of heart disease total more than $320 billion. Consequently, early and cost-effective diagnosis of heart diseases is an important requirement for clinical practitioners worldwide.

Cardiac Tomography Angiography (CTA), a non-invasive imaging technique, is becoming increasingly popular for cardiac examination, mainly due to its superior spatial resolution compared to MRI. This imaging modality is currently widely used for the diagnosis of Coronary Artery Disease (CAD) but it is not commonly used for the diagnosis of ventricular and atrial function. The primary reason for that is the lack of autonomous image processing techniques for cardiac medical imagery. Medical imagery, in general, suffers from poor contrast, low signal to noise ratio (SNR), weak or missing edges or smearing of edges due to patient movement. These among other reasons make it difficult to apply standard image processing techniques to medical image processing tasks and obviate the need to incorporate as much prior information about the objects of interest as possible.

In our work, we use shape prior model-based implicit parametric representations of each anatomical structure of interest (LV, RV, epicardium), and calculate parameters of each segmenting curve via gradient descent procedure to minimize an energy functional designed for the segmentation task.

1.1. Relationship to prior work

To overcome the shortcomings of the image acquisition process and inherent complexity of the heart anatomy, many different classes of techniques have been proposed over the years to aid diagnosis and prognosis of cardiac diseases in various imaging modalities and data acquisition protocols. Model based segmentation methods have become particularly prominent in literature and many different formulations have been proposed with many including some form of weak or strong shape prior. Active Shape and Appearance Models (Cootes and Taylor 1992; Cootes et al. 1995; Cootes and Taylor 1998; Cootes et al. 2001) based techniques have been very influential in the field of medical image processing, and have been used for both LV and RV segmentation (van Assen et al. 2006, 2008; Mitchell et al. 2001). A further enhancement of model based techniques is in using deformable models (Weese et al. 2001) which provides additional flexibility by allowing the models to freely deform in response to local image features while restraining the global shape. Ecabert et al. (2008) used this technique to present a model-based approach for fully automatic segmentation of the whole heart (four chambers, myocardium and great vessels) in cardiac CT imagery. Their model consisted of multi-compartment triangulated mesh of vertices and triangles. The authors used a 3D Global Hough Transform for whole heart initialization and estimated global pose using similarity transformations. This model was then allowed to adapt to local individual patient anatomy using shape-constrained deformable models.

Atlas based segmentation is another class of techniques that have become increasingly popular in recent years. An atlas describes the different structures present in a given type of image. It can be generated by manually segmenting an image or by integrating information from multiple segmented images from different individuals. Given an atlas, an image can be segmented by mapping its coordinate space to that of the atlas using a registration process (Rohlfing et al. 2003). Initially applied to brain segmentation, it is now being widely used for cardiac segmentation as well. Recently Shahzad et al. (2017) presented a fully automatic segmentation framework to segment the whole heart, aortic root, left atrium, right atrium, LV, and RV including both the blood-pool and the myocardial tissue in the ventricles from non-contrast-enhanced calcium scoring CT scan. Based on the work of Kirişli et al. (2010), this work used a multi-atlas-based segmentation process to register multiple atlas scans with corresponding manually annotated labels to an unseen subject’s scan. The registration process used a mutual-information (Mattes et al. 2003) based similarity measure as a cost function to be minimized with respect to a set of image transformation parameters.

While most methods referenced here require some form of training images or prior information, Zhu et al. (2014) presented a training free method for extraction of LV myocardium from CT images. The authors used the blood pool surface to approximate the heart surface and automatically detected the LV by examining the distribution of levelsets starting from the LV apex and further refining it by utilizing a geometric active contour model (Kichenessamy et al. 1995; Caselles et al. 1997) on the blood pool surface. After locating the endocardial surface, the authors used a variational region growing method (Gao et al. 2012) to locate the epicardial surface. These two surface estimates are then refined using an active contour model with shape constraints and myocardium extracted from the voxels between these two surfaces. While this method was used to obtain only LV myocardium, Zhu et al. (2013a) used a similar strategy to segment both the ventricles and myocardium. The RV is identified on heart surface constructed by using the LV as a seed region for the variational region growing method.

Active contour models have been widely used in medical image segmentation because of their flexibility and robustness. Prior information can be incorporated as well to restrict the optimization space. Chen et al. (2001) used the concept of an average shape model within the framework of geometric active contour models. Staib and Duncan (1992) introduced a parametric point model based on an elliptic Fourier decomposition of the land-mark points. Chakraborty et al. (1994) extended this approach to a hybrid segmentation model that incorporates both gradient and region-homogeneity information. Leventon et al. 2000 proposed a model-based segmenter that incorporated shape information as a prior model to restrict the flow of the geodesic active contour (Yezzi et al. 1997; Caselles et al. 1997). Their prior parametric shape model is derived by performing PCA on a collection of signed distance maps of the training shape. The segmenting curve then evolves according to two competing forces: 1) the gradient force of the image, and 2) the force exerted by the estimated shape where the parameters of the shape are calculated based on the image gradients and the current position of the curve. In contrast to these edge-based active contour models which utilize image gradient to stop the evolving contours on the object boundaries, region-based models (Chan and Vese 1999), which minimizes the Mumford-Shah (Mumford and Shah 1989; Chan and Vese 2001) energy functional have shown to be more robust to noise and initial placement of contours. These methods are more global in nature and avoid taking image intensity derivatives.

Tsai et al. (2003) developed a parametric model using pose and shape parameters for segmentation, where they describe the segmenting curve as a linear combination of eigenvectors that are obtained by performing PCA on the variations from the mean shape. Vikram et al. (2010) later applied a variation of this technique which incorporated region of confidence (ROC) labels indicating the reliability of edges in certain object regions, to the task of myocardial segmentation in pigs cardiac MR datasets.

1.2. Our contribution

Vikram et al. (2010) and Tsai et al. (2003) use the chan-vese image model which models the image as piece-wise smooth function and the evolution of the segmenting curve depends upon the pixel intensities within entire regions. That is, region-based models regard an image as the composition of a finite number of regions and rely on regional statistics for segmentation. The statistics of entire regions (such as sample mean and variance) are used to direct the movement of the curve toward the boundaries of the image.

More formally, the Chan-Vese energy is defined as:

$$E_{\mathit{cv}}={\int }_{R^u}{(I-\mu )}^{2}d\mathbf{x}+{\int }_{R^v}{(I-\nu )}^{2}d\mathbf{x}$$ (1)

where R^u and R^v are the regions inside and outside the segmenting curve. It has been shown that the optimal choice for the constants μ and ν are the region means. This model can be used either on single contour with two regions, inside and outside, or multiple regions. It can also be easily extended from 2D to 3D. In case of 3D model, the image statistics are calculated on entire volumes rather than regions.

We have customized these frameworks for the task of 3D Cardiac CT imagery by introducing several high level constraints and making use of shape priors for three anatomical regions, LV, RV, and Epi simultaneously. We have developed a Chan-Vese like model for the interior region of the Epicardium i.e. RV, LV and Myocardium. To customize the model for myocardial segmentation, we have derived a mathematical formulation for enforcing a strict ordering of the image statistics of different regions in the appearance model which makes the segmentation even more robust to initial contour placement.

Additionally, the region surrounding the epicardium in cardiac imagery is very complex with different types of tissues exhibiting different intensity ranges. This makes it unreasonable to model the background as a coherent region with a single mean intensity. Consequently, we have developed a dynamic adaptive background model. We model the background as a binary cluster, one cluster exhibiting very low and the other very high image intensities. We have derived and present the mathematical formulation for doing so.

Instead of using the model separately for each region¹ which doesn’t allow the shapes to be constrained to not overlap with each other, we use all three priors simultaneously with high level constraints. Our model includes a coupling energy functional with an overlap penalty between distinct regions which effectively couples the three priors. The shape models we have developed are defined directly in 3D space instead of operating in a slice-by-slice manner.

Note that our binary clustering strategy for background region together with strict ordering of the image statistics within the four regions imposes more prior information on the known aspects of the image appearance than other methods. These same adaptations together with the overlap constraints between regions however rendered the problem non-convex and made the optimization much more challenging compared to other models. We have successfully addressed this complication by using a relaxation of the overlap constraint which permits small amounts of overlap during the optimization process but not in the final result. This relaxation together with momentum based gradient descent has resulted in a robust and reliable optimization process on our models.

Our formulation also overcomes several shortcomings of some of the latest methods outlined in the previous section. In particular, the multi-atlas based method fails to respond to local image variations. Consequently, it fails to differentiate between LV endo and epicardium. Zhu et. al’s region growing method uses generic moment based priors and does not take into account the expected ordering of the average appearance intensities between various regions being segmented. In contrast, our formulation is based on anatomically specific PCA based shape priors and region based image appearance models highly customized to contrast enhanced cardiac CT images. This customized appearance model combined with customized PCA based shape priors is able to adequately capture both global and local image information and outperforms the other approaches as shown in later sections.

The rest of the paper is organized as follows. Section 2 provides a high level overview of our segmentation models and approach. Section 3 describes a variational gradient-based approach to align all the training shapes in the database to eliminate variations in pose. Based on this aligned training set, we show in Section 4 the development of an implicit parametric representation of the segmenting curve using principal component analysis. Section 5 describes the region-based models we have developed for image segmentation. In section 6 we show the experimental results obtained on human CTA datasets and experimentally illustrate the salient features of our algorithmic framework. Finally, in section 7 we conclude with a summary and some possible future research directions of this work.

2. Approach

In collaboration with Nuclear Cardiology R&D Laboratory at Emory University School of Medicine, we have applied our algorithm for myocardial segmentation in a 30 patient 3D CTA study. We use part of the studies for the training phase of our algorithm in which we extract mean shapes and principal modes of shape variation of each of the 3 anatomical structures, namely LV, RV and Epi. The training phase is based on manual expert segmentation of the anatomical structures by experts at Emory University Hospital. The result of the manual segmentation process is a binary mask for each region which we then convert to a signed distance function (SDF) (Osher and Sethian 1988) where the zero level set of the SDF represents shape boundary. If we define the image domain to be Ω, the boundary of the binary shape as ∂Ω and x as any point in the image domain, then the signed distance function SDF(x) is defined as follows:

$$\mathrm{SDF}(\mathbf{x})=\{\begin{array}{cc}+d(\mathbf{x},\partial \Omega ),\hfill & \mathbf{x}\in {\Omega }^{+}\hfill \\ -d(\mathbf{x},\partial \Omega ),\hfill & \mathbf{x}\in {\Omega }^{-}\hfill \end{array}\phantom{\}}$$ (2)

where d(x,∂Ω) is the euclidean distance from a point x to the nearest point on the shape boundary and Ω⁺ and Ω⁻ are, respectively, the parts of the domain outside and inside the shape boundary. Fast Marching methods (Sethian 1996) provide very efficient ways of computing the SDF from a binary shape.

Since shape is defined to be invariant to Euclidean similarity transformations (Dryden and Mardia 1998), we first perform image alignment on the set of training shapes before proceeding further. Once we have a set of aligned training SDF’s, we perform PCA to extract mean shape representations and principal modes. In the actual segmentation phase, we allow optimization of a set of free parameters, pose and shape, for each region. The shape parameters correspond to a set of weights in a linear combination of mean shape and its principal modes. This linear combination is then allowed to deform by a set of pose parameters including translation, rotation and scale.

The parameter optimization process is based on the gradients of a region based energy functional designed to achieve the task of myocardial segmentation. This image modeling functional is based on highly successful and well understood image modeling frameworks developed over the years. We have added several high level modifications to these models leading to a highly customized and integrated 3D anatomical model for automatic myocardial segmentation in full 3D Cardiac CT Imagery.

3. Image Alignment

The various images in the training set vary in size and orientation. We need to employ an alignment technique as a preprocessing step to allow us to capture shape variations in our database without interference from pose variations. There are many works dealing with image alignment problem (Dryden and Mardia 1998; Goodall 1991; Cootes and Taylor 2001). For our purposes, we approach the task of image alignment in variational framework.

Let the training set $\mathbb{T}$ consist of n 3D binary images {I¹,I²,⋯,Iⁿ}, each with values of one inside and zero outside. The goal is to calculate a set of pose parameters {p¹,p²,⋯,pⁿ} used to jointly align the binary training images, and hence remove any variations in shape due to pose differences. We focus on using similarity transformations to align these binary images to each other. That is, in three dimensions, each set of pose parameters, pⁱ, comprises of translation in x,y,z directions, uniform scale h in 3 dimensions, and three rotations, yaw,pitch and roll. We represent the translation as a vector with three components $T={[xyz]}^T$ and scale as a single parameter h.

Space rotations have three degrees of freedom, and admit several ways to represent and operate with them (Murray et al. 1994). Each representation has advantages and disadvantages. In our work we chose to use exponential or twist coordinates, also known as Euler-Rodrigues parameters (Cheng and Gupta 1989; Murray et al. 1994, p. 33), a compact representation requiring only three real numbers. The Euler-Rodrigues formula (Murray et al. 1994, p. 28) states that the rotation matrix represented by $v\in {\mathbb{R}}^3$ is given by:

$$R(v)=\cos (\Vert v\Vert )I+\sin (\Vert v\Vert ){\left[\frac{v}{\Vert v\Vert }\right]}_{\times }+(1-\cos (\Vert v\Vert ))\left[\frac{v}{\Vert v\Vert }\right]{\left[\frac{v}{\Vert v\Vert }\right]}^T$$ (3)

where v is the axis of rotation and its magnitude is the amount of rotation in radians and

$${[a]}_{\times }≔\left(\begin{array}{ccc}\hfill 0\hfill & \hfill -a_3\hfill & \hfill a_2\hfill \\ \hfill a_3\hfill & \hfill 0\hfill & \hfill -a_1\hfill \\ \hfill -a_2\hfill & \hfill a_1\hfill & \hfill 0\hfill \end{array}\right)\in {\text{Skew}}_3$$ (4)

is the cross-product (skew-symmetric) matrix such that [a]_×b = a×b, for all $a={(a_1{a}_2{a}_3)}^T,b\in {\mathbb{R}}^3$.

The transformed image of I, based on pose parameters, p, is defined as:

$$\widehat{I}(\widehat{\mathbf{x}})=I(\mathbf{x})$$ (5)

where $\widehat{\mathbf{x}}$ and x represent any point in the transformed and original un-transformed image respectively. $\widehat{\mathbf{x}}$ and x are related by the following relation:

$$\widehat{\mathbf{x}}=\mathit{hR}(v)(\mathbf{x}-O)+(T+O)$$ (6)

where O is the center of rotation. The inverse transformation given by the relation:

$$\mathbf{x}=\frac{1}{h}R{(v)}^T\left(\widehat{\mathbf{x}}-\left(T+O\right)\right)+O$$ (7)

While transforming any image (or its signed distance representation), we go over the transformed image and use the inverse relation from equation 7 to get location from original image where we can use tri-linear interpolation for sub-pixel locations.

In order to apply a variational framework for this process, and achieve sub-pixel accuracy in the alignment process we first convert the binary training images to signed distance functions. We then align the 3D SDF masks of the original binary images to a reference image by minimizing an energy functional w.r.t. the set of pose parameters. The objective of the alignment process is to maximize the overlap between a training image and a reference image. An intuitive way to formulate an energy functional representing this task could be as follows:

$$E_{\text{align}}(\text{Training Image})={\int }_{\Omega }\mid {\chi }_{\text{train}}-{\chi }_{\mathit{ref}}\mid d\widehat{\mathbf{x}}$$ (8)

In equation 8, Ω is the training image domain and χ_train and χ_ref are the characteristic functions of the training and reference images respectively. In any image domain, the characteristic function is equal to 1 inside a shape and 0 outside it. The energy functional essentially measures the overlap between the reference image and image being aligned. If the boundary of the image being aligned matches exactly with that of the reference image then the energy would be zero. Any deviation from that would lead to an increase in energy. Hence, taking the gradient of alignment energy w.r.t. the pose parameters and running a gradient descent procedure leads to the desired set of final optimized pose parameters which reduce the mismatch due to pose variations in the training images. The problem of aligning a group of images to each other is under determined. In order to get around this problem we can use one of the training images as a reference by keeping its pose parameters fixed and align all other images to this reference.

At each iteration of gradient descent process, we use the following pose parameter update equation:

$${\mathbf{p}}_i^{n+1}={\mathbf{p}}_i^n-\delta t\cdot {\nabla }_{{\mathrm{p}}_i}{E}_{\text{align}}$$ (9)

where ∇_piE_align represents the gradient of the alignment energy w.r.t. i_th component of the pose parameters. Following the discussion in appendix A, the expressions for individual derivatives w.r.t. translation (T), scale (h) and rotation (v) are given by the following set of equations:

$${\nabla }_T{E}_{\text{align}}={\int }_{\widehat{S}}\left(1-2{\chi }_{\mathit{ref}}\right){\left(\frac{\partial \widehat{P}}{\partial T}\right)}^T\widehat{N}d\widehat{\mathbf{S}}$$ (10a)

$${\nabla }_h{E}_{\text{align}}={\int }_{\widehat{S}}\left(1-2{\chi }_{\mathit{ref}}\right){\left(\frac{\partial \widehat{P}}{\partial h}\right)}^T\widehat{N}d\widehat{\mathbf{S}}$$ (10b)

$${\nabla }_v{E}_{\text{align}}={\int }_{\widehat{S}}\left(1-2{\chi }_{\mathit{ref}}\right){\left(\frac{\partial \widehat{P}}{\partial v}\right)}^T\widehat{N}d\widehat{\mathbf{S}}$$ (10c)

In equations 10, $\widehat{S}$ refers to the transformed zero levelset² of the image being aligned and $\widehat{N}$ is the outward unit normal at each point $\widehat{P}$ on the surface. The detailed derivations and final expressions that can be implemented for equation 10 are provided in equation 30 derived in section 5.5.

Based on the experience of clinical experts we chose a set of ten training images from a total of thirty datasets. Out of the ten training images we used one image as a reference image and aligned the rest using the gradient descent formulation of equation 9 and gradient expressions of equation 10 separately for each region LV, RV and Epi.

Figure 1, shows a subset of the training images for Epicardium. The binary black and white shape is the reference image. Each small image represents a 2D slice through the 3D volume which helps to better visualize the images. The red curve is the zero levelset of the corresponding signed distance function of the reference image. The other colored curves represent the boundaries of five other training images before alignment. We can clearly see that these images are not exactly overlapping indicating the pose differences in the dataset. Figure 2 shows the same set of images after the alignment process is finished. Again we are showing 2D slices through the 3D volume. The final shapes are much closer to the reference image but clearly differences remain. These differences represent the shape variability in the dataset which we intend to capture through the PCA process and use as prior information in our segmentation models. This alignment process is run separately for each region namely, Epicardium, Left and Right Ventricles.

Figure 1.

Set of Epicardium training images before alignment. The black and white image is the reference image with the red curve being the zero levelset of its corresponding signed distance function representation. Different color boundaries are other images that are aligned to this reference image. 2D slices of a 3D volume are shown for better visualization

Figure 2.

Set of Epicardium training images after alignment. The amount of overlap increases after the alignment process. The differences that still remain represent the shape variability we wish to capture through the PCA technique. 2D slices of a 3D volume are shown for better visualization

Using these aligned training image SDF’s we use PCA to generate shape priors as described in the following section.

4. Shape Priors

After the Image Alignment phase, we use the aligned signed distance representation of training images for each region, LV, RV and Epi, for running a PCA algorithm (Leventon et al. 2000; Tsai et al. 2003; Vikram et al. 2010). The boundaries of each of the n aligned training shapes are embedded as the zero level set of n separate signed distance functions {Ψ₁, Ψ₂,, Ψ_n} with negative distances assigned to inside and positive distances assigned to the outside of the object. We compute the mean level set, Ψ_mean, as the average of the n aligned level set representations, ${\Psi }_{\text{mean}}(\mathbf{x})=\frac{1}{n}{\sum }_{i=1}^n{\Psi }_i(\mathbf{x})$. To extract the shape variability, the mean shape, Ψ_mean, is subtracted from each of the aligned signed distance functions, Ψ_i, to create n mean-offset functions, ΔΨ₁,ΔΨ₂,⋯,ΔΨ_n. These mean-offset functions are then used to run PCA procedure to extract the shape priors which represent the shape variabilities.

We form n column vectors, $\Delta {\overrightarrow{\Psi }}_i$, consisting of N samples of each ΔΨ_i. The most common approach is to vectorize the N₁×N₂×N₃ grid of the training images into a large vector of N = N₁×N₂×N₃ samples in row major order by looping over all slices. Next we define the shape variability matrix as:

$$S=\left[\Delta {\overrightarrow{\Psi }}_1,\Delta {\overrightarrow{\Psi }}_2,\cdots ,\Delta {\overrightarrow{\Psi }}_n\right]$$ (11)

We then employ eigen-value decomposition as follows:

$$\frac{1}{n}{\mathit{SS}}^T=U\Sigma U^T$$ (12)

where $U=[{\overrightarrow{u}}_1,{\overrightarrow{u}}_2,\cdots ,{\overrightarrow{u}}_n]$ is a N×n matrix whose columns represent the n modes of variations in the shape and Σ is a n×n diagonal matrix whose diagonal elements represent the corresponding non-zero eigen-values or singular values. The N values of the i_th column of U, ${\overrightarrow{u}}_i$, are arranged back into N₁×N₂×N₃ grid by undoing the earlier vectorization operation to yield u_i(x), the i_th principal mode of variation which we also refer to as eigen-shape. Based on this PCA procedure we can extract maximum n eigen-shapes, u₁(x),u₂(x),⋯,u_n(x).

Since we are dealing with 3D images, the dimensions of the matrix $\frac{1}{n}{\mathit{SS}}^T$, N×N are very large. Running singular value decomposition on such a large matrix is very computationally expensive and also requires large amount of computer resources. In our implementation we compute the eigen-vectors and eigen values of $\frac{1}{n}{\mathit{SS}}^T$ from a much smaller n×n matrix given by:

$$W=\frac{1}{n}{S}^{T}S$$ (13)

If d is an eigen-vector of W with eigen-value λ then Sd is an eigen-vector of $\frac{1}{n}{\mathit{SS}}^T$ with eigen-value λ (Leveton 2000).

Using the mean level set function and selecting k ≤ n eigen-shapes we introduce a new level set function expressed as a linear combination of mean and k eigen shapes as follows:

$$\Psi (w,\mathbf{x})={\Psi }_{\text{mean}}(\mathbf{x})+\sum _{i=1}^k{w}_i{u}_i(\mathbf{x})={\Psi }_{\text{mean}}(\mathbf{x})+w\cdot U_k(\mathbf{x})$$ (14)

where w = [w₁,w₂,⋯,w_k] are the weights associated with the k eigen-shapes U_k(x) = [u₁(x),u₂(x),⋯,u_k(x)] with corresponding variances, $\{{\sigma }_1^2,{\sigma }_2^2,\cdots ,{\sigma }_k^2\}$ given by the eigen-values calculated from PCA procedure. Equation 14 gives a finite representation of a level set function whose zero level set represents the boundary of the evolving shape. While the PCA analysis behind equation 14 was based on SDF’s, it will not, in general, yield a strict SDF but only an approximation³.

As an illustration of how our shape prior model captures the shape variability from the aligned training images, Figure 3 shows how the mean RV shape changes by adding weighted combination of eigen-shapes to the mean. Each small image is a 2D slice through a 3D volume. The white pixels represent the inside of the mean RV shape while black pixels represent outside. The red curve which is the zero levelset of the mean, shows the boundary of the RV shape without any contribution from the eigen-shapes. The blue curve shows how the shape changes when we add σ₁ times the first principal mode of RV variation while magenta curve shows the shape with σ₁ times the first principal mode subtracted from the mean. We can clearly see how this model captures the shape variability. The RV shape generally varies a lot from patient to patient in terms of both size and orientation. The magenta curve particularly highlights the large variation in shape captured by the model.

Figure 3.

Illustration of how the shape prior model captures the shape variability in case of RV. The red curve is the mean RV shape boundary and bright pixels show the interior for easier visualization. Blue curve shows Ψ_mean + 1σ₁u₁ while magenta curve is Ψ_mean − 1σ₁u₁. 2D slices through the 3D volume are shown here

Similarly, figure 4 shows the shape variability of Epicardium. In this blue and magneta curves show σ₂ times the second mode of variation being added and subtracted from the mean respectively. Again we can see the shape variability especially at the base slices where the shape of the epicardium changes dramatically.

Figure 4.

Illustration of how the shape prior model captures the shape variability in case of Epicardium. The red curve is the mean Epicardium shape boundary and bright pixels show the interior for easier visualization. Blue curve shows Ψ_mean + 1σ₂u₂ while magenta curve is Ψ_mean − 1σ₂u₂. 2D slices through the 3D volume are shown here

The segmentation also need to accommodate the variations shape due to pose differences. To be able to handle pose variations p we modify our shape prior model in equation 14 as follows:

$$\widehat{\Psi }(w,\widehat{\mathbf{x}})={\widehat{\Psi }}_{\text{mean}}(\widehat{\mathbf{x}})+\sum _{i=1}^k{w}_i{\widehat{u}}_i(\widehat{\mathbf{x}})={\widehat{\Psi }}_{\text{mean}}(\widehat{\mathbf{x}})+w\cdot {\widehat{U}}_k(\widehat{\mathbf{x}})$$ (15)

where $\widehat{\mathbf{x}}$ is related to x by equation 6.

$$\widehat{\mathbf{x}}(\mathbf{x},\mathbf{p})=\mathit{hR}(v)(\mathbf{x}-O)+(T+O)$$

We will use the zero level set of $\widehat{\Psi }(w,\widehat{\mathbf{x}})$ as the new representation of our shape. By varying the shape weights as well as pose parameters, we can capture a broad class of shape variations. Figure 5 shows an overview of the entire training process used to obtain the shape priors and summarizes the notation used at each step.

Figure 5.

Overview of the training process and notation used for elements of the shape prior model.

Since our ultimate goal is myocardial segmentation, we combine 3D shape prior models for LV, RV and Epi which implictly divides the image into 4 regions as myocardium can now be defined as the region bounded by LV, RV, and Epi and the Background (BG) as the region outside Epi shape. We make use of the 3 shape prior models simultaneously instead of individually. This formulation allows us to impose high level constraints to customize the model for the challenging task of myocardial segmentation.

5. Image modeling

As described previously in section 1.1, there are two broad class of segmentation models used in variational computer vision, edge based and region based. We use region based models where the evolution of the segmenting curve depends upon the pixel intensities within entire regions. That is, region-based models regard an image as the composition of a finite number of regions and rely on regional statistics for segmentation. The statistics of entire regions (such as sample mean and variance) are used to direct the movement of the curve toward the boundaries of the image. This is in contrast to edge-based models where the evolution of the curve depends strictly on nearby pixel intensities (i.e., gradient information). As a result, region-based models are more global than edge-based models. Furthermore, because of the global nature of region-based models, these models do not require the use of inflationary terms commonly employed by edge-based techniques to drive the curve toward image boundaries. Region-based models are also more robust to noise since they do not employ gradient operators, which are inherently sensitive to noise, to explicitly detect the location of edges.

5.1. Region-Based model

In cardiac CT imagery, the interior of Epicardium consists of RV, LV and Myocardium which is the region of Epicardium excluding RV and LV. Additionally, everything excluding Epicardium i.e. the background consists of various anatomical structures. We have developed a region based model defined using an energy functional consisting of two parts:

$$E_{\text{coupled}}=\text{Interior Model}+\text{Background Model}$$

The Interior Model is a Chan-Vese like model that would appear as follows:

$$E_{\text{coupled}}=w_{\mathit{LV}}{\int }_{\mathit{LV}}{(I-{\mu }_{\mathit{LV}})}^{2}d\widehat{\mathbf{x}}+w_{\mathit{RV}}{\int }_{\mathit{RV}}{(I-{\mu }_{\mathit{RV}})}^{2}d\widehat{\mathbf{x}}+w_{\mathit{Myo}}{\int }_{\mathit{Myo}}{(I-{\mu }_{\mathit{Myo}})}^{2}d\widehat{\mathbf{x}}+\text{Background Model}$$ (16)

Due to the complexity of the background we treat the Background Model separately as described in the next section.

Note that in the above formulation, whenever the three regions plus the background (BG) are disjoint then these data fidelity terms function the same way as in the Chan-Vese formulation. This occurs when there is no overlap between any of the surface pairs. However, overlap could occur in which case these regions do not remain automatically disjoint. In such a scenario we have to more precisely define the domains of each potentially overlapping region. Accordingly, we define the region domains for LV and RV explicitly as the interior of their respective shape models. The background (BG) is defined explictly as the exterior of Epicardium shape model. Myocardium (Myo) region domain, is defined using all three shape models implicitly as the region inside Epicardium shape and outside both LV and RV shapes. Based on these definitions whenever overlap occurs between the LV and RV the data fidelity penalty gets double counted along the overlap through the first two terms in equation 16. A similar double counting would occur between LV/BG, RV/BG pairs using the background model described in section 5.2 This acts as natural in-built penalty for overlap and thus discourages overlapping shapes. This definition of potentially overlapping region domains introduces an extra level of coupling between the regions and making equation 16 different from standard Chan-Vese energy. In section 5.3, we introduce an additional explicit overlap penalty to ensure the absence of overlap in the final segmented result.⁴

In equation 16, μ_LV, μ_RV, μ_Myo are the respective region mean intensities and w_LV, w_RV, and w_Myo are a set of weighting factors for each region. Using these weighting factors we can bias the evolving shapes to favor the expansion of a particular region. These weighing factors are real numbers and the values relative to each other lead to differing penalties on regions. For example, if w_Myo is more than w_RV and their respective means are same then a voxel being in Myocardium region would increase the energy more than being in RV region. So using these weights we can bias the expansion of one region compared to others. These weights could be subtly tuned to the specific parameters of a given CT system and contrast protocol. For the system and protocol used in our study we have found the set of weights as shown in section 6.2.

In the standard Chan-Vese model there is no restriction on the ordering of the region means. Our model differs in this respect by introducing a strict ordering of the means of each region as shown in equation 17:

$${\mu }_{\mathit{Myo}}\le {\mu }_{\mathit{RV}}\le {\mu }_{\mathit{LV}}$$ (17)

This enhancement allows added robustness against initial placement of contours. As an example, if the LV is initialized in the vicinity of the RV in the actual image the segmentation may drift towards RV boundaries. Enforcing this ordering prevents this drift and helps drive the contours towards the correct boundaries even if initialized poorly. While segmenting a dataset, we check and enforce this ordering at each gradient descent iteration. For example if the Myo mean exceeds the RV mean then we enforce the mean ordering constrain by setting both μ_Myo and μ_RV to be equal to the weighted average of their means.

5.2. Background model

In cardiac imaging the tissue surrounding the epicardium is very complex. There are different types of tissues, muscles, blood pools, calcified vessels and other organs. Each of these regions exhibit different intensity ranges. This leads to the presence of both bright and dark regions. Some regions can be even brighter than the LV. This makes it difficult to model the whole background as a single region with a mean intensity. The shape prior model tries to capture these variations and effectively bleeds the myocardium into the background region.

To overcome these difficulties, we model the background as a combination of two clusters, one darker than myocardium, and other brighter than LV. This leads to the following modification of energy functional:

$$E_{\text{coupled}}=\text{Interior Model}+w_{\mathit{BG}}{\int }_{\mathit{BG}}\min \left({(I-c_{\mathit{lo}})}^2,{(I-c_{\mathit{hi}})}^2\right)d\widehat{\mathbf{x}}$$ (18)

The above formulation is a standard two level c-means clustering. By iterating over the background pixels, the background region is divided into two clusters where c_lo and c_hi are the cluster centers of the low and high intensity regions respectively.

The overall energy functional now becomes:

$$E_{\text{coupled}}=w_{\mathit{LV}}{\int }_{\mathit{LV}}{(I-{\mu }_{\mathit{LV}})}^{2}d\widehat{\mathbf{x}}+w_{\mathit{RV}}{\int }_{\mathit{RV}}{(I-{\mu }_{\mathit{RV}})}^{2}d\mathbf{x}+w_{\mathit{Myo}}{\int }_{\mathit{Myo}}{(I-{\mu }_{\mathit{Myo}})}^{2}d\widehat{\mathbf{x}}+w_{\mathit{BG}}{\int }_{\mathit{BG}}\min \left({(I-c_{\mathit{lo}})}^2,{(I-c_{\mathit{hi}})}^2\right)d\widehat{\mathbf{x}}$$ (19)

Accordingly we modify the mean intensity ordering given in equation 17:

$$c_{\mathit{lo}}\le {\mu }_{\mathit{Myo}}\le {\mu }_{\mathit{RV}}\le {\mu }_{\mathit{LV}}\le c_{\mathit{hi}}$$ (20)

We start the c-means clustering algorithm by setting c_lo equal to μ_Myo and c_hi equal to μ_LV and then run the clustering iteration. In cases where the constraints in eq. 20 are violated at the end of clustering we again enforce the constrain by setting c_lo or c_hi equal to Myo or LV before running gradient descent iteration. This ensures segmentation better than one achieved using a single mean for background region without mean ordering constraints.

5.3. Overlap Penalty

As described in section 5.1, the basic energy term has an in-built overlap penalty which discourages overlap between regions. On top of that we introduce a separate overlap penalty term to further bias the different regions to not overlap. The overlap penalty is purely regularization term which induces a repulsive force whenever regions overlap. This extra overlap penalty term further couples the separate shape priors into a single higher level shape model.

$$E_{\text{overlap}}={\beta }_{\mathit{LR}}\left({\int }_{\mathit{LV}\cap \mathit{RV}}d\widehat{\mathbf{x}}\right)+{\beta }_{\mathit{L}}\left({\int }_{\mathit{LV}\cap \mathit{BG}}d\widehat{\mathbf{x}}\right)+{\beta }_{\mathit{R}}\left({\int }_{\mathit{RV}\cap \mathit{BG}}d\widehat{\mathbf{x}}\right)$$ (21)

This overlap penalty energy term penalizes the amount of intersection between LV and RV, LV and BG, and RV and BG with tunable penalty factors. This formulation is made possible by our framework to use three shape priors simultaneously rather than individually.

5.4. Parameter Optimization

Based on the energy formulations described in the image modeling section, the task now is to find an optimal set of pose and shape parameters which minimize the energy functionals designed to segment the four regions. This goal is achieved using parameter optimization procedure which involves running a gradient descent algorithm and updating the pose(p) and shape(w) parameters for each region at every iteration which can be formulated as:

$$p_i^{(n+1)}=p_i^n-\delta t\cdot {\nabla }_{p_i}{E}_{\text{total}}$$ (22)

$$w_i^{(n+1)}=w_i^n-\delta t\cdot {\nabla }_{w_i}{E}_{\text{total}}$$ (23)

where we take the gradients of the total energy which includes the coupled energy with binary background (E_coupled) and overlap penalty energy (E_overlap), w.r.t. each pose and shape parameter. We choose a time step such that segmenting curves don’t move by more than one pixel at every step. Additionally, we apply the rotation around the centroid of the evolving shape so as to orthogonalize the translation and scale as much as possible. Since we also update the shape weights every iteration, we update the centroid and recenter the pose parameters every iterations.

Since our entire formulation is in 3D and the energy integrals are volume integrals, the gradient expressions involve surface integrals for each shape respectively. At every iteration we recalculate the updated PCA model with the latest pose and shape parameters and extract the zero crossings of the shapes levelset surfaces. We then calculate the required energy gradients as described in the next sections and update the pose and shape parameters using equation 22 and equation 23 respectively.

5.5. Energy gradients

In this section we derive the expressions for the gradients required for the gradient descent procedure with respect to both pose and shape parameters. In our approach we segment the cardiac images implictly by varying the pose and shape parameters and represent the evolving shape boundary as $\widehat{S}$ which is the zero levelset surface of shape model $\widehat{\Psi }$ given by equation 15. Let $\widehat{P}$ and P represent a point on the zero levelset surface of the pose transformed and un-transformed shape models given by equation 15 and equation 14 respectively. $\widehat{P}$ and P are related to each other by the relation:

$$\widehat{P}(w,\mathbf{p})=\mathit{hR}(v)(P(w)-O)+(T+O)$$

Similar to the discussion in image alignment section 3 and based on the derivations in appendix A, the expressions for individual derivatives of total energy E w.r.t. translation (T), scale (h) and rotation (v) are given by the following equations:

$${\nabla }_{T}E={\int }_{\widehat{P}}f{\left(\frac{\partial \widehat{P}}{\partial T}\right)}^T\widehat{N}d\widehat{\mathbf{S}},{\nabla }_{h}E={\int }_{\widehat{P}}f{\left(\frac{\partial \widehat{P}}{\partial h}\right)}^T\widehat{N}d\widehat{\mathbf{S}},{\nabla }_{v}E={\int }_{\widehat{P}}f{\left(\frac{\partial \widehat{P}}{\partial v}\right)}^T\widehat{N}d\widehat{\mathbf{S}}$$ (24)

The expressions for the scalar force term f for each region are derived in appendix B. Here we derive the gradient expressions ${\left(\frac{\partial \widehat{P}}{\partial \lambda }\right)}^T\widehat{N}$ where λ could be T, h or v.

5.5.1. Rotation Gradient

We have chosen to use the Euler-Rodrigues representation for rotation due to its compact nature. Gallego and Yezzi (2015) provide a number of derivations related to this particular representation of space rotations. In particular we make use of the following result to derive the expressions for gradient w.r.t. to rotation:

$$\frac{\partial R(v)u}{\partial v}=-R[u\times ]\frac{{\mathit{vv}}^T+\left(R^T-I\right)\left[v\times \right]}{{\Vert v\Vert }^2}=-[\mathit{Ru}\times ]\frac{{\mathit{vv}}^T+\left(I-R\right)\left[v\times \right]}{{\Vert v\Vert }^2}$$ (25)

where $u\in {\mathbb{R}}^3$ is a vector independent of v.

$$9{\left(\frac{\partial \widehat{P}}{\partial v}\right)}^T\widehat{N}={\left(\frac{\partial }{\partial v}(\mathit{hR}(P-O)+(T+O))\right)}^T\widehat{N}$$ (26a)

$$={\left(h\frac{\partial }{\partial v}R(P-O)\right)}^T\widehat{N}$$ (26b)

$$={\left(-h[R(P-O)\times ]\frac{{\mathit{vv}}^T+(I-R)[v\times ]}{{\Vert v\Vert }^2}\right)}^T\widehat{N}$$ (26c)

$$={\left(-\left[\left(\widehat{P}-T-O\right)\times \right]\frac{{\mathit{vv}}^T+(I-R)[v\times ]}{{\Vert v\Vert }^2}\right)}^T\widehat{N}$$ (26d)

$$=\left(\frac{{\mathit{vv}}^T+[v\times ](R^T-I)}{{\Vert v\Vert }^2}\right)\underset{\widehat{z}}{\underbrace{\left(\left(\widehat{P}-\widehat{O}\right)\times \widehat{N}\right)}}$$ (26e)

$$=\frac{{\mathit{vv}}^T\widehat{z}+(v)\times (R^T\widehat{z})-v\times \widehat{z}}{{\Vert v\Vert }^2}=\frac{{\mathit{vv}}^T\widehat{z}+(R^T\mathit{Rv})\times (R^T\widehat{z})-v\times \widehat{z}}{{\Vert v\Vert }^2}$$ (26f)

$$=\frac{{\mathit{vv}}^T\widehat{z}+R^T((\mathit{Rv})\times \widehat{z})-v\times \widehat{z}}{{\Vert v\Vert }^2}=\frac{{\mathit{vv}}^T\widehat{z}+R^T(v\times \widehat{z})-v\times \widehat{z}}{{\Vert v\Vert }^2}$$ (26g)

$$=\frac{{\mathit{vv}}^T\widehat{z}+(R^T-I)(v\times \widehat{z})}{{\Vert v\Vert }^2}=\frac{{\mathit{vv}}^T+(R^T-I)[v\times ]}{{\Vert v\Vert }^2}(\widehat{P}-\widehat{O})\times \widehat{N}$$ (26h)

$$=\underset{M_v}{\underbrace{\frac{{\mathit{vv}}^T+(R^T-I)[v\times ]}{{\Vert v\Vert }^2}}}\underset{\widehat{Q}}{\underbrace{(\widehat{P}-\widehat{O})}}\times \widehat{N}=M_v\left(\widehat{Q}\times \widehat{N}\right)$$ (26i)

Here in going from equation 26b to equation 26c we have made use of the derivative of rotation result provided in equation 25.

5.5.2. Translation gradient

$${\left(\frac{\partial \widehat{P}}{\partial T}\right)}^T\widehat{N}={\left(\frac{\partial }{\partial T}(\mathit{hR}(P-O)+(T+O))\right)}^T\widehat{N}$$ (27a)

$$={\left(\frac{\partial T}{\partial T}\right)}^T\widehat{N}=\widehat{N}$$ (27b)

Going from equation 27a to equation 27b, the only term dependent on the translation is T itself.

5.5.3. Scale gradient

$${\left(\frac{\partial \widehat{P}}{\partial h}\right)}^T\widehat{N}={\left(\frac{\partial }{\partial h}\left(\mathit{hR}(P-O)+\underset{\widehat{O}}{\underbrace{(T+O)}}\right)\right)}^T\widehat{N}$$ (28a)

$$={\left(R(P-O))\right)}^T\widehat{N}={\left(R\left(\frac{1}{h}{R}^T\left(\widehat{P}-(T+O)\right)\right)\right)}^T\widehat{N}$$ (28b)

$$=\frac{1}{h}\underset{\widehat{Q}}{\underbrace{{\left(\widehat{P}-\widehat{O}\right)}^T}}\widehat{N}$$ (28c)

$$=\frac{1}{h}\widehat{Q}\cdot \widehat{N}$$ (28d)

In the above set of equations we have used the inverse transform relation equation 7 and the fact that the rotation matrix R is orthogonal such that R^TR = RR^T = I.

5.5.4. Eigen-shape weights gradient

If $\widehat{P}(w)$ is an evolving point on the evolving zero levelset of $\widehat{\Psi }(w,\widehat{\mathbf{x}})={\widehat{\Psi }}_{\text{mean}}(\widehat{\mathbf{x}})+w\cdot {\widehat{U}}_k(\widehat{\mathbf{x}})$ (where ${\widehat{\Psi }}_{\text{mean}}(\widehat{\mathbf{x}})\doteq {\Psi }_{\text{mean}}(\mathbf{x})$ and ${\widehat{U}}_k(\widehat{\mathbf{x}})\doteq U_k(\mathbf{x})$), then we may compute the total derivative in w

$$\frac{\partial }{\partial w}\left(0=\widehat{\Psi }(w,\widehat{P}(w))={\widehat{\Psi }}_{\text{mean}}\left(\widehat{P}(w)\right)+w\cdot {\widehat{U}}_k\left(\widehat{P}(w)\right)\right)$$

$$0=\stackrel{\text{chain rule}}{\overbrace{\underset{\nabla \widehat{\Psi }}{\underbrace{\frac{\partial \widehat{\Psi }\text{(}w\mathrm{,}\widehat{P}\text{)}}{\partial \widehat{x}}}}\frac{\partial \widehat{P}}{\partial w}}}+\stackrel{\text{direct partial}}{\overbrace{\frac{\partial \widehat{\Psi }}{\partial w}\text{(}w\mathrm{,}\widehat{P}\text{)}}}={\left(\frac{\partial \widehat{P}}{\partial w}\right)}^T\underset{\nabla \widehat{\Psi }}{\underbrace{\widehat{N}\text{(}\widehat{P}\text{)}\Vert \nabla \widehat{\Psi }\text{(}w\mathrm{,}\widehat{P}\text{)}\Vert }}+\underset{\frac{\partial \widehat{\Psi }}{\partial w}}{\underbrace{{\widehat{U}}_k\text{(}\widehat{P}\text{(}w\text{))}}}$$

which consists of the direct partial derivative with respect to w together with a chain rule term which links the spatial movement of the zero level set with the partial derivatives in space.⁵ Finally from this equation we obtain:

$${\left(\frac{\partial \widehat{P}}{\partial w}\right)}^T\widehat{N}(\widehat{P})=-\frac{{\widehat{U}}_k(\widehat{P})}{\Vert \nabla \widehat{\Psi }(w,\widehat{P})\Vert }=-\frac{U_k(P)}{\Vert \nabla \widehat{\Psi }(w,\widehat{P})\Vert }$$ (29)

At the end of equation 29, we use the last equality for computational efficiency such that we do not have to transform all the principal components.

5.5.5. Full gradient expression

In this section we just present the summary of the last sub-sections showing the consolidated gradient expressions that are eventually used in the gradient descent algorithm’s update equations presented in section 5.4. If we denote the full vector of pose and shape parameters by λ then we may write more compactly:

$$\lambda =\left[\begin{array}{c}\hfill v\hfill \\ \hfill h\hfill \\ \hfill T\hfill \\ \hfill w\hfill \end{array}\right],\frac{\partial \widehat{P}}{\partial \lambda }=\left[\frac{\partial \widehat{P}}{\partial v}\frac{\partial \widehat{P}}{\partial h}\frac{\partial \widehat{P}}{\partial T}\frac{\partial \widehat{P}}{\partial w}\right],{\left(\frac{\partial \widehat{P}}{\partial \lambda }\right)}^T\widehat{N}=\left[\begin{array}{c}\hfill M_v\left(\widehat{Q}\times \widehat{N}\right)\hfill \\ \hfill \frac{1}{h}\left(\widehat{Q}\cdot \widehat{N}\right)\hfill \\ \hfill \widehat{N}\hfill \\ \hfill -\frac{U_k(P)}{\Vert \nabla \widehat{\Psi }(w,\widehat{P})\Vert }\hfill \end{array}\right]$$

If we denote the total energy i.e. the sum of E_coupled and E_overlap as simply E then we may write:

$${\nabla }_{\lambda }E=\left[\begin{array}{c}\hfill \frac{\partial E}{\partial v}\hfill \\ \hfill \frac{\partial E}{\partial h}\hfill \\ \hfill \frac{\partial E}{\partial T}\hfill \\ \hfill \frac{\partial E}{\partial w}\hfill \end{array}\right]=\left[\begin{array}{c}\hfill {\int }_{\widehat{P}}f{\left(\frac{\partial \widehat{P}}{\partial v}\right)}^T\widehat{N}d\widehat{\mathbf{S}}\hfill \\ \hfill {\int }_{\widehat{P}}f{\left(\frac{\partial \widehat{P}}{\partial h}\right)}^T\widehat{N}d\widehat{\mathbf{S}}\hfill \\ \hfill {\int }_{\widehat{P}}f{\left(\frac{\partial \widehat{P}}{\partial T}\right)}^T\widehat{N}d\widehat{\mathbf{S}}\hfill \\ \hfill {\int }_{\widehat{P}}f{\left(\frac{\partial \widehat{P}}{\partial w}\right)}^T\widehat{N}d\widehat{\mathbf{S}}\hfill \end{array}\right]=\left[\begin{array}{c}\hfill M_v{\int }_{\widehat{P}}f\left(\widehat{Q}\times \widehat{N}\right)d\widehat{\mathbf{S}}\hfill \\ \hfill \frac{1}{h}{\int }_{\widehat{P}}f\left(\widehat{Q}\cdot \widehat{N}\right)d\widehat{\mathbf{S}}\hfill \\ \hfill {\int }_{\widehat{P}}f\widehat{N}d\widehat{\mathbf{S}}\hfill \\ \hfill -{\int }_{\widehat{P}}f\frac{U_k(P)}{\Vert \nabla \widehat{\Psi }(w,\widehat{P})\Vert }d\widehat{\mathbf{S}}\hfill \end{array}\right]$$ (30)

Since we are optimizing a set of shape and pose parameters for three shapes, LV, RV, and Epi, we would have three sets of expressions given by equation 30, one for each shape. For example, for LV’s scale derivative expression, we would have a set of zero crossing points $\widehat{P}$ representing the LV zero levelset surface with corresponding unit normals $\widehat{N}$ at each point. The force term f would be given by equation B5a. Scale h would be the current LV scale and $\widehat{Q}$ would be calculated at each point for LV. Two similar instances of scale derivative would in turn apply to Epi and RV.

5.5.6. Momentum accelerated gradient descent

Standard gradient descent algorithms can generally suffer from slow convergence or can get stuck in a shallow valleys leading to sub-optimal results. There are several methods that help ameliorate these drawback (Sebastian 2017). Most common method employed to overcome slow convergence and oscillations in standard gradient descent is called momentum (Qian 1999) based accelerated gradient descent. In this scheme a fraction of previous update vector is added to the current update vector. This approach allows the descent procedure to pick up “speed” and helps overcome “shallow valleys” and approach better local minimums. Essentially, momentum increases for parameters whose gradient points in the same directions and reduces updates for parameters whose gradient changes directions. Our shape based approach can be combined with this method to further boost the gradient descent procedure in natural dynamic way and leads to better segmentation results.

6. Experimental results

We applied our integrated anatomical model for cardiac segmentation on 30 cardiac tomography angiography (CTA) patient studies obtained at Nuclear Cardiology R&D Laboratory at Emory University School of Medicine. Each of the studies were hand segmented by clinical experts to obtain binary masks as gold standard segmentations for each anatomical region of interest. We used 10 out of the 30 studies as training images to derive the shape models.

6.1. Choosing number of principal components

As discussed in section 4, each of the training datasets produces a principal component (eigen shape) and during the automatic segmentation phase we can represent the model using k of those eigen shapes. As such, there is no automatic procedure for choosing an optimal k, instead it is a trial and error process. We applied our PCA model to the original binary hand segmented images. We conducted experiments by using different numbers of eigen shapes, and compared the accuracy of the segmentations obtained using a statistical technique of binary classification (FMe 2017) called F-Measure. Figure (6) shows the plot in increasing order of accuracy for the Epicardium region for different numbers of modes. We observed a prominent spike in performance going from 3 to 5 modes but a rather smaller increases for using additional eigen shapes. The left and right ventricle segmentations also exhibited a similar trend. Such an analysis would be beneficial for a future commercial application where hundreds of training images might be used with limited computational resources.

Figure 6.

Accuracy for the Epicardium region for different numbers of modes.

6.2. Automatic segmentation results

Figure 8 shows a sample CT image along with the shape prior model initialized with mean shape for each of the three regions. The red, yellow and blue curves are the zero level sets of Epi, RV and LV respectively. Each small image shows a 2D slice through the 3D volume where the curves are actually surfaces. The region enclosed by these curves is Myo and everything outside of the red (Epi) curve is BG.

Figure 8:

Shape prior model initialized with mean shape for each of the three regions. The red, yellow and blue curves are the zero level sets of Epi, RV and LV respectively. Each small image shows a 2D slice through the 3D volume

As illustrated in figure 7, the segmentation process starts with this mean shape model where the eigen-shape weights are initially all set to zero. The pose parameters are also initialized to unity (translation and rotation are 0 but scale is 1) i.e. no transformations are applied yet to the model. The segmentation then proceeds with the momentum based gradient descent algorithm using the mathematical formulation derived in section 5. We run several iterations until convergence when the energy stops decreasing substantially. In each iteration, the shape and pose parameters are updated to yield progressively better segmentation.

Figure 7.

Flowchart for the segmentation process.

Figure 9 shows the same test case which was initialized with the shape prior mean shape model after the gradient descent procedure is finished. During this process an optimized set of pose and shape parameters are obtained for each of the three regions. The difference in the starting and final shape for RV and Epi is especially stark. The RV region initially partially overlaps with myocardium and is at the edge of LV region. The gradient descent process pushes it to its own region leading to a much better final myocardial segmentation. We used w_LV = 0.5, w_RV = 0.5, w_Myo = 1.5 and w_BG = 1 for all experiments. As explained in section 5.1, tuning these weights can lead to more expansion of some regions compared to others. Having higher weights for Myo compared to LV and RV leads to expansion of these regions compared to Myo. This is in line with our knowledge that most of the region inside Epi should be occupied by LV and RV compared to Myo. With these weights we can overcome the lack of contrast especially between RV and Myo regions.

Figure 9:

Myocardial segmentation after gradient descent is finished. The red, yellow and blue curves are the zero level sets of Epi, RV and LV respectively. Each small image shows a 2D slice through the 3D volume

Figure 10 shows the region-based model that corresponds to the final segmentation results in the previous example. The darkest and brightest region (visible in some 2D slices) outside the epicardium (red curve) correspond to the two clusters of the background region. Without this binary background formulation the Epicardium shape gets attracted towards these bright regions, generally calcified vessels, especially if they are located very close to epicardium. This problem is avoided due to our binary background and means ordering formulation.

Figure 10:

The model corresponding to the final segmentation in example test case. The darkest and brightest regions are the two background clusters. In the interior of Epicardium (red curve), LV is the brightest region followed by RV and finally myocardium

Finally, figure 11 shows the 3D surfaces representing the manual segmentation generated by clinicians compared to the final result of the optimized PCA model generated by our algorithm. Compared to manual segmentation, our model is also much smoother.

Figure 11.

3D rendering of manual(left) vs automatic(right) segmentation

After running our automatic segmentation algorithm, we convert the final segmenting curves of each shape back to binary masks, again one for each region. Clinical experts at Emory university calculated and compared the results with manual tracings and found an error (mean±std) of 1.46±0.41mm for LV, 2.06±0.65mm for RV, and 2.88±0.59mm for Epi. The results for LV are most accurate as this region usually has the best contrast. The Epi and RV results are slightly lower than LV. Currently, there are very few, if any, tools that can perform fully automatic cardiac segmentation available in the market. Hence it is difficult to compare the improvements to any industry standard.

As mentioned in section 1.1, there have been many different segmentation methods reported in the literature. There are several issues that prevent a direct comparison of our algorithm with these other state-of-the-art methods. Firstly, all the methods reviewed in this paper reported segmentation accuracy results in trans-axial orientation of the data. Since our imaging experts are trained to recognize cardiac anatomy based on the hearts natural axes (short, vertical and horizontal long axes) they are more confident in their manual tracings done in short axis orientation compared to trans-axial orientation. Thus biventricular short axis segmentation leads to more accurate manual segmentation and reduced inter and intra observer variability of the boundaries. In practically all of cardiac imaging this natural axes approach has been adopted as the gold standard for manual segmentation of myocardial boundaries. Consequently, we have developed our algorithms based on short-axis orientation of all our datasets. Hence a direct comparison with techniques implemented in trans-axial orientation would generate a larger error due to the gold standard rather than the automatic segmentation techniques and thus not a fair comparison. Secondly, the method used to compare the accuracy of automated segmentation to manual tracings is not clear in some of the reviewed papers. This further makes it difficult to compare the results directly. Finally, comparison between techniques is further complicated by some techniques reporting their results only for the LV, some treat LV and RV myocardium separately while others treat them as a single component.

For these reasons, the best way to make a fair quantitative comparison between different myocardial segmentation algorithms is to use same datasets for testing as well as use the same segmentations as gold standard. To this end, we now use the same image datasets and manual segmentations standards to compare the automatically segmented results from our new algorithm to those using the variational region growing method presented in Zhu et al. (2013b).

In order to compare Zhu’s method and our new method, we used six out of the twelve pigs datasets as training data to generate our shape prior models for each shape. We then used our segmentation algorithm to generate automatic segmentation masks for all twelve datasets. Our clinical expert collaborators at Emory University then used the same error measurement algorithm to compare the results of both methods to the manual segmentations. Note that these manual segmentations were originally drawn from trans-axial slices since it was the only original data we had in common for comparisons between techniques.

Table 1, summarizes the mean errors and standard deviation for each shape. Note that our proposed algorithms shows a trend of smaller mean error compared to the variational region growing method for all three shapes. A one-tail paired t-test confirms the trend albeit not yet reaching statistical significance with p = .055 for the epicardial boundaries, p = .20 for the LV endocardium and p = .41 for the RV. Improvements in these results will require improvements in the reference gold standard by generating the manual segmentation from natural axis of the heart as well as averaging the results of multiple experts. In Zhu’s paper the authors showed better results compared to the localized PCA based method presented in Vikram et al. (2010, 2011). The authors also showed their results to be competitive with the methods of Ecabert et al. (2008) and Zheng et al. (2008). Now we have shown a trend of improved results over Zhu’s method.

Table 1.

Comparison between proposed method and the variational region growing method (Zhu et al.) on same twelve pigs datasets (mean mm ± std mm)

Method	LV	RV	Epi
Proposed method	1.38 ± 0.70	1.88 ± 0.93	1.63 ± 0.74
Zhu et al.	1.64 ± 0.63	1.97 ± 0.63	3.21 ± 2.84

Additionally, we ran our tests on a quad core 3.2 GHz Intel core i7 CPU with 16 GB RAM. Segmenting a new test case takes approximately 50 seconds on average. However, in our algorithms a lot of computation can be parallelized and the running time can be cut to about one third of current times.

7. Conclusion

We have presented a novel combination and extension of existing frameworks to automatically segment myocardial boundaries in cardiac CT images. Among their various applications, the proposed methodologies have been developed also in the context of a project for multimodality image fusion (Piccinelli et al. 2014). Function and anatomy are equally important in the assessment of Coronary Artery Disease. A powerful imaging strategy for a comprehensive evaluation of the patient status is to fuse a nuclear imaging test, i.e. single-photon emission computed tomography (SPECT) or positron emission tomography (PET), with an anatomical imaging test, i.e. a CTA. Currently, there are very few, if any, tools that can perform full automatic cardiac segmentation available in the market, which hampers the translation of a multimodality fusion approach to the clinical environment. The results obtained so far are particularly promising and a validation of a fully automated fusion procedure is underway to assess its feasibility and clinical reliability. The algorithmic framework can also be easily adapted to work with cardiac magnetic resonance imagery (MRI), which is increasingly becoming a prominent cardiac imaging modality.

Appendix A. Region Based Active Contours and Active Surfaces

For its various advantages, in our work we have used region-based active surfaces for the task of 3D Cardiac CT segmentation. Here we provide some basic concepts related to active contours/surfaces frequently used in our mathematical formulations. Figure A1 shows a typical active contour in 2D where the curve C divides an image in two regions inside (R_in) and outside (R_out). In 3D it would be a surface (S) which divides a volume into inside and outside. Suppose we devise some energy functional for a particular task as follows:

$$E={\int }_{R_{\text{in}}}{f}_{\text{in}}d\mathbf{x}$$ (A1)

We can modify equation A1 using the well known divergence theorem. Suppose ${\overrightarrow{F}}_{\text{in}}$ is a vector field such that $\nabla \cdot \overrightarrow{F}=f_{\text{in}}$, then equation A1 can be written as:

$$E={\int }_{R_{\text{in}}}{f}_{\text{in}}d\mathbf{x}={\int }_{R_{\mathit{in}}}(\nabla \cdot {\overrightarrow{F}}_{\text{in}})d\mathbf{x}={\int }_S{\overrightarrow{F}}_{\text{in}}\cdot N\mathit{dS}$$ (A2)

where the last equality follows by using the divergence theorem which relates the volume integral of divergence of flux $\overrightarrow{F}$ to an equivalent surface integral. In equation A2, N is the outward unit normal to the surface S (curve C in 2D) and dS is a unit surface area element. As shown in Yezzi et al. (2003), the derivative of energy w.r.t. a parameter λ is given by the following expression:

$$\frac{\partial E}{\partial \lambda }={\int }_S\nabla \cdot {\overrightarrow{F}}_{\text{in}}\left(\frac{\partial S}{\partial \lambda }\cdot N\right)\mathit{dS}$$ (A3)

Figure A1.

Typical active contour

With our assumption that $\nabla \cdot {\overrightarrow{F}}_{\text{in}}=f_{\text{in}}$ the last equation becomes:

$$\frac{\partial E}{\partial \lambda }={\int }_S{f}_{\text{in}}\left(\frac{\partial S}{\partial \lambda }\cdot N\right)\mathit{dS}$$ (A4)

Now if we are given an integral over an external region R_out as $E={\int }_{R_{\text{out}}}$ f_out dx, we can rewrite it as follows:

$$E={\int }_{\Omega }{f}_{\text{out}}d\mathbf{x}-{\int }_{R_{\text{in}}}{f}_{\text{out}}d\mathbf{x}$$ (A5)

where Ω is the complete image domain. Noting that the first term doesn’t depend on λ and using equations A1 through A4, the derivative of this energy w.r.t. parameter λ is given by the following equation:

$$\frac{\partial E}{\partial \lambda }=0-{\int }_S{f}_{\text{out}}\left(\frac{\partial S}{\partial \lambda }\cdot N\right)\mathit{dS}=-{\int }_S{f}_{\text{out}}\left(\frac{\partial S}{\partial \lambda }\cdot N\right)\mathit{dS}$$ (A6)

In our work we have energy functionals with integrals of both inside and outside regions of the form:

$$E={\int }_{R_{\text{in}}}{f}_{\text{in}}d\mathbf{x}+{\int }_{R_{\text{out}}}{f}_{\text{out}}d\mathbf{x}$$ (A7)

By using equation A4 and equation A6, the derivative w.r.t. λ of the previous energy is easily seen to be:

$$\frac{\partial E}{\partial \lambda }={\int }_S{f}_{\text{in}}\left(\frac{\partial S}{\partial \lambda }\cdot N\right)\mathit{dS}-{\int }_S{f}_{\text{out}}\left(\frac{\partial S}{\partial \lambda }\cdot N\right)\mathit{dS}$$ (A8a)

$$={\int }_S(f_{\text{in}}-f_{\text{out}})\frac{\partial S}{\partial \lambda }\cdot N\mathit{dS}$$ (A8b)

In equation A8, λ could be any parameter such as translation (T), scale (h), rotation (v) or shape weight (w). We have derived the partial derivatives of the surface $\frac{\partial S}{\partial \lambda }$ in the relevant places in the main text.

Although equation 8 in section 3 represents the alignment energy functional, it is more convenient to rewrite it as follows:

$$E_{\text{align}}(\text{Training Image})={\int }_{R_{\text{in}}}\left(1-{\chi }_{\mathit{ref}}\right)d\widehat{\mathbf{x}}+{\int }_{R_{\text{out}}}{\chi }_{\text{ref}}d\widehat{\mathbf{x}}$$ (A9)

where R_in and R_out are the inside and outside regions of the training image shape being aligned with respect to the reference image. In this alignment energy, f_in = 1 − χ_ref and f_out = χ_ref. Hence f_in − f_out = 1 − 2χ_ref which leads to the derivative expressions in equation 10.

Appendix B. Outward Normal Force For Segmentation Energy

In section 5, we described our image segmentation model consisting of an interior model, a background model and an overlap penalty term. While these models are more complicated than the image alignment model, they are nevertheless of the same form as equation A7 consisting of integrals over an interior and exterior region. Consequently, the derivatives of the segmentation energy w.r.t. the pose and shape parameters of each shape, namely, LV, RV and Epi, are of the same form as equation A8. Here we derive the expressions for f_in − f_out for each component of the segmentation energy for all three shapes.

Equation 16 and 18 defined the interior and background model respectively and the resulting region based energy functional, E_coupled, was defined as follows:

$$E_{\text{coupled}}({\widehat{S}}_{\mathit{LV}},{\widehat{S}}_{\mathit{RV}},{\widehat{S}}_{\mathit{Epi}})=w_{\mathit{LV}}{\int }_{\mathit{LV}}{(I-{\mu }_{\mathit{LV}})}^{2}d\widehat{\mathbf{x}}+w_{\mathit{RV}}{\int }_{\mathit{RV}}{(I-{\mu }_{\mathit{RV}})}^{2}d\widehat{\mathbf{x}}+w_{\mathit{Myo}}{\int }_{\mathit{Myo}}{(I-{\mu }_{\mathit{Myo}})}^{2}d\widehat{\mathbf{x}}+w_{\mathit{BG}}{\int }_{\mathit{BG}}\min \left({(I-c_{\mathit{lo}})}^2,{(I-c_{\mathit{hi}})}^2\right)d\widehat{\mathbf{x}}$$ (B1)

Following the same procedure as in appendix A, when taking the derivatives of E_coupled w.r.t. LV parameters we note that the second and fourth integral terms do not depend on LV shape. Similarly, in case of RV the first and fourth terms do not depend on RV and for Epi the first and second terms are independent of Epi shape. Hence,

$$f_{\mathit{LV}}^{\text{coupled}}=\underset{f_{\text{in}}}{\underbrace{w_{\mathit{LV}}{(I-{\mu }_{\mathit{LV}})}^2}}-\underset{f_{\text{out}}}{\underbrace{w_{\mathit{Myo}}{(I-{\mu }_{\mathit{Myo}})}^2{\chi }_{\mathit{epi}}(1-{\chi }_{\mathit{RV}})}}$$ (B2a)

$$f_{\mathit{RV}}^{\text{coupled}}=\underset{f_{\text{in}}}{\underbrace{w_{\mathit{RV}}{(I-{\mu }_{\mathit{RV}})}^2}}-\underset{f_{\text{out}}}{\underbrace{w_{\mathit{Myo}}{(I-{\mu }_{\mathit{Myo}})}^2{\chi }_{\mathit{epi}}(1-{\chi }_{\mathit{LV}})}}$$ (B2b)

$$f_{\mathit{epi}}^{\text{coupled}}=\underset{f_{\text{in}}}{\underbrace{w_{\mathit{Myo}}{(I-{\mu }_{\mathit{Myo}})}^2(1-{\chi }_{\mathit{LV}})(1-{\chi }_{\mathit{RV}})}}-\underset{f_{\text{out}}}{\underbrace{w_{\mathit{BG}}{(I-c(I))}^2{\chi }_{\mathit{epi}}(1-{\chi }_{\mathit{RV}})}}$$ (B2c)

where χ_epi, χ_RV, and χ_LV are characteristic functions of Epi, RV and LV shapes respectively. The characteristic function terms can be explained by how we have defined the region domains due to the possibility of overlap between regions in section 5.1. Keeping with our modified background model described in 5.2, in equation B2c, we define the background parameter c(I) as:

$$c(I)=\{\begin{array}{cc}{c}_{\mathit{lo}},\hfill & I\le \frac{c_{\mathit{lo}}+c_{\mathit{hi}}}{2}\hfill \\ c_{\mathit{hi}},\hfill & I>\frac{c_{\mathit{lo}}+c_{\mathit{hi}}}{2}\hfill \end{array}\phantom{\}}$$ (B3)

The overlap penalty is defined as:

$$E_{\text{overlap}}({\widehat{S}}_{\mathit{LV}},{\widehat{S}}_{\mathit{RV}},{\widehat{S}}_{\mathit{Epi}})={\beta }_{\mathit{LR}}\left({\int }_{\mathit{LV}\cap \mathit{RV}}\text{dmagenta}\widehat{\mathbf{x}}\right)+{\beta }_L\left({\int }_{\mathit{Lv}\cap \mathit{BG}}d\widehat{\mathbf{x}}\right)+{\beta }_R\left({\int }_{\mathit{RV}\cap \mathit{BG}}d\widehat{\mathbf{x}}\right)$$

which may be rewritten as follows:

$$E_{\text{overlap}}={\beta }_{\mathit{LR}}{\int }_{\mathit{LV}}{\chi }_{\mathit{RV}}d\widehat{\mathbf{x}}+{\beta }_{\mathit{LR}}{\int }_{\mathit{RV}}{\chi }_{\mathit{LV}}d\widehat{\mathbf{x}}+{\beta }_L{\int }_{\mathit{LV}}\left(1-{\chi }_{\mathit{epi}}\right)d\widehat{\mathbf{x}}+{\beta }_L{\int }_{\mathit{BG}}{\chi }_{\mathit{LV}}d\widehat{\mathbf{x}}+{\beta }_R{\int }_{\mathit{RV}}\left(1-{\chi }_{\mathit{epi}}\right)d\widehat{\mathbf{x}}+{\beta }_R{\int }_{\mathit{BG}}{\chi }_{\mathit{RV}}d\widehat{\mathbf{x}}$$

Following the same procedure as above and noting that for LV and RV f_out = 0 and for Epi f_in = 0, we get the following expressions:

$$f_{\mathit{LV}}^{\text{overlap}}={\beta }_{\mathit{LR}}{\chi }_{\mathit{RV}}+{\beta }_L(1-{\chi }_{\mathit{epi}})$$ (B4a)

$$f_{\mathit{RV}}^{\text{overlap}}={\beta }_{\mathit{LR}}{\chi }_{\mathit{LV}}+{\beta }_R(1-{\chi }_{\mathit{epi}})$$ (B4b)

$$f_{\mathit{Epi}}^{\text{overlap}}=-{\beta }_{\mathit{L}}{\chi }_{\mathit{LV}}+{\beta }_R{\chi }_{\mathit{RV}}$$ (B4c)

The overall force f that is used in the gradient expressions in equation 30, is the sum of the forces corresponding to E_coupled and E_overlap and is given by the following:

$$f_{\mathit{LV}}=f_{\mathit{LV}}^{\text{coupled}}+f_{\mathit{LV}}^{\text{overlap}}$$ (B5a)

$$f_{\mathit{RV}}=f_{\mathit{RV}}^{\text{coupled}}+f_{\mathit{RV}}^{\text{overlap}}$$ (B5b)

$$f_{\mathit{Epi}}=f_{\mathit{Epi}}^{\text{coupled}}+f_{\mathit{Epi}}^{\text{overlap}}$$ (B5c)