A Deep-Learning Semantic Segmentation Approach to Fully Automated MRI-based Left-Ventricular Deformation Analysis in Cardiotoxicity

Abstract

Left-ventricular (LV) strain measurements with the Displacement Encoding with Stimulated Echoes (DENSE) MRI sequence provide accurate estimates of cardiotoxicity damage related to breast cancer chemotherapy. This study investigated an automated LV chamber quantification tool via segmentation with a supervised deep convolutional neural network (DCNN) before strain analysis with DENSE images. Segmentation for chamber quantification analysis was conducted with a custom DeepLabV3+ DCNN with ResNet-50 backbone on 42 female breast cancer datasets (22 training-sets, eight validation-sets and 12 independent test-sets). Parameters such as LV end-diastolic diameter (LVEDD) and ejection fraction (LVEF) were quantified, and myocardial strains analyzed with the Radial Point Interpolation Method (RPIM). Myocardial classification was validated against ground-truth with sensitivity-specificity analysis, the metrics of Dice, average perpendicular distance (APD) and Hausdorff-distance. Following segmentation, validation was conducted with the Cronbach’s Alpha (C-Alpha) intraclass correlation coefficient between LV chamber quantification results with DENSE and Steady State Free Precession (SSFP) acquisitions and a vendor tool-based method to segment the DENSE data, and similarly for myocardial strain analysis in the chambers. The results of myocardial classification from segmentation of the DENSE data were accuracy = 97%, Dice = 0.89 and APD = 2.4 mm in the test-set. The C-Alpha correlations from comparing chamber quantification results between the segmented DENSE and SSFP data and vendor tool-based method were 0.97 for LVEF (56 ± 7% vs 55 ± 7% vs 55 ± 6%, p=0.6) and 0.77 for LVEDD (4.6 ± 0.4 cm vs 4.5 ± 0.3 cm vs 4.5 ± 0.3 cm, p=0.8). The validation metrics against ground-truth and equivalent parameters obtained from the SSFP segmentation and vendor tool-based comparisons show that the DCNN approach is applicable for automated LV chamber quantification and subsequent strain analysis in cardiotoxicity.

1. Introduction

The purpose of this study was the development of a MRI tool with a deep convolutional neural network (DCNN) for automated segmentation of the left-ventricle (LV) in chemotherapy patients who are susceptible to cardiotoxicity development [6, 9, 10, 12–17]. Our primary goal was LV chamber quantification by segmenting the true extent of the myocardium, within which we estimate cardiac motion and strains from data acquired with the Displacement Encoding with Stimulated Echoes (DENSE) sequence [18–22]. The ultimate vision of our study is monitoring LV function (via chamber quantification and strain analysis) with a single-scan MRI in patients who undergo treatment with chemotherapeutic agents (CTA) such as anthracyclines or trastuzumab [6–11]. In this respect, DCNN training of DENSE images for single-scan monitoring is a novel study that has not been pursued by the scientific community previously. The classification-based segmentation of the LV and surrounding anatomy was conducted on 42 patient datasets with the tool’s DeepLabV3+ DCNN architecture and underlying ResNet-50 backbone [12–15, 23–27]. The deep-learning tool’s segmentation outputs are processed for LV chamber quantification parameters essential for strain analysis, such as end-diastolic diameter (LVEDD) and ejection fraction (LVEF). The DCNN layers consist of convolution, rectified linear units (ReLU), batch-normalization (BN) (regularization), addition, atrous spatial pyramid pooling (ASPP) and softmax as shown in the schematic in Fig. 1 [13–15, 27]. The network conducts semantic segmentation by learning the weights and biases for its various layer-based functions and improves accuracy by recomputing gradients and errors via backpropagation approximations.

Fig. 1.

The DeepLabV3+ deep convolutional neural network (DCNN) trained to segment the LV myocardium in DENSE images, which is shown with output strides, atrous convolution, atrous spatial pyramid pooling (ASPP), upsampling, scorer and softmax. A similar DCNN was trained on the SSFP data.

Artificial intelligence (AI) and machine-learning tools (convolutional neural networks (CNN) and DCNNs are subsets of both) for motion and deformation analysis in cardiac images provide vital information on pathologies related to myocardial dysfunction [1–4, 28–31]. Key parameters based on population samples are built during learning to provide detailed spatiotemporal information at each point on the myocardium and at any time during the cardiac cycle [1, 30–33]. Thus, deep-learning approaches such as DCNNs can provide valuable clinical information (for example, estimates of LVEF) related to specific cardiac pathologies, particularly when supervised learning techniques are used [3, 14, 33–36]. This study’s supervised learning technique (a DCNN model) is specifically designed to transform input data to new identities, which follow training the network to perform classification tasks according to labels and annotations via multi-scale convolutional kernels [33, 35, 37–39]. It has an encoder-decoder architecture built for classification, with a dimensionality reduction technique that leads to upsampling to a higher dimension, and includes evaluation of classification accuracy via a loss function (at the softmax layer in Fig. 1) [2, 3, 35, 40–42]. In effect, the strength of the DCNN lies in the encoder (with ASPP for capturing multi-scale contextual information) and decoder (with bilinear upsampling) that together recover the spatial information and detect sharp object boundaries [12, 13, 15]. Thus, when provided with labels and annotations as supervision to differentiate data, such DCNN-based approaches (with its multi-scale edge detection capability) can meet the challenges of estimating motion variations that occur in pathological cardiac imaging [1, 15, 27, 31–33, 36, 42].

This study validated a novel LV chamber quantification technique via segmentation with a DeepLabV3+ DCNN prior to strain analysis with DENSE acquisitions. Validations were conducted with comparisons to ground-truth and to chamber quantification via a similar Steady State Free Precession (SSFP) DCNN and vendor tool, Circle cvi⁴² (Version 5.11.4, Circle Cardiovascular Imaging Inc., Calgary, Alberta). Additionally, we compared LVEF computations from the DENSE-based results to corresponding clinical measurements in patients with transthoracic echocardiography (TTE) exams.

2. Materials and Methods

2.1. Human Subjects Database

We tested the DCNN-based network for LV segmentation and the possibility of LV dysfunction in 42 DENSE datasets acquired on adult female breast cancer survivors. To validate the DENSE chamber quantification results, DCNN training with the same DeepLabv3+ architecture and segmentation followed by chamber quantification were conducted on 42 SSFP datasets acquired in the same patients [12, 13, 15, 27]. All patients had undergone CTA treatment consisting of one of two regimens, with either anthracyclines or trastuzumab, with details on drug-specific regimens given in our previous studies [19, 43]. Recruitment consisted of carefully screening patients to reduce the effect of other various cardiac comorbidities on cardiotoxicity analysis. Patients were recruited only if they had non-acute cardiac complications that existed before chemotherapy or developed afterward, as done previously [19, 43]. They signed informed consents based on Institutional Review Board (IRB) guidelines and volunteered access to their MRI data and medical histories. Since this is a study on LV chamber quantification in CTA treated breast cancer patients, a count of patients with clinically diagnosed cardiotoxicity is provided. The clinical diagnosis of cardiotoxicity in these patients was primarily based on the appearance of cardiovascular dysfunction documented by TTE exams with either a decline > 15% from baseline with the final LVEF still remaining ≥ 50% or a decline of LVEF to < 50% [7]. The timeframe for this detection was either during chemotherapy or at a routine post-chemotherapy TTE follow-up within three to six months of completing treatment. Hence, it is noted that while we measured LVEF with the SSFP acquisitions as part of the research protocol for this study, clinical cardiotoxicity was detected with echocardiography.

2.2. Background Theory

In this work, we performed the task of semantic segmentation of the DENSE and SSFP cardiac images by taking advantage of the latest contribution of atrous convolution in DCNNs, with proven merit for sharper feature detection [13, 15, 44]. Without atrous convolution, the commonly deployed down-sampling in DCNNs achieve invariance but has a toll on localization accuracy. We apply the atrous convolution in two different ways, of which the first is by applying a rate, r, corresponding to the stride with which we sample the input signal, as shown in Fig. 1. This atrous rate allows explicit control of the resolution at which the model computes feature responses without increasing the number of parameters or amount of computation [15]. Secondly, by use of ASPP which is an atrous version of the concept of spatial pyramid pooling used in SPP-Net [15, 27, 44] (Fig. 1). ASPP probes convolutional features with filters (or pooling) at multiple sampling rates and effective fields-of-view to segment objects at multiple scales. The ASPP is followed by the decoder’s upsampling layers, a scorer function for computing the pixel-wise probabilities for classification, and a softmax layer for cross-entropy loss computation. Detailed theory on the layers implemented (or modified) in the network, are outlined next. Explanations are provided on the functioning of convolutional (including atrous), batch normalization (BN), rectified linear unit (ReLU) and softmax layers, as well as the computation of gradients corresponding to them.

To explain atrous convolution, consider a layer with general cost function, $\mathcal{C}$, of convolution of an input signal, z_j, with filters γ_,j, bias β_i, predicted output $y={\sum }_j{\gamma }_{ij}{z}_j+\beta =\mathbf{\gamma }{\mathbf{z}}_j+\beta$ and true output label $\widehat{y}$ of length N defined by [15],

$$\mathcal{C}=\frac{1}{N}\sum _{=1}^N{(\widehat{y}-y)}^2=\frac{1}{N}\sum _{=1}^N{\left({\widehat{y}}_i-\left(\mathbf{\gamma }{\mathbf{z}}_j+\beta \right)\right)}^2$$ (1a)

If a rate parameter, r, is defined that corresponds to the stride of sampling the input signal, then $y={\sum }_j{\gamma }_{ij}{Z}_{rj+j_0}+\beta$, and we define a modified z_j as $z_{j^{\prime }}=Z_{rj+j_0}$. It is equivalent to convolving the input with upsampled filters via inserting r-1 zeros between two consecutive filter values along each dimension in space. Once we modify the cost function according to r, the change in the gradients for updating the output, input, weights and biases are given by,

$$\frac{d\mathcal{C}}{dy}=-\frac{2}{N}\left({\widehat{y}}_i-\left(\mathbf{\gamma }{\mathbf{z}}_{j^{\prime }}+\beta \right)\right)$$ (1b)

$$\frac{d\mathcal{C}}{d{z}_{j^{\prime }}}=-\frac{2}{N}\sum _{z_{rj+j_0}}{\gamma }_{ij}\left({\widehat{y}}_i-\left({\mathbf{\gamma }}_i{\mathbf{z}}_{j^{\prime }}+{\beta }_i\right)\right)$$ (1c)

which is summed over only those components of $\mathcal{C}$ with $z_{rj+j_0}$.

$$\frac{d\mathcal{C}}{d{\gamma }_{ij}}=-\frac{2}{N}\sum _{=1}^N\left({\widehat{y}}_i-\left(\mathbf{\gamma }{\mathbf{z}}_{j^{\prime }}+\beta \right)\right)z_{j^{\prime }}$$ (1d)

$$\frac{d\mathcal{C}}{d\beta }=-\frac{2}{N}\sum _{=1}^N\left({\widehat{y}}_i-\left(\mathbf{\gamma }{\mathbf{z}}_{j^{\prime }}+\beta \right)\right)$$ (1e)

Segmentation by edge detection is an essential tool in digital image analysis involving the identification and subsequent classification of certain objects and their edges. To find edges the algorithm looks for contours in the image where there is abrupt changes in intensity, discontinuities or illumination in a scene. One such method for edge detection is by using the Laplacian of Gaussian (LoG) filter where the Laplacian is a 2D isotropic measure of the second derivative of an image [45]. We applied the LoG at the first convolution kernel following input with a goal to sharpen edge detection and enhance the DCNN’s capacity to learn the variances and invariances inherent to the image set. We also preserve the full learning capacity of the network weights by maintaining backpropagation parameter optimization. The LoG is a multidimensional generalization of the Ricker wavelet and derived from a function, f(i,j) = I(i,j)*G(x,y), which is the convolution of an image, I(i,j), with the Gaussian filter, G(x,y). Here (i,j) are the pixel coordinates and (x,y) are local scale-space coordinates. The Laplacian kernel function, L(i,j), which is the second derivative of f(i,j) is given by,

$$L(i,j)={\nabla }^{2}f(i,j)=\frac{{\partial }^{2}f(i,j)}{\partial x^2}+\frac{{\partial }^{2}f(i,j)}{\partial y^2}$$ (2a)

The LoG filter is then given by,

$$LoG(x,y)=-\frac{1}{\pi {\sigma }^4}\left(1-\frac{\left(x^2+y^2\right)}{2{\sigma }^2}\right)e^{\frac{-\left(x^2+y^2\right)}{2{\sigma }^2}}$$ (2b)

where σ is the filter radius. Therefore, the layer weights are initialized as, γ_ij = LOG(x, y), and the gradients, $\Delta {\gamma }_{ij}$, are learned using back-propagation in its kernel.

BN is the step following convolution that reduces the internal covariate shift, via regularizing the change in the distribution of network activations. The shifts occur due to the change in network parameters during training. [25, 46, 47]. As BN generally has a slowing effect, the internal covariate shift problem is addressed by normalizing each mini-batch for layer inputs [25, 47]. Consider a $\mathcal{D}$-dimensional input to a layer, $x=\left[x^{(1)}\dots x^{(\mathcal{D})}\right]$, where x^(k) is each input to the layer. Then, the BN scaled and shifted output is given by, $y^{(k)}=B{N}_{\gamma ,\beta }{\widehat{x}}^{(k)}={\gamma }^{(k)}{\widehat{x}}^{(k)}+{\beta }^{(k)}$, where $B{N}_{\gamma ,\beta }{\widehat{x}}^{(k)}$ denotes the BN function, and the learnable parameters are γ^(k) and β^(k). The input normalization, ${\widehat{x}}^{(k)}$(the input is x^(k)), is given by,

$${\widehat{x}}^{(k)}=\frac{x^{(k)}-E\left[x^{(k)}\right]}{\sqrt{Var\left[x^{(k)}\right]}}$$ (3a)

where E[x^(k)] is the expectation (mean) and Var[x^(k)] is the variance. These parameters are learned along with the original model parameters, boost the representation power of the network and speed it up. Thus, in the batch setting of the stochastic gradient descent (SGD) training that follows, we have B = 1 … m values of activations in a mini-batch with the output defined as, $y=B{N}_{\gamma ,\beta }{\widehat{x}}_i$, in which ${\widehat{x}}_l$ is the normalization of the input, x_i. To calculate the Loss, $\mathcal{L}$, and its gradients, and to learn the weight and bias parameters, γ and β, the functions implemented for the layer are,

$$\widehat{x_l}=\frac{x-{\mu }_B}{\sqrt{{\sigma }_B^2+ϵ}}$$ (3b)

$${\mu }_B=\frac{1}{m}\sum _{=1}^{m}x$$ (3c)

$${\sigma }_B^2=\frac{1}{m}\sum _{=1}^m{\left(x-{\mu }_{\beta }\right)}^2$$ (3d)

where μ_B is the mean and σ_B the standard deviation for the mini-batch. The loss, $\mathcal{L}$, and its gradients with respect to the normalization, input, weight and bias are given by,

$$\mathcal{L}=\frac{1}{m}\sum _{=1}^m{\left(\widehat{y}-\gamma {\widehat{x}}_i-\beta \right)}^2=\frac{1}{m}\sum _{=1}^m{\left(\widehat{y}-\gamma \frac{x-{\mu }_B}{\sqrt{{\sigma }_B^2+ϵ}}-\beta \right)}^2$$ (3e)

$$\frac{\partial \mathcal{L}}{\partial {\widehat{x}}_l}=\frac{\partial \mathcal{L}}{\partial y}\cdot \gamma =-\frac{2}{m}\left(\widehat{y}-\gamma \frac{x-{\mu }_B}{\sqrt{{\sigma }_B^2+ϵ}}-\beta \right)$$ (3f)

$$\frac{\partial \mathcal{L}}{\partial x}=\frac{\partial \mathcal{L}}{\partial {\widehat{x}}_i}\frac{1}{\sqrt{{\sigma }_B^2+ϵ}}+2\frac{\partial \mathcal{L}}{\partial {\sigma }_B^2}\frac{x-{\mu }_{\beta }}{m}+\frac{\partial \mathcal{L}}{\partial {\mu }_B}\frac{1}{m}$$ (3g)

$$\frac{\partial \mathcal{L}}{\partial \gamma }=\sum _{=1}^m\frac{\partial \mathcal{L}}{\partial y}\cdot \widehat{x}$$ (3h)

$$\frac{\partial \mathcal{L}}{\partial \beta }=\sum _{=1}^m\frac{\partial \mathcal{L}}{\partial y}$$ (3i)

The BN transform can be added to a network to manipulate any activation, which depends on the training and other examples in the mini-batch [25, 46, 47].

The scaled and shifted output y^(k) is then passed to the ReLU layer. The ReLU is the default activation function in ResNet-50 that transforms the summed weighted input via a piecewise linear function [48, 49]. The output, y_i, will be the input, z_i, if is positive, otherwise zero. ReLU’s rectified output is given by,

$$y=ReLU(z)=\{\begin{array}{ll}z\hfill & z>0\hfill \\ 0\hfill & z<0\hfill \end{array}$$ (4a)

The gradient of ReLU is given by,

$$\frac{d(y)}{dz}=\frac{dReLU(z)}{dz}=\{\begin{array}{ll}1\hfill & z>0\hfill \\ 0\hfill & z<0\hfill \end{array}$$ (4b)

ResNet-50’s blocks of convolution are followed by a series of deconvolution layers, scoring, softmax and labeling in the DCNN’s DeepLabV3+ architecture. These later layers produced score maps and semantic label predictions, with short-range conditional random fields (CRF) employed to smooth noisy segmentation maps [13, 23, 50]. For a CRF-based model, the entropy function, E, employed by the softmax layer just before output is,

$$E(\mathbf{t},\mathbf{\theta })=-\sum _{=1}^{N}t\log (\theta )$$ (5a)

where $\theta =\frac{e^{z_i}}{{\sum }_j{e}^{z_j}}$ is the softmax function defining the probability for the ith input pixel, z_i, and t_i is one hot label for pixels of a class. Here the derivative of the softmax, ${\theta }_k=\frac{e^{z_k}}{{\sum }_{j=1}^C{e}^{z_j}}$, for classes k = [1 … C], and entropy, E, in relation to any input, z_i, is given by,

$$\frac{d{\theta }_k}{dz}={\theta }_k\left({\delta }_{k,i}-\theta \right)$$ (5b)

$$\frac{dE}{dz}=\frac{d}{dz}\left[-\sum _{k=1}^c{t}_k\log \left({\theta }_k\right)\right]=t-\theta$$ (5c)

Note that our network does not have learnable parameters for the softmax layer.

2.3. Protocols for MRI Acquisition

Navigator-gated, spiral 3D DENSE data were acquired on a 1.5T MAGNETOM Espree (Siemens Healthcare, Erlangen, Germany) scanner with displacement encoding applied in two orthogonal in-plane directions and one through-plane direction [18, 21, 51, 52]. Typical imaging parameters included a field of view (FOV) of 360 × 360 mm, matrix size of 128 × 128, echo time (TE) of 1.04 ms, repetition time (TR) of 15 ms, flip angle (FA) of 20°, voxel size of 2.81 × 2.81 × 5 mm, 21 cardiac phases, an encoding frequency of 0.06 cycles/mm and simple 4-point encoding and 3-point phase cycling for artifact suppression [18, 53]. For receiving patient signals from the Espree an anterior 18-channel array coil in combination with elements of the table-mounted spine array coil was used. The SSFP acquisition consisted of a FOV of 340 × 276 mm, TE of 1.48 ms, TR of 51.15 ms, FA of 80°, matrix size of 192 × 156, 1.77 × 1.77 mm pixel size, slice thickness of 7 mm and 25 cardiac phases. Heart rates (HR) and diastolic and systolic blood pressures (DBP and SBP) were continuously monitored during the scans.

2.4. Tailoring the DCNN for Cardiac Image Segmentation

Modifications were made to some layers of the original ResNet-50 architecture while keeping receptive field sizes of different layers the same. The hierarchical information in the original network is then hypothesized to help edge detection with the addition of our modifications as outlined in the following,

• The weights in the first convolution layer (Conv1 in Fig. 1) following the RGB input are initialized using a LoG function for a filter of size 5X5. The Laplacian is used for zero-crossing edge detection that highlight the regions of rapid intensity change in the image.

• Network speed was improved by cutting out the max-pooling layer for convolutional addition (Add1 in Fig. 1), which can similarly be used to summarize data. It is done with the same output size of the old pooling layer to improve accuracy while reducing model size. Additionally, the learning optimization (accuracy) is improved via convolutional addition with weights and biases, which a parameter-less pooling layer cannot achieve.

• The weights in the final convolution block (B4 in Fig. 1) are initialized using a LoG function for a filter of size 3X3. This LoG initialized convolution layer remains connected to two convolution layers with kernel size 1X1, and channel depths remain the same as the original one from ResNet-50.

• Following the fusion of feature maps from each convolution stage, we modified the classification weights of the cross-entropy layer (Softmax in Fig. 1) for loss computation. The classification weights were assigned according to the median frequency of pixels belonging to: (1) chest-cavity (CC), (2) chest wall (CW) and surrounding-anatomy (including liver and others), (3) myocardium, and (4) LV-cavity (LVC) as shown in Fig. 1.

Note that being a fully convolutional network for dense prediction tasks, DeepLabV3+ does not have a final fully-connected layer. The last DeepLabV3+ block using atrous convolutions via ASPP and different dilation rates to capture multi-scale context was retained (ASPP in Fig. 1). Following the ASPP and decoder layers, a convolution-based scorer function computes the pixel-wise probabilities and the softmax cross-entropy function (with class weights for the myocardium, LVC, CW and CC) computes loss before the final output layer.

To validate the DENSE-based results, corresponding SSFP data in the same patients were trained using an identical DeepLabV3+ DCNN, similarly initialized DCNN parameters, and machine specifications as DENSE, which are described next. Following imaging, the systolic period, apex-to-base DENSE and SSFP data were each arranged into a training cohort (22 datasets), a validation cohort (eight datasets), and a testing cohort (12 datasets). This arrangement is approximately a 52%−20%−28% division and similar to how previous studies with the DeepLabV3+ network divided data [15]. Hence, the total number of systolic period DENSE frames processed with DCNN segmentation was 8449 (42 datasets), which were divided into the training (22 datasets, N = 4420), validation (8 datasets, N = 1632) and independent testing (12 datasets, N = 2397) cohorts. Similar DCNN segmentation, with the custom DeepLabV3+ network, was conducted on a total of 7560 (42 datasets) SSFP frames divided into 22 training-sets (N = 3960), 8 validation-sets (N = 1440) and 12 test-sets (N = 2160). The end-diastolic to end-systolic 2D images in every slice-position in the respective training cohorts were augmented and trained via the DeepLabv3+ network. The Dicom data were converted into three-channel RGB images in Portable Network Graphics (PNG) format, which is the default input format for ResNet-50. During training with a mini-batch size of 50 for every epoch, the weights are updated via the SGD optimization algorithm. From the fine-tuning of parameters, via the gradient noise-scale plotting method outlined by Smith et al., the learning rate and the number of max epochs determined for both protocols were 0.001 and 60, respectively, to ensure efficient training [54]. The network was implemented with the Deep-learning Toolbox™ Model for DeepLabV3+ with underlying ResNet-50 backbone in MATLAB (Version 2020a, MathWorks Inc., Natick, MA). Our training experiments were performed on a machine with specifications: Intel Xeon E5–2690 Processor at 2.60 GHz CPU, NVIDIA Pascal Titan X GPU, and 128-GB RAM. The performance metrics evaluated following the training of the DCNN were the average perpendicular distance (APD), percentage of good contours (GC) that are contours with APD less than 5 mm, Dice, Hausdorff-distance and accuracy [55, 56]. For additional assessments, the receiver-operating characteristics curve (ROC) and area under the curve (AUC) from the ground-truth labels and the softmax scores for each class were generated. Finally, we generated pixel-based confusion matrices for both training-sets and test-sets that provide one-to-one comparison between true and predicted classes.

2.5. Chamber Quantification and Strain Analysis

Following image-based reconstruction of the full 3D LV, chamber quantification included measuring the LVEF, LVEDD, LV end-systolic diameter (LVESD), end-diastolic volume (LVEDV), end-systolic volume (LVESV), stroke volume (LVSV) and mass (LVM) with both DENSE and SSFP data. For validation, we compared chamber quantification results via the DCNNs for segmenting the DENSE data (DENSE-DCNN), and SSFP data (SSFP-DCNN), and thirdly, similar DENSE-based estimates with Circle cvi⁴². Henceforth, it is also noted that the abbreviations DENSE-DCNN and SSFP-DCNN are used to indicate the respective DCNNs for segmenting data with the two sequences. The results from the three evaluations were compared with intraclass correlation coefficients (ICC) and repeated measures analysis as outlined in the next section. Global radial (GRS), circumferential (GCS) and longitudinal (GLS) strains and torsion (apical) were analyzed in the LV myocardium with the meshfree Radial Point Interpolation Method (RPIM), which is a methodology detailed in several of our previous studies [19, 21, 22, 57]. For this study, RPIM strains were computed from the DENSE phase images in the LV chambers generated via the DENSE-DCNN and Circle cvi⁴² and statistically compared as detailed in the next section. Additionally, we compared the LVEF results from the DENSE-DCNN to the LVEF recorded from post-chemotherapy TTE exams in patients, with the MRI scans scheduled either on the day of their TTE or within 1–3 days.

2.6. Statistical Validation

Means and standard deviations were computed on the patients’ demographic data, results of ground-truth validation from segmenting the LV images, and results of chamber quantification and strains. The results of chamber quantification were compared via repeated measures analysis, following segmentation with the DENSE-DCNN and SSFP-DCNN and analysis with Circle cvi⁴². Repeatability (reliability) tests were also conducted between the three sets of chamber quantification parameters, by estimating the Cronbach’s Alpha (C-Alpha) ICC index. All repeated measures and ICC tests were conducted with SPSS (Version 26, IBM Corp., Armonk, NY). Agreement was assessed with Bland-Altman analysis between LVEF from chamber quantification via the DENSE-DCNN and SSFP-DCNN. Similar agreement was assessed between the LVEF results from chamber quantification with the DENSE-DCNN and analysis with Circle cvi⁴². Repeated measures analysis and ICC tests were conducted on the estimates of RPIM-based strains (GRS, GCS and GLS) in LV chambers reconstructed from the DENSE-DCNN results and Circle cvi⁴² analysis.

3. Results

Table 1 shows the demographic information on patients who volunteered their DENSE and SSFP cardiac data for the DCNN-based and Circle cvi⁴²-based chamber quantification and strain analyses. Table 1 also details their existing comorbidities and the number of cardiotoxicity cases, which was determined clinically via the LVEF-based criteria outlined in the Methods section. The time to recruiting patients from the end of chemotherapy was 4.7 ± 2.5 months.

Table 1:

Demographics, comorbidities and detected cardiotoxicity cases in N=42 post-chemotherapy breast cancer patients.

Parameter	Value
Demographics
Age (years)	55.5 (8.6)
DBP (mmHg)	73.2 (12.0)
SBP (mmHg)	125.0 (14.3)
HR (bpm)	73.5 (11.1)
Body Mass (kg)	72.5 (13.9)
BMI (kg/m2)	27.0 (4.4)
BSA (m2)	1.8 (0.2)
Comorbidities
Hypertension	9
Hypercholesterolemia	7
Diabetes Mellitus	9
Radiotherapy	15
Detected cases*	8

Abbreviations: DBP: diastolic blood pressure, SBP: systolic blood pressure, HR: heart rate, BMI: body mass index, BSA: body surface area.

^*Clinician detected cardiotoxicity.

(): Standard deviation.

3.1. Deep-learning Output with Multi-class Classification

With the completion of training the DENSE-DCNN, we achieved an overall validation accuracy of 94% (Table 2a) for the four classes and a final cross-entropy loss < 0.1 after 60 epochs and 5304 iterations. This was similar to the validation accuracy achieved with the SSFP-DCNN, which was 95% (Table 2b) after a similar 60 epochs, 4752 iterations and final cross-entropy loss < 0.1. The training took 12.6 hours to complete for the DENSE-DCNN and 11.3 hours to complete for the SSFP-DCNN, as performed in a Windows environment with specifications outlined earlier in the Methods section. Additional computational time spent on semantic segmentation of the independent test-set took approximately 0.33 seconds per image. The ROC curves acquired cumulatively across all images in the DENSE validation-set are shown in Fig. 2a, with AUCs of 1.0, 0.99, 1.0, and 0.98 for CC, CW, myocardium, and LVC, respectively. The AUCs from the DENSE test-set images were 1.0, 0.98, 1.0, and 0.98 on predicting CC, CW, myocardium, and LVC, respectively (Fig. 2a). The ROC curves acquired cumulatively across all images in the SSFP validation-set are shown in Fig. 2b, with AUCs of 1.0, 0.99, 0.98, and 0.98 for CC, CW, myocardium, and LVC, respectively. The AUCs from the SSFP test-set images were 1.0, 0.99, 0.98, and 0.97 on predicting CC, CW, myocardium, and LVC, respectively (Fig. 2b). Additionally, we determined the predictive accuracy of the DCNN model for labeling pixels in each image by confusion matrix. It is seen from the confusion matrix in Fig. 2a that the DENSE validation cohort exhibited a sensitivity of 97.1% and precision of 81.9% for predicting the myocardium, and a sensitivity of 89.9% and precision of 94.6% for predicting LVC (Fig. 2a). The testing cohort showed sensitivities of 96.9% and 89.7% and precisions of 81.9% and 94.3% for myocardium and LVC, respectively (Fig. 2a). Most misclassifications were observed where the extent of myocardial tissue is visually inseparable from the surrounding anatomy, such as chest-wall or liver. Similar predictive results were obtained from segmenting the SSFP validation-set and test-set data, as shown in Fig. 2b. Figs. 3a and 3b show a series of systolic period, mid-ventricular DENSE images in a validation-set patient and a test-set patient, respectively, superposed with the DENSE-DCNN predicted labels. Figs. 4a and 4b show systolic period, mid-ventricular SSFP images in a validation-set patient and a test-set patient, respectively, superposed with labels predicted by the SSFP-DCNN. The corresponding RGB formatted input images to both DCNNs for Figs. 3 and 4 are given in Appendix A. Shown in Fig. 5a are some of the activations following the LoG convolution layer (block 1 in Fig. 1) for edge detection, which are implemented by the atrous convolution of Eqn. 1. Fig. 5b shows the corresponding RGB image of the LV slice (from the DENSE validation-set) and its output label. Fig. 5c shows the scorer layer output activations, which are upsampled for input to the final softmax layer, and Fig. 5d shows the final output labels from the softmax layer (with its cross-entropy function given by Eqn. 5).

Table 2:

Validation results in reference to ground-truth with Deep Convolutional Neural Network (DCNN) segmentation of (a) DENSE and (b) SSFP left-ventricular MRI from N=42 post-chemotherapy breast cancer patient datasets.

Segment	Accuracy§	Good Contour** (%)	Dice Score	APD (mm)	Hausdorff Dist. (mm)	Sensitivity	Specificity
Test Set
Chest cavity	93	88.6	0.95 (0.02)	4.60 (0.91)	15.84 (4.33)	93 (3)	95 (3)
Surrounding frame	96	99.4	0.90 (0.04)	3.22 (0.55)	9.49 (3.39)	94 (3)	93 (3)
LV myocardium	97	100.0	0.89 (0.04)	2.36 (0.56)	5.15 (0.94)	97 (4)	99 (<1)
LV cavity	89	100.0	0.90 (0.09)	2.03 (0.47)	5.60 (1.28)	88 (9)	99 (<1)
All	94	97.0	0.91 (0.03)	3.10 (0.37)	9.02 (1.34)	93 (3)	97 (1)
Validation Set
Chest cavity	94	81.1	0.95 (0.02)	4.93 (0.97)	16.73 (4.54)	92 (3)	95 (3)
Surrounding frame	96	99.6	0.90 (0.04)	3.23 (0.52)	9.49 (3.36)	94 (3)	93 (3)
LV myocardium	97	99.9	0.89 (0.04)	2.40 (0.54)	5.48 (1.61)	96 (4)	99 (<1)
LV cavity	90	100.0	0.89 (0.09)	2.06 (0.46)	5.55 (1.30)	87 (8)	99 (<1)
All	94	95.2	0.91 (0.03)	3.16 (0.40)	9.31 (1.46)	92 (3)	97 (1)

TABLE 2B.

VALIDATION METRICS^* FROM DCNN SEGMENTATION OF SSFP LV DATA (N=1440 VALIDATION-SET IMAGES AND N=2160 TEST-SET IMAGES)

Segment	Accuracy§ (%)	Good Contour** (%)	Dice Score	APD (mm)	Hausdorff Dist. (mm)	Sensitivity (%)	Specificity (%)
Test Set
Chest cavity	86	89.8	0.91 (0.03)	4.61 (0.76)	12.52 (3.71)	86 (4)	96 (1)
Surrounding frame	96	100.0	0.92 (0.02)	1.41 (0.44)	5.47 (3.65}	96 (4)	87 (4)
LV myocardium	97	100.0	0.93 (0.02)	0.90 (0.33)	5.32 (1.51)	97 (4)	100 (<1)
LV cavity	95	100.0	0.94 (0.08)	0.73 (0.30)	4.92 (2.33)	94 (6)	100 (<1)
All	94	97.4	0.92 (0.03)	1.91 (0.30)	7.06 (1.42)	93 (6)	95 (5)
Validation Set
Chest cavity	86	77.3	0.91 (0.03)	4.87 (0.77)	11.42 (3.49)	86 (4)	97 (1)
Surrounding frame	96	100.0	0.92 (0.02)	1.26 (0.38)	5.45 (3.32)	96 (3)	87 (4)
LV myocardium	99	100.0	0.94 (0.02)	0.57 (0.19)	4.46 (1.26)	98 (3)	100 (<1)
LV cavity	97	100.0	0.97 (0.03)	0.53 (0.18)	3.67 (1.89)	96 (4)	100 (<1)
All	95	94.3	0.93 (0.02)	1.80 (0.24)	6.26 (1.36)	94 (6)	96 (5)

Abbreviations: DCNN: Deep convolutional neural network, APD: Average perpendicular distance.

^*In comparison to ground-truth.

^§Overall pixel accuracy: percentage of correctly identified pixels for each class.

^**Calculated as % of countours with APD < 5mm.

(): Standard deviaitons.

Fig. 2.

Pixel-wise confusion matrices and ROC-AUC of LV images in the (a) DENSE validation-set (N = 1632 images) and test-set (N = 2397 images) and (b) SSFP validation-set (N = 1440 images) and test-set (N = 2160 images).

Fig. 3.

Output labels from the trained DENSE-based DCNN superposed on DENSE grayscale, systolic period, mid-ventricular LV images of (a) a validation-set patient, and (b) a test-set patient. The corresponding RGB input images to the DCNN for both patients are given in Appendix A.

Fig. 4.

Output labels from the trained SSFP-based DCNN superposed on SSFP grayscale, systolic period, mid-ventricular LV images of (a) a validation-set patient, and (b) a test-set patient. The corresponding RGB input images to the DCNN for both patients are given in Appendix A.

Fig. 5.

The DENSE-based DCNN output of (a) activations (16 of 64) at the first convolution layer with LoG filters for edge detection (Conv1 in Fig. 1), (b) final label shown with input RGB image, (c) activations (16 of 256) at the scorer layer (Scorer in Fig. 1), (d) the four class-based activations at the softmax layer (Softmax in Fig. 1).

3.2. Comparisons of Chamber Quantification Parameters

Table 3 shows the three chamber quantification results, comprising of the DENSE-DCNN, SSFP-DCNN and Circle cvi⁴²-based LV segmentation approaches, and the repeated measures analysis and ICC results between them. Significant differences were not found between the three measurements, which is an important finding related to validating the methodology for chamber quantification via the DENSE-DCNN. Table 3 also shows the ICC results between strains computed in the LV chambers estimated with the DENSE-DCNN and Circle cvi⁴². The ICC of 0.97 in Table 3 indicates similarities between LVEF results with the three approaches toward chamber quantification. Fig. 6a shows the full-LV end-diastolic and end-systolic chamber and cavity wall dimensions estimated in a patient following segmentation with the DENSE-DCNN. Fig. 6b shows the Bland-Altman agreements between LVEF estimated from chamber quantification with the DENSE-DCNN and SSFP-DCNN, and also between the DENSE-DCNN and Circle cvi⁴²-based estimates.

Table 3.

Left-ventricular chamber quantification via Deep Convolutional Neural Network (DCNN) segmentation of DENSE and SSFP MRI, and analysis with Circle cvi⁴² on N = 42 post-chemotherapy breast cancer patient datasets.

LV chamber quantification and strains with new and existing tools
Parameter	DENSE Deep Learninga	SSFP Deep Learninga	Vendor toolb	p-valuecde	IOCc
LV EDD (cm)	4.6 (0.4)	4.5 (0.3)	4.5 (0.3)	0.8	0.77
LV ESD (cm)	3.2 (0.3)	3.2 (0.3)	3.2 (0.3)	0.7	0.89
LV EDV (ml)	113 (14)	113 (13)	112 (16)	0.8	0.75
LV ESV (ml)	50 (9)	51 (9)	50 (11)	0.9	0.89
LV SV (ml)	63 (11)	62 (9)	62 (12)	0.7	0.80
LV EF (%)	56 (7)	55 (7)	55 (6)	0.6	0.97
LVM (gm)	123 (9)	123 (8)	121 (9)	0.5	0.73
Global Eer (%)	32 (4)	–	32 (5)	0.3	0.90
Global Ecc (%)	−20 (3)	–	−21 (4)	0.3	0.93
Global Ell (%)	−15 (3)	–	−15 (3)	0.2	0.89
Apical Torsion (°)	6.7 (2)	–	7.0 (2)	0.3	0.87

Abbreviations: EDD: end-diastolic diameter, ESD: end-systolic diameter, EDV: end-diastolic volume, ESV: end-systolic volume, SV: stroke volume, EF: ejection fraction, LVM: LV mass, Err: Global radial strain, Ecc: Global circumferential strain, Ell: Global longitudinal strain.

^*Please note that (e) implies all numbers entered in ()

^aEstimated with chamber quantification via the DCNN for each protocol.

^bEstimated with Circle cvi⁴².

^cFrom three-samples repeated measures analysis.

^dICC: Intraclass-correlation coefficient computed with the Cronbach’s Alpha index, range: 0–1. (): Standard deviation.

^e(): Standard deviation.

Fig. 6.

(a) Reconstructed LV systolic and diastolic geometries from the labeled myocardium in stacked short-axis DENSE slices, (b) Bland-Altman agreement on LVEF estimates between the DENSE-DCNN and vendor tool, and between the DENSE-DCNN and SSFP-DCNN.

4. Discussion

In this study, we designed and validated a new and automated DCNN-based methodology for chamber quantification during LV contraction that is required for strain analysis in DENSE images. The training and testing were conducted in a subpopulation of breast cancer patients whose LV strains are tracked to observe the possibility of cardiotoxicity occurrence. Our study on developing a chamber quantification tool via applying DCNN technology to segment DENSE images is novel, and was conceptualized from a necessity to define the LV chamber before strain analysis. Since this approach to chamber quantification is unprecedented, we first validated our encoder-decoder model with extensive comparisons to ground-truth, which is the traditional method for validating segmentation studies, and secondly, by comparisons to chamber quantification with a SSFP-DCNN trained with the same DeepLabv3+ network. It is seen from Table 2 that the validation results of segmenting with the two DCNN models are consistently similar according to the metrics used. The above similarities in validation metrics in addition to significant differences not found between chamber quantification with the DENSE-DCNN and SSFP-DCNN infer that both DENSE and SSFP approaches can estimate the LV myocardium before strain analysis. The validation results noticeably different between the two DCNN models correspond to the APD and Hausdorff distance metrics, and this difference could be attributed to the many sources of variations in protocols and a lack of generalizing data [58, 59]. Indeed, variations in the acquired data from two different sequences, such as characteristics particular to a pulse-sequence, FOV, image quality and differences based on breath-holds, can influence image intensities and contrast and therefore, segmentation results [58, 60]. Next, we review the validation techniques used by previous studies that have used different machine-learning approaches to segment SSFP data and analyze myocardial contraction. Previous examples include DCNN-based approaches in combination with auto-encoders for multi-class classifications, recurrent convolutional neural networks (RCNN) and other techniques [1, 25, 28, 29, 32, 33, 35, 36, 39]. Some of the earlier neural network approaches for segmenting the myocardium in cine SSFP MRI have earned top results at the Automated Cardiac Diagnosis Challenge (ACDC) contest [31, 32, 38, 61]. State-of-the-art segmentation results verified on the ACDC cardiac dataset include reporting the fundamental metric of accuracy (as we report), with values of 0.96 achieved by Khened et al. (Random Forest algorithm), and 0.92 by both Cetin et al. (Support Vector Machine (SVM) algorithm) and Isensee et al. (Random Forest) [31, 32, 38, 61]. Among other earlier studies, one by Emad et al. targeted improving the automatic localization of differently sized LVs in short-axis MRI images by using a six-layer DCNN and a fully-connected softmax layer, and tested it on a publically available database of 33 patients [36]. They used pyramids of scales to augment the differently sized LVs (similar to the ASPP approach in our study) and obtained 98.7%, 83.9% and 99.1% for accuracy, sensitivity and specificity, respectively, which are also units of ground-truth validation that we report. Machine learning has also been applied to generate accurate cardiac segmentation that guide diagnosis and therapy management, with several studies aimed at detecting myocardial infarction (MI). In a MRI study by Zhang et al. to develop a fully automatic framework for chronic MI delineation via deep-learning on non-contrast cardiac cine data, differences were not found between the non-enhanced and late gadolinium-enhanced (LGE) analyses in per-patient MI area (6.2cm² vs 5.5cm², p = 0.3) [62]. Recently, Bai et al. conducted a study on automated segmentation with fully convolutional network on a large-scale cardiac dataset from the UK Biobank [30]. In a 600-subjects test-set, the authors showed that the Dice score from LV segmentation was similar between a group with cardiovascular diseases (Dice = 0.87) and the entire set of healthy subjects and patients (Dice = 0.88). Like previous studies, we validated a novel approach with the DENSE-DCNN via extensive comparisons to ground-truth, and additionally, by statistical validation for chamber quantification in reference to similar analysis with SSFP (the standard protocol in medical imaging).

The outcome of our study is the development of a novel and validated chamber quantification tool, with a fully automated deep-learning framework, which detected myocardial extent and contraction in an authentic breast cancer dataset. Using an independent test-set from the DENSE dataset, the Dice score of 0.89, APD of 2.4 mm, GC of 100% and accuracy of 97% for the myocardium show that our deep-learning approach detects the presence, position, and size of the LV myocardium in a cohort of breast cancer patients. Detailed results of validation with the DENSE and SFFP validation-sets and test-sets are given in Table 2, which shows that the new DCNN-trained technique is applicable for time-saving and automated segmentation. Hence, our study has benefited from the vast development in deep-learning techniques, which have demonstrated superior performance in medical image analysis by leveraging a broad range of data [1, 4, 5]. Like previous studies that harnessed the precision of cardiac deep-learning, we have shown that accuracy can be achieved on frame-wise evaluations over the systolic period, in a heterogeneous group of patients, a fraction of whom have confirmed cardiotoxicity. Overall, the DCNN-based approach to segmenting the DENSE data achieved high sensitivity, specificity, and AUC for detecting the myocardium, which we believe is due to our framework’s output of a dense vector with classification probabilities for each input pixel. Therefore, we obtained the desired output resolution (same as that of the input layer) by applying the decoder’s upsampling to features extracted from the ResNet-50 backbone and ASPP block [13]. In this context, our MATLAB-based DCNN code is open-source and accompanied by sample datasets, with which (or similarly acquired data) one can assess the performance of our network. In addition to evaluating the ground-truth-based validation metrics, the LV chamber quantification and strain analyses were statistically validated once we computed the LV extent and parameters with the DENSE-DCNN, SSFP-DCNN and Circle cvi42, as shown in Table 3 [20–22, 57]. The similarities seen in results from the three different techniques support conducting a single-scan MRI with DENSE to assess LV function in patients who undergo CTA treatment. It is also noted that we achieved the similarities between the three different techniques for chamber quantification (Table 3) after processing several thousand image frames. However, while a myocardium segmentation accuracy in greater than 90% of the pixels indicates a high success rate, there is potential for failure under certain circumstances. These failures can occur at a later phase of the systolic period (in both DENSE and SSFP images) when the DCNN is unsuccessful at segmenting due to a lack of contrast between myocardium and surrounding tissue or cavity [63]. The low Bland-Altman agreement limits (less than 5%) for LVEF in Fig. 6b and the ICCs achieved for global strains (Table 3) within the identified geometry also show our methodology’s consistency. Furthermore, significant differences were not found with the paired t-test comparison between LVEF estimated with the DENSE-DCNN and post-chemotherapy TTE exams in patients (56 ± 7% vs 56 ± 6%, p=0.1), which were the exams for clinically detecting the cardiotoxicity cases in Table 1. From background research we found previous studies that reported similarly reduced strains (GLS ≤ 18%) when CTA doses of anthracyclines and trastuzumab were administered in breast cancer patients, which was seen as indicative of LV dysfunction related to cardiotoxicity [8–11, 64]. Additionally, the GLS found in this study is considered indicative of LV dysfunction based on research-to-practice guidelines on monitoring chemotherapy outcome [65]. These guidelines recommend 16% ≤ GLS < 18% as borderline and GLS < 16% as indicative of LV dysfunction, which can be assessed independent to LVEF measurements (or reductions). Previous studies have shown similar cardiotoxicity-induced reductions in apical torsion to values ≤ 6^○, where ≥ 8^○ is considered normal [64, 66]. The reduced GLS and torsion seen in our previous study with the semi-automated, quantization-based chamber quantification approach also support the reductions seen with the current DCNN-based approach [19].

The first limitation of this study is that backbone networks such as Xception, Inception, ResNet-101, U-Net and others were not tested for LV segmentation [13, 15, 26, 59]. The authors of DeepLabv3+ have shown that some of these other networks (as standalone or in combination) can improve the accuracy of feature recognition with sharper edge detections. A second limitation is that with cine DENSE only a portion of the cardiac cycle is imaged, which is during the systolic period [18, 19, 43, 52]. Hence, we could not determine our network’s performance accuracy for detecting the diastolic contours, despite having SSFP data [5, 30, 31, 41, 61]. We may accomplish this goal if more phases from the full cardiac cycle are available with future DENSE sequences. A third limitation is that ground-truth contours of the endocardium and epicardium were initially delineated with a semi-automated quantization technique, which could have biased expert decision when they modified the contours for accuracy [19, 22, 43]. A final limitation is that we did not aggregate images from both DENSE and SSFP and generalize the training with a single DCNN. However, considerable network modifications are required to homogenize the wide variations in image intensities between the protocols (for example, the contrast between black-blood in DENSE and white-blood in SSFP acquisitions). The homogenization could be implemented in a new study with histogram mapping techniques that enforce the histogram distribution in images to be similar to a target histogram or histogram equalization techniques that reduce the heterogeneous effect of contrast and brightness [67].

5. Conclusion

This study introduced a novel and automated, DCNN architecture-based chamber quantification methodology for detecting the extent of LV myocardium in DENSE acquisitions. The critical requirements were accurate segmentation, chamber quantification and subsequent strain analysis in the myocardium, via a single-scan DENSE MRI, in breast cancer patients susceptible to chemotherapy-related cardiotoxicity. Automated DCNN segmentation of the DENSE acquisitions was validated with comparisons to ground-truth, and chamber quantification was validated compared to DCNN-based analysis with SSFP and a vendor tool. Myocardial strains computed from DENSE displacements were compared in the LV chambers from DCNN segmentation and vendor tool and validated by significant differences not found. Additionally, the study findings show cardiac dysfunction in the patient subpopulation that may follow chemotherapy with agents like anthracyclines. In conclusion, the findings emphasize that our DCNN-based segmentation approach provides accurate estimates of the LV chamber quantification required in strain analysis.

Supplementary Material

Appendix A

RGB versions of patient DICOM images that were processed with the DENSE-DCNN and SSFP-DCNN to generate the validation and test labels shown in Figures 3 and 4.

Availability of Data

Data related to this project can be obtained from the Open Science Framework repository at URL: https://osf.io/enjbu/ DOI: 10.17605/OSF.IO/ENJBU