JOINT DEEP LEARNING FOR IMPROVED MYOCARDIAL SCAR DETECTION FROM CARDIAC MRI

Abstract

Automated identification of myocardial scar from late gadolinium enhancement cardiac magnetic resonance images (LGE-CMR) is limited by image noise and artifacts such as those related to motion and partial volume effect. This paper presents a novel joint deep learning (JDL) framework that improves such tasks by utilizing simultaneously learned myocardium segmentations to eliminate negative effects from non-region-of-interest areas. In contrast to previous approaches treating scar detection and myocardium segmentation as separate or parallel tasks, our proposed method introduces a message passing module where the information of myocardium segmentation is directly passed to guide scar detectors. This newly designed network will efficiently exploit joint information from the two related tasks and use all available sources of myocardium segmentation to benefit scar identification. We demonstrate the effectiveness of JDL on LGE-CMR images for automated left ventricular (LV) scar detection, with great potential to improve risk prediction in patients with both ischemic and non-ischemic heart disease and to improve response rates to cardiac resynchronization therapy (CRT) for heart failure patients. Experimental results show that our proposed approach outperforms multiple state-of-the-art methods, including commonly used two-step segmentation-classification networks, and multitask learning schemes where subtasks are indirectly interacted.

1. INTRODUCTION

Myocardial scarring is an important factor that leads to cardiac dysfunction and poor prognosis of patients with heart diseases [1, 2]. Identifying the presence and extent of myocardial scars is important to many clinical interventions. For instance, it can significantly increase the response rate of cardiac resynchronization therapy (CRT), since contractile improvements are often shown on scar-free sites [3]. Another example is that the quantification of scar tissue can provide prognostic information about the risk for arrhythmia in patients with nonischemic cardiomyopathy undergoing cardioverter-defibrillator implantation [4]. With rapid development of advanced imaging techniques, LGE-CMR has become one of the most promising and noninvasive methods to determine myocardial scar burden and guide therapeutic decisions [5]. However, the variable quality of LGE images due to image noises, motion artifacts, and partial volume effect makes the task of automated scar detection challenging.

Inspired by the great performance of deep learning methods, recent research has made the first attempt to address this issue by employing deep neural networks to segment scar directly from LGE-CMR images [6, 7]. Despite relatively low Dice scores, these works demonstrated that the predicted scar segmentations can be substantially improved with provided myocardium segmentation labels. Later, two-step approaches were developed to achieve better performance by cascading the task of myocardium and scar segmentation sequentially in two separate networks [8]. While the scar segmentation accuracy was improved, the performance of two-step methods heavily depends on the quality of myocardium segmentation. Another mainstream of incorporating myocardium segmentations is through multitask learning (MTL). In contrast to two-step approaches, MTL performs both tasks simultaneously and allows mutual benefit from each other [9, 10]. However, MTL methods are fully data-driven and the learned relationship between tasks is unclear. With little to no knowledge on how the task relationship is utilized, the performance of MTL can be unstable even with fine-tuned parameters. In addition, MTL heavily depends on the quality of training data. It may learn an inaccurate task relationship that misleads the learning towards suboptimal models when the data is low quality with high noises.

In this paper, we propose a novel joint deep learning (JDL) framework that efficiently solicits the knowledge of spatial relationships between the myocardium and myocardial scars from jointly learned networks. In contrast to previous methods treating related tasks in a separate or parallel manner, our model (i) explicitly enforces a mutual guidance between the myocardium and scar segmentation tasks in training, and (ii) directly incorporates the prior information provided by predicted myocardium segmentations to identify the scars. We develop an interactive joint deep network that connects myocardium segmentation and myocardial scar detection by a message-passing module. More specifically, we use the predicted myocardium probability maps as weighting masks on the LGE images to eliminate the negative effects of non-ROI regions in scar detection. Meanwhile, the myocardium segmentation network receives extra supervision from the scar maps, which captures the scar region more accurately.

To the best of our knowledge, we are the first to develop JDL models for myocardial scar detection with learned myocardium segmentations from LGE-CMR images. Experimental results show that our model outperforms a direct segmentation network on original LGE images [7], fully automatic two-step approaches [8], and recent MTL schemes [10]. The outcome of this research has great potential to impact various image-guided clinical systems, such as locating optimal left-ventricular-lead sites that are scar-free for CRT [11], providing more accurate prognostic information in patients with known or suspected coronary artery disease [1], and making the best possible therapeutic decisions for coronary revascularization [12].

2. A JOINT DEEP NETWORK FOR SCAR DETECTION

Consider a set of training images I₁, … , I_N with ground truth labels of myocardium and myocardial scars, where N is the number of images. Our JDL framework simultaneously learns the tasks of myocardium and scar segmentation in an interactive manner. With a joint optimization scheme, our model produces two different outputs: a myocardium segmentation probability map and a binary scar map. The myocardium probability map is used as a ROI-mask that is iteratively passed to the learning process of scar detection. Next, we will introduce the network architecture and optimization schemes.

2.1. Joint Network Architecture

Our designed network is conceptually simple and modifies two-step approaches [8] to jointly optimized networks through a message passing module. The first myocardium segmentation network takes LGE-CMR as an input and generates a probability map. A message passing module then combines the information of probability maps with the original images to reproduce a masked input data for the scar segmentation network. This effectively suppresses noises or artifacts from non-ROI regions in scar detection. It is worthy to mention that our framework is highly flexible to various types of segmentation networks for both tasks. In this paper, we employ pre-trained TransUNet [13], which shows superior performance on medical image segmentation tasks over other commonly used networks such as U-Net [14] An overview of our proposed approach is presented in Fig. 1.

Fig. 1:

The overview of our proposed framework, where the myocardium segmentation network communicates with the scar segmentation network via the message-passing module, and the two networks are jointly trained. The ⊙ means element-wise multiplication and refers to masking operation, and ⊕ means summation for the joint optimization.

2.2. Loss and Optimization

Consider a set of ground truth segmentation labels Y, we use a weighted sum of cross entropy and Dice loss L(·,·) for both segmentation networks, i.e.,

$$L(Y,\widehat{Y}(\theta ))=-\sum Y\text{ log }\widehat{Y}(\theta )-w\sum \left(1-\frac{2Y\widehat{Y}(\theta )+ϵ}{Y+\widehat{Y}(\theta )+ϵ}\right),$$

where Ŷ (θ) represents network prediction with parameter θ, w is a weighting parameter, and ϵ is a small smoothing constant. In this paper, we set ϵ = 1e − 6 in all our experiments.

Supervised by manually delineated myocardium M and myocardial scars S, our model jointly learns to predict myocardium segmentation probability maps $\widehat{M}\left({\theta }_M\right)$ and binary scar maps $\widehat{S}\left({\theta }_S\right)$. A message-passing module effectively utilizes the predicted myocardium segmentations to mask out non-ROI regions from original images, which provides new input data to the scar network. The total loss function is then defined by a weighted sum of the regularized myocardium segmentation loss L_m and the scar segmentation loss L_s

$$L\left({\theta }_M,{\theta }_S\right)=\lambda \sum _{i=1}^N{L}_m\left(M_i,{\widehat{M}}_i\left({\theta }_M\right);I_i\right)+L_s\left(S_i,{\widehat{S}}_i\left({\theta }_S\right);{\widehat{M}}_i\left({\theta }_M\right)\odot I_i\right)+{\beta }_M\text{Reg}\left({\theta }_M\right)+{\beta }_S\text{Reg}\left({\theta }_S\right),$$ (1)

where {λ, β_M, β_s} are weighting parameters, ⊙ denotes an element-wise production), and Reg(·) represents network regularization, which is set to L₂ weight decay in our experiments.

Joint network optimization.

We use a joint optimization scheme to minimize the loss in Eq. (1). Our scar segmentation network is trained using the output of the myocardium segmentation network, whereas the myocardium segmentation network is trained by taking a sum of weighted gradients from both loss functions L_m and L_s. The optimization of Eq. (1) splits into two parts:

$${\widehat{\theta }}_M=\underset{{\theta }_M}{\text{arg min}}\lambda \sum _{i=1}^N{L}_m\left(M_i,{\widehat{M}}_i\left({\theta }_M\right);I_i\right)+L_s\left(S_i,{\widehat{S}}_i\left({\theta }_S\right);{\widehat{M}}_i\left({\theta }_M\right)\odot I_i\right)+{\beta }_M\text{Reg}\left({\theta }_M\right),{\widehat{\theta }}_S=\underset{{\theta }_S}{\text{arg min}}{L}_s\left(S_i,{\widehat{S}}_i\left({\theta }_S\right);{\widehat{M}}_i\left({\theta }_M\right)\odot I_i\right)+{\beta }_S\text{Reg}\left({\theta }_S\right).$$

This shows that our model explicitly allows two tasks mutually guided through the message-passing module. In practice, we train the two subnetworks simultaneously. A detailed description of parameter settings will be introduced in the experiment section.

3. EXPERIMENTAL RESULTS

3.1. Experimental Settings

We test the proposed JDL framework on a newly collected dataset of LGE-CMR images in our institute. To validate the performance of our model, we compare it with three types of baseline algorithms: a direct scar segmentation network on original image data, a two-step myocardium and scar segmentation network, and a multitask learning network with hard-parameter sharing. For fair comparison, the backbone network of all methods is TransUNet [15] with the transformer part pre-trained on imageNet-21K [13]. To evaluate volume overlap between the predicted segmentation A and the manual segmentation B, we compute the Dice similarity coefficient $DSC(A,B)=2(|A|\cap |B\left|)/(\right|A\left|+\right|B|)$, where ⋂ denotes an intersection of two regions.

Data.

We have been collecting LGE-CMR images with the size of 224 × 224 from 168 heart failure patients who had CMR studies prior to undergoing the treatment of CRT. All ground truth labels, including segmentation of myocardium and scar maps involved in our experiment were manually delineated by experienced medical doctors. 510 LGE-CMR images from 140 patients are used for training, and 125 images from 28 different patients are used for testing. All images are normalized to [0, 1] before sending to the network. We generated random affine transforms, including scaling, rotating, translating and shearing, for data augmentation. While the data used in this paper is not publicly available online, upon the acceptance of this manuscript, we commit to share all images along with myocardium and scar labels if an institutional data use agreement is in place.

Network training and hyper-parameter tuning.

The joint network is trained on GPU units with Adam optimizer [16]. We set the batch size and the number of epochs to 40 and 500 respectively. We have the learning rate η, drop-out rate p, loss weight λ and L₂ weight decay weights β_M and β_S as hyper-parameters to be tuned using Bayesian optimization with Gaussian processes implemented by wandb [17] package. The empirical optimal hyper-parameters for JDL are $\widehat{\eta }=4.73e-4,p=0.39,\widehat{\lambda }=1.02$ and ${\beta }_M={\beta }_S=5.58e-6$.

Compared methods.

We compared our proposed approach with 3 other approaches: the direct segmentation approach, the two-step approach and the multi-task learning approach. In the direct segmentation approach a segmentation network is directly trained to predicted the scar segmentation mask given one LGE MR image. In the two-step approach, we train a myocardium segmentation network, and separately train a scar segmentation network that takes MR image masked by manual labeled myocardium segmentation mask. In the testing stage, we first predict the myocardium mask using the myocardium segmentation network and use them to mask the LGE MR images. Then, the masked images are sent to the scar segmentation network to generate scar segmentation mask. All the networks mentioned above use the TransUNet [13] as backbone. For the multi-task learning approach, we adopted the multi-head network structure, where the network has on shared sub-network followed by two sub-branch networks, one of which generates myocardium segmentation mask and the other generates scar segmentation mask. In the implementation, we duplicated the decoder (i.e. the upscaling convolutional layers) of the TransUNet [13] as two sub-branch networks. The loss function of (sub-branch) networks is the same with our method as Eq.1.

3.2. Results & Discussion

The comparison of Dice scores of all methods are reported in Fig. 2. It shows that our proposed JDL framework outperforms all other approaches in terms of both accuracy and robustness. Examples of segmentation results are shown in Fig. 3, which also demonstrates that our method achieves the closest scar segmentations to manual delineations (ground truth) provided by experts.

Fig. 2:

Scar Segmentation Dice score of Different Methods

Fig. 3:

A comparison of predicted scar segmentations between our method and other approaches. Left to right: input images, manually delineated ground truth labels, direct segmentation on original images, two-step approach, multitask learning (MTL), and our joint deep learning (JDL) framework.

4. DISCUSSION

4.1. Interpretation of JDL Framwork

To further understand why our JDL achieves better performance, we also computed the precision and recall of the segmentation results. For each segmentation result, they are computed as $\text{Precision}(S,\widehat{S})=(S\cap \widehat{S})/\parallel \widehat{S}\parallel$ and $\text{Recall}(S,\widehat{S})=(S\cap \widehat{S})/\parallel S\parallel$ where S and $\widehat{s}$ are the ground-truth predicted masks, respectively. The segmentation model with high precision tends to capture all possible areas and misses less, while the model with high recall tends to avoid false positive prediction and tries to ensure all the captured regions are indeed the target region. The scar segmentation precision and recall of different methods are shown in Fig. 4. While our JDL model has both the best precision and recall performance, its recall is significantly higher than all other methods, which indicates that our JDL model is less likely to wrongly report the existence of scar at a scar-free region, and this also matches the examples given in Fig.3.

Fig. 4:

Scar segmentation precision and recall of all methods

4.2. Effect on Myocardium Segmentation Task

While in general, when multiple tasks are trained jointly, the performance improvement of all tasks is expected. However, the mean Dice scores of the myocardium segmentation of the two-step approach, the MTL approach and our approach are 0.894, 0.900 and 0.891, respectively. In other words, the joint training strategy has little effect on the myocardium segmentation task, while it does improve the scar segmentation task. This raise further interpretability question to those approaches, which is crucial to build trustable algoroithms and could be the future work of the proposed JDL method.

5. CONCLUSION

In this paper, we proposed a novel joint deep neural network for an improved performance of myocardial scar detection with learned myocardium segmentations. In contrast to existing deep learning based approaches that train these tasks in a separate or parallel manner, our methods incorporates a message-passing module that effectively passes information within the subnetworks. Therefore, our model is able to explicitly use the information of learned myocardium segmentations to guide the scar detection. Meanwhile, the myocardium segmentation network receives additional supervision from the scar labels, which improves the predicted segmentation accuracy. Experimental results show that our method achieved the state-of-the-art performance on myocardial scar segmentation. Our work has great potential to impact various image-guided clinical systems, such as identifying optimal pacing sites that are scar-free for efficient treatment of CRT [11]. We commit to make code produced in this work publicly available online upon the acceptance of this manuscript.