Simultaneous Segmentation and Motion Estimation of Left Ventricular Myocardium in 3D Echocardiography Using Multi-task Learning

Abstract

Motion estimation and segmentation are both critical steps in identifying and assessing myocardial dysfunction, but are traditionally treated as unique tasks and solved as separate steps. However, many motion estimation techniques rely on accurate segmentations. It has been demonstrated in the computer vision and medical image analysis literature that both these tasks may be mutually beneficial when solved simultaneously. In this work, we propose a multi-task learning network that can concurrently predict volumetric segmentations of the left ventricle and estimate motion between 3D echocardiographic image pairs. The model exploits complementary latent features between the two tasks using a shared feature encoder with task-specific decoding branches. Anatomically inspired constraints are incorporated to enforce realistic motion patterns. We evaluate our proposed model on an in vivo 3D echocardiographic canine dataset. Results suggest that coupling these two tasks in a learning framework performs favorably when compared against single task learning and other alternative methods.

1. Introduction

Reductions in myocardial blood flow due to coronary artery disease (CAD) can result in myocardial ischemia or infarction and subsequent regional myocardial dysfunction. Echocardiography provides a non-invasive and cost-efficient tool for clinicians to visually analyze left ventricle (LV) wall motion and assess regional dysfunction in the myocardium. However, such a qualitative method of analysis is subjective by nature and, as a result, is prone to high intra- and interobserver variability.

Motion estimation algorithms provide an objective method for characterizing myocardial contractile function, while segmentation algorithms assist in localizing regions of ischemic tissue. Traditional approaches treat these tasks as unique steps and solve them separately, but recent efforts in the medical image analysis and computer vision fields suggest that these two tasks may be mutually beneficial when optimized simultaneously [1–3].

In this paper, we propose a multi-task deep learning network to simultaneously segment the LV and estimate its motion between time frames in a 3D echocardiographic (3DE) sequence. The main contributions of this work are as follows: 1) We introduce a novel multi-task learning architecture with residual blocks to solve both 3D motion estimation and volumetric segmentation using a weight sharing feature encoder with task-specific decoding branches; 2) We incorporate anatomically inspired constraints to encourage realistic cardiac motion estimation; 3) We apply our proposed model to 3DE sequences which typically provides additional challenges over Magnetic Resonance (MR) and Computed Tomography (CT) due to lower signal-to-noise ratio as well as over 2DE due to higher dimensionality and smaller spatial and temporal resolution.

2. Related Works

Classic model-based segmentation methods such as active shape and appearance models usually require a priori knowledge or large amounts of feature engineering to achieve adequate results [4,5]. In recent years, data-driven deep learning based approaches have shown promising results, but still suffer challenges due to inherent ultrasound image properties such as low signal-to-noise ratio and low image contrast [4,6]. This becomes increasingly problematic as cardiac motion estimation approaches often rely on accurate segmentations to act as anatomical guides for surface and shape tracking or for the placement of deformable grid points [7,8]. Errors in segmentation predictions can be propagated to motion estimations. While it is conceivable that an expert clinician can manually segment or adjust algorithm predictions prior to the motion estimation step, this is a tedious and infeasible workaround. Several deep learning based approaches have been shown to be successful in estimating motion in the computer vision field, but the difficulty in obtaining true LV motion in clinical data makes supervised approaches challenging. Unsupervised approaches which seek to maximize intensity similarity between image pairs have been successful in MR and CT, however there are still limited applications in 3DE [9–12].

In recent years, efforts have been made to combine the tasks of motion estimation and segmentation. Qin et al. [2] proposed a Siamese-style joint encoder network using a VGG-16 based architecture that demonstrates promising results when applied to 2D MR cardiac images. The work done in [13] adapts this idea to 2D echocardiography by adopting a feature bridging framework [1] with anatomically inspired constraints. This is further expanded to 3DE in [3] through the use of an iterative training approach where results from one task influence the training of the other [14]. In this work, we propose an alternative novel framework for combining motion estimation and segmentation in 3DE that uses a shared feature encoder to exploit complementary latent representations in the data, which the iterative style of [3] is not capable of doing. In addition to being able to estimate 3D motion and predict volumetric LV segmentations, our model further differs from [2] through the implementation of a 3D U-Net-style architecture [15] with residual blocks [21] for the encoding and decoding branches as opposed to a VGG-16 and FCN architecture (Fig. 1).

Fig. 1.

The proposed network and its components. (A) Motion estimation and segmentation tasks are coupled in a multi-task learning framework. (B) An overview of the residual block.

3. Methods

3.1. Motion Estimation Branch

Motion estimation algorithms aim to determine the voxel-wise displacement between two sequential images. Given a source image I_source and a target image I_target, motion estimation algorithms can be described by their formulation of the mapping function F such that F (I_source, I_target) → U_x,y,z where U_x,y,z is the displacement along the x-y-z directions. Supervised deep learning formulations of F seek to directly learn the regression between the image pairs and ground truth displacement fields. However due to the scarcity of ground truth in cardiac motion, the motion branch of our network is designed and trained in an unsupervised manner similar to the framework presented in [10], which utilizes a spatial transformation to maximize a similarity metric between a warped source image and a target image.

Our proposed motion branch is comprised of a 3D U-Net inspired architecture with a downsampling analysis path followed by an upsampling synthesis path [15,16]. Skip connections are used to branch features learned in the analysis path with features learned in the synthesis path. Our model also uses residual blocks in order to improve model performance and training efficiency [21]. The downsampling analysis path serves as a feature encoder which shares its weights with the segmentation branch. The input to the motion branch is a pair of 3D images, I_source and I_target. The branch outputs a displacement field U_x,y,z which describes the motion from I_source to I_target. The displacement field is then used to morph I_source to match I_target as described in [10]. The objective of the network is to maximize the similarity between the morphed I_source and I_target by minimizing the mean squared error between each corresponding voxel p in the two frames. This can be described as follows:

$$I_{\mathit{\text{morphed}}}=\mathcal{T}\left(I_{\mathit{\text{source}}},U_{x,y,z}\right)$$ (1)

$${\mathcal{L}}_{sim}=\frac{1}{\Omega }\sum _{p\in \Omega }{\left(I_{\mathit{\text{target}}}(p)-I_{\mathit{\text{morphed}}}(p)\right)}^2$$ (2)

where $\Omega \subset {ℝ}^3$ and $\mathcal{T}$ describes the spatial transforming operator that morphs I_source using U_x,y,z.

Anatomical Constraints.

In order to enforce realistic cardiac motion patterns, we incorporate some anatomical constraints. Cardiac motion fields should be generally smooth. That is, there should be no discontinuities or jumps within the motion field. To discourage this behavior, we penalize the L²-norm of the spatial derivatives in a manner similar to [8,10] as follows:

$${\mathcal{L}}_{\mathit{\text{smooth}}}=\frac{1}{\Omega }\sum _{p\in \Omega }{\Vert \nabla U_{x,y,z}(p)\Vert }_2^2$$ (3)

Additionally, it is expected that the LV myocardium preserves its general shape through time. In order to enforce this notion, a shape constraint is added which morphs the manual segmentation of I_source using U_x,y,z and compares it against the manual segmentation of I_target in a manner similar to [11] as follows:

$${\mathcal{L}}_{\mathit{\text{shape}}}=\left(1-\frac{2\left|S_{\mathit{\text{target}}}\cap \mathcal{T}\left(S_{\mathit{\text{source}}},U_{x,y,z}\right)\right|}{\left|S_{\mathit{\text{target}}}\right|+\left|\mathcal{T}\left(S_{\mathit{\text{source}}},U_{x,y,z}\right)\right|}\right)$$ (4)

where S_source and S_target are the manual segmentations of I_source and I_target, respectively. We can then define the full motion loss as:

$${\mathcal{L}}_{\mathit{\text{motion}}}={\lambda }_{sim}{\mathcal{L}}_{sim}+{\lambda }_{\mathit{\text{smooth}}}{\mathcal{L}}_{\mathit{\text{smooth}}}+{\lambda }_{\mathit{\text{shape}}}{\mathcal{L}}_{\mathit{\text{shape}}}$$ (5)

3.2. Segmentation Branch

The objective of segmentation is to assign labels to voxels in order to delineate objects of interest from background. The segmentation branch of our proposed model follows the same 3D U-Net architectural style of the motion branch [15]. A downsampling analysis path shares weights with the motion branch and separates to a segmentation-specific upsampling synthesis path. The goal of this branch is to minimize a combined dice and binary cross entropy loss between predicted segmentations and manual segmentations as follows:

$${\mathcal{L}}_{\mathit{\text{dice}}}=\left(1-\frac{2|S\cap M|}{\left|S\right|+\left|M\right|}\right)$$ (6)

$${\mathcal{L}}_{bce}=-(y_{i}log\left(P_i\right)+\left(1-y_i\right)\left(log\left(1-P_i\right)\right)$$ (7)

$${\mathcal{L}}_{seg}={\lambda }_{\mathit{\text{dice}}}{\mathcal{L}}_{\mathit{\text{dice}}}+{\lambda }_{bce}{\mathcal{L}}_{bce}$$ (8)

where M is the model prediction of the LV mask, S is the manually traced ground truth mask. y is a binary indicator for if a voxel is correctly labeled, and P is the predicted probability a voxel is part of the LV mask.

3.3. Shared Feature Encoder

Multi-task learning is a popular method for combining closely related tasks in a unified framework. In this work, we adopt a soft parameter sharing approach [18]. Inspired by the success of [2] on 2D MR images, we employ a similar Siamese-style model (using a 3D U-Net style architecture with residual blocks) for feature encoding by sharing the weights between the downsampling analysis path of the motion estimation and segmentation branches. These shared weights are then concatenated to each task-specific upsampling synthesis path (feature decoding), thereby using features learned in both tasks to influence the final output. In this way, the model allows each branch to exploit the complementary latent representations for each task during training. Both branches are trained simultaneously and optimized using a composite loss function, weighted by α and β:

$${\mathcal{L}}_{\mathit{\text{total}}}=\alpha {\mathcal{L}}_{\mathit{\text{motion}}}+\beta {\mathcal{L}}_{seg}$$ (9)

4. Experiments and Results

4.1. Datasets and Evaluation

In vivo studies were conducted on 8 anesthetized open-chested canines, which were each imaged under 5 conditions: healthy baseline, mild LAD stenosis, mild LAD stenosis with low-dose dobutamine (5 μ g/kg/min), moderate LAD stenosis, and moderate LAD stenosis with low-dose dobutamine [19]. Images were captured using a Philips iE33 scanner with an X7–2 probe. In total, we had 40 3D echocardiographic sequences which we then sampled into image pairs to be inputted to the network. Image pairs were sampled in a 1-to-Frame manner. In other words, for each sequence, we used the first time frame (which roughly corresponds to end-diastole) as I_source and all subsequent time frames as I_target. All experiments conducted in support of this work were approved under Institutional Animal Care and Use Committee policies.

Due to the scarcity of true cardiac motion, quantitative evaluation of motion performance is often done by comparing propagated labels [25]. In this work, we employ a similar evaluation strategy by warping the endocardial (endo) and epicardial (epi) contours of the source mask and evaluating the mean contour distance (mcd) from the expected (manually traced) target mask contours. We compare our model against a conventional motion estimation approach (denoted as Optical flow - as formulated in [20]) as well as a state-of-the-art deep learning based model (denoted as Motion only - which resembles the VoxelMorph framework described in [10]). Results displayed on Fig. 2 and Table 1a suggest that the proposed model performs favorably against the alternative methods. Wilcoxon rank sum test indicates significant increase in performance (p < 0.05) of the proposed model [24].

Fig. 2.

Motion estimations from end-disatole to end-systole for healthy baseline canines. From left to right for both (a) and (b): optical flow, motion only, proposed model.

Table 1.

Quantitative evaluation. (a) Lower mcd indicates better performance. (b) Higher Jaccard (up to 1) and lower HD indicates better performance

Methods	Endo (mm)	Epi (mm)
Optical flow	0.78 ± 0.41	1.89 ± 1.37
Motion only	0.66 ± 0.31	1.62 ± 1.20
Proposed model	0.49 ± 0.19	0.61 ± 0.32
(a) Motion estimation: mean contour distances (mcd)

Methods	Jaccard	HD (mm)
Seg only	0.61 ± 0.06	4.00 ± 0.52
Proposed model	0.71 ± 0.04	3.65 ± 0.29
(b) Segmentation: Jaccard index and Hausdorff distance (HD)

To evaluate the segmentation predictions, we compare the Jaccard index and Hausdorff distance (HD) between model predictions and manually traced segmentations [22]. We evaluate the performance of the proposed model against a segmentation specific branch without feature sharing (denoted as Segmentation only - which resembled the 3D U-Net architecture described in [15]). Additionally, since both the segmentation and motion branches of the proposed model produce segmentation predictions as either part of their main task or in support of the shape constraint, we report the average values from these predictions. Results displayed on Fig. 3 and Table 1b suggest that the proposed model performs favorably in predicting LV myocardium segmentation. Wilcoxon rank sum test indicates significant improvement in performance (p < 0.05) of the proposed for both Jaccard and HD over the segmentation only model [24].

Fig. 3.

Predicted left ventricular masks. From left to right for both (a) and (b): segmentation only, proposed model.

4.2. Implementation Details

Of the 8 canine studies, we set aside 1 entire study, which consisted of all 5 conditions, for testing. Of the 7 remaining studies, we randomly divided the image pairs such that 90% were used for training and 10% were used for validation and parameter searching. The acquired images were resampled from their native ultrasound resolutions so that each voxel corresponded to 1 mm³. During training and testing, the images were further resized to 64³ due to computational limitations. Images were resized back to 1 mm³ resolution prior to evaluation. An Adam optimizer with a learning rate of 1e⁻⁴ was used. The model was trained with a batch size of 1 for 50 epochs. Due to the small batch size, group normalization was used in place of standard batch normalization [23]. Hyperparameters and loss weights were empirically selected. The model was developed using PyTorch and trained on an NVIDIA GeForce RTX 2080 Ti. Pre and post processing were done using MATLAB 2019b.

5. Conclusion

In this paper, we proposed a novel multi-task learning architecture that can simultaneously estimate 3D motion and predict volumetric LV myocardial segmentations in 3D echocardiography. This is accomplished through a weight-sharing feature encoder that is capable of learning latent representations in the data that is mutually beneficial to both tasks. Anatomical constraints are incorporated during training in order to encourage realistic cardiac motion patterns. Evaluations using an in vivo canine dataset suggest that our model performs favorably when compared to single task learning and other alternative methods. Future work includes further evaluation such as cross-validation using our existing dataset or validation against a larger or different dataset. Furthermore, we will explore the potential clinical applications of our model in estimating cardiac strain and detecting and localizing myocardial ischemia.