Unsupervised Motion Tracking of Left Ventricle in Echocardiography

Abstract

Accurate motion tracking of the left ventricle is critical in detecting wall motion abnormalities in the heart after an injury such as a myocardial infarction. We propose an unsupervised motion tracking framework with physiological constraints to learn dense displacement fields between sequential pairs of 2-D B-mode echocardiography images. Current deep-learning motion-tracking algorithms require large amounts of data to provide ground-truth, which is difficult to obtain for in vivo datasets (such as patient data and animal studies), or are unsuccessful in tracking motion between echocardiographic images due to inherent ultrasound properties (such as low signal-to-noise ratio and various image artifacts). We design a U-Net inspired convolutional neural network that uses manually traced segmentations as a guide to learn displacement estimations between a source and target image without ground-truth displacement fields by minimizing the difference between a transformed source frame and the original target frame. We then penalize divergence in the displacement field in order to enforce incompressibility within the left ventricle. We demonstrate the performance of our model on synthetic and in vivo canine 2-D echocardiography datasets by comparing it against a non-rigid registration algorithm and a shape-tracking algorithm. Our results show favorable performance of our model against both methods.

1. INTRODUCTION

Cardiovascular disease is the leading cause of mortality in the United States.¹ To help in the diagnosis, echocardiography, an ultrasound imaging technique that is non-invasive and cost efficient, is widely used by physicians to study the heart function in patients. Cardiologists evaluate the motion of the myocardium as it provides valuable information into cardiac health as damaged tissue will behave differently from healthy tissue. Therefore, it is important to be able to accurately track the motion of the left ventricle in order to localize myocardial damage after an injury, such as myocardial ischemia or infarction. One popular method for modeling motion is non-rigid registration in which a lattice of control points is placed along a source image and transformed to match a target image.^2–4 Another method is shape tracking in which points sampled along the myocardial surface in one time frame are matched to points in subsequent time frames based on trajectory-likelihood and similarity metrics.^5,6 Efforts with deep learning models show promising motion estimation results using convolutional neural networks (CNN).⁷ Fischer et al. presented two CNN architectures, a single stream and a double stream architecture, that learns to estimate optical flow between pairs of images.⁷ However, their work requires ground-truth displacement fields to train the network which is difficult to obtain for medical images. In order to address this lack of ground-truth data, recent work has been done to develop an unsupervised deformable registration framework.⁸ Balakrishnan et al. developed a learning-based framework to register medical images in an unsupervised manner using a spatial transformer to morph a source image to match a target image.⁸ However, this work is focused on MR images, which has considerably higher signal-to-noise ratio (SNR) than ultrasound images. To our knowledge, there has been no success in implementing an unsupervised deep learning framework in the ultrasound domain for motion tracking.

In the work presented in this paper, we propose an unsupervised motion tracking framework using a U-Net inspired CNN architecture that learns to estimate displacement between sequential pairs of two-dimensional B-mode echocardiography image sequences. We introduce physiological constraints as regularization terms in order to enforce realistic cardiac behavior and alleviate the issues caused by low SNR. More specifically, we choose to enforce the incompressibility of the myocardium by modeling displacement as a divergence-free vector field and we choose to enforce myocardial shape by guiding displacement estimation using anatomical landmarks from manually traced left-ventricular segmentations. We demonstrate the performance of our method against a non-rigid registration based algorithm² and a shape-tracking based algorithm⁶ using two 2-D echocardiography image sequences: a synthetic dataset generated by Alessandrini et al.⁹ and an in vivo canine dataset.

2. METHODS

2.1. Network Architecture

Our goal is to estimate the motion of the left ventricle between two time frames (a source frame and a target frame) in an echocardiography sequence by finding a displacement field that maps the target frame to the source frame in an unsupervised learning manner. Inspired by the success of Balakrishnan et al in applying an unsupervised learning model to track the deformation of images in MR sequences⁸ and the success of the Ronneberger et al U-Net in analyzing medical images,¹⁰ we construct an end-to-end unsupervised network that inputs two time frames from an echocardiography sequence into a U-Net like convolutional network, which outputs a displacement field that is then inputted into a spatial transformer that uses the displacement map to transform one time frame to match the other (Fig. 1, 2). Convergence is achieved when the mean squared error between the transformed source time frame and the target time frame is minimized. We incorporate physiological constraints such as incompressibility and shape in the form of divergence-free and segmentation-based regularization on the displacement field (Section 2.2) in order to introduce biomechanical properties of the myocardial movement.

Figure 1.

Overview of the model.

Figure 2.

U-net based CNN architecture for unsupervised motion tracking.

2.2. Loss Function

Let X = (x₁ … x_n) be an echocardiography sequence comprised of n time frames and let Y = (y₁ … y_n) be the corresponding left ventricular segmentations of each time frame. There exists $N=\left(\begin{array}{l}n\hfill \\ 2\hfill \end{array}\right)$ possible pairs of possible pairs of time frames within this sequence. We define a model F that learns to calculate the displacement field U_t that characterizes the motion of the left ventricle from arbitrary time frame x to some subsequent time frame x_t such that F (x, U_t) transforms x to match x_t using bilinear interpolation. The goal is to minimize the transformed x and the target x_t (denoted as L_reg) while penalizing divergence (denoted as L_div) in the displacement field and imposing shape constraints using Y (denoted as L_seg.):

$$L=L_{reg}+{\lambda }_1{L}_{seg}+{\lambda }_2{L}_{div}$$ (1)

where λ₁ and λ₂ are weighting parameters for the regularization terms.

2.2.1. Deformation

The U-Net portion of the network seeks to find a displacement field U_t that minimizes x_t and F (x, U_t). To train the network to find the most accurate displacement field, we calculate the mean squared error between the transformed time frame, x, and the target time frame, x_t, over all time frame pairs, N. This can be described as follows:

$$L_{reg}=\frac{1}{N}\sum _{i=1}^N{\left(x_i-F\left(x,U_i\right)\right)}^2$$ (2)

2.2.2. Divergence-Free

As seen in Parajuli et al¹¹ we choose to penalize divergence as a method of enforcing incompressibility within the myocardium of the left ventricle. Penalizing divergence prevents sources and sinks in the displacement field, which are unrealistic motion patterns of the left ventricle. This penalization allows us to incorporate the biomechanical properties of the left ventricle motion. The divergence of the displacement field U can be calculated by taking the gradient as follows:

$$\nabla U=\frac{\partial U}{\partial x}+\frac{\partial U}{\partial y}$$ (3)

where the partial derivative terms are all displacements in their respective x-y directions. In the presence of sinks and sources, the magnitude of the divergence would be maximized, thus we seek to minimize the magnitude of the divergence. This regularization term can be described as follows:

$$L_{div}=\sum _{i=1}^N\nabla U_i$$ (4)

2.2.3. Segmentation

In order to incorporate a shape-based regularization term, we enforce tracking along and within the epicardial and endocardial border of the left ventricle using manually segmented masks, Y, which correspond directly to the time frame pairs being inputted into the network. The same learned displacement field which maps the time frames in X is used to transform y to match y_t by minimizing the mean square error between the transformed mask and the mask of the target time frame. This can be described as follows:

$$L_{seg}=\frac{1}{N}\sum _{i=1}^N{\left(y_i-F\left(y,U_i\right)\right)}^2$$ (5)

Thus, the complete loss function of the model can be described by combining Equations 2, 4, 5 into 1 to obtain:

$$L=\frac{1}{N}\sum _{i=1}^N{\left(x_i-F\left(x,U_i\right)\right)}^2+{\lambda }_1\frac{1}{N}\sum _{i=1}^N{\left(y_i-F\left(y,U_i\right)\right)}^2+{\lambda }_2\sum _{i=1}^N\nabla U_i$$ (6)

3. EXPERIMENTS AND RESULTS

We tested our method on two different sets of data. First, we used a synthetic echocardiography dataset developed by Alessandrini et al.⁹ The synthetic dataset contains eight volumetric cardiac sequences which is generated by incorporating realistic ultrasound features that simulated the difficulty in tracking real ultrasound image sequences as well as the electro-mechanical model of the human heart. The 8 individual sequences from the dataset simulated different physiological conditions, including 1 normal, 4 sequences with occlusions in the proximal (ladprox) and distal (laddist) left anterior descending artery, left circumflex (lcx) artery, right coronary artery (rca), and 3 sequences with dilated geometry with 1 synchronous (sync) and 2 dyssynchronous activation of the left ventricle (lbbb, lbbbsmall).

Second, we used in vivo canine datasets that included 8 canine echocardiography studies. Each study contained three different physiological conditions: baseline image of the animal, severe stenosis in the mid-left anterior descending artery, and a dobutamine-induced stress condition in the continued presence of severe stenosis. Each canine was implanted with 16 sonomicrometer crystals in the left ventricle, which were used in order to obtain information regarding location and motion of the myocardium. The motion information from the 16 sonomicrometers were used as ground-truth displacement fields for the in vivo dataset. All canine studies were acquired using a Philips iE33 scanner (Philips Medical Systems, Andover, MA) and X7–2 probe and conducted in compliance with Institutional Animal Care and Use Committee policies. We used imaging frequency that ranges from 50–60 frames per second and typically produced around 15–30 3-D volumes for each 4-dimensional sequence.

For the 8 synthetic datasets, we set up our input data to be sample number × number of channels × width × length. Each sample number represents a pair of two frames in the image sequence. For these experiments, we take a mid-cavity slice of the heart at each time frame in order to observe the maximal movements during the cardiac cycle. We train our unsupervised framework using 6 synthetic datasets, validate on 1, and test on the last remaining set. Training set has a dimension of 3672 × 2 × 128 × 128. Validation and Testing sets both have dimensions of 528 × 2 × 128 × 128.

For the 8 in vivo canine datasets with 3 sequences each, we have a total of 24 sequences. The set up was similar to the synthetic dataset where we take the mid-cavity slice in order to generate our 2-D images at each time frame. We trained our model using 18 sequences which represents 6 animal studies, validated on 3 sequences (1 animal study), and tested on 3 sequences (1 animal study). In summary, the training set has a dimension of 7058 × 2 × 128 × 128. Validation set was 913 × 2 × 128 × 128 and testing set was 738 × 2 × 128 × 128.

In order to evaluate our model, we quantitatively assess our model by calculating the root mean squared error (RMSE) of our predicted displacement fields in the x-y directions with the ground truth or sonomicrometer crystals. We compared our model against a non-rigid registration based method² and a shape-tracking based method.⁶ We further compare the effects of our regularization terms on our model by evaluating three scenarios: No constraints, segmentation regularizer (SR), and segmentation regularizer with divergence-free (SR + DF). Figure 3 and 4 show the results for one of the testing sequences for synthetic dataset and in vivo data set, respectively. We can visually see a gradual increase in accuracy in our unsupervised model when compared against the ground truth as each of the regularization term is added. We list the complete RMSE results in Table 1. For both the in vivo and synthetic studies, the SR + DF model outperformed both the SR and no constraints models. In the in vivo study, both the SR and SR + DF models outperformed the non-rigid registration and shape tracking methods. In the synthetic study, the SR and SR + DF models outperformed the non-rigid registration study, but not the shape tracking algorithm. Overall, our unsupervised model show improvements in both x- and y-displacement fields as contraints on the biomechanical properties and anatomical shape are incorporated into the loss term.

Figure 3.

Estimated displacement vector field for a normal (healthy) synthetic sequence: (a) Ground Truth, (b) Non-rigid Registration, (c) Shape Tracking, (d) Our Model (No Constraints), (e) Our Model (SR), (f) Our Model (SR + DF).

Figure 4.

Estimated displacement vector field for an LAD stenosis in vivo canine sequence: (a) Ground Truth, (b) Non-rigid Registration, (c) Shape Tracking, (d) Our Model (No Constraints), (e) Our Model (SR), (f) Our Model (SR + DF).

Table 1.

Root Mean Squared Error (RMSE) for the x and y displacements (Ux, Uy) through a sequence

Methods	Synthetic		In Vivo
	Ux	Uy	Ux	Uy
Non-rigid Registration	1.06±0.87	0.73±0.31	1.19±0.70	1.12±0.72
Shape Tracking	0.39±0.05	0.24±0.05	1.01±0.07	0.83±0.37
Our Model (No Constraints)	1.11±0.43	1.10±0.34	1.15±0.55	0.86±0.31
Our Model (SR)	0.77±0.29	0.80±0.31	0.70±0.31	0.52±0.18
Our Model (SR + DF)	0.61±0.14	0.69±0.19	0.61±0.36	0.51±0.20

4. CONCLUSION

In conclusion, our work seeks to demonstrate the feasibility of an end-to-end unsupervised learning-based framework on tracking displacement in echocardiography, specifically of the left ventricle. Our framework exploits the incompressible nature of the myocardium and the general circular shape of the left ventricle in the short axis (SAX) view in order to enforce realistic cardiac motion by incorporating them as regularization terms on the displacement field. We compare our proposed model to nonrigid registration,² which calculates a transformation matrix between two images by minimizing an objective function, and shape tracking,⁶ which focuses on tracking the motion of the epicardium and endocardium of the left ventricle as opposed to the myocardium. We also perform an evaluation on the effect of the regularization terms on overall performance. Our model with both divergence-free incompressibility and the segmentation regularization terms perform favorably against the competing methods. This shows that incorporating the biomechanical and anatomical information is critical in developing an unsupervised motion tracking in medical imaging.

It is important to note that while this work does not require any ground truth displacement fields, it does require segmentations (obtained manually or otherwise) to help guide tracking. This may serve as a practical limitation. Future work involves reducing the reliance on densely segmented data as well as expanding to 3-dimensional volume over all time frames.