Native-Resolution Myocardial Principal Eulerian Strain Mapping Using Convolutional Neural Networks and Tagged Magnetic Resonance Imaging

Abstract

Background:

Assessment of regional myocardial function at native pixel-level resolution can play a crucial role in recognizing the early signs of the decline in regional myocardial function. Extensive data processing in existing techniques limits the effective resolution and accuracy of the generated strain maps. The purpose of this study is to compute myocardial principal strain maps ϵ_p1 and ϵ_p2 from tagged MRI (tMRI) at the native image resolution using convolutional neural network (CNN) models.

Methods:

For network training, validation, and testing, realistic tMRI datasets were generated and consisted of 53,606 cine images simulating the heart, the liver, blood pool, and backgrounds, including ranges of shapes, positions, motion patterns, noise, and strain. In addition, 102 in-vivo image datasets from three healthy subjects, and three Pulmonary Arterial Hypertension patients, were acquired and used to assess the network's in-vivo performance.

Four convolutional neural networks were trained for mapping input tagging patterns to corresponding ground-truth principal strains using different cost functions. Strain maps using harmonic phase analysis (HARP) were obtained with various spectral filtering settings for comparison. CNN and HARP strain maps were compared at the pixel level versus the ground-truth and versus the least-loss in-vivo maps using Pearson correlation coefficients (R) and the median error and Inter-Quartile Range (IQR) histograms.

Results:

CNN-based strain maps at a phantom resolution of 1.1mm×1.1mm and in-vivo resolution of 2.1mm×1.6mm were artifact-free with multiple fold improvement with ϵ_p1 ground-truth median error of 0.009(0.007) vs. 0.32(0.385) using HARP and ϵ_p2 ground-truth error of 0.016(0.021) vs. 0.181(0.08) using HARP. CNN-based strain maps showed substantially higher agreement with the ground-truth maps with correlation coefficients R>0.91 for ϵ_p1 and ϵ_p2 compared to R<0.21 and R<0.82 for HARP-generated maps, respectively.

Conclusion:

CNN-generated Eulerian strain mapping permits artifact-free visualization of myocardial function at the native image resolution.

Graphical Abstract

1. Introduction

Regional myocardial dysfunction is a primary hallmark of diffused myocardial and focal ischemic heart diseases; the primary causes of death from cardiovascular disease worldwide [1, 2]. Heart disease, such as myocardial ischemia, secondary to coronary artery disease, may be identified and localized by analyzing the cardiac contractile function [2]. Quantitative assessment is essential for diagnostic evaluation and management of disease severity. These procedures employ regional parameters that include cardiac deformation analysis and mechanical tissue strain quantification in different regions of the heart[3-5].

Tagged Magnetic Resonance Imaging (tMRI) has been extensively used to assess regional tissue deformation in many organs, primarily myocardial mechanical function [5, 6]. Spatially selective saturation pulses are utilized for tissue labeling to create single or multidimensional patterns of signal voids that, when placed on the image at end-diastole, persist throughout the cardiac cycle [2]. This tissue-labeled magnetization pattern is then tracked by acquiring a cine MRI to monitor the tissue's deformation.

Although tMRI provides images reflecting myocardial motion, extracting a reliable artifact-free dense motion field map from these images has been a challenge for decades. Several postprocessing methods, such as model-based, optical flow-based, and filter-based estimation, have been developed in order to capture this motion and deformation field, leading to the quantification of the myocardial strain[3, 7].

Model-based techniques track tag-pattern features using specific underlying motion or deformation model assumptions. Examples of modeling left ventricular deformation include geometrical models [1], finite element deformable meshes [8], three-dimensional (3-D) B-spline analysis [9], and multidimensional stochastic models [10]. Most model-based techniques require manual feature and parameter selection. Moreover, due to discontinuity, interpolation is usually employed to achieve dense motion estimation. Finally, such manual intervention requirements are considered tedious and time-consuming tasks that limit the techniques' generalizability and clinical applicability [11].

Optical flow methods employ minimization of various energy functions where the image gradients and temporal derivatives are used to track tagging lines or grids. The deformed lines are interpolated to create the displacement and strain field maps. These methods cannot measure large motions such as those occurring between end-diastole and end-systole. However, motion fields between subsequent image timeframes are calculated to measure more extensive multi-frame displacements [12].

Bandpass filtering methods in the spatial domain, including Gabor filter-bank methods, employ a set of bandpass filters of different shapes and orientations to extract local tag spacings and directions from which displacement and strain maps are calculated. The main disadvantage of using spatial bandpass filter-based analyses is the unavoidable over-smoothing that occurs in the spatial filtering application. This smoothing reduces the effective or the Fourier resolution of the estimated myocardial displacement and strain maps [3, 13-17].

Harmonic phase (HARP) analysis is a spectral domain-based bandpass filtering method to compute regional myocardial strain and track cardiac motion with minimal manual intervention. This method relies on computing the local myocardial tissue displacement encoded in the phase map of the tMRI spectral harmonic peak [7, 11]. Like other filter-based methods, the resolution and bandwidth of the HARP-based strain maps are limited by the size and shape of the filters used to localize the harmonic spectra [11, 15, 16].

The performance of filter-based methods is therefore critically sensitive to the filtering parameters. While narrowband filters will result in underestimated strain maps, broader-band filters will be more susceptible to strain overestimation and more artifacts, particularly near the edges.

Another challenge in conventional tMRI analysis methods for reliable displacement and strain mapping is their poor adaptation to imaging sequence-dependent time-varying tagging signal and contrast to noise ratios (SNR) and (CNR). Additionally, various MRI artifacts, including motion artifacts, T1-dependent signal fading, magnetic field inhomogeneity, and inter-organ susceptibility, often lead to severely artifactual or even failed strain calculations [18, 19].

Recently, Deep Learning (DL) techniques have shown effectiveness in medical image analysis, as they can extract, represent, and integrate highly semantic features without manual intervention. These DL techniques are rapidly gaining importance in different fields such as organ segmentation [20-22], classification [22], image enhancement [23, 24], Antibiotic discovery [25], cardiomyopathy detection [26], and mortality prediction in Critical care [27]. The main advantage of DL-based techniques is their ability to learn underlying data features in an unsupervised manner meaning that they do not need any handcrafting of discriminative features [25]. Moreover, DL systems have been highly successful due to the availability of sizeable high-quality training datasets, in addition to low-cost, powerful graphics processing unit (GPU) hardware and open-source development libraries.

Convolutional Neural Network (CNN) is a class of deep learning neural networks that specializes in learning a set of multidimensional convolutional features maps and utilizes these feature maps in identifying the association between input and output training sets [26]. CNN-based algorithms have been employed in multiple medical imaging applications, including heart localization [21, 28], segmentation [29, 30], tissue characterization [31, 32], and landmark tracking [28, 33-35].

To our knowledge, DL algorithms have not been used before to determine native pixel-level resolution strain maps. DL has been recently utilized by Ferdian et al. [28] as a preprocessing tool to extract the specific landmarks at which the strain values are calculated. The process is limited to the short-axis view of the heart and cannot be generalized to other views or other organs. Moreover, the algorithm cannot estimate the native resolution maps since it only calculates the circumferential and radial strains at limited landmark locations.

This paper proposes a local-patch CNN-based framework (CNN) to estimate the myocardial pixel-to-pixel native-resolution principal strain maps using the vertical and horizontal tMRI images. The proposed framework overcomes the previously mentioned shortcomings of existing tMRI strain mapping methods.

For CNN training and performance assessment, we developed Monte-Carlo simulated tMRI cine datasets of the left ventricle and the liver with a wide range of realistic geometry, noise, anatomical background, and MRI parameters variations. The CNN loss function was formulated to minimize the error in regions with zero and large strain magnitudes. Moreover, the in-vivo performance of the proposed approach is demonstrated in tMRI datasets of a group of 6 healthy subjects and pulmonary hypertension patients.

2. Theory

2.1. Strain Estimation using tMRI

In line-tag tMRI, strain is calculated at timeframe t based on the amount of deformation occurring in the applied perpendicular sinusoidal line tags with the tagging frequencies (ω_d).

The tagging lines in direction d at time t=0 are given by:

$$T_d(X)=\cos ({\omega }_{d}X{n}_d),$$ (1)

where n_d is a unit vector in the direction d. The tagging image I_d can be calculated using equation (2) as follows.

$$I_d(x,t)=I_o(x,t)[A_d(x,t)+A_T(x,t)T_d(x,t)]e^{j{\phi }_e(x,t)},$$ (2)

where I_o is the intensity of the anatomical image and φ_e(x, t) is the artifactual inhomogeneity phase. A_d(x, t) and A_T(x, t) are the effects of tag-fading relaxation time T₁ on the effective spin density (I_o) and tagging pattern (T_d), respectively.

In the short-axis orientation, myocardial strain is typically represented by the circumferential and radial strain components. Circumferential strain represents myocardial shortening in the short axis plane in a direction tangential to the epicardial surface. In contrast, radial strain represents myocardial thickening in the radial direction, perpendicular to the wall and towards the ventricle center. Negative circumferential strain approximates the macroscopic myocyte contraction while positive radial strain indicates myocardium wall thickening, and negative radial strain during this period implies local wall thinning [11]. Meanwhile, myocardial long-axis strain is typically represented by radial strain and longitudinal shortening.

The principal strains ϵ_p1 and ϵ_p2 represent the maximum normal expansion and contraction strains at a given point, respectively, where the shear strain is zero. Principal strains are a more accurate representation of myocardial tissue deformation because they do not depend in their estimation on the characteristic ring shape of the myocardial cross-section that is only approximately realistic in healthy subjects with the perfect short-axis image positioning. More generally, principal strains can be estimated in any tissue of arbitrary shape and orientation. ϵ_p1 and ϵ_p2 are calculated by solving the eigenvalue problem of the strain tensor where the associated eigenvalues are the amount of strain in the direction of maximum stretch (ϵ_p1) and maximum shortening (ϵ_p2) at that point, and the eigenvectors representing the directions of ϵ_p1 and ϵ_p2 [7, 36].

2.2. Convolutional Neural Networks

CNN are feed-forward artificial neural networks for deep feature learning [37-39] inspired by the natural vision process performed within the visual cortex in which the receptive fields of the sensory neurons perceive and process overlapping regions in the visual field. A typical CNN comprises multiple layers, consisting of a sequence of alternating convolutional layers and subsampling layers followed by a fully connected layer. The network also includes pooling layers, employing a spatial down-sampling. Though, the network gains a shift-invariance property. Dropout layers and regularization were introduced to decrease the overfitting problem that may occur due to the training dataset's limited size. Batch normalization layers are employed to standardize the inputs to the next convolution layer, dramatically reducing the number of training epochs required to train deep networks. The rectified linear unit (ReLU) activation function layer utilizes a rectifier function to add non-linearity in the processing of the input features and to overcome the vanishing gradient problem seen in deep networks. Finally, a fully connected layer is used immediately after the last subsampling layer to perform classification or regression tasks.

3. Materials And Methods

All in-vivo scans were approved by the local institutional review board (IRB) with necessary informed consent obtained from all subjects before the scans.

3.1. Myocardial tMRI Data Modeling

For the CNN to determine the strain maps, it needs to learn this association using tagged datasets with their ground-truth principal strain values. To achieve that goal, we developed a numerical Monte-Carlo motion phantom to simulate realistic tMRI cine images of the heart. Phantom tagged images combined with corresponding ground-truth strain maps represented the labeled datasets utilized for network learning and testing.

Two moving objects, namely the LV and the liver, were simulated within an anatomical image, where a background image I_g was included. Each simulation contained a unique I_g randomly selected and rotated from a pool of background images. This pool comprised 250 images, previously acquired from 6 different subjects, using 3-Tesla MRI 3D- TFE or cine b-FFE sequences. The LV was simulated as a complete annulus with a center, initial inner and outer radii, named (${\mathit{c}}_{\mathit{x}}^{\mathit{h}}$, ${\mathit{c}}_{\mathit{y}}^{\mathit{h}}$), ${\mathit{r}}_0^{\mathit{ih}}$, and ${\mathit{r}}_0^{\mathit{oh}}$, respectively. The liver was simulated as an incomplete annulus sector with its center, initial inner and outer radii, named (${\mathit{c}}_{\mathit{x}}^{\mathit{l}}$, ${\mathit{c}}_{\mathit{y}}^{\mathit{l}}$), ${\mathit{r}}_0^{\mathit{il}}$, and ${\mathit{r}}_0^{\mathit{ol}}$, respectively. The liver was positioned with an angular position θ₁, and width θ₂, as shown in Figure 1. The regions around the LV and inside its lumen were simulated as static tissue with rapid intensity fading factors to mimic the surrounding epicardial fat and the washed-out blood in the blood pool. The values of all anatomical and imaging parameters were randomly chosen from their corresponding pool of potential values, as shown in Figure 1.

Figure 1:

Simulated left ventricle and liver shapes, the pool of values of the geometric and functional parameters, and their simulated tMRI sequence and image quality parameters.

The reference time, representing end-diastole when the tissue of region k expresses no motion or strain, occurs at the first time-point t=0. During systole, the annulus' deformation follows a radial shortening, wherein the displacement of a material point p, at radius $r_n^k$ from its original radius r₀ at a specific timeframe t=n can be calculated using equation (3),

$$r_n^k\left(r_0^k\right)=\sqrt{r_0^{k^2}+\left(2{\epsilon }_n^k+{\epsilon }_n^{k^2}\right)r_0^{i{k}^2}}.$$ (3)

Here, $r_0^{ik}$ is the inner radius of the annulus k at timeframe t=0, and ${\epsilon }_n^k$ is the circumferential principal strain along the annulus' innermost rim at timeframe t=n such that ${\epsilon }_n^k=\left(r_n^{ik}-r_0^{ik}\right)∕r_0^{ik}$.

Given a thin tissue ring of an initial radius $r_0^k$ and thickness of $d{r}_0^k$, its area before and after deformation can be expressed as $d{A}_0^k=2\pi r_0^{k}d{r}_0^k$, and $d{A}_n^k=2\pi r_n^{k}d{r}_n^k$, respectively. Assuming the ring is incompressible, then $d{A}_n^k=d{A}_0^k$. Consequently, $d{r}_0^k∕d{r}_n^k=r_n^k∕r_0^k$ and the principal strain along the ring's circumference; ${\epsilon }_{p2}^k(n,r_n^k)$, or simply ${\epsilon }_{p2}^k$, is defined by:

$${\epsilon }_{p2}^k\stackrel{\text{def}}{=}\frac{(r_n^k-r_0^k)}{r_0^k}.$$ (4)

Similarly, the principal strain across the ring ${\epsilon }_{p1}^k(n,r_n^k)$, or simply ${\epsilon }_{p1}^k$ is defined by:

$${\epsilon }_{p1}^k\stackrel{\text{def}}{=}\frac{(d{r}_n^k-d{r}_0^k)}{d{r}_0^k},$$

or,

$${\epsilon }_{p1}^k=\frac{(r_0^k-r_n^k)}{r_n^k}.$$ (5)

At the n^th time frame and radius $r_n^k$, the two-dimensional incompressibility relation between the two principal strains, can be represented by,

$${\epsilon }_{p2}^k=\frac{-{\epsilon }_{p1}^k}{1+{\epsilon }_{p1}^k}.$$ (6)

Based on the simulation geometry and strain choices, Eq. (3) was used to create the full-image displacement field map at the n^th time frame ${\mathit{u}}_n^k$. Eqs. (4) and (5) were used to create the underlying ground-truth principal strain maps. The corresponding n^th time frame vertical and horizontal tagged images were created using Eqs. (7) and (8), respectively, where their original location at 0^th time frame was calculated using Eq. (3).

$$\begin{gathered} I_{vert}(n)=\sum _{k\in \{g,h,l,b\}}{I}_k\left[2+(1-{\mathcal{B}}_k{)}^{n-1}\left[\cos \left(\omega \left(x-u_{x_k}(n)\right)\right)-1\right]\right]\times \\ \left(1+{\mathcal{S}}_x(n)\right)+{\mathcal{N}}_x(n), \end{gathered}$$ (7)

and

$$\begin{gathered} I_{horiz}(n)=\sum _{k\in \{g,h,l,b\}}{I}_k\left[2+(1-{\mathcal{B}}_k{)}^{n-1}\left[\cos \left(\omega \left(y-u_{y_k}(n)\right)\right)-1\right]\right]\times \\ \left(1+{\mathcal{S}}_y(n)\right)+{\mathcal{N}}_y(n), \end{gathered}$$ (8)

where k is the index of the specific organ mask or region, represented as either the background g, left ventricle h, liver l, or blood pool b, $\mathcal{S}$ is the multiplicative speckle noise [40], I_g is the background image or the object mask, $\mathcal{B}$ is the tag fading factor, u_x and u_y are the displacement fields across the vertical and horizontal tag directions, respectively, and $\mathcal{N}$ is the additive white noise. Figure 2 shows the process of generating the simulated datasets, where the first row shows the input images I_g, I_b, I_h, I_l that are randomly generated using the criteria mentioned earlier. The second row shows the different regions after including speckle noise. Regions were then combined to form the anatomical images, wherein the displacement fields were included, and tags were applied to form the horizontal and vertical tagging images along with the ground-truth strain. Finally, white noise maps were added to the tagged images.

Figure 2:

The steps of creating a simulated tMRI cine dataset. First, a background image is randomly picked and rotated. Liver and left ventricle are created using randomized geometrical and deformation properties. Second, the corresponding ground-truth principal stain maps ϵ_p1 and ϵ_p2 are generated. Third, multiplicative speckle noise is added, and the cine anatomical image is created. Fourth, cine tagged patterns and white noise are superimposed based on the formerly prescribed deformation pattern.

3.2. Tagging MRI Data Acquisition

The in-vivo datasets comprised 102 images from six adult subjects, including healthy and pulmonary hypertension patients. A standard line sinusoidal 1-1 SPAMM gradient echo (GRE) tagging sequence was applied using a 3.0T Siemens Magnetom Verio scanner (Siemens Medical Solutions, Erlangen, Germany). Each subject underwent SPAMM scans of the short axis view of the heart, in which horizontal and vertical tags were applied separately with TR/TE = 42/2.51 ms, 13° RF excitation flip angle, image resolution of 2.1 mm×1.6 mm×7 mm, reconstructed matrix size of 192 x 108 mm, tag spacing of 8.0 mm, eight k-space segments per image, and 17 heart phases were acquired per cine.

In this study, the network learned tagging-to-strain mapping using the simulated datasets with 8 mm tag spacing and 1.1 mm pixel spacing. The tag spacing was kept the same in in-vivo imaging, at 8 mm.

3.3. CNN Strain Mapping

We propose employing localized patch-level learning rather than whole image level learning to decrease network size in terms of the number of neurons and filters used within each layer. The input to the network was set to 21×21×2, with the third channel dimension representing the extracted patch for both the vertical and horizontal tags. The patch size was selected as such to include at least three tag lines exhibiting minimum strain to provide the network with a significant number of lines describing the deformation due to the heart motion. The proposed network comprises four convolutional layers connected to three fully connected layers. The filter sizes used in the convolutional layers were 7×7 for the first and second layers and 5×5 for the third and fourth layers. Three fully connected layers were then employed before the final regression layer. Within each convolutional layer, batch normalization and nonlinear activation were applied. Dropout was not employed in the third and fourth layers in order to achieve a tagging translation-insensitive architecture, which improved strain estimation accuracy. The tanh function was used in the decision layer to allow for both positive and negative output needed for stretching and contraction strain estimation, and the Adam optimizer was used to accelerate training convergence with a learning rate of 0.1 [41].

Three different cost functions were investigated; mean squared error (MSE), mean absolute error (MAE), and weighted mean squared error (WMSE). The latter WMSE function differentiates the estimated strain calculated for zero strain patches and non-zero strain patches. The different cost functions were defined accordingly as in the following Eqs. (9).

$$\begin{gathered} MS{E}_i=\frac{1}{n}{\sum }_{j=1}^n{\left({ϵ}_{Pij}^T-{ϵ}_{Pij}^E\right)}^2, \\ MA{E}_i=\frac{1}{n}{\sum }_{j=1}^n\mid {ϵ}_{Pij}^T-{ϵ}_{Pij}^E\mid , \\ WMS{E}_i({\sigma }^2)=\frac{1}{n}{\sum }_{j=1}^n\{({ϵ}_{Pij}^T-{ϵ}_{Pij}^E{)}^2\times \phantom{\}} \\ \phantom{\{}{e}^{-\frac{{\left({ϵ}_{Pij}^T\right)}^2}{{\sigma }^2}}\}. \end{gathered}$$ (9)

Here, ${ϵ}_{pi}^T$ is the ground-truth i^th principal strain, and ${ϵ}_{Pi}^E$ is the estimated i^th principal strain, with i= (1, 2). WMSE was designed to specifically reduce low-strain errors through the weighting part that is exponentially inversely proportional to the ground-truth of the strain. In the case of zero strain, the exponent term becomes one, and as the absolute strain increases, the contribution of the exponential term to the overall error estimation decreases. Two weighted error functions, WMSE₁ and WMSE₂, were tested with different variances: ${\sigma }_1^2=0.05$ and ${\sigma }_2^2=0.1$, respectively.

3.4. CNN Training

A total of 391,051 simulated timeframes were used to extract the patches used for network training. To prevent the network from converging towards minimizing the error in the abundant zero strain regions, we kept the ratio between zero and non-zero strain patches in the training sets to a 1:1 ratio. As for testing, 200 random timeframes were set aside to evaluate network performance throughout the entire ROI, including the heart and liver regions.

Batch learning was utilized with a batch size of 2000 per epoch for 100 epochs and early stopping to prevent overfitting. The ground-truth strain for each image patch was set to the ground-truth strain value at the center of that patch.

The experiments were carried out on a Linux Ubuntu 18.0 Intel Core i9 3.3 GHz workstation using the Keras Python ML library on a Nvidia Quadro RTX 5000 graphics card with a memory size of 16 GB.

3.5. Statistical Analysis

The performance of the proposed CNN principal stain mapper using the four different loss functions, as mentioned earlier, was evaluated and compared to the performance of HARP analysis. For HARP, Fourier-domain bandpass filtering was centered at the tagging-induced positive spectral harmonic peak with six different full widths at half maximum (FWHM) sized filters of 8, 12, 16, …, and 28 pixels, named as HARP₈, HARP₁₂, HARP₁₆, …, HARP₂₈, respectively. These filter widths corresponded to Fourier resolutions Δ_f = 35, 23, 17.5, …, and 10 mm, respectively, according to the following formula,

$${\Delta }_f(inmm)=\frac{FOV(inmm)}{FilterFWHM(inpixels)}.$$ (10)

Phantom comparisons involved 200 simulated cines with 17 frames/cine covering the cardiac cycle's systole period. Principal stain maps ϵ_p1 and ϵ_p2 estimated using CNN and HARP were compared to the ground-truth maps on a per-pixel basis, and correlation and scatter plots were generated. To evaluate the bias and variance in the estimation, error histogram bar graphs were created. The whole strain range [−0.5 to 0.5] was evenly split into 20 bands, and the median and Inter-Quartile Range (IQR) were then calculated for each band. Similarly, stain mapping results were also compared against the best-performing CNN method.

In-vivo comparisons proceeded similarly to their simulation counterparts except that there was no ground-truth map to use as a reference. Therefore, all techniques were compared against the CNN method that showed the lowest loss during the testing.

4. Results

All four CNN loss functions examined in this study, namely MSE, MAE, WMSE₁, and WMSE₂, converged successfully during training. This section will demonstrate results qualitatively, followed by the quantitative assessment.

4.1. Qualitative Assessment

Overview examples of the outcome from the simulated testing dataset, healthy, and PAH subjects are demonstrated in Figure 3, Figure 4, and Figure 5, respectively. Animated cine representations can be found in the included supplementary video clips. Part (a) of these figures show the vertically and horizontally tagged images at end-diastole; the time of no strain, as well as the corresponding images at end-systole; the time of peak strain, together with zoomed-in ROIs of the heart and liver regions. Straight and curved tag lines at end-systole and end-diastole, respectively, indicate the amount of strain exerted. Figure 3b, Figure 4b, and Figure 5b show the different HARP principal strains ϵ_p1 and ϵ_p2 at end-systole. Figure 3b also includes the simulated data ground-truth principal strains. HARP-generated strains exhibit that narrower bandpass filtering as in HARP₈ produces smooth yet severely underestimated strain maps. As the filter size increases, we realize the strain's spatial details increase as we move from the endocardium to the epicardium. However, that level of detail is associated with an increasing amount of artifacts, as in the case of HARP₂₄. This trend is evident in both the phantom shown in Figure 3, and in the healthy and PAH in-vivo cases shown in Figure 4 and Figure 5.

Figure 3:

Simulated dataset example; (a) Tagging MR cine images including horizontally and vertically tagged images on the left ROIs showing the simulated left ventricle and liver at end-diastole and end-systole time-points. (b) The ground-truth principal strains ϵ_p1 and ϵ_p2 at end-systole, on the left, followed by the calculated principal strain maps using HARP with different filter sizes (8×8), (16×16), and (24×24). (c) The estimated principal strain maps obtained using CNN using four different loss functions, MSE, MAE, and weighted MSE; WMSE₁ and WMSE₂, with standard deviations 0.1 and 0.05, respectively. * WMSE₂ is the method with the least testing error and highest correlation. A more comprehensive animated version of this figure can be found in the supplementary clip supplV01_Phantom.mp4.

Figure 4:

Normal adult tMRI example. (a) Vertically and horizontally tagged end-diastole and end-systole images as well as ROIs around the left ventricle and the liver. (b) Principal strain maps obtained using HARP with different filter sizes (8×8), (16×16), and (24×24). (c) Principal strain maps obtained using CNNs, separately utilizing the four different loss functions, MSE, MAE, and weighted MSE, WMSE₁ and WMSE₂ with standard deviations 0.1 and 0.05, respectively. Notice in CNN methods the clear sharp RV and LV boundaries and the absence of the artifacts that keep worsening with larger HARP filter sizes. * WMSE₂ is the method with the least testing error and highest correlation. A more comprehensive animated version of this figure can be found in the supplementary clip supplV02_Healthy.mp4.

Figure 5:

PAH patient example. (a) Vertically and horizontally tagged end-diastole and end-systole images as well as a zoomed-in ROIs including the right and left ventricles. (b) Calculated principal strain maps using HARP with different filter sizes (8×8), (16×16), and (24×24) demonstrating the strain underestimation at small filter sizes versus the maps with severe artifacts as filter size increases. (c) principal strains ϵ_p1 and ϵ_p2 using the proposed CNN method with four different loss functions, MSE, MAE, and weighted MSE; WMSE₁ and WMSE₂ with standard deviations 0.1 and 0.05, respectively. The red arrow in (a) points to the anterior insertion point at which CNN method successfully demonstrates the lack of strain, which is common in PAH. *WMSE₂ is the method with the least testing error and highest correlation. A more comprehensive animated version of this figure can be found in the supplementary clip supplV03_PAH.mp4.

In comparison, the corresponding CNN-generated strain maps with different loss functions are demonstrated in Figure 3c, Figure 4c, and Figure 5c, respectively. Notice in Figure 3c the substantial improvement in the amount of detail and clarity, as well as the close pixel-by-pixel resemblance to the ground-truth. In the in-vivo cases, shown in Figure 4 and Figure 5, CNN -based methods were capable of producing strain maps of conspicuous clarity and detail, particularly evident in the PAH case, in Figure 5 even at the borders of the organs where all HARP schemes failed. As shown later, CNN-WMSE₂ had the overall best performance among the four experimented loss functions and will thus be used for in-vivo comparisons.

Full cine maps using CNN-WMSE₂ compared to those of HARP₁₆, are shown in Figure 6 for a phantom case and in Figure 7 and Figure 8 for the healthy and PAH subjects, respectively. Comprehensive animated cine results are also available in the online video clips. Results reveal the following enhancements. The first is high strain accuracy. As shown in Figure 6, one can appreciate the almost exact resemblance of the artifact-free CNN-WMSE₂ maps to the ground-truth at all cine frames. In contrast, HARP₁₆ maps suffer from a substantial deviation from the ground-truth and an increasing amount of artifact, particularly in the ϵ_p1 maps. The second enhancement is the remarkable sensitivity to temporal strain variations.

Figure 6:

Phantom dataset principal strain maps, calculated using HARP₁₆ and CNN-WMSE₂, compared to the ground-truth maps. (a) Selected tMRI cine frames from end-diastole to end-systole. (b) The corresponding ground-truth principal strain maps ϵ_p1 and ϵ_p2. (c) and (d) show ϵ_p1 and ϵ_p2 using HARP₁₆ and CNN-WMSE₂, respectively. The full cine can be viewed in the supplementary media

Figure 7:

In-vivo principal strain maps calculated using HARP₁₆ and CNN-WMSE₂ for a healthy adult subject. Selected tMRI frames within a single cardiac cycle from systole to after end-systole are shown in (a). (b) and (c) show the corresponding ϵ_p1 and ϵ_p2 using HARP₁₆ and CNN-WMSE₂, respectively. The full cine can be viewed in the supplementary media

Figure 8:

In-vivo principal strain maps calculated using HARP₁₆ and CNN-WMSE₂ for a PAH adult patient. (a) The tMRI cine at selected frames. (b) and (c) The corresponding ϵ_p1 and ϵ_p2 maps using HARP₁₆ and CNN-WMSE₂, respectively. The full cine can be viewed in the supplementary media

Figure 7 shows an example of a CNN-based maps' ability to track and visualize both the anatomy and the temporal progression of strain throughout the cardiac cycle in both the LV and right ventricle (RV).

As expected, HARP₁₆ is only suitable for lower resolution strain quantification in large regions, which was the original premise and typical utility of HARP analysis. The third enhancement provided by CNN strain mapping is the preservation of organ boundaries. As shown in Figure 8 and previous figures, the boundaries of RV and LV are entirely distinguished. Unlike in the normal subjects, strain wave propagation clearly demonstrates the desynchrony between LV and RV. The strain propagation wave is seen to be delayed in RV, stronger, and longer-living than in LV, while HARP analysis hardly showed these subtle anatomical and functional characteristics.

In all, these results provide essential insights into the quality of CNN-based strain maps compared to HARP as a representative of the bandpass filtering methods. The results also lay the foundations for the following quantitative assessment.

4.2. Quantitative Assessment

The bar-graphs in Figure 9, left column from top to bottom, show the error of HARP₈, HARP₁₆, HARP₂₄, CNN-MSE, CNN-WMSE₁, and CNN-WMSE₂, respectively, compared to the ground-truth for the simulation test patches. Error bar-graphs help visualize the accuracy and precision of each of the tested methods. First, the bias seen in the bar-graphs is an indicator of a method's strain mapping level of inaccuracy or how far the estimates deviate from the ground-truth. In low strain ranges, the median absolute error is lower using HARP₈ compared to HARP₁₆ and HARP₂₄, while in high strain bands, it is the opposite; HARP₈ introduces substantially larger errors. This finding indicates information loss and, consequently, strain underestimation due to the small filter size used in HARP₈. Likewise, IQR lines represent the fluctuations and artifacts in every strain band and are considered an indicator of strain mapping precision. As expected, HARP₈ had the smallest IQRs among all HARP methods: the narrower the filter, the smoother the maps, and the lesser the rippling or fluctuational artifacts.

Figure 9:

Bar-graphs showing the median and interquartile range (IQR) of the median error in the HARP and CNN strain maps ϵ_p1(red bars) and ϵ_p2 (blue bars). Each error bar represents a strain range of 0.05. Columns (a) and (b) are for the simulated data, where column (a) shows the error in the various methods compared to the ground-truth and column (b) shows the error in the same methods compared to CNN-WMSE₂. Column (c) shows the difference between the strain estimated by the different methods compared to that estimated using CNN-WMSE₂ for the in-vivo dataset. The top three rows compare the HARP-based methods, whereas the bottom three rows correspond to the CNN-based methods, in which the vertical axes ranged from 0 to 0.2 instead of 0.5 to magnify the differences.

In contrast, all CNN methods, as shown in the bottom three bar-graphs of Figure 9a, had significantly less estimate bias (error bars) and fluctuations (IQRs). Note that the HARP plot's vertical axes ranged from 0 to 0.5, while for the CNN methods, the axes ranged from 0 to 0.2 only. The CNN-WMSE₂ scheme had the smallest error and artifacts among CNN methods, as evident from Figure 9a. When CNN-WMSE₂ was used as the reference for comparison, as shown in Figure 9b, strain accuracy and precision trends were similar to those in Figure 9a, in which methods were compared to the ground-truth. The similarities of the results between Figure 9b and Figure 9a are another indication of the strong resemblance between CNN-WMSE₂ and the ground-truth and will help in interpreting the in-vivo results in Figure 9c.

Figure 9c shows the in-vivo comparisons of strain mapping methods against CNN-WMSE₂. As shown in Figure 9b, HARP₈ had the highest precision in all strain ranges and the least error in the low strain range, while the error increased substantially at high strain values compared to HARP₁₆ and HARP₂₄. Although the mean error in the case of the in-vivo dataset is higher than that for the simulated test data, the maximum error is 10% in the case of CNN-MSE compared to the maximum error in HARP₂₄.

The scatter and correlation diagrams are illustrated, in Figure 10, to demonstrate the relationship between different methods against the ground-truth and CNN-WMSE₂ strain maps.

Figure 10:

Scatter-plots and correlation coefficients of the pixel-to-pixel strain maps ϵ_p1(red) and ϵ_p2 (blue) calculated using various HARP and CNN based methods versus ground-truth and best-performing CNN method.

Columns (a), (b), (d), and (e) are for simulated data points while columns (c) and (f) display in-vivo data points. Columns (a-c) compare the HARP-based methods to the ground-truth (a) and to CNN-WMSE₂ in (b) and (c), whereas columns (d-f) correspond to the CNN-based methods. Columns (d-f) compare the CNN-based methods to the ground-truth (d) and to CNN-WMSE₂ in (e) and (f).

Columns (a) and (d) plot the simulated data estimated strain against the ground-truth. Columns (b) and (e) plot the phantom results using various methods against the CNN-WMSE₂. Columns (c) and (f) plot the in-vivo results using CNN-MSE, and CNN-WMSE₁ versus CNN-WMSE₂.

Figure 10a shows scatterplots comparing the calculated strains using HARP with different filter sizes versus the ground-truth strains, using the test data. Red and blue points in the plots as well as red and blue colored correlation coefficient (R) measures correspond to ϵ_p1 and ϵ_p2, respectively. The most striking observation in HARP methods is that the scattering in ϵ_p1 is much worse than in ϵ_p2. This pattern is also evident in the correlation coefficients and is because ϵ_p1 direction is across the myocardium and, therefore, more prone to inter-boundary ripple and other artifacts while the ϵ_p2 direction is predominantly along the circumference.

The corresponding CNN comparisons are demonstrated in Figure 10(d-f). Points of CNN methods are distributed along the diagonal 45° unity correlation line with minimal scattering. The correlation coefficients of the estimated ϵ_p1 to the ground-truth are 0.99, 0.9, and 0.989 for CNN-MSE, CNN-WMSE₁, and CNN-WMSE₂, respectively. The correlation coefficients of the estimated ϵ_p2 to the ground-truth are 0.981, 0.988, and 0.993 for CNN-MSE, CNN-WMSE₁, and CNN-WMSE₂, respectively. Based on these findings, CNN-WMSE₂ is considered the best performing CNN scheme. Correlation coefficients are significantly improved over the HARP methods, with the best correlations are of HARP₁₆: R=0.819 for ϵ_p2 and only 0.214 for ϵ_p1.

Further comparisons of HARP and CNN methods against CNN-WMSE₂ are demonstrated in Figure 10b and Figure 10e, respectively, revealing a remarkable resemblance of CNN-WMSE₂ strain estimates to that of the ground-truth in both the pattern of scattering and correlation to other techniques.

Figure 10c and Figure 10f show in-vivo scatterplots of HARP with different filter sizes and CNN with different loss functions versus the CNN-WMSE₂, respectively. As expected, scattering worsened, and correlation decreased in all methods compared to the simulated test comparisons in Figure 10b and Figure 10e. The deterioration of HARP performance was so staggering as the correlation coefficients almost halved for both ϵ_p1 and ϵ_p2 for most situations. On the contrary, for the CNN methods, the reduction in correlation was minor as the correlation coefficient decreased by only 0.093 and 0.090 in CNN-MSE and CNN-WMSE₁, respectively.

Finally, it is important to note that scattering in HARP around the unity line was not random. The strain underestimation pattern was evident, particularly in HARP₈, as shown in the top subplot of Figure 10a. The pattern of underestimation is marked by the dashed ellipses L and M for ϵ_p1 and ϵ_p2, respectively. HARP estimates deviate away from the unity line and towards zero. The density of points in these regions decreased as the filter size increased, as shown in HARP₁₆ and HARP₂₄ plots. However, at the same time, artifact influenced increases, as evident in the regions marked by the ellipses K and N for ϵ_p1 and ϵ_p2, respectively, do not show any sign of these underestimations or non-random artifact patterns.

5. Discussion

We have presented a deep-learning framework for tMRI principal strain quantification at an unprecedented pixel-resolution of the input tagged image set, allowing excellent visualization of the progressive alteration of myocardial mechanical function and providing a potentially important tool for the non-invasive assessment of early signs of cardiac mechanical wave desynchrony and microfiber and viability abnormalities.

Previous tMRI analysis methods, whether based on feature detection and tracking, bandpass filtering, or optical flow methods, are likely to be sensitive to parameter selection and image quality. Moreover, the computations involved in these methods inevitably limit the effective resolution of the obtained strain maps. We chose to compare the network performance against HARP because it is a widely recognized tMRI strain mapping bandpass-filtering method and validated both scientifically and clinically [42]. Like other bandpass-filtering methods, HARP-calculated maps are sensitive to the filtering parameters. Wider bandpass filters accommodate larger information bandwidth to produce accurate high-resolution strain maps but fail to accomplish that due to the inability to exclude noise, harmonic interference, and artifacts. Ultimately, the maps suffer from low precision. On the other hand, narrow bandwidth filtering fails to include the necessary strain information and eventually results in inaccurate underestimated low-resolution maps.

Recently, Ferdian et al. [28] proposed a DL framework that employed two cascaded networks to estimate the left ventricular (LV) circumferential and radial strain on short-axis tMRI images at specific landmark locations. Their average reported errors were about −0.001 in basal and mid slices and 0.004 in apical slices relative to manually determined strain values by trained readers in the absence of ground-truth comparisons. Building on Ferdians' work, Loecher et al. [35] went several steps further with their rigorous study design and comparisons, developing CNN-based motion tracking and Lagrangian strain mapping. Their network was trained entirely from synthetically generated MRI data obtained from an elegantly built computational phantom using natural images, simulated motion, and Bloch tMRI simulation. Strains, however, were calculated after approximation of the deformation mapping, fitting, and regularization, substantially lowering, making it hard to determine the final Fourier resolution of the obtained strain maps. This approach is different from ours in which the strain maps are created directly from the network at the native image resolution without fitting or derivative computations. While their algorithm allowed for completely automatic tag tracking, user input was still required to segment the LV and to guide the selection of tag line intersections. On the contrary, our direct native-resolution Eulerian strain mapping approach does not require any manual interaction yet produces strain maps everywhere in the image, including both the right and the left ventricles.

DL techniques have not been employed before in automatic Eulerian strain mapping directly from, and at the native image resolution of the raw images. In the network learning phase, our method used local-patch CNN architecture to determine and extract all the necessary features directly from the tagged image and assign the mapping parameters to calculate the associated principal strain maps. With the support of an in-house developed versatile computational phantom, an infinite number of unique realistic tMRI cine datasets were generated for training, validating, and testing purposes. The simulated tMRI data included a variety of actual anatomical images used as background images, various parameters to control the shapes and deformation of objects, as well as additive and multiplicative noise and tissue-dependent signal fading, coving a wide range of realistic tMRI situations. Pure synthetic samples were used for training and testing the reconstruction network to solve the problem because it is difficult to obtain actual ground-truth in-vivo training mapping samples. This method has already been used in some related works in the field of magnetic resonance imaging [35, 43-46].

These studies focused on tracking the motion profile of the material points for later calculation of Lagrangian strain. It was, therefore, crucial to model all motion components as close as possible. The presented study focused explicitly on Eulerian principal strains pixel-to-pixel mapping in each frame independent of the other timeframes, contrary to other studies that focused primarily on motion tracking of tag zero-crossing points. However, such comprehensive models may be valuables to implement to further improve the precision and accuracy of the results.

Our model included white and speckle noises, wide ranges of geometries and deformations, and different tissue properties to model actual tMRI as much as possible. In reality, there are several major sources of artifacts and distortion that challenge any computational MRI phantom. First, our model did not include field inhomogeneity effect, inter-organ susceptibility artifacts, blood flow, nor residual breathing motion. These factors distort the pattern and continuity of tag lines in ways too challenging to incorporate in the current study's simulations but could be valuable to model in future improvements. An additional challenge is inter-scan misregistration. Since the vertical and horizontal data were acquired at different breath-hold scans, there is often a possibility of dataset misregistration due to patient motion and multi-breath-hold fatigue causing in-plane and through-plan shifts. Heart rate variability between scans is also a common cause of temporal misregistration among the timeframes leading to image quality reduction and artifact increase. Given these challenges and thanks to the versatility of generated synthetic data, the performance of the proposed CNN-based method is at least 5-fold better than the best HARP performance in both phantom and in vivo data, as shown in Figure 9.

In the heart beating process, the liver may also be accompanied by a certain degree of movement and deformation. Several research groups studied this relation and the resulting strain in the liver [47-52]. In this work, we did include the effect of cardiac motion into the simulations and network training and testing. In the simulation, we included a separate stretching region representing the liver with its distinct motion pattern and tissue MR-relaxation properties, as shown in Figure 1. In the network training and testing phases, we included this region in the generated patches for both the training and testing phases to ensure the network is exposed to the motion and deformation seen in the liver and not only to those of the heart. However, in the comparisons in both simulation and in-vivo data, we chose to restrict the point-to-point strain comparison to the heart's LV and RV only because the focus of validation was on assessing the CNN’s performance in measuring myocardial strain.

As was shown in Figure 9 and Figure 10, we compared the computational phantom results of all methods against the ground truth and CNN-WMSE₂ and compared the in-vivo results to those of CNN-WMSE₂ even though CNN-WMSE₂ is not itself a gold standard. We did that because there is no way to obtain a ground truth for the in-vivo scans. By the in-vivo comparisons, we wanted to demonstrate that the differences in performance of the other methods compared to CNN-WMSE₂ were not tied to the phantom results only. Obtaining similar improvements when using CNN-WMSE₂ on both synthetic data and in-vivo experiments makes it unlikely that these observed improvements are simply simulation-overfitting or limited to synthetic data. With the lack of a gold standard for the in-vivo scans, we thought this was the best possible approximation of a quantitative evaluation of the relative merits of both methods in in-vivo experiments.

Another critical point of consideration in this work is the effect of image pixel resolution on the accuracy of the calculated map. From a signal sampling perspective, in the presence of noise and with all other factors kept the same, the quality of estimated strain Q_s (accuracy and precision) is expected to be influenced by the ratio of the sampling rate to the bandwidth of the information sought after, i.e. Q_s ∝ (Sampling Rate)/(Expected Strain Dynamic Range) in our case. This is because a large change in the tagging frequency (a.k.a. high strain) is more likely to be severely disturbed by noise and thus more challenging to detect than a low change in the tagging frequency. From the perspective of pure simulation, the accuracy of deformation mapping is limited as well, even in the absence of noise or artifacts due to two main limiting factors. The first factor is the loss of information involved in discretizing the image and the corresponding strain maps. Higher resolution will lead to higher fidelity and thus accuracy. However, the simulated image resolution cannot be infinitesimal. The second limiting factor is the network computations. The approximate finite-difference gradient calculations, the associated truncation errors throughout the network operations, and the limits on the number of training iterations to avoid the risk of overfitting will ultimately limit the accuracy.

As shown in Figure 9, the higher the strain, either stretching or contraction, the lower Q_s, represented by both precision and accuracy, became. Nevertheless, the reduction in quality in CNN-based measurements is substantially minimal compared to the enormous worsening in the HARP measurements. Although not tested in this work, but the same thing can be said about fixing the strain and reducing the sampling rate or equivalently increasing pixel size.

Finally, our HARP results were not an exact reproduction of those presented by Osman et al.[7]. In that work, the strain maps were displayed after being smoothed by a 7 x 7 averaging filter and then restricted to a simple threshold segmentation obtained from the magnitude images. That smoothing filter was used after HARP strain calculations to compensate for noise amplification in the gradient computations [7]. These artifacts are further accentuated with the use of a large filter size. A wide bandpass filter admits more noise and interference from the conjugate spectral peak, adding more artifacts into HARP strain computations. In comparison, we chose to display the strain maps using only the HARP spectral peak filtering and CNN outputs without any postprocessing averaging filtering or thresholding. We wanted the reader to appreciate that CNN mapping automatically recognized the no-strain regions in the lung and other regions and the low strain values in the liver without any ripple or ringing artifacts and without any segmentation or thresholding involved. This is in comparison to the HARP results that suffered from severe artifacts.

The proposed DL framework demonstrated the ability to provide highly accurate and precise strain maps compared to HARP. This method was evaluated on large-diverse simulated data and in-vivo tMRI cine datasets from normal healthy adults and PAH patients. The proposed architecture provided promising results on a wide range of image quality and strain variations without the need to change any parameters. It has the advantage of being completely automatic.

6. Conclusion

Pixelwise Eulerian principal strains can be obtained directly from tagged MRI at the image native resolution using deep learning CNN. The proposed framework was trained and validated using extensive realistic datasets from an in-house developed computational phantom and in-vivo datasets obtained from healthy controls and pulmonary arterial hypertension patients. The performance of the proposed method with various loss-function settings was evaluated and compared to that of the bandpass filtering harmonic phase analysis method, where the CNN with weighted mean square loss function outperformed the other methods at the native image pixel level both qualitatively and quantitatively in terms of precision and accuracy.

Highlights:

• Assessing heart function at native resolution is key in spotting early disease signs

• Eulerian mechanical strain is a metric for quantification of cardiac function

• Magnetic resonance imaging labels and acquires cine images of the contracting heart

• Neural networks are trained to map heart contraction onto corresponding strain

• Neural network’s strain maps showed superior agreement with ground truth