Deriving New Soft Tissue Contrasts from Conventional MR Images Using Deep Learning

Abstract

Versatile soft tissue contrast in magnetic resonance imaging is a unique advantage of the imaging modality. However, the versatility is not fully exploited. In this study, we propose a deep learning-based strategy to derive more soft tissue contrasts from conventional MR images obtained in standard clinical MRI. Two types of experiments are performed. First, MR images corresponding to different pulse sequences are predicted from one or more images already acquired. As an example, we predict T_1ρ weighted knee image from T₂ weighted image and/or T₁ weighted image. Furthermore, we estimate images corresponding to alternative imaging parameter values. In a representative case, variable flip angle images are predicted from a single T₁ weighted image, whose accuracy is further validated in quantitative T₁ map subsequently derived. To accomplish these tasks, images are retrospectively collected from 56 subjects, and self-attention convolutional neural network models are trained using 1104 knee images from 46 subjects and tested using 240 images from 10 other subjects. High accuracy has been achieved in resultant qualitative images as well as quantitative T₁ maps. The proposed deep learning method can be broadly applied to obtain more versatile soft tissue contrasts without additional scans or used to normalize MR data that were inconsistently acquired for quantitative analysis.

Introduction

Magnetic Resonance Imaging (MRI) provides superior and versatile soft tissue contrasts. However, the versatility of MRI contrast is not fully exploited. In a standard clinical MRI examination, only few images with certain contrasts are acquired due to limited scan time. Pulse sequences that are not included in standard imaging protocols do not have opportunities to make real impacts on clinical practice. For example, in knee MRI, T_1ρ weighted imaging is a valuable disease indicator [1, 2], but not conventionally performed. Obtaining unconventional contrasts without prolonging scan time would be desirable. In addition, emulating different imaging parameter values in commonly used pulse sequences would also be meaningful. One application is to predict multiple images (with the same type of weighting) from a single MR image, which paves a way for deriving quantitative MR parametric map from standard clinical MR images without extra data acquisition. Another utility is to normalize MRI data acquired using inconsistent imaging parameter values, which enlarges data sets for quantitative analysis [3]. Therefore, retrospective change of MRI contrasts is of clinical significance.

Synthesizing MR images with alternative soft tissue contrasts has been investigated in multiparametric mapping [4–6], MAGiC [7], and MR fingerprinting [8]. In these approaches, the Bloch equations are applied on quantitative tissue parametric maps (T₁, T₂ and ρ), which typically require additional data acquisition using a unique pulse sequence (e.g. QRAPTEST [5], QRAPMASTER [6, 7], MRF-compatible pulse sequences with pseudo-random acquisition parameters [8]). Due to the potential change of clinical protocol as well as demand on the special pulse sequence, translation of these synthetic MRI methods into clinical practice is not straight forward.

New hopes have been brought to medical imaging by recent thriving of deep learning [9]. With its superior capability in learning complex relationships and incorporating existing knowledge into inference model, deep learning has changed the landscape of medical physics [10]. There has been tremendous progress in deep learning-based image reconstruction [11–16], super-resolution [17, 18], quantitative imaging [19–21], and synthetic MRI [22]. Particularly in quantitative MRI, deep learning has been used to support parameter quantification from highly undersampled data [19], to derive quantitative parametric map without extra data acquisition [20], or to accelerate parameter fitting from MR fingerprinting or multi-pathway multi-echo data [21, 22]. These deep learning-based quantitative MRI approaches facilitate synthesis of MR images when used with the Bloch equations.

In this study, we propose a novel deep learning strategy to derive new contrasts from MR images already acquired in a standard clinical examination by establishing direct mapping between variable contrast images that reflect the same tissue properties. A specially designed self-attention convolutional neural network architecture is employed, which demonstrates superior network performance in image-to-image translation tasks. The proposed contrast changing method is validated in knee MRI data sets that are retrospectively collected, where T₁ weighted, T₂ weighted and T_1ρ weighted images were acquired from 56 subjects using a consistent imaging protocol. We predict T_1ρ weighted image from corresponding T₂ weighted image and/or T₁ weighted image to investigate the feasibility of achieving different type of weighting from MR images already acquired. Furthermore, we change the contrast of T₁ weighted image by emulating variable flip angles, where prediction accuracy is evaluated on resultant T₁ weighted images as well as on quantitative T₁ maps subsequently derived.

Method

In this study, we use deep learning to obtain new soft tissue contrasts from MR images already acquired in standard MRI. As is known, the signal intensity of an MRI image is determined by both tissue properties (T₁, T₂, ρ) and external imaging protocol (pulse sequence and acquisition parameter values). While impacts from both categories of factors are intertwined in the MR images actually acquired, deep learning has the capability to separate them from each other [20] and further provide direct mapping between different image sets that correspond to alternative pulse sequences or acquisition parameters. The scheme is illustrated in Figure 1.

Figure 1.

Scheme of derivation of new MRI contrasts from MR images already acquired in clinical practice. Deep neural network is employed to provide an end-to-end mapping from one or more images with certain contrasts to new images with different weighting.

In the preliminary study, the proposed strategy is validated in knee MRI data sets that are retrospectively collected, where four variable flip angle T₁ weighted images, a T₂ weighted image and a T_1ρ weighted image were acquired from each of 56 subjects [23–26]. Two types of experiments are performed to derive new contrasts from conventional MR images. First, we synthesize images corresponding to alternative pulse sequence (e.g. obtaining T_1ρ weighted image from T₁ / T₂ weighted images). In another case, we predict images corresponding to different values of imaging parameter (e.g. deriving variable flip angle images from a single T₁ weighted image, which are subsequently used to extract quantitative T₁ map).

Image Acquisition

For training and testing of deep learning models that transform the contrast of MR images, we retrospectively collect knee MR image sets that were acquired on a 3T MR750 scanner (GE Healthcare Technologies, Milwaukee, WI) using a consistent imaging protocol with Institutional Review Board (IRB) approval. For every subject, T₁ weighted, T₂ weighted and T_1ρ weighted images were acquired using 3D Ultrashort TE (UTE) sequences [23–26]. More specifically, four T₁ weighted images were acquired using variable flip angles of 5°, 10°, 20° and 30° with a TE of 32μs and a TR of 20ms [24]; a T₂ weighted image was measured with a TE of 4.4ms, a TR of 500ms, and a flip angle of 16° [25]; and a T_1ρ weighted image was obtained using a 3D adiabatic inversion recovery spin-lock prepared UTE sequence with a TE of 32μs, a TR of 500ms, and a flip angle of 10° [26]. The images with different weightings all had a matrix size of 256×256×36 and were aligned with each other. In addition, a B₁ map (with a matrix size of 128×128×36) was measured to compensate for radiofrequency field inhomogeneity using the actual flip angle method [27]. Finally, quantitative T₁ map was extracted from four variable flip angles images using least square fitting, and then corrected by B₁ map [24].

Deriving MR Images with Different Type of Weighting

As an example of obtaining new contrast corresponding to different pulse sequence, T_1ρ weighted images are predicted from conventional MR images using deep learning. Based on the high correlation between T_1ρ and T₂ weighted images [1], T_1ρ weighted image is first predicted from T₂ weighted image. Then T_1ρ weighted image is also predicted from T₁ weighted image. Furthermore, both T₂ and T₁ weighted images are employed as the input to investigate whether redundant information embedded in both images leads to improved prediction accuracy. In these scenarios, deep neural networks are employed to provide direct mappings from conventional T₁ and/or T₂ weighted images to new images with T_1ρ weighting, achieving different contrast without additional data acquisition performed. The scheme is illustrated in Figure 2.

Figure 2.

Obtaining T_1ρ weighting from T₂ and/or T₁ weighted images. Deep neural networks are employed to provide direct mappings from conventional images to new images with different types of weighting.

Changing Image Contrast by Adopting Alternative Imaging Parameter Values

Using the same strategy, more contrasts can be obtained by adopting different imaging parameter values in a given pulse sequence. In a representative case, deep learning models are trained to provide end-to-end mappings from a single T₁ weighted image (e.g. acquired using flip angle of 20°) to variable flip angle images (corresponding to 5°, 10° or 30°). The accuracy of the predicted images is further evaluated on the T₁ map subsequently derived, where ground truth T₁ map was extracted from four variable flip angle images and compensated by B₁ map using conventional approaches. In our method, T₁ mapping is accomplished via deep learning, which improves the robustness of parameter quantification as compared to conventional nonlinear fitting [19, 22]. In particular, compensation for radiofrequency inhomogeneity can be automatically achieved in the resultant T₁ maps, even though B₁ map is not incorporated as an input [19]. Similarly, a baseline T₁ map is predicted from three images actually acquired at the same flip angles (5°, 10° or 30°) using deep learning. The difference between the proposed and baseline T₁ maps indicates the accuracy of the predicted variable flip angle images. The T₁ weighted image prediction and T₁ mapping scheme is illustrated in Figure 3.

Figure 3.

Deriving alternative T₁ weighted images that correspond to different flip angles and validating the accuracy of predicted images in quantitative T₁ mapping. (a) Three T₁ weighted images corresponding to flip angle of 5°, 10° and 30° are predicted from a single T₁ weighted image acquired with 20° using deep neural networks, (b) Using deep learning, T₁ map is derived from the three predicted images (as proposed) or from the corresponding images actually measured (baseline); the resultant T₁ maps are compared with the ground truth T₁ map (extracted from four variable flip angle images using least square fitting and compensated by a B₁ map actually measured).

Deep Neural Network

To accomplish the tasks of contrast changing and T₁ mapping, a specially designed deep neural network architecture is employed, as illustrated in Figure 4. In brief, it is a hierarchical network where feature maps are extracted at various scales with receptive field gradually enlarged. Global shortcuts, which connect the corresponding levels of the encoder and the decoder, compensate for details lost in down-sampling [28]. Local shortcut connections, which forward the input of a hierarchical level to all the subsequent convolutional blocks at the same level [29], facilitate residual learning [30, 31]. Moreover, the self-attention mechanism [32–35] is integrated into the network design to make more efficient use of non-local information. Basically, all the voxels in a feature map make contributions to the signal prediction at a given voxel in a self-attention map, and the influence is determined by the relevance between the voxel of interest and remote voxels as well as the feature representation of remote voxels. The relevance between two voxels is quantified by

$$s\left(X_i,X_j\right)=e^{{\left(W_f{X}_i\right)}^T\left(W_g{X}_j\right)}$$ (1)

and the feature representation of remote voxels is given by

$$h\left(X_j\right)=W_h{X}_j$$ (2)

where W_f, W_g, and W_h are weight matrices (1×1 convolution) learned in the training of the network.

Figure 4.

Self-attention convolutional neural network employed for contrast changing and parametric mapping. (a) The hierarchical network architecture composed of an encoder and a decoder. Global shortcut connections are established between the corresponding levels of the two paths to compensate for details lost in down-sampling, whereas local shortcut connections are established within the same level of a single path to facilitated residual learning. (b) The composition of a convolutional block, which has a novel self-attention layer in addition to convolutional layer and activation layer.

Several deep neural networks are trained, each for deriving specific contrast or T₁ mapping. The network parameters are initialized using the He method [36] and updated using the Adam algorithm [37] with an adaptive learning rate (α of 0.001, β₁ of 0.89, β₂ of 0.89, and ∊ of 10⁻⁸). A loss function defined as

$$loss=l_1+\lambda \ast l_{SSIM}$$ (3)

is employed [38], where l₁ norm minimizes uniform biases, l_SSIM preserves local structure and contrast in high-frequency regions, and λ is empirically chosen as 5. Here, l_SSIM = 1 − SSIM, and SSIM (structural similarity index measure) is calculated as

$$\mathit{SSIM}(x,y)=\frac{\left(2{\mu }_x{\mu }_y+c_1\right)\left(2{\sigma }_x{\sigma }_y+c_2\right)}{\left({\mu }_x^2+{\mu }_y^2+c_1\right)\left({\sigma }_x^2+{\sigma }_y^2+c_2\right)},$$ (4)

where μ_x, μ_y,σ_x, and σ_y correspond to the mean and standard deviation of signal intensity in the reconstructed image and the ground truth, C₁ and C₂ are constants [38]. This is a simplification of the three-component SSIM definition [39, 40]. The simplified version, which was proposed in the original specification of SSIM [39], has been widely used to evaluate the performance of deep neural networks. The correlation coefficient is specified as

$$Corr(x,y)=\frac{2{\sigma }_{xy}}{{\sigma }_x^2+{\sigma }_y^2+{\left({\mu }_x-{\mu }_y\right)}^2}$$ (5)

The models are trained using 1104 images from 46 subjects. After training, 240 images from 10 other subjects are tested, and the results are evaluated using quantitative metrics −l₁ error, SSIM, and correlation coefficient. The mean and standard deviation of these quantitative metrics are calculated over all the testing images.

Results

Using 1104 images as the training data, deep learning models are established to predict T_1ρ weighted images from the corresponding T₂ weighted images, T₁ weighted images, or a combination of them. High fidelity is achieved in T_1ρ weighted images predicted from T₂ weighted images, which confirms the capability of deep learning to exploit the correlation between T_1ρ and T₂ weighted images. An example is shown in Figure 5. Alternatively, T_1ρ weighted images are derived from T₁ weighted images, which, however, are less accurate. When both T₂ weighted and T₁ weighted images are incorporated as the input to the prediction model, the performance is slightly improved than single-input predictions. This is not surprising because every input image, which reflects tissue properties from a unique perspective, make its own contribution. The overall quantitative results for the prediction of T_1ρ weighted images are presented in Table 1. In all the test images, the mean of l₁ error ranges from 0.10 to 0.15, the averaged SSIM is between 0.83 and 0.90, and the correlation coefficient is around 0.97.

Figure 5.

Prediction of T_1ρ weighted image from corresponding T₂ weighted image, T₁ weighted image, or a combination of them, as shown in the upper, middle and lower rows. The T_1ρ weighted image predicted from T₂ weighted image has high fidelity to the ground truth, whereas the multi-input prediction (with T₁ weighted image incorporated) is slightly more accurate.

Table 1.

Quantitative results (mean and standard deviation of l₁ error, SSIM and correlation coefficient) over all test images for the prediction of T_1ρ weighted images from T₂ weighted images, T₁ weighted images, or the combination of them.

	Predicting T1ρ-w image from T 2 -w image	Predicting T1ρ-w image from T1 image	Predicting T1ρ-w image from T 2 -w and T 1-w images
l1 Error	0.138 ± 0.011	0.151 ± 0.013	0.107 ± 0.009
SSIM	0.846 ± 0.029	0.829 ± 0.027	0.898 ± 0.018
Correlation	0.973 ± 0.008	0.966 ± 0.011	0.985 ± 0.007

Similarly, using deep neural networks trained with 1104 images, variable flip angle images (that correspond to 5°, 10°, and 30°, respectively) are predicted from single T₁ weighted images (obtained using a flip angle of 20°) with high accuracy achieved. An example is shown in Figure 6. The overall quantitative results for the prediction of variable flip angle images are given in Table 2. In all the test images, the mean of l₁ error ranges from 0.04 to 0.08, the averaged SSIM is between 0.90 and 0.96, and the correlation coefficient is around 0.99. Subsequently, T₁ maps derived from predicted images are evaluated against baseline T₁ maps obtained from corresponding images actually acquired, where very similar results are obtained −l₁ errors (0.12 vs 0.09), SSIMs (0.82 vs 0.88), correlation coefficients (0.96 and 0.98). This validates the accuracy of the predicted variable flip angle images in the application of T₁ mapping. The high fidelity of the predicted T₁ maps to the ground truth maps indicates the feasibility of deriving quantitative T₁ maps from single T₁ weighted images.

Figure 6.

Predicting variable flip angle T₁ weighted images from a single T₁ weighted image and validating the prediction accuracy in quantitative T₁ mapping. (a) In the proposed method, variable flip angle T₁ weighted images (corresponding to 5°, 10°, and 30°) are predicted from a single T₁ weighted image (acquired using 20°) with high image fidelity achieved; the differences between the predicted images and the ground truth images are shown in the next row (rescaled to ½ of the full scale of the ground truth images); subsequently, T₁ map is derived from the three predicted images, and the difference between the predicted map and the ground truth map is shown in the rightmost column, (b) In a baseline method, T₁ map is derived from three images actually measured using 5°, 10° and 30°, (c) The ground truth T₁ map was extracted from four variable flip angle images using least square fitting and compensated by a B₁ map actually acquired. High similarity is achieved between the proposed, baseline, and ground truth T₁ maps.

Table 2.

Quantitative results (mean and standard deviation of l₁ error, SSIM and correlation coefficient) over all test images in the prediction of various flip angle images (from single T₁ weighted images) and T₁ maps (from predicted or measured images).

	Predicting variable flip angle T 1weighted images from a single input image (20°)			Deriving T1 map from different input images
	Predicting 5° image	Predicting 10° image	Predicting 30° image	Using 3 predicted images as input	Using 3 measured images as input
l1 Error	0.084 ± 0.010	0.057 ± 0.006	0.044 ± 0.004	0.124 ± 0.024	0.092 ± 0.019
SSIM	0.901 ± 0.024	0.941 ± 0.017	0.964 ± 0.006	0.824 ± 0.048	0.879 ± 0.016
Correlation	0.987 ± 0.005	0.994 ± 0.002	0.998 ± 0.001	0.965 ± 0.018	0.982 ± 0.007

A region of interest (ROI) in cartilage is delineated in T₁ maps as shown in Figure 7. In all test cases, the mean and standard deviation of T₁ within the ROI in the ground truth maps, predicted maps, and baseline maps are 974.3 ± 114.5, 942.9 ± 118.1, and 980.1 ± 138.9, respectively.

Figure 7.

A region of interest (ROI) in the predicted, baseline, and ground truth T₁ maps with the corresponding error maps given in the lower row.

Discussion and Conclusions

In this study, we demonstrate the feasibility of a deep learning-based strategy to gain new contrasts from conventional MR images already acquired in standard clinical MRI. Two clinically useful cases are presented. One is to derive unconventional T_1ρ weighted images from T₂ weighted and/or T₁ weighted images. The other is to predict variable flip angle images from a single T₁ weighted image, which can subsequently be used for quantitative T₁ mapping.

The underlying reason why this strategy would work is that deep learning has the capability to separate the influence of inherent tissue properties from the effect of acquisition protocol. Other studies also indicate the feasibility of using deep learning to obtain quantitative parametric maps from conventional MR images [20]. For this reason, the deep learning-based approach can be broadly applied in other MRI applications, such as generating contrast enhanced image or FLARE (fluid-attenuated inversion recovery) from T₁ and/or T₂ weighted images.

In the proposed framework, single or multiple images may be employed as the input. As demonstrated, a single T₁ weighted image provides sufficient information for the prediction of variable flip angle images and T₁ map. Meanwhile, T_1ρ weighted images can be accurately derived from single T₂ weighted images, but not from single T₁ weighted images, whereas multi-contrast images lead to more accurate prediction of T_1ρ weighted images. To reliably obtain other contrasts, further investigation should be conducted.

To accomplish the MRI contrast changing and T₁ mapping tasks, convolutional neural network and self-attention mechanism are employed. In fact, the encoder path of the network extracts feature maps, where inherent tissue parameters are separated from image acquisition parameters; and the decoder path of a network combines various features in a comprehensive pattern. When several output images (e.g. variable flip angle images) are expected in the system, a multi-output network similar to StarGAN [41] may be adopted instead of using several parallel single-output networks, since features representing inherent tissue parameters can be shared to produce different output images. The use of convolution operator, hierarchical architecture, and self-attention mechanism helps to incorporate influence from other pixels, which improves the robustness of prediction and enables exploitation of a priori information in spatial-parameter space [19].

The proposed approach provides direct mapping between various images, although synthetic MR images used to be derived from quantitative parametric maps. Quantitative parametric maps are hard to obtain, which typically require additional data acquisition. In a recent deep learning-based quantitative MRI approach that requests no extra scan, the establishment of quantitative parametric mapping model is still challenging due to the need for a large number of quantitative parametric maps in training (as the ground truth maps) [20]. In the propose method, the only exceptional effort is the acquisition of images with target contrast for model training, which is more acceptable in clinical practice.

The proposed MRI contrast changing strategy has a variety of potential clinical utilities. It can be used to derive unconventional contrasts or quantitative parametric maps from standard clinical MRI. While some pulse sequences are not performed in routine MRI, they are virtually reactivated in the proposed framework - after the acquisition of training data and establishment of prediction model, target contrasts can be automatically obtained from conventional images. In addition, the proposed method can be used to normalize MRI data acquired with inconsistent imaging parameters within or across medical centers, thus enlarging data sets for quantitative radiomics analysis.

In conclusion, we propose a deep learning strategy that enables the derivation of new soft tissue contrasts from MR images already acquired.