Improving Quantitative Magnetic Resonance Imaging using Deep Learning

Introduction

Deep learning is one particular type of machine learning which uses a combination of simple transformations to represent complex learning functions. The learning process is featured by a multiple level representation with each level followed by a higher, slightly more abstract level (1). Although conventional machine learning techniques have been investigated for decades (2-4), the applications of deep learning methods have recently become more popular owing to the dramatic increase in computational efficiency and advanced algorithm implementation. In conventional machine learning, hand-crafted features are designed in order to analyze and capture patterns in the data. Such an engineering-based approach typically requires domain knowledge and might not result in the extraction of the correct information when discrepancies between the data being analyzed and applied assumptions occur. Deep learning utilizes a specific algorithm architecture, a neural network, for processing information. Inspired by the anatomy of neurons and how they function in the brain to perform cognitive tasks, the neural network relies on multiple hidden layers with artificial neurons interconnecting with each other by ‘weights’ which can represent the intensity of the connection between neurons in the brain. To analyze multi-dimensional image data for computer vision tasks, a set of processing modules was introduced into the neural network. A process to characterize the inter-correlation among image pixels is called convolution, where a set of the extracting ‘filters’ are used to identify intensity variations, edges, and patterns at low-level layers. The activation function, similar to real neuron activation, is a non-linearity multiplier which is added to the output of the convolutional layer to increase model complexity and enhance the learning capability. A pooling process can also be performed in the network to reduce the size of the features and remove redundant noise and nuisance factors while maintaining only essential image information. This convolutional neural network (CNN) consisting of multiple convolutional layers each of which is interconnected with each other is a widely used modern neural network architecture in computer vision. The unique structure in CNN allows highly flexible, scalable, and efficient learning of multi-level structural features that are inherently embedded in the training data and otherwise not be capable to be learned via other machine learning methods. Therefore, the modern application of deep learning in multi-dimensional data analysis is typically referred to applying CNN for learning features upon which subsequently more abstract purposes are performed such as image recognition, region-of-interest detection, and object classification.

Since 2010, deep learning has revolutionized computer vision, and has quickly expanded into a much broader range of disciplines, including but are not limited to speech recognition, natural language processing, robotics, drug discovery, financial analysis, chemical engineering and biotechnology(1,5). In particular, there has been much recent interest in applying deep learning to a wide variety of medical imaging subspecialties, including cancer, neuro, lung, abdomen, breast, and cardiac imaging (6-8). The applications in musculoskeletal imaging have emerged, and great success has been made for tissue segmentation (9-15), and image reconstruction (16-18). More recently, the use of deep learning for lesion detection, progression, and prediction of musculoskeletal disease on X-ray radiography, computed tomography (CT), and magnetic resonance (MR) imaging has further demonstrated the potential of deep learning to maximize diagnostic performance while reducing subjectivity and errors induced by the human interpretation(19). One another important image subspecialty in musculoskeletal imaging is quantitative imaging which, however, is less studied and remains very limited to the use of deep learning. Quantitative mapping of MR parameters, such as the spin-spin relaxation time (T2) and the spin-lattice relaxation time in the rotating frame (T1rho), have been shown as valuable methods for improved assessment of a range of musculoskeletal diseases(20-24). In contrast to conventional anatomical MR, quantitative mapping can provide increased sensitivity and specificity to different pathologies and delivered information that is complementary to anatomical images and relevant to tissue composition and microstructure(25-29). A few recent review articles have focused on the use of deep learning for musculoskeletal image reconstruction(30), tissue segmentation and disease diagnosis and prediction(19). This paper aims to review the current deep learning applications with a primary focus on the improvement of quantitative imaging using deep learning and the use of deep learning on quantitative imaging analysis in musculoskeletal imaging.

Rapid Quantitative Imaging: Acquisition and Reconstruction

Despite the demonstrated value of quantitative MR imaging for the assessment of musculoskeletal diseases(25), these techniques are not part of clinical routine, mainly because the low resolution and exhaustive scan times associated with these mapping methods prevent their use in clinical settings. Multiple anatomical images are required with varying parameters of imaging, for example, the multiple echo times (TEs) for T2 mapping, in order to provide adequate information for subsequent parameter map generation. The parametric maps are then calculated by fitting the acquired data with a signal model on a pixel-by-pixel basis. Accelerated quantitative imaging is highly desirable and remains an active topic with a great interest in musculoskeletal imaging research. In particular, many pioneer works in MR image reconstruction have shown that conventional reconstruction methods, including parallel imaging(31-34) and compressed sensing(35,36) can provide decreased acquisition time while maintaining acceptable image quality at a low acceleration rate(37-42). More recently, there have been a few attempts to dramatically increase the acquisition speed of quantitative imaging using deep learning and have shown deep learning-based methods can yield better image quality in comparison to conventional state-of-the-art reconstruction methods.

Liu et al. developed a deep learning-based image reconstruction approach called Model-Augmented Neural neTwork with Incoherent k-space Sampling (MANTIS) for efficient MR parameter mapping and evaluated this method for T2 mapping in knee MRI(18). The proposed method directly reconstructed T2 maps from a set of undersampled MR images at different echo times using an end-to-end CNN learning. The retrospective experiments were performed on fully sampled image datasets from 110 symptomatic patients on a 3T scanner using a multi-echo spin-echo T2 mapping sequence in the sagittal orientation with a total of eight echoes. The undersampled images were retrospectively synthesized using 1D variable density Cartesian sampling masks over the full k-space, and the reference T2 maps were generated by fitting fully sampled image data. From the 110 patient image datasets, 100 patient datasets were randomly selected for network training and validation, while the remaining 10 patient datasets were used for model evaluation. MANTIS was found to enable up to eight-fold acceleration with acceptable reconstruction performance and accurate T2 estimation in the cartilage and meniscus in comparison to the fully sampled reference. Compared with conventional iterative reconstruction approaches, MANTIS showed improved reconstruction performance with a lower errors indicated by normalized root mean square error (nRMSE) of 6.1% for acceleration factor R=5 and 7.1% for R=8 and higher similarity indicated by structural similarity index (SSIM) of 86.2% at R=5 and 82.1% at R=8 to the reference in the T2 estimation (Figure 1). The performance improvement of MANTIS is largely attributed to two key components. One is the application of multi-level spatial-temporal filters in the CNN training to characterize the latent spatial-temporal correlation embedded in the training datasets that helps restore the missing k-space information. The other is the added MR signal model into the CNN training that enables more robust learning and ensures data fidelity between acquired k-space and the learned parameter maps. The direct end-to-end mapping from undersampled images to the parameter map in MANTIS also avoids the time-consuming pixel-wise fitting process, which dramatically reduces the computation time from tens of minutes for standard fitting methods to a few seconds for the deep learning T2 map generation.

Figure 1:

Comparison of the T2 map estimated from MANTIS with T2 maps produced using different conventional sparsity-based iterative reconstruction approaches at R=5 (top) and R=8 (bottom). MANTIS generated a nearly artifact-free T2 map with well-preserved sharpness and texture comparable to the reference T2 maps. Other methods created suboptimal T2 maps with either reduced image sharpness or residual artifacts, as indicated by the white arrows. GLR: global low rank; LLR: local low rank; k-t SLR: k-t sparsity and low-rank structure; ALOHA: annihilating filter-based low-rank Hankel matrix; (Image reprinted from Figure 3 in Liu et al. MANTIS: Model-Augmented Neural neTwork with Incoherent k-space Sampling for efficient MR parameter mapping. Magn Reson Med 2019 Jul;82(1):174-188; with permission)

Another pilot study by Li et al. attempted to reconstruct T1rho and T2 maps for knee joint simultaneously using a deep learning algorithm(43). The method also used an end-to-end CNN learning strategy to map the undersampled input MR images directly into combined T1rho and T2 maps. The full k-space datasets of ten knees were acquired at a 3T scanner using a magnetization-prepared angle-modulated partitioned k-space spoiled gradient echo snapshots (MAPSS) T1rho and T2 quantification sequence with a total of 16 acquisitions at different preparation echo times and spin lock times (TSLs). Eight datasets were used to train a 3D CNN and the remaining two datasets were used for evaluation. A 2D Poisson random sampling mask was used with an acceleration factor R=2 for generating the undersampled input MR images. The deep learning method was found to provide equivalent T1rho and T2 quantification using only 3 undersampled echoes in contrast to the gourd truth parametric maps generated by fitting the full image datasets. The preliminary result indicates the possibility of acquiring simultaneous T1rho and T2 maps at a single scan within 5 min on the use of deep learning reconstruction algorithm.

An alternative approach to accelerate quantitative imaging is to reduce the scan time for each image acquisition at varying imaging parameters, then followed by the standard model fitting process. Zibetti et al. investigated this approach to accelerate the T1rho mapping in knee imaging. This study incorporated a Variational Network (VN), an unrolled network for optimizing constrained iterative reconstruction(17), into the deep learning reconstruction method to accelerate the MR acquisition at different TSLs(44). Seven datasets of in vivo human knee 3D T1rho-weighted images were acquired with ten different TSLs using a modified 3D Cartesian turbo-Flash sequence at a 3T scanner. The network was trained on three image datasets and the remaining four image datasets were used for evaluation. The study found that VN provided superior image reconstruction performance than the conventional compressed sensing methods for an individual image at an acceleration factor R=2 to 6, which then resulted in improved T1rho maps with reduced errors and improved image details for knee cartilage (Figure 2). A further study from the same group also investigated the performance of VN in generating bi-component T1rho maps(45). They found the improved image quality at the individual image using VN can also benefit advanced tissue modeling of knee cartilage using multi-compartmental T1rho analysis, the quality of which is highly dependent on the image signal-to-noise ratio.

Figure 2:

Comparison of T1rho map estimated from VN with T1rho map produced using SENSE at R=4. The fully sampled SENSE was used as a reference standard. The advantages of VN over SENSE were faster image reconstruction with low reconstruction error. VN also performed better than SENSE with low error and bias in cartilage T1rho quantification. SENSE: sensitivity encoding; (Image courtesy of Marcelo V.W. Zibetti, Ph.D.).

Chaudhari et al. proposed to accelerate T2 mapping in knee imaging using a deep learning-based super-resolution technique to learn interslice information for low-resolution thick-slice images using high-resolution thin-slice images(46). In contrast to accelerating in-plane image acquisition through k-space undersampling like that of Liu et al.(18), Li et al. (43) and Zibetti et al. (44), the trained deep network allowed the restoration of through-plane resolution from thick slices which enables a large coverage of image volume with thin slices, at reduced acquisition times. The study initially used 3D dual-echo steady-state image datasets with 0.7mm slice thickness from 124 subjects in the Osteoarthritis Initiative (OAI) to train a 3D CNN, with an additional 17 patients used for network evaluation. The super-resolution algorithm was able to convert down-sampled thick-slice images into an image with a 0.7mm slice thickness. The super-resolution algorithm was found to outperform a conventional interpolation-based method, Fourier interpolation method, and a state-of-the-art sparse-coding super-resolution method at most down-sampling factors up to 8. This technique was also shown not to bias cartilage segmentation metrics, cartilage image quality, and osteophyte detection when compared with original high resolution images(47). Subsequently, transfer learning was used to adapt this CNN trained on 3D dual-echo steady-state images for 3D quantitative dual-echo steady-state images, that can simultaneously generate morphological images and T2 relaxation time measurements(48). On the comparison of T2 values, the super-resolution algorithm using deep learning is robust enough to generate minimally biased cartilage T2 relaxation time measurements, achieving the closet value of T2 in contrast to the thin reference slice (Figure 3).

Figure 3:

Example images comparing T2 relaxation measurements of the medial femoral cartilage computed using a quantitative dual-echo steady-state sequence with 1.6mm slice thickness (a) and the same sequence enhanced with deep learning super-resolution to two-fold enhance slice thickness to 0.8mm (b). Both techniques demonstrated a clear delineation between the deep and superficial cartilage (figure insets). The difference between the T2 relaxation measurements (c) demonstrated minimal systematic bias induced by the deep learning super-resolution technique. (Image courtesy of Akshay Chaudhari, Ph.D.).

Improved Tissue Segmentation on Parametric Maps

The segmentation of medical images plays a fundamental role in many imaging-based applications. The accurate segmentation of tissue structures forms a critical step in enabling quantification in medical research. Manual segmentation is highly dependent on human expertise and its repeatability and reproducibility are influenced by many factors including the level of human experience, the applied segmentation tools and the image quality, contrast and geometry(49). Besides, manual segmentation is typically time-consuming and susceptible to errors due to fatigue and distraction associated with human operation. Semi-automatic approaches that involve a certain amount of human interaction for initial automatic segmentation results, despite alleviating these problems, remain mostly influenced by subjective adjustment. There have been high needs in recent years to develop fully automatic segmentation methods in musculoskeletal imaging. In many medical image segmentation tasks, deep learning-based methods have achieved state-of-the-art performance for a variety of tissue structures(6,7). With the rapid advance of those techniques, a few studies have described CNN methods for musculoskeletal tissue segmentation on parametric maps.

Liu et al. implemented an efficient CNN for performing cartilage and bone segmentation on multi-contrast knee MR image datasets (9,10). This CNN uses a pair of convolutional encoder and decoder networks (50,51), where the encoder network performs data compression and feature extraction, and the decoder network combines useful image features to generate tissue binary masks (Figure 4). The network was evaluated on segmenting three image datasets with different contrasts including sagittal T1-weighted 3D spoiled gradient recalled-echo images in 100 subjects, sagittal fat-suppressed proton density-weighted 3D fast spin-echo images in 60 subjects, and sagittal multi-echo spin-echo T2 mapping images in 100 subjects. In contrast to the conventional state-of-the-art model or atlas-based segmentation methods, the convolutional encoder-decoder network provides a significant improvement in the segmentation accuracy and the reduction of segmentation time. In the assessment of T2 values for knee cartilages, the average T2 values of femoral, tibial, and patellar cartilage were 39.8ms, 36.4ms, and 32.0ms for the manual segmentation and 38.4ms, 37.2ms, and 33.1ms for the fully automated segmentation using deep learning. No significant difference between manual segmentation and deep learning method was found for the T2 values for femoral cartilage (p=0.641), tibial cartilage (p=0.512), or patellar cartilage (p=0.407) using paired Student’s t-test.

Figure 4:

An illustration of a deep learning-based segmentation framework using a convolutional encoder-decoder network for bone and cartilage segmentation. The convolutional encoder-decoder contains an encoder network and a decoder network where the encoder network performs data compression and feature extraction, and the decoder network combines useful image features to generate tissue binary masks. The output of the network is high resolution pixel-wise tissue classification, which is further processed with a 3D deformable process and regularized with image intensity-based tissue boundaries to produce accurate and desirable tissue surface. BN: Batch Normalization; ReLU: Rectified Linear Unit. (Image reprinted from Figure 1 in Liu et al. Deep convolutional neural network and 3D deformable approach for tissue segmentation in musculoskeletal magnetic resonance imaging. Magn Reson Med 2018;79(4):2379–2391; with permission.)

In another study, Norman et al. analyzed the automatic segmentation of cartilage and meniscus using deep learning and compared with manual segmentation to determine the accuracy and precision for morphology and relaxometry to study knee osteoarthritis (OA)(11). The authors performed an analysis using a convolutional encoder-decoder network (UNet(51)) for segmenting knee MR image datasets from two large subject cohorts, including 3D spoiled gradient recalled-echo images with T1rho and T2 weighting and 3D dual-echo steady-state images. The first data set consisted of 464 spoiled gradient-recalled acquisition in the steady-state T1rho-weighted image volumes for the relaxometry analysis for patients with ACL injuries, patients with OA (Kellgren-Lawrence [KL] = 1)(52), and healthy subjects. The baseline and 12-month follow-up studies from 49 of these subjects whose KL = 0 across time points were used to assess the precision of the deep learning method in extracting T1rho and T2 values. Only the baseline images were used to train CNN, and the 12-month follow-up data set was used as a holdout test for evaluating the precision of the automatic segmentation longitudinally. No significant difference was found between manual and deep learning segmentation for T1rho and T2 values across all cartilage compartments (Figure 5). They found a strong correlation with an average correlation coefficient of 0.8767 and 0.894 between manual and deep learning method for T1rho and T2, respectively, and the mean absolute difference between segmentation methods for T1rho and T2 were 2.1ms and 1.54ms, respectively. They also found no significant difference between the manual and deep learning method based relaxometry time point changes, showing the equivalence of manual and automatic procedures in assessing longitudinal disease progression. These findings indicate the potential of the use of deep learning to accurately and efficiently extract relaxation times that can be used to predict and monitor joint degeneration in OA.

Figure 5:

Scatterplots and Bland-Altman plots show a comparison of T1rho (top) and T2 (bottom) relaxation times produced from manual and automatic deep learning segmentation methods. There was a strong correlation with an average correlation coefficient of 0.8767 and 0.894 between manual and deep learning methods for T1rho and T2, respectively. FC: femoral cartilage, LTC: lateral tibial cartilage, MTC: medial tibial cartilage, PC: patellar cartilage, SD: standard deviation, Train: training. (Image reprinted from Figure 3 in Norman et al. Use of 2D U-Net Convolutional Neural Networks for Automated Cartilage and Meniscus Segmentation of Knee MR Imaging Data to Determine Relaxometry and Morphometry. Radiology 2018 Jul;288(1):177-185; with permission)

Byra et al. developed a deep learning method to segment knee meniscus in assessing knee OA with the automatic extraction of T1, T1rho, and T2*(13). The whole knee imaging was acquired for 61 subjects on a 3T scanner using 3D UTE-Cones sequences including 3D UTE-Cones-VFA for T1 mapping, 3D UTE-Cones-AdiabT1rho for T1rho mapping and 3D UTE-Cones-T2* for T2* mapping. A 2D attention UNet CNN was trained and validated on 36 and 10 subjects, respectively, and evaluated on the remaining 15 holdout datasets. A strong linear correlation was found between the manual and automatic deep learning method for calculating relaxometry values with the correlation coefficient ranging between 0.90 and 0.97. No significant difference was observed for calculating the average meniscal relaxometry parameters between manual and automatic deep learning methods.

Quantitative Imaging for Disease Diagnosis

There have been recent significant interests in using deep learning to perform lesion detection, progression, and prediction of musculoskeletal disease on X-ray radiography, CT, and MRI(19). Many pioneer studies have shown the potential of deep learning to maximize diagnostic performance while reducing subjectivity and errors induced by human interpretation. For example, Liu et al. developed a fully automatic cartilage lesion detection system using a cascaded deep learning algorithm to perform cartilage segmentation followed by localized cartilage lesion detection on MRI (53). The system had an averaged area under the curve (AUC) on receiver operating characteristic (ROC) analysis of 0.92 for detecting cartilage lesions with an averaged sensitivity and specificity of 83% and 87%, respectively. Later, the same group used a similar cascaded system to detect anterior cruciate ligament (ACL) tears on MRI with an AUC of 0.98 and sensitivity and specificity of 96% and 98%, respectively(54). Pedoia et al. used a UNet to segment patellar cartilage, which was then analyzed by a custom-made classification CNN to detect cartilage lesions(55). The machine had an AUC of 0.88 for detecting cartilage lesions with sensitivity and specificity of 80%. Roblot et al. used a fast-CNN to first segment meniscus and then determined the presence or absence of meniscal tears(56). Their machine had an AUC of 0.94 for detecting meniscal tears. Bien et al. used a single custom-made MRNet classification CNN to analyze multi-contrast MR images to detect the presence or absence of ACL and meniscal tears(57). Their machine had a sensitivity and specificity of 76% and 97%, respectively for detecting ACL tears and a sensitivity and specificity of 71% and 74%, respectively for detecting meniscal tears. In general, deep learning was found to provide equivalent or better diagnostic performance in most reported studies comparing between the automatic machine systems and human interpreters in the detection of musculoskeletal diseases. A comprehensive summary of reported studies can be found in a recent review article(19).

While most of these studies focused on the use of deep learning to detect tissue abnormality with anatomical MR images, there is very limited research that attempted to utilize the information from quantitative images. One pioneer study by Pedoia et al. investigated the diagnostic difference to distinguish knees with and without radiographic OA using conventional and deep learning-based T2 relaxometry measures(58). In their study, the T2 maps from 4384 subjects from the baseline OAI datasets were extracted and analyzed. A deep learning method using DenseNet(59) was trained to directly diagnose OA from the T2 maps with training, validation, and holdout testing set of a 65-20-15% split of the entire datasets. The deep learning was compared with shallow models, which combined the dominant principal components from the voxel-based relaxometry (VBR) analysis(60), demographic features and knee pain scores. They found the shallow model using VBR T2 maps outperformed the averaged T2 values and the models with demographic features outperformed the ones without demographics. The deep learning method, despite used the raw T2 maps, was found to provide the best diagnostic performance with an AUC of 0.824 and a sensitivity and specificity of 77% and 78%, respectively, in contrast to the best shallow model with an AUC of 0.778 and a sensitivity and specificity of 67% and 72%, respectively (Figure 6). The results showed that the deep learning system could learn useful T2 features that can potentially provide better diagnostic performance for OA than simple machine learning methods.

Figure 6:

Receiving operator characteristic curve comparing the Random Forest results between different feature combinations (a). Receiving operator characteristic curve comparing the best performant shallow classifier with the deep learning model (b). The shallow model using VBR T2 maps in combination with demographics, outperformed all other shallow models. The deep learning method that used the raw T2 maps in the combination of demographics provided the best diagnostic performance with an AUC of 0.824, outperforming the best shallow model with an AUC of 0.778. VBR: voxel-based relaxometry (Image reprinted from Figure 6 in Pedoia et al. Diagnosing osteoarthritis from T2 maps using deep learning: an analysis of the entire Osteoarthritis Initiative baseline cohort. Osteoarthritis and Cartilage 2019; 27:1002-1010; courtesy of Valentina Pedoia, Ph.D.)

Conclusion

There has been much recent interest in applying deep learning to a wide variety of imaging applications. While surveys on deep learning in medical image analysis have shown emerging applications in musculoskeletal imaging(19,30), the use of deep learning in quantitative musculoskeletal imaging remains somewhat limited. Deep learning methods have shown promising results for accelerating quantitative musculoskeletal magnetic resonance imaging for T2 and T1rho relaxometry, which might allow the incorporation of such imaging techniques into clinical practice. Deep learning methods have been shown to improve musculoskeletal tissue segmentation from T2 and T1rho maps, which allows efficient and accurate relaxometry analysis for monitoring and predicting musculoskeletal diseases. More importantly, multiple studies have documented the potential of using deep learning methods for fully automatic relaxometry analysis with diagnostic performance comparable to human readers for identifying pathologic abnormalities. However, much more technical development and clinical evaluation are warranted to investigate the applicability, reliability, and repeatability of deep learning methods to create a fully automated interpretation of musculoskeletal diseases. Furthermore, deep learning methods have shown promising results for disease detection on quantitative MR images with diagnostic performance superior to conventional machine learning methods for identifying knee osteoarthritis. The diagnostic performance of those methods must be thoroughly evaluated in large scale prospective studies before they can be implemented in clinical practice.

Key Points.

• Deep learning methods have shown promising results for accelerating quantitative musculoskeletal magnetic resonance imaging for T2 and T1rho relaxometry.

• Deep learning methods have been shown to improve musculoskeletal tissue segmentation on parametric maps, which allows efficient and accurate T2 and T1rho relaxometry analysis for monitoring and predicting musculoskeletal diseases.

• Deep learning methods have shown promising results for disease detection on quantitative magnetic resonance imaging with diagnostic performance superior to conventional machine learning methods for identifying knee osteoarthritis.

Synopsis: There has been recently great interest in the use of deep learning in musculoskeletal magnetic resonance imaging. However, applications in quantitative musculoskeletal imaging have remained relatively limited. This article will review the current use of deep learning in quantitative musculoskeletal imaging, including accelerated parametric mapping, image segmentation, and musculoskeletal disease diagnosis and prediction.