Prospective Deployment of Artificial Intelligence in MRI: A Framework for Important Considerations, Challenges, and Recommendations for Best Practices

Abstract

Artificial intelligence algorithms based on principles of deep learning (DL) have made a large impact on the acquisition, reconstruction, and interpretation of MRI data. Despite the large number of retrospective studies using DL, there are fewer application of DL in the clinic on a routine basis. To address this large translational gap, we review the recent publications to determine three major use cases that DL can have in MRI, namely that of model-free image synthesis, model-based image reconstruction, and image or pixel-level classification. For each of these three areas, we provide a framework for important considerations that consists of appropriate model training paradigms, evaluation of model robustness, downstream clinical utility, opportunities for future advances, as well recommendations for best current practices. We draw inspirations for this framework from advances in computer vision in natural imaging as well as additional healthcare fields. We further emphasize the need for reproducibility of research studies through the sharing of datasets and software.

The advent and success of artificial intelligence and machine learning has energized the realm of medical imaging (1). Within MRI specifically, machine learning has impacted a multitude of pulse sequences and nearly all anatomies (2). Deep learning (DL), a subset of machine learning algorithms, has provided the spark necessary to fuel all these innovations (3). Most DL algorithms rely on the use of convolutional neural networks (CNN), which are non-linear feature extractors that can represent images and other signals at varying levels of hierarchical detail. By aggregating these multi-scale local and global patterns that exist within images, DL networks are capable of complex tasks, such as classifying the content of images and image transformations. Given the high-dimensional and multi-channel nature of MRI data along with the variations that can be introduced by different imaging hardware, artifacts, and patient anatomies, DL becomes a promising choice to interpret these highly heterogeneous images.

Currently, a handful of deep learning based solutions for MRI have been approved by the FDA under the software as a medical device designation for tasks such as image reconstruction, enhancement, segmentation, and classification (4). However, there is a dearth of studies comparing clinical utility of such DL models against the standard of care in large multi-center trials. Consequently, a large translational gap currently exists between technology development and technology deployment. To address the challenges in overcoming this translational chasm, in this review article, we provide a framework for important considerations for training and prospective deployment of DL models, beyond simply maximizing the accuracy DL models. For a comprehensive review of the fundamentals of DL, we refer to several different recent articles that provide an excellent description of the mechanisms and strategies commonly used in DL (5–7). In contrast, the goal of this review is to evaluate critically the current state of translational MRI DL research and to provide a high-level summary of the different opportunity areas and challenges for future use cases.

FRAMEWORK FOR CONSIDERATIONS AND OPPORTUNITIES:

The ability to apply DL to a variety of MRI techniques and anatomies is simultaneously a benefit and a challenge. Each technique and clinical use-case presents a unique need and optimization challenge, especially because the exact mechanisms underlying DL training and generalizability are known only through a limited set of empirical observations. We broadly categorize the different application areas of DL in MRI into model-free image synthesis, model-based reconstructions, and either pixel- or image-level classification tasks (Fig. 1). To mitigate prospective deployment challenges for these techniques, we will draw inspiration from computer vision for natural imaging and different medical imaging modalities to focus on the following areas important for translation considerations in this manuscript:

Figure 1:

An overview of the three generalized use-cases of deep learning (DL) and convolutional neural network (CNNs) in MRI. The first category relies on model-free contrast synthesis that synthesizes one image contrast or artifact status into another (a). A model-based reconstruction uses k-space data and embeds physics of Fourier encoding and data consistency into an iterative deep learning model to act as an additional regularizer (b). Images can be input into DL networks for assessment of imaging abnormalities (classification) or for segmentation specific tissues (c).

1. Training Paradigm: This notion entails the curation of the datasets for DL, CNN architectures, associated labels for training data, and loss functions that are used to solve a task of interest.

2. Robustness: After a model has been trained, how well can it be deployed in a useful clinical and research setting in light of variations in hardware, software, scan parameters, along with patient anatomies and populations?

3. Downstream Analysis: After training and evaluating robustness, how does the model inference affect the knowledge required to perform the clinical task at hand or to guide future treatment?

4. Opportunities: Considering the current status of the MRI and computer vision, are there specific short-term questions or challenges that can be addressed to provide efficient and robust training techniques and to maximize information available to enhance patient care?

5. Recommendations: What evidence can be provided to increase clinical confidence and drive adoption of the proposed techniques?

MODEL-FREE IMAGE SYNTHESIS:

Prior to DL, learning features or representations of images was based on developing hand-crafted image features such as gradient operators and filter banks (8). With the advent of DL-based models, more complex features can be automatically designed and extracted from images by building models that are more expressive to determine subtle variations in images. A variety of such DL tasks in MRI incorporate learning a mapping from an input image space to an output image space by learning the representation of the input data in a fully data-driven manner. Such end-to-end models do not utilize any external priors and do not consist of any external considerations or constraints in the intermediate and final outputs of the network. The benefit of such networks is directly related to the expressive power of CNNs and their ability to rapidly and simultaneously learn low-level image features (such as edges and shapes) as well as higher level features (such as the complex organization of multiple shapes and textures in images) (9).

These data-driven DL models have been used for many common image synthesis tasks, including image super-resolution, image denoising, artifact reduction, and contrast synthesis. Super-resolution is a long-studied task in computer vision that entails the recovery and estimation of high-resolution features from a low-resolution image (10). In MRI, super-resolution has recently been used to improve the resolution of a rapidly acquired low-resolution image by more than what is achievable through common interpolation algorithms (11, 12). Similarly, image denoising can model the distribution of image noise in order to remove the noise without blurring the true underlying image signal (13, 14). Analogous to denoising, DL can reduce the number of averages or directions required for accurate diffusion weighted imaging (15–17). Reducing MRI artifacts arising from motion, off-resonance, and Gibb’s ringing has also been addressed using a data-driven DL approach (18–20). DL has also allowed promising research to synthesize one medical imaging modality contrast to another; for example, DL algorithms can use MRI to estimate the appearance of a computed tomography (CT) image, which is beneficial for positron emission tomography attenuation correction (21, 22). Additionally, model-free contrast synthesis methods have estimated the appearance of contrast-enhanced brain MRI scans with a low-dose gadolinium input (23) and even a virtual contrast enhancement without any gadolinium input (24).

Training Paradigm:

Most image synthesis tasks employ fully supervised machine learning, which requires that every input image has a corresponding output image. However, acquiring a perfectly registered high-quality input and output image is not always feasible clinically, which limits CNN training to small datasets. Further, since this class of algorithms creates output images, there is a chance that additional image details not fundamentally present in the fully sampled imaging may be inadvertently created by the algorithm. Protecting against such image hallucinations is an active area of research (25). One possible technique to mitigate hallucination artifacts is through the use of residual images, which only generate sparse feature differences between an input and output image (26, 27). This allows the residual image to utilize the input image to learn a fundamental set of priors that constrain the output of images (Fig. 2). Generating only a residual output also implicitly forces network weights to be sparse and to automatically regularize themselves, similar to L2-regularization techniques (28, 29). Moreover, depending on the clinical use case and the amount of sparse residual content expected, an additional regularization term to limit the magnitude of the output of DL predictions may also mitigate hallucinations. In contrast to the problem of ‘false-positive’ hallucinations in images, there is also a risk of false-negatives by excluding true anatomical details. For examples, subtle features may accidentally be smoothed out if considered to be image noise. A common approach to mitigate this entails training with adequate representation of expected patient anatomies, which requires careful curation of training datasets that represent an approximate probability of expected anatomical variations.

Figure 2:

A residual-based CNN uses a low-quality input image (a low-resolution image in this case) and tries to create a sparse difference image between the low-resolution and high-resolution image. The sparsity of the model output limits the creation of hallucinated artifacts. A computation of the loss is performed by comparing the high-resolution ground truth to the sum of the low-resolution input and the sparse residual images.

Using model-free DL to solve ill-posed contrast-synthesis problems for MRI has the benefit of being entirely data-driven with minimal constraints placed on the models during optimization. However, without a defined model, it may be challenging to distinguish between true image signals and artifactual signals. In such an approach, incorporating the MRI encoding process and placing constraints on the conceivable outputs of a DL model may act as an additional regularization parameter to mitigate the impact of hallucinations or non-probable image signal.

An additional challenge with model-free approaches is the creation of the retrospective paired input-output datasets. For example, to create low-resolution images in super-resolution, high-resolution DICOM images are commonly Fourier transformed and cropped in k-space with a rectangular window or high-resolution images may be simply blurred with Gaussian kernels (12, 30). While these techniques may produce adequate results for the DL network, the downsampling process may not always fully reflect MRI encoding and does not take into effect complex-valued k-space domain data, ringing, or filtering. As a result, such approaches may have limited generalizability when applied to prospectively-acquired low-resolution datasets since the DL models have not learned an accurate representation of true low-resolution data. Thus, for model-free approaches, it is critical to develop an accurate processing pipeline to generate realistic high-quality and low-quality image pairs for supervised training.

Robustness

While training model-free algorithms, the concept of adversarial robustness, which is the ability of a trained network to withstand small perturbations in network inputs while maintaining accurate representations of input datasets, should be carefully considered (31). The need for adversarial robustness is highlighted in natural imaging when a correctly classified image is misclassified with high confidence following the addition of an imperceptible amount of noise. Sensitivity to adversarial noise can often be linked to out-of-distribution (OOD) datasets, wherein a network trained on a narrow distribution of input data cannot accurately represent data sampled from varying distributions (32). Adversarial robustness may be especially important in the context of DL MRI, where algorithms trained using images with a particular signal-to-noise ratio (SNR) may not generalize to images that have a lower SNR (33). Being robust to such adversarial examples implies lower susceptibility to OOD effects. For model-free systems, it is challenging to ensure data consistency between inputs and outputs, which further emphasizes the need for training robust networks through realistic data augmentations that minimize deleterious OOD effects. Common MRI artifacts such as motion or SNR corruption can be incorporated for data augmentation with the acquisition of k-space data, which can be intentionally degraded to build network robustness.

Downstream Analysis

The goal of medical imaging is not simply to image anatomy non-invasively, but rather, to use the images to assess any underlying abnormalities and organ status in either a qualitative or quantitative manner. However, most image generation algorithms conventionally use optimization metrics such as the structural similarity (SSIM), or L1 and L2 norms during the DL training process (34). While these metrics provide acceptable image quality, they do not always correspond to perceived image quality or diagnostic utility as assessed by human readers (35). This notion was recently highlighted during the evaluations of the fastMRI challenge, which provided a large-scale dataset of raw k-space data for developing DL-based undersampled 2D fast-spin-echo MRI reconstruction algorithms. There was considerable discordance between optimization of metrics such as SSIM and peak SNR (pSNR) that are used during image reconstruction, as compared to the opinions of expert radiologists in ascertaining image quality (36). In such scenarios, it becomes important not only to perform image quality optimization, but also to simultaneously evaluate the downstream utility and biases emanating from the reconstructed images. For example, it was shown that cardiac MRI super-resolution can be used to improve segmentation outcomes (37), knee MRI super-resolution can improve segmentation of cartilage as well as detection of small osteophytes (Fig. 3), and brain MRI super-resolution can improve depiction of cortical parcellation (Fig. 4) (38). Prior work has demonstrated that prospective super-resolution can simultaneously enhance image quality while maintaining quantitative accuracy of parametric T2 mapping (39) as well as diagnostic accuracy of rapid MRI scans (40). Consequently, while developing new loss functions for optimizing image quality remains an active research area, using model-free reconstruction approaches and evaluating their downstream clinical utility has the potential to provide an understanding of the overall utility and clinical applicability of the proposed techniques.

Figure 3:

Example images from a super-resolution technique used for threefold improving the resolution of sagittal double-echo steady-state knee MRI scans. Ground-truth original high-resolution (a,d), super-resolution images (b,e), and tricubic interpolated images (c,f) of axial (a-c) and coronal (d-f) reformations show the improvement that super-resolution brings about, especially in the case of subtle features such as cartilage (dashed arrow), thin collateral ligaments (dotted arrow), and effusion (solid arrow).

Figure 4:

Enlarged views of axial slices from ground-truth (a,b), down-sampled (b,c), spline up-sampled (e,f) and deep-learning super-resolution scans (g,h) in the primary visual cortex, with reconstructed gray matter-white matter surfaces and gray matter-cerebrospinal fluid (CSF) surfaces visualized as colored contours. All scans are 0.7mm isotropic resolution nominally, with the exception of the down-sampled scan that has a resolution of 1.0mm. Reconstructed surfaces displayed in (i,j) are overlaid on the ground-truth 0.7-mm isotropic T1-weighted images. Magenta arrowheads highlight locations where super-resolution images provided improved cortical surface estimates.

Opportunities

One of the largest opportunities for potential advances in clinically useful model-free DL algorithms is the development of quantitative metrics that estimate image quality as perceived by humans. Instead of requiring reader studies to evaluate image quality, automatically incorporating image quality knowledge as priors in the DL model may lead to improved reconstructions. A particularly exciting approach entails the use of generative adversarial networks (GANs). Instead of a single DL network producing an output image, GANs consist of a generator network, which creates a plausible output image, along with a discriminator network, which seeks to evaluate whether the created image is realistic or not. While GANs may produce higher quality image features, they pose a greater risk of hallucinations since the exact optimization mechanisms have not been extensively explored. Feature (or perceptual) losses compare two images by computing the difference between high-level features extracted by a pre-trained CNN at different layers. By evaluating the feature-wise difference instead of a pixel-wise difference, they are better able to capture perceptual differences between two images (41). Feature losses have shown promise in maintaining high image quality and may serve as a compromise method that does not require the use of GANs (42). Most current techniques typically require the presence of a paired high- and low-quality images to determine the quality of the low-quality image. Developing blind image quality metrics for assessing blurring and SNR based on a single test image itself could greatly reduce the need for large paired datasets (38). Novel CNNs based on CycleGAN, which learns invertible mappings from one domain to another, may also provide an additional technique for mitigating the challenge of paired datasets (43).

Recommendations:

DL has shown great promise for a variety of image synthesis tasks that increase the value of MRI in medicine for patients, clinicians, and for hospital systems. However, most early work has been performed on retrospective datasets and as proof-of-concept studies. Translating these promising contrast synthesis techniques to prospectively-acquired datasets and understanding the variations in the performance of the algorithms on non-ideal and real-world datasets will be tantamount in ascertaining the clinical impact of image synthesis techniques. This would rely upon an improved understanding of the impact of MRI acquisition and relaxation processes and how they influence the low quality or fast input images versus the ground-truth that the algorithms were trained with. In addition to focusing on generalizability of algorithms across different MRI scanners and institutions, enhancing the generalizability of algorithms trained on retrospective data to perform well on prospective datasets will be the first step required prior to widespread clinical adoption.

MODEL-BASED RECONSTRUCTION

The fundamental goal of DL-based MRI reconstruction is to enable more rapid scanning by using deep neural networks to reconstruct high-quality images from relatively few k-space measurements. DL image reconstruction can be performed using a model-free image synthesis approach as discussed in the previous section. In such a model-free approach, a generative neural network is trained to learn a direct mapping between undersampled k-space data and corresponding high-quality reference images (44). More commonly, the undersampled k-space data is first naively reconstructed by zero-filling missing k-space points and a low-resolution aliased image is consequently mapped to the reference images (45–48). However, a more robust approach embeds physics-based models into the CNN to enforce consistency between intermediate network outputs and the acquired k-space data samples. Inspired by iterative algorithms used to solve constrained image reconstruction problems, model-based DL reconstructions iteratively process the raw k-space data by interleaving data consistency steps with data-driven regularization functions (49–54). During data consistency steps, the network assumes an MRI signal model comprised of coil sensitivity and Fourier encoding operators to project back and forth between image and k-space domains. During regularization steps, shallow CNNs are used to model and learn data-driven regularization functions instead of using hand-crafted regularization functions, such as sparsity or low-rank. This iterative process is unrolled to form a model-based DL reconstruction network which is trained end-to-end using standard gradient descent algorithms.

It should however, be noted that in certain scenarios, the line between model-based and model-free deep learning may be nebulous. For example, for some techniques such as artifact removal, artifacts can be generated for a supervised training dataset using a model of MRI physics (18). However, the training for this model can be performed in an end-to-end manner without an explicit model being incorporated into the network.

Training Paradigm

Many model-based DL reconstruction approaches are trained in a fully supervised manner. For instance, the most common approach is to collect fully sampled raw k-space data and perform retrospective undersampling by systematically excluding k-space lines to simulate a rapid acquisition. Corresponding input-output pairs are then formed using the retrospectively undersampled k-space data and fully sampled reference images. Training is performed by backpropagating through the entire network to iteratively minimize a loss function that aims to capture the dissimilarity between the network’s output images and the reference images in the training dataset. The loss function is most commonly chosen to be a pixel-wise L2 or L1 loss due to their stable training properties (55). However, these loss functions often cannot capture perceptual differences between images. For extreme undersampling rates, however, networks trained on these loss functions can produce images with excessive spatial and temporal blurring. Using advanced loss functions, such as perceptual and adversarial losses, to train model-based DL reconstructions may enable sharper images for high undersampling factors (46, 56).

When a limited amount of fully sampled training data is available, various forms of data augmentation can be used to artificially increase the number of training datasets. Common augmentation strategies include image translation, flipping, and cropping. For DL reconstruction tasks, many instances of the same raw data can be generated by pseudo-random undersampling as long as the sampling scheme is consistent with prospective data collection. In certain scenarios, however, it can be difficult or impossible to obtain fully sampled k-space data with high-resolution due to the physical limitations of MR acquisition. In these scenarios, scan-specific neural networks have been proposed to reconstruct missing lines of k-space without large databases by instead treating a fully-sampled calibration region as the training dataset (57). Other approaches include unsupervised learning to leverage large databases of undersampled k-space data, which can be much easier to collect than fully-sampled data, in order to train model-based DL reconstruction networks (58–60).

Robustness

Incorporating physics-based modeling into neural networks has the potential to act as a form of regularization, much like L2 weight regularization or dropout, by constraining the sub-space of possible output images (61). Ultimately, this improves generalization to data that was not seen during the training phase. For this reason, model-based DL reconstruction approaches require significantly less training data than model-free approaches. For example, many model-based DL reconstruction works describe only using 10 to 20 3D datasets to train their networks (50, 51, 62). However, the lack of large-scale datasets has currently made it challenging to verify the impact of additional training data. Moreover, model-based DL approaches have demonstrated remarkable generalization to data acquired with varied sequence parameters. For example, variational networks demonstrate high-quality reconstructions of images depicting tissue contrast not reflected in the training dataset. While robust to changes in tissue contrast, they also demonstrated residual artifacts in images with varied SNR not included in the training set (33). When varied reconstruction accuracy is observed for OOD data, transfer learning and model fine-tuning can be used to adapt domain knowledge learned on a larger dataset to a smaller dataset acquired with different sequence parameters (33, 63).

Downstream Analysis

Model-based DL image reconstruction has demonstrated potential to enable rapid MRI scan times without sacrificing resolution or SNR, as is necessary with more conventional image reconstruction techniques. For certain applications, faster scan times can result in more accurate downstream analysis. For example, Liu et al demonstrated that model-based DL reconstruction networks can be used to directly reconstruct pixel-wise T2 maps from 8-fold accelerated k-space data without compromising quantitative accuracy (64). Additionally, comprehensive scans which simultaneously acquire anatomic and quantitative information may not have the luxury of deferring to other sequences in case of motion artifacts (65, 66). Faster imaging enabled by model-based DL reconstruction may mitigate motion artifacts and improve the accuracy of downstream quantitative analyses, such as segmentation or parametric mapping. Prior work has shown that the total number of breath-holds necessary to acquire 2D cardiac cine images with whole heart coverage can be significantly reduced, and therefore ventricular volumes can be segmented and measured more accurately (Fig. 5) (67).

Figure 5:

To reduce the total number of breath-holds for 2D cardiac cine, a parallel imaging acquisition is typically used to enable 2-fold scan time acceleration (a). The segmentations of the left ventricular endocardial (blue) and epicardial (yellow) borders, and the right ventricular endocardial border (red) are shown for parallel imaging (b). With model-based DL reconstructions, 12-fold scan time acceleration can be used to achieve single breath-hold scan times without significantly impacted image quality (c). Additionally, DL-accelerated cardiac cine scans can help to reduce inevitable variations between breath-holds which can bias ventricular volume measurements in multi-breath-hold scans (d).

For both model-free and model-based DL tasks, it is important to ensure a framework for real-time reconstruction and quality control of images prior to a patient leaving the imaging facility. If a sequence relies on DL reconstruction or image enhancement and the prospective acquisition of data has image artifacts or is OOD for the trained networks, it is essential to either re-scan the subject with the same protocol or to defer to a protocol that does not rely on the DL post-processing. This necessitates near-real-time inference of the input data in a manner that the MRI technologist or automated algorithm can perform a rough quality-control assessment and choose to re-scan, if necessary. Such a built-in redundancy system may help ensure that patients are not required to return for additional imaging.

Opportunities

Although many works have demonstrated robustness in small cohorts, the robustness of model-based DL reconstructions is not yet fully understood in large, diverse cohorts. Antun et al demonstrated that model-based DL reconstructions can become unstable in the presence of small perturbations and structural changes in the underlying images (68). Using data consistency layers as described above in all models can mitigate the artifacts described in this instability study. Moreover, modeling plausible image corruptions that are likely to arise in real-world deployment instead of simulated patterns that will not arise clinically may provide an increased understanding of the robustness of the method. Opportunities also remain for developing mathematical tools to assess uncertainty in the DL reconstructed images, which can further apprise users of the robustness of the technique (69).

A particularly exciting opportunity that remains for improving model-based DL robustness is the incorporation of other intrinsic signal structures such as complex-valued convolutional layers (70, 71). Most deep learning frameworks do not support complex-valued deep learning and networks generally decompose the real and imaginary components into two separate real-valued channels as inputs into reconstruction models. MRI is inherently complex-valued and conserving phase information is important during reconstruction as well for downstream fat/water separation, phase-contrast imaging, displacement encoding, and quantitative susceptibility mapping applications. In addition, complex-valued convolutional operations have greater expressivity, possibly reducing the number of model parameters and the likelihood of overfitting on training data (70, 71). Novel activation functions such as the zReLU, cReLU, modReLU, and cardioid have been developed for maintaining phase data along with magnitude activations during gradient descent of the DL networks (71, 72). These activations ensure that not only is the network input is complex-valued, but also that the internal representation of the network is complex-valued and consistent with the acquired data (Fig. 6).

Figure 6:

Comparisons of model-based reconstruction networks, with example input zero-filled, real-valued convolutions only, complex-valued convolutions, compressed sensing (CS) with L1-wavelet regularization, and ground truth fully-sampled reconstructions for undersampling factors (R) of 2.3 and 7.40. The difference maps, scaled by a factor of 40, are displayed under each reconstruction. The complex-valued magnitude images appear sharper than the real-valued images and with reduced noise than the CS images. The complex-valued phase maps appear smoother, with red arrows indicating differences in visibility of subtle details.

DL for image reconstruction also provides an additional opportunity to improve the efficiency of reconstruction algorithms. Even if DL algorithms may not surpass image accelerations beyond compressed sensing (CS) for example, training an algorithm to implement CS as a CNN would allow for faster inference durations. Instead of requiring a time-consuming iterative reconstruction, which has limited the clinical impact of CS, a simpler forward pass of a trained CNN could achieve similar reconstructions in near real-time.

Recommendations

Prospective data collection is often necessary for training since most institutions do not store fully-sampled multi-coil k-space data. While large quantities of reconstructed images in DICOM format are stored in hospital databases, the authors recommend that these should not be used for simulating training data. After reconstruction, complex-valued multi-channel images are often interpolated, filtered, coil-combined, converted to magnitude, corrected for gradient non-linearity, and intensity-corrected before being written out as DICOM files. Through these various post-processing steps, valuable information that can be leveraged during the reconstruction, such as phase, is discarded. Moreover, the distribution of training data simulated from DICOM images may differ from that of prospectively collected k-space data, which may impact performance of deployed algorithms. Combining the advances in the representation abilities of CNNs along with physical constraints to which all MRI processes are inherently subjected to, likely improves stability of DL in MRI reconstruction. Using approaches such as data consistency during network training along with the leveraging complex data may enable robust deployment in prospective clinical use-cases.

To address the lack of readily available fully-sampled k-space data, there have been initiatives to collect and release large databases of fully-sampled 2D and 3D fast spin echo k-space data (73, 74). A practical challenge that may arise in sharing raw k-space data, especially in the case of high-resolution multi-coil dynamic imaging, is that of storage capacity. For example, approximately 1500 fully-sampled knee MRI scans from the fastMRI dataset require over 1TB of hard disk capacity (74). Leveraging cloud storage providers that have different pricing options as a function of access rates may help alleviate this challenge. Further, to offset the challenges in formatting and handling of raw k-space data, standardized approaches such as those adopted by the International Society for Magnetic Resonance in Medicine can help facilitate easier data sharing (75). Combining these formats with publicly available databases have not only served to enable development of novel DL image reconstruction frameworks, but also provided baselines to compare performance across different networks, much like ImageNet has done for the computer vision community.

CLASSIFICATION AND SEGMENTATION

Classification is perhaps one of the most common computer vision tasks performed in medical imaging with the advent of DL. Classification tasks can be grouped into two different levels: image-wise and pixel-wise classification, also known as segmentation. The former image-wise method may be used to answer questions such as “Is there a stroke in a brain MRI dataset?” to determine a single variable pertinent to the diagnostic status of the subject from either 2D MRI slices, 3D volumes, or multiple 3D sequences. A recent manuscript even highlighted the use of ultrashort echo-time MRI for evaluating COVID-19 imaging findings, for which DL may be able to assist with patient triage and prognostics predictions (76). Such image-level classification is the fundamental underpinning of computer aided detection (CAD) systems. On the other hand, a pixel-wise classification task may help answer “Which pixels in this brain MRI dataset correspond to a region of stroke?” by determining whether each pixel in a given image belongs to a specific class or not. Such pixel-wise classification, tasks have been used to report or isolate the morphology of specific image features of tissues. Numerous review articles have been published recently that summarize the findings from a variety of classification and segmentation tasks for medical imaging (77), MRI in general (78), as well as specialized MRI applications including musculoskeletal imaging (79, 80), breast imaging (81) and neuroradiology (82). In the following sections, we will critically analyze important considerations prior to deploying such algorithms clinically and will provide a perspective on some unaddressed challenges and opportunities to make a tangible clinical impact.

Training Paradigm:

Unlike model-based reconstructions where a physics-based model can lower the need for training data, classification and segmentation tasks typically require larger training datasets to learn accurate image representations and outcomes. To balance the need for large high-quality annotated datasets for MRI tasks, a primary consideration should be to set a minimum acceptable level for the requisite accuracy of the DL model. While setting a precise threshold can be challenging for many clinical scenarios, an approximate technique could entail comparing the performance of a DL model with that of multiple expert radiologists for the task in question. The goal of such an exercise should be to match the variability among the DL models and the expert radiologists to that of the inter-reader variability among the radiologists. Such an approach was applied in a study evaluating the performance of a DL model to detect tears of the anterior cruciate ligament in the knee (83). In this study, the authors further stratified their findings by including human readers with varying levels of expertise, such as residents, fellows, and experienced attending radiologists.

Once such an acceptable performance level of DL systems is established, the training datasets could be progressively subsampled to demonstrate a relationship between the size of an input training dataset versus the accuracy of the model. Prior studies have shown that the accuracy of DL networks scales with a power-law relationship with the size of the input training datasets, and is consistent amongst tasks in computer vision and natural language processing (84). This finding was further validated in a study for segmenting articular cartilage from knee MRI scans that demonstrated that independent of network architectures, increasing the size of the training dataset can predictably estimate the accuracy of the network using a power-law relationship (Fig. 7) (85). Moreover, this study also demonstrated the accuracy of the network architectures by re-training identical models with random seed initializations, which has further been replicated in natural language processing tasks also (86). Including error bars in the accuracy of the models is an important step in understanding the repeatability of the network to random perturbations, especially if multiple different networks will be trained using transfer learning strategies.

Figure 7:

Segmentation of femoral cartilage from knee double-echo steady-state scans shows that the accuracy of the model on an identical test set evaluated with a Dice Score Coefficient (DSC) follows a power-law relationship with the extent of training data with very high correlation coefficients. A power law function for y=f(x) is defined as y = α*x^β.

Robustness:

In contrast to image synthesis and reconstruction tasks, which receive supervision from local regions of images to reduce blurring, noise, etc; classification and segmentation tasks typically rely on a larger receptive field and require image cues from different non-local regions of the image. This distinction makes it important to ensure that all training images are free of artifact to avoid biasing network representation. In addition, a simulated stress-testing pipeline that mimics common artifacts that will likely be present during prospective deployment could help determine whether the model is appropriate for clinical use. For example, failed breath holds during myocardial infarct imaging and image ringing on brain MRI are common artifacts that may be encountered in clinical and research applications and can help determine the robustness of a segmentation network. Additionally, developing methods for detecting OOD images as described above could also help automatically detect if a particular image may not be suitable for analysis by a given trained network (87).

Downstream Analysis:

The goal of most classification and segmentation systems is to aid the radiologist in making a rapid and accurate diagnostic decision. Thus, to ensure that DL models can be as efficacious as possible, they should be tightly integrated into the routine workflow of a radiology practice. Without integration into the picture archiving and communications systems (PACS), the promise of DL models may not fully be realized because obtaining information from the DL model would require time-consuming consultation with a system that is not a part of routine workflow. Recent guidelines provided through the Integrating the Health Enterprise (IHE) address workflow management and the incorporation of DL algorithms for medical images in clinical settings (88, 89). These encourage the use of common standards such as the DICOM format for imaging data and Fast Healthcare Interoperability Resources (FHIR) for electronic medical record data. They emphasize AI results being presented to radiologists in a standard reading environment in a supplementary manner to existing workflows, taking into account worklist prioritization and continual learning algorithms that learn from previous radiologist-AI shared interactions.

Further, even if a successful and seamless integration with PACS is possible, not enough is known about how the information from models should be presented to the radiologists. Researchers have only recently started to study how best to pair a human expert with a DL classification model (90). In the case of a pathology images, one study showed that when a DL pathology classification model was correct, it increased the aggregate accuracy of 11 pathologists, but when the model was incorrect, it decreased the accuracy of the pathologists, compared to baseline (91). Experiences in interaction between human users and CAD software from mammography may provide powerful lessons in determining how to best integrate decision support tools clinically (92, 93). A recent meta-analysis found that CAD in mammography did not always increase sensitivity of studies and actually decreased specificity in some studies (94). Additionally, compared to radiologists who read large volumes of mammograms, CAD was more beneficial for lower volume readers. These findings suggest that designers of DL CAD models should critically understand the tradeoffs between positive and negative predictive values of models, based on specific use-cases and prospective users. In lower resource or high-volume settings, CAD could also be used as a second reader as a part of double read systems.

Opportunities

A common criticism of DL algorithms, especially those used for image classification, is their ‘black-box’ nature, alluding to the difficulty of users to understand how they function. Unlike traditional signal processing tools, no transfer function or point-spread function can specify the optimal behavior of an arbitrary model for a given set of inputs. To remedy this challenge, concepts such as integrated gradients and saliency maps have been proposed (95, 96). These powerful approaches provide a pixel-wise attribution to estimate how important a particular pixel was for the final model classification. Instead of only performing such interpretability analysis on a few examples from a testing dataset, there lies a large opportunity in performing this technique on the entire training and testing datasets to uncover latent but clinically meaningful relationships. Pairing saliency with uncertainty analysis can also provide additional context and confidence to the interpreting radiologist. Similar to the model-based reconstruction approaches, embedding prior human knowledge with the data-driven capabilities of DL networks may also help achieve high model and decision interpretability.

DL also has an opportunity to show clinical impact by performing entirely in-silico retrospective research studies. When a large external dataset is available, it can be used to show that replacing manual assessments with automated segmentation and classification provides the same study and patient outcomes, thereby enabling model deployment with higher confidence. For example, a prior study showed that an automated segmentation algorithm for thigh muscle areas can replicate effect sizes determined with manual segmentation on a clinical study of subjects experiencing osteoarthritic pain (97). Similarly, DL can further complement and enhance the high-value sequences in MRI studies where automated segmentation methods can be used for analysis of the entire imaging volumes, instead of single slices only (98).

In such clinical and research studies, there are often circumstances when either a longitudinal timepoint or a specific scan sequence is missing from the overall analysis. DL models can be made robust to such incomplete datasets by augmenting training datasets by randomly excluding network inputs. Such an approach, termed DropOut, which randomly excluded MRI sequences during network training for segmenting brain metastasis showed superior results compared to an approach that used complete MRI contrasts (Fig. 8) (99).

Figure 8:

Case examples from a post-gadolinium-enhanced T1-weighted image series (a) for the MRI sequence DropOut method showing the segmentation predictions using (b) DeepLab V3 and (c) the proposed method. Segmentation predictions are shown as voxel-wise probability maps (ranging from 0.5 to 1) and performance maps classified as true negative, false positive, and false negative as specified by the color code. The first row shows a 53-year-old female patient with malignant melanoma, while the second and third row show a 55-year-old female with malignant melanoma. The arrows indicate true positive lesions (blue) and false positive lesions (yellow). Note that in the last row, the DeepLab V3 show false positive lesions which are not reported by the DropOut method. Figure courtesy Dr. Endre Grøvik from the ongoing TREATMENT clinical study (NCT03458455).

Recommendations

For determining beneficial use cases for MRI classification and segmentation, we recommend developing a robust pipeline to ensure that common MRI artifacts do not cause networks to perceive real-world images as OOD. By sub-sampling the training data, the amount necessary for the given clinical task may be coarsely extrapolated. This extrapolation can support an analysis to ensure that all studies using DL are adequately powered to detect subtle changes. Developing approaches to quantify the uncertainty of model outputs also can help radiologists during image interpretation (100, 101). Combining multiple models for classification and segmentation models tailored to specific clinical protocols may also assist in differential diagnoses for radiologists, as was demonstrated in the case of neuroradiology (102).

Lastly, Occam’s Razor suggests that when multiple models are being used for a downstream task, the best approach is to use the model with the fewest assumptions. Especially with newer DL models using millions of trainable free parameters, developing the simplest models may provide the best results for troubleshooting and during implementation. A study of a segmentation challenge organized in 2019 showed that simpler models often perform on par with larger and more complex models (103).

REPRODUCIBILITY:

One of the largest differences between computer vision for natural imaging and for medical imaging, is likely that of reproducible experiments. This difference can be further broken down into reproducible data and reproducible experimental setups. The computer vision community in natural imaging has flourished due to the existence of large scale datasets such as ImageNet, Microsoft Common Objects in Context (MS-COCO), etc (104, 105). Concerns with patient health information and adequate anonymization techniques make data sharing of MRI considerably more challenging than in natural imaging. However, the presence of large-scale efforts such as those in fastMRI and MRNet that provide high-quality data with appropriate annotations (fully-sampled k-space for fastMRI and acute injuries for knee MRI in MRNet) are encouraging efforts for the entire research community (74, 106). While sharing of datasets is essential to advance the DL in medical imaging community as whole, sharing of data must be governed with a strong ethical conduct (107).

Although sharing of code faces much fewer hurdles than sharing data, it is still uncommon, with only a handful of publications openly releasing software resources. Code sharing can create a powerful generalized framework for the entire research community, both to replicate experiments with varying datasets and to improve the speed and efficiency of research. Furthermore, since DL can involve a large variety of hyperparameters, there is a significant need for improved and standardized reporting of the procedures used to train DL algorithms. The recently developed Checklist for Artificial Intelligence in Medical Imaging (CLAIM) checklist provides a solid foundation for encouraging adequate reporting of parameters (108). Going further than CLAIM, we recommend that all manuscripts also report their hyperparameter optimization strategies, along with a summary of which parameters were successful and which were not. Reporting unsuccessful parameters can help other researchers avoid similar mistakes and can mitigate the economic and environmental impacts of machine learning research (109). For DL to advance into the clinic faster, such code and data sharing practices should have a higher priority than they do currently.

CONCLUSION AND SUMMARY

Deep learning has the potential to improve every step of the MRI diagnostic workflow and to provide value for every user, all the way from the technologists performing the scan, the physician ordering the imaging, the radiologist providing the interpretation, and most importantly, the patient who is receiving the medical care. Before these benefits are realized, there is a pressing need to systematically characterize the robustness of our current deep learning processes and to improve our understanding of the success and failure modes of these algorithms. We have addressed some of the major challenges to bridge the gap between technology development in deep learning and clinical deployment of these methods.

Our overview of DL applications in MRI focused on techniques that have potential and utility for clinical deployment. We paired these techniques with a framework for important considerations prior to the beginning of new deep learning studies in order to fully understand key questions and the challenges that can arise in the deployment of these tools. We hope that by summarizing the current challenges and opportunities, by and recommending best practices for the use of deep learning, MRI becomes an even higher-value modality.

Table 1:

A summary of the framework for the analysis of deep leaming (DL) algorithms for the application areas of model-free image synthesis, model-based reconstruction, and classification and segmentation problems.

	Model-Free Image Synthesis	Model-Based Reconstruction	Classification and Segmentation
Training Paradigm	• Creation of paired datasets • Guarding against hallucinations or missing details	• Choice of appropriate loss functions • Challenge in acquiring fully sampled data	• Understanding the accuracy necessary for clinical task • Determining an adequate sample size for required accuracy • Impact of random seeds
Robustness	• Adversarial robustness • Out-of-domain detection	• Incorporation of data consistency and physics-embeddings into models	• Adversarial robustness to commonly experienced imaging artifacts
Downstream Analysis	• Biases in diagnoses and tissue quantification	• Accurate quantification and interpretation of images • Real-time inference for quality-control and minimizing call-backs	• Integration into routine clinical workflow • Evaluation how to best present outputs
Opportunitles	• Quantitation of perceptual image quality mètrics • Blind image quality mètrics • Improveddata augmentations	• Model uncertainty and adversarial robustness • Use of complex-valued neural networks • Speeding up iterative reconstruction processes	• Going beyond analysis of saliency on single test images • Performing in-silico studies to determine similarity of outcomes • Guard against missing input images
Recommendations	• Demonstrating models on prospective real-world data	• Using complex-valued input data with complex convolutional layers and data consistency	• Developing simulation pipeline to improve robustness to OOD data • Using the simplest model