Upstream Machine Learning in Radiology

Introduction

Machine learning approaches have many applications in radiology. Over the last several years, the majority of attention has been garnered by applications focused on assisted interpretation of medical images. However, a significant component of radiology encompasses the generation of medical images prior to interpretation, the upstream component. These elements include modality selection for an exam, hardware design, protocol selection for an exam, data acquisition, image reconstruction, and image processing. Here, we give the reader insight into these aspects of upstream Artificial Intelligence (AI), conveying the breadth of the emerging field, some of the techniques, and the potential impact of the applications.

Discussion

The standard radiology workflow can undergo seven steps; 1 - conversion of a patient’s clinical question into a radiology exam order, usually performed by a non-radiologist physician, 2 – conversion of an exam order into an exam protocol, usually performed by a radiologist or radiology technologist based on institution-specific guidelines, 3 – scheduling of an exam onto a specific scanner, usually performed by a radiology schedular or technologist, 4 – adaption of a protocol to a specific device, usually performed by a radiology technologist, 5 – acquisition and reconstruction of acquired imaging data into images, usually performed automatically with software algorithms directly on the imaging hardware, 6 – image processing, to retrospectively improve image quality and produce multiplanar reformations and quantitative parameter maps, typically performed directly on the imaging hardware, and 7 – evaluation of images to render a diagnosis related to the patient’s clinical question, usually performed by a radiologist (Fig. 1). Applications and research areas for AI algorithms in radiology have typically focused on the final 2 steps in the radiology workflow.¹ Several AI techniques exist to perform automated classification, detection, or segmentation of medical images, many of which are even approved for clinical use by the Food and Drug Administration (FDA). Furthermore, the acquisition of raw data for generating diagnostic quality images is also being used for reducing scan time or radiation dosage of medical images, or retrospectively improving resolution or signal to noise ratio (SNR) of images.

Figure 1:

A summary of the radiology workflow starting from the ordering of an examination and ending with a radiologist interpretation of the acquired imaging. Errors during each of the seven stages of this workflow can lead to delays or the rendering of an incorrect final diagnosis decision.

Despite a dearth of research studies, there exist substantial challenges for all steps in the ‘pre-image-acquisition’ workflow (steps 1-4) that may be alleviated using data-driven benefits that AI techniques can offer. For example, a patient’s initial exam order may not always be correctly defined by a radiology specialist, which can lead to delays in insurance authorizations and the schedule of the correct examination. Next, conversion of an order to a protocol is commonly performed by radiology fellows at many academic institutions, while many private practice organizations may not have such a luxury. Furthermore, with ‘protocol creep’ of new protocols being aggregated in an unorganized manner, choosing the appropriate protocol for the patient becomes a more challenging task.² The scheduling of patients to different scanners also requires considerations for complex cases that may require a radiologist presence. Following successful protocoling and scheduling, tuning the scanner parameters to cater to the specific exam order needs and patient requirements (such as image resolution, coils, and breath holds for MRI, dose optimization and contrast needs for CT, etc.) can directly affect image quality, and consequently, downstream utility for eventual image interpretation.

Errors can occur during any part of the radiology pre-image-acquisition workflow and can lead to delayed diagnoses, or, in the worst case, missed diagnoses. Each of these four steps is repetitive, but because of the innate variability between patients, these tasks have not lent themselves to automated techniques. Using manual techniques to complete such tasks is prone to user error, leads to delays in the workflow, and is an inefficient use of human capital particularly when these tasks require radiologist oversight. The application of pre-image-acquisition AI can lead to potential gains in efficiency and patient care. There are several factors that these tasks ripe for automation with AI techniques. First, all the tasks are already performed on computer systems, entailing that all inputs to and output of the tasks are already digitized, recorded, and can be retrospectively extracted. Second, as large volumes (hundreds to thousands) of most exam types are performed every year, even at a single academic center, large institution-specific data sets needed to train AI algorithms are already available. Finally, a stepped pathway for incorporating these algorithms into the radiology workflow can be achievable. Initial versions of the ML algorithms can suggest or pre-select choices for use as a clinical decision support system with continued human oversight. Following this, continual-learning strategies can be used to improve such algorithms to have higher reliability than humans for eventual transition to full automation.

Recent advances in natural language processing (NLP) have been utilized to perform automated assignment for CT and MRI protocols using procedure names, modality, section, indications, and patient history as inputs to guide a clinical decision support model.³ Similar NLP systems have also been previously used to use unstructured data for performing automated protocoling for musculoskeletal MRI examinations^4,5 as well as brain MRI examinations.⁶ Newer AI techniques for NLP that utilize the Bidirectional Encoder Representations from Transformers (BERT) model⁷ have also been used in the context of automated protocoling.⁸ Advances in multi-modality data integration presents an exciting opportunity to improve the upstream workflow by further combining data from a patient’s medical record, along with AI models that rely on conventional computer vision and NLP techniques.⁹

Although the techniques and applications of upstream AI are broad, below we discuss some representative examples across various commonly utilized medical imaging modalities.

Improved Image Formation in Ultrasound

In ultrasound image reconstruction, raw echo data is received by an array of transducer sensors. This “channel data” is processed to produce an image of the underlying anatomy and function. Traditional ultrasound image reconstruction can include various noise artifacts that degrade signal and image quality. Rather than attempting to correct these using already-reconstructed images, upstream ultrasound research utilizes the raw channel data to address the artifacts at their source. AI has found a wide range of early uses in upstream ultrasound, a few of which we highlight here.

A major source of ultrasound image degradation is reverberation clutter.¹⁰ Clutter arises from shallow reverberations that are incorrectly superimposed upon echoes from deeper tissue, resulting in decorrelated channel data. This decorrelation manifests in ordinary B-mode images as a dense haze and also degrades the ability to perform Doppler and other advanced imaging techniques. Several non-AI techniques have been proposed to remove clutter from channel data directly, improving channel data correlations and B-mode image contrast, albeit at great computational cost.^11-13 Recently, AI has been used to achieve the same goal using a data-driven approach.^14,15 In our work, we used matched simulations of ultrasound sensor data with reverberation clutter (input) and without reverberation clutter (output) to train a convolutional neural network (CNN).¹⁵ Figure 2 demonstrates how the CNN removes high-frequency clutter from simulated channel data. In Figure 3, the CNN removes clutter from in vivo channel data, resulting in B-mode images with improved structure visualization and contrast. The CNN restores correlations in the raw channel data and thus enables subsequent advanced imaging techniques, such as sound speed estimation and phase aberration correction.

Figure 2:

Images of simulated channel data comparing cluttered, CNN-filtered, and reference uncluttered images. Reverberation clutter, which appears as a high frequency noise across the channels, is removed by the CNN while preserving the structure of reflections from true targets. Data from Brickson LL, Hyun D, Jakovljevic M, Dahl JJ. Reverberation Noise Suppression in Ultrasound Channel Signals Using a 3D Fully Convolutional Neural Network. IEEE Trans Med Imaging. 2021;PP.

Figure 3:

B-mode images of a longitudinal cross-section of a carotid artery and thyroid. The CNN-filtered image visualizes several hypoechoic and anechoic targets that were originally obscured by clutter. Data from Brickson LL, Hyun D, Jakovljevic M, Dahl JJ. Reverberation Noise Suppression in Ultrasound Channel Signals Using a 3D Fully Convolutional Neural Network. IEEE Trans Med Imaging. 2021;PP.

Another major noise source in ultrasound images is a pervasive grainy texture called speckle. Speckle is an unavoidable artifact arising from the finite resolution of ultrasound imaging systems. While speckle can be useful for certain applications, it is largely treated as an undesirable noise that degrades B-mode images of the underlying tissue echogenicity. Rather than filtering speckle from already-reconstructed B-mode images, we trained a convolutional neural network (CNN) to estimate the tissue echogenicity directly from raw channel data using simulations of ultrasound speckle and known ground truth echogenicity.¹⁶ In simulations, the trained CNN accurately and precisely estimated the true echogenicity. In phantom and in vivo experiments, where the ground truth echogenicity is unknown, the CNN outperformed traditional signal processing and image processing speckle reduction techniques according to standard image quality metrics such as contrast, contrast-to-noise ratio, and signal-to-noise ratio. Figure 4 shows an in vivo example of a liver lesion acquired using a clinical scanner. This CNN has been further demonstrated in real-time on a prototype scanner.¹⁷

Figure 4:

CNN-based ultrasound speckle reduction and detail preservation. (left) A traditional B-mode image of a complex focal liver lesion. (right) The CNN output when provided with the same raw data. Data from Hyun D, Brickson LL, Looby KT, Dahl JJ. Beamforming and Speckle Reduction Using Neural Networks. IEEE Trans Ultrason Ferroelectr Freq Control. 2019;66(5):898-910.

Another emerging application in ultrasound is contrast-enhanced imaging, including with targeted contrast agents, to detect the presence of disease. Contrast-enhanced images are presented alongside ordinary B-mode images to highlight any disease-bound contrast agents in the field of view. A key challenge in contrast-enhanced imaging, particularly molecular imaging, is to detect echoes from contrast agents while suppressing echoes from background tissue. Current state-of-the-art imaging identifies molecular signals retrospectively after a strong destructive ultrasound pulse eliminates them from the field of view, and thus cannot be used for real-time imaging. We recently used CNNs to nondestructively emulate the performance of destructive imaging, enabling real-time molecular imaging for early cancer detection.¹⁸ Rather than treating B-mode and contrast-enhanced modes as separate images, the CNN combines raw data from both to improve the final molecular image. As shown in Fig. 5, the trained CNN (panel D) was able to mimic the performance of the state-of-the-art destructive approach (panel C), enabling high quality real-time ultrasound molecular imaging.

Figure 5:

Ultrasound molecular imaging of a breast cancer tumor in a transgenic mouse. (A) B-mode image, with the tumor in the center. (B) Contrast-enhanced image overlaid on the B-mode image. (C) Destructive state-of-the-art molecular image. (D) Output of the trained CNN using only nondestructive input data. The nondestructive CNN closely matched the destructive image, showing the potential for AI-enabled real-time molecular imaging.

In addition to these, AI has found numerous other upstream ultrasound applications, including adaptive beamforming,¹⁹ ultrasound localization microscopy,²⁰ surgical guidance,²¹ vector flow Doppler imaging, ^22,23 as well as many others.²⁴ The range and variety of applications highlight the extraordinary adaptability of AI techniques to ultrasound signal and image processing.

CT Dose Optimization

Over 75 million computed tomography (CT) scans are performed annually in the United States, with the number expected to continue increasing.²⁵ Despite clear clinical value, ionizing radiation dose has been a major concern for CT. In addition, there are significant variations in radiation dose and image quality across institutions, protocols, scanners, and patients. While CT has evolved from the manual adjustment of dose control parameters such as tube current, to automatic tube current modulation, CT dose optimization still has major challenges in achieving the most efficient use of radiation dose for each patient and clinical task.^26,27 An overview of existing technologies for controlling dose and image quality is shown in Fig. 6.

Figure 6:

Various technologies for controlling CT dose and image quality are listed under the categories of system design, acquisition, and reconstruction. Sub-figures show (left to right): overall scanner geometry, x-ray spectra from different tube voltages, and two reconstructions with different kernels showing sharper image but higher noise (top) and lower noise but smoother image (bottom).

Reconstructed CT images contain quantum noise when the ALARA (as low as reasonably achievable) principle is followed to reduce radiation exposure. As a result, important details in the image could be hidden by the noise, leading to degradation in the diagnostic value of CT. It is therefore crucial to assess the quality of reconstructed CT images at reduced dose. Dose and image noise have a complex dependency on patient size (ranging from pediatrics to obese adults), patient anatomy, CT acquisition, and reconstruction. As can be seen in Fig. 7, CT images at lower doses relative to the routine dose induce higher noise. Unfortunately, there is no consistent image quality assessment measure that has been established to objectively quantify images from different imaging protocols, vendors, sites, etc.²⁸ Having such capabilities would help balance image quality and radiation dose. At every step of the CT scan, from system design to acquisition to reconstruction, efforts could then be made to lower radiation dose.²⁹ As a result, the quality of the acquired CT images could be varied and assessed to guide dose reduction. In addition, effective dose to the patient needs to account for the different radiosensitivity of different organs (for example using the ICRP tissue weights).³⁰ Therefore, an accurate, reliable, and region-focused assessment of dose and image quality is extremely important.

Figure 7:

CT slice of an abdomen scan showing higher noise at lower dose levels. Mayo patient CT projection data library⁹³ provides routine (100%) dose and low-dose (25%) after inserting noise in the projection data. The additional four dose levels are synthesized from these two, assuming only quantum noise.

In an effort to combat increased noise in low-dose settings, CT image reconstruction has been going through a paradigm shift, from conventional filtered backprojection (FBP) to model-based iterative reconstruction (MBIR), and now to deep learning image reconstruction (DLIR).³¹ FBP, although fast and computationally efficient, incurs higher noise and streak artifacts at low dose. Towards improving image quality at low dose, iterative reconstruction has emerged which preserves diagnostic information, although low-contrast detail remains a challenge and can impact detectability of subtle pathologies. With the advent of machine learning and deep learning in particular, high diagnostic quality is created with reduced noise at low dose, with the additional benefit of faster reconstruction than iterative reconstruction.³² For example, 1) generating standard-dose images from low-dose images in deep learning-based CT denoising³³; 2) through incorporating Deep CNN in the complex iterative reconstruction process, less noisy images can be obtained at low dose³⁴; 3) improved image quality of dual energy CT, particularly for large patients, low-energy virtual monoenergetic images, and material decomposition³⁵; 4) reduction of metal artifacts through deep learning techniques.^36,37 Deep learning-based image reconstruction enables significantly lower image noise and higher image quality (IQ score and better CNR) over FBP and MBIR, which can be traded off for dose reduction.³⁸ However, dose reduction needs to be balanced against an acceptable level of diagnostic accuracy. Therefore, not only phantom data but also patient data are needed to understand the impacts of dose reduction on image quality (using objective metrics) as well as the task-specific diagnostic performance (such as detecting liver lesions).³⁹

Conventionally, medical images are assessed subjectively requiring manual labor, long times, and observer variations, which is impractical for use in a clinical workflow. It is therefore desirable to perform fast, automatic, and objective image quality assessment. Objective assessment can be of two types: reference-based and non-reference-based. A number of reference-based image assessment metrics have been widely used in medical imaging. For example, PSNR, MSE, CNR, SSIM, and RMSE are often used. However, these metrics rely on the availability of high-quality reference images. Since our goal is to acquire CT scans at reduced dose, we cannot assume to have access to high-quality (and high-dose) images of the same patient.⁴⁰ Therefore, non-reference assessments are preferred so that we can quantify any CT image without requiring the corresponding reference images at high dose. In addition to the established image quality assessment metrics, a few non-reference assessment methods are also available, such as a blind/referenceless image spatial quality evaluator based on natural scene statistics that operates in the spatial domain.⁴¹ A growing body of research has recently emerged focusing on no-reference and automated image quality assessment using deep learning. For example, Patwari et al. used a regression CNN to learn mapping noisy CT images to GSSIM scores, as an assessment of the reconstructed image quality.⁴² Kim et al. proposed a no-reference image quality assessment framework via two-stage training: regression into objective error map and subjective score.⁴³ Another deep learning-based non-reference assessment has recently been proposed leveraging self-supervised prediction of noise levels.⁴⁴

In summary, reduction of CT dose directly impacts image quality. Preserving image quality at low dose has been an active area of research. Deep learning-based methods have achieved some promising results in CT image reconstruction, denoising, artifact reduction, and image quality assessment. However, to bring dose optimization to CT acquisition, prediction of the expected dose and image quality has to be real time, patient specific, organ specific, and task specific. This is a complex and challenging problem for which deep learning algorithms are likely to excel.

Optimized Data Acquisition in MRI via Deep Learning

In MRI, data is acquired in the Fourier domain, or k-space, which represents spatial and temporal frequency information in the object being imaged. Sampling of k-space is necessarily finite. Therefore, optimizing sampling patterns and designing k-space readout trajectories that enable collecting the most informative k-space samples for image reconstruction can improve the resulting diagnostic information in images. Besides improved image quality, the design of optimized sampling trajectories can lead to shortened data acquisition for improved efficiency. Recently, several deep learning methods have been proposed for learning k-space sampling patterns jointly with image reconstruction algorithmic parameters. These methods have shown improved reconstructed image quality compared to reconstructions obtained with predetermined trajectories such as variable density sampling. Deep learning methods for data sampling can be separated into two categories: active and non-active (fixed) strategies. These two strategies differ primarily in whether the learned sampling schemes are fixed or not at inference time.

Given a target k-space undersampling or acceleration factor, the non-active (fixed) strategies produce undersampling masks or sampling trajectories using a set of fully-sampled images. After the training procedure is completed, the learned sampling pattern is fixed, and new scans can be acquired using the learned trajectory (Figs 8A&B). Image reconstruction can then be performed using the reconstruction network learned as a part of the training process. Several studies^45,46 focus on the Cartesian sampling case and model binary sampling masks probabilistically. These methods can be applied to either 2D or 3D Cartesian sampling to determine the optimal set of phase encodes. As an extension to more general parameterized, or non-Cartesian, k-space trajectories,⁴⁷ one can directly optimize for k-space coordinates by restricting the optimization to a set of separable variables such as horizontal and vertical directions in 2-D plane. The framework presented in Weiss et al. additionally incorporates gradient system constraints into the learning process.⁴⁸ This enables finding the optimal gradient waveforms in a data-driven manner.

Figure 8:

Non-active and active data strategies for deep learning based data sampling methods. Non-active (fixed) strategies optimize sampling trajectories and reconstruction networks at training time (A). At inference time (B), the sampling trajectory is fixed, and the corresponding gradient waveforms are programmed in the scanner for acquisition. The optimized reconstruction network is then used for reconstructing images from undersampled measurements. Active strategies (C&D) use an additional neural network that suggests the next sample to collect using the reconstruction obtained from existing samples. The process is repeated until a desired metric or uncertainty threshold is met. Due to their sequential nature, active strategies require an additional mechanism that generates gradient waveforms on-the-fly for acquiring the samples proposed by the sample selection network.

Active acquisition strategies,^49-52 on the other hand, attempt to predict in real-time the next k-space samples to be acquired using information from existing samples from the same acquisition, i.e. the same patient/scan (Figs 8 C&D). These methods employ an additional neural network that suggests the next sample to collect by measuring the reconstruction quality or uncertainty during the acquisition. Active techniques therefore have the benefit of obtaining subject-specific data acquisition strategies. In other words, active sampling strategies tailor the sampling pattern to new scans and patients.

In general, the non-active acquisitions are easier to implement in scanner hardware. However, such acquisitions can be suboptimal on new types of data or scans. For example, a sampling pattern that is optimal for brain imaging might not be the best trajectory for abdominal imaging. On the other hand, active acquisitions could adapt to new kinds of data at the expense of added complexity in determining and generating gradient waveforms on-the-fly.

Optimization of data sampling using deep learning is an active area of research and many open questions exist for further investigation. The effects of imaging non-idealities such as off-resonance, relaxation effects and motion on the optimized sampling pattern have not yet been fully investigated. In addition, the findings on deep learning-based data sampling approaches in MR literature typically rely on retrospective studies and simulations. Their applicability to prospective studies and clinical settings should be investigated to understand the full potential and applicability of learned sampling patterns.

Image Reconstruction in MRI: Supervised Deep Learning

To reconstruct rapidly acquired k-space data into high-quality images, advanced image reconstruction methods are necessary. Previously, parallel imaging and compressed sensing (PI-CS) methods^53,54 have been used to iteratively reconstruct highly undersampled data, thus enabling rapid MRI scans. However, due to their iterative nature, PI-CS methods suffer from impractically long reconstruction times which limit their clinical applicability. More recently, deep learning-based reconstruction approaches have demonstrated much faster reconstruction times, and in some cases, better image quality than PI-CS methods. These approaches are based on machine learning models which must first be trained to learn the process of reconstructing raw data into images.

The most common way of training a model to reconstruct MR images is through supervised learning. In the case of reconstruction, the inputs are undersampled k-space data, which the model uses to output a reconstructed image. Supervision here is enabled by utilizing fully sampled k-space data, which may have either been directly acquired through a lengthy scan or synthesized from under-sampled data via more conventional image reconstruction techniques. The model itself is learned by updating the model parameters such that the difference between the model output and the fully sampled ground truth is minimized. This process, known as stochastic gradient descent, is done repeatedly for each example in the training dataset until the model has fully converged (Fig. 9).

Figure 9:

(a) MRI reconstruction model training pipeline. If fully-sampled k-space data is available, a linear reconstruction is performed to generate the high-quality, ground truth image. The fully-sampled k-space data is then retrospectively undersampled by throwing away k-space lines, simulating how the scanner would undersample data in a true, accelerated acquisition. This undersampled data is given to the network for reconstruction. The network is then trained to output a predicted image, which is enforced to be “close” to the ground truth image via a loss function. A simple mean-squared-error loss is shown here. The model parameters are then iteratively updated by a stochastic gradient descent (SGD) algorithm, which intends to minimize the loss function, thereby minimizing the difference between the predicted image and the ground truth image. (b) Once the model is fully trained, it can be used to “infer” or reconstruct images from prospectively undersampled data in an efficient manner.

Choosing the type of model to use for reconstruction tasks remains an active area of research. Similar to image classification approaches, the most common type of model used to learn MR image reconstruction is a deep convolutional neural network (CNN).^55-57 CNNs consist of convolutions, cascaded with non-linear activation functions, which are able to extract features from images at various hierarchical levels of detail and use them for reconstruction (Fig. 10A). As in any other application of deep learning, deeper networks (i.e. more cascades) are capable of learning better, more accurate reconstruction models. However, deeper networks also consist of more learnable parameters, and are therefore more likely to overfit to the training dataset. In other words, if not provided with enough training data to support the number of network parameters, the network will begin to memorize the data, degrading performance on data which was not seen during training.

Figure 10:

(a) An example of a deep convolutional neural network (CNN) for MRI reconstruction. There are many degrees of freedom in designing CNNs. One popular architecture is called a U-Net, which repeatedly applies convolutions and down-sampling layers to extract both low-resolution and high-resolution features. (b) Unrolled neural networks instead apply shallow CNNs (again U-Nets are shown) cascaded with simulated MRI model projections which ensure that the output of each CNN does not deviate from the raw, undersampled k-space data. These projections also make use of Fourier transforms and coil sensitivity information to convert intermediate network outputs back to the original multi-channel k-space domain where the projection is performed.

Recently, it has been demonstrated that deep CNN methods benefit from incorporating additional information about the MRI acquisition as an aid for image reconstruction. This is done by interleaving shallow CNNs with projections of a simulated MRI scanning model (Fig. 10B).^58-60 This interleaved model, known as an unrolled neural network (UNN), is treated as one deep model and trained end-to-end in the same way as a conventional CNN. By incorporating additional acquisition information, the network is never allowed to produce images that are relatively inconsistent with the raw k-space data. As a result, UNN reconstruction methods reduce the overfitting problem, and allow for better generalizability to unseen data. Initial proof-of-concept studies have demonstrated that UNNs outperform conventional reconstruction methods across a wide variety of clinical applications, such as 2D cardiac cine MRI scans⁶¹ for example (Fig. 11). However, further research is necessary to demonstrate their robustness and effectiveness across large clinical populations.

Figure 11:

Two 2D cardiac cine scans are performed on a pediatric patient with 17 short-axis view (SAX) slices covering the heart and scan parameters: TE=1.4ms, TR=3.3ms, matrix size=200x180. Reformats are shown to visualize 4-chamber (4Ch), 2-chamber (2Ch) views along with a 3D rendering. (a) The first scan is performed with 2X undersampling and reconstructed using a standard parallel imaging technique. (b) The second scan is performed with 12X undersampling and reconstructed using a deep learning approach. With deep learning-powered acceleration, the scan time is shortened from 6 breath-holds down to a single breath-hold. This not only has important implications for patient comfort, but also for the accuracy of volumetric assessments from these images, since the inevitable variations between breath-holds is significantly reduced in the DL images (yellow arrows).

Besides undersampled MRI reconstruction that utilizes raw k-space data, an additional class of techniques perform image-to-image translations (for example, low-resolution to high-resolution, low SNR to high SNR, etc) in order to overcome the challenge that raw k-space data is not always archived from clinical MRI scans. Such techniques have previously been utilized to transform low-resolution images into higher-resolution images, under the constraints of maintaining segmentation, quantitative parameter mapping, and diagnostic accuracy.^62-64 These methods have been demonstrated across varying anatomies, including neuro, cardiac, and musculoskeletal MRI.^65-68 Deep learning based approaches have also been utilized to alleviate the constraints of long scan times for accelerating quantitative MRI^69-71 scans.

MR Image reconstruction: Unsupervised Deep Learning

As described above, supervised DL reconstruction methods^{56,58,59,72-78} may provide more robustness, higher quality, and faster reconstruction speed than conventional image reconstruction approaches. However, the need for fully-sampled data for supervised training poses a problem for applications such as dynamic contrast enhancement (DCE), 3D cardiac cine, 4D flow, low-dose contrast agent imaging, and real-time imaging, where collecting fully-sampled datasets is time-consuming, difficult, or impossible. As a result, supervised DL-based methods often cannot be used in these applications. There are two main possible ways to address this problem. First, PI-CS reconstructions can be used as ground truth for DL frameworks.⁷⁵ However, reconstructions of the DL model are unlikely to surpass PI-CS reconstruction methods with respect to image quality. Another way is to re-formulate DL training in a way that only leverages undersampled datasets for training, otherwise known as unsupervised learning.^79-86

In unsupervised learning, the goal is to train a network to reconstruct an image that is consistent with a distribution of images. Therefore, paired inputs and outputs are no longer necessary like they were for supervised learning. This can be achieved using generative adversarial networks (GANs),⁸⁷ which in the past, have been used to learn and model complex distributions of images.^88,89 GAN training involves jointly training two networks: a generator network whose goal is to reconstruct an image from undersampled k-space data, and a discriminator whose goal is to discern whether or not the generator output came from the same underlying distribution as the training data. During training, these two networks are pitted against each other such that the generator continuously learns to reconstruct higher quality images, while the discriminator becomes better at discerning outputs from the generator.

GANs can be used to train MRI reconstruction models in an unsupervised fashion in two situations. If fully-sampled data is available, the generator can be trained to output images that match the distribution of a training dataset comprised of fully-sampled images⁹⁰ (Figure 12a). If only undersampled data is available, a slightly different framework known as AmbientGAN⁹¹ can be used (Figure 12b). In AmbientGAN, the generator output is converted back to k-space and is undersampled through simulation. Then the discriminator is tasked with determining if the undersampled output is consistent with the undersampled training data distribution. This approach has been demonstrated to be effective for unsupervised DL reconstruction, when the undersampling is sufficiently varied throughout the training dataset.⁹²

Figure 12:

A conventional supervised learning system (a) and an unsupervised system (b). (a) Framework overview in a supervised setting with a conditional GAN when fully-sampled datasets are available. (b) Framework overview in an unsupervised setting. The input to the generator network is an undersampled complex-valued k-space data and the output is a reconstructed two-dimensional complex-valued image. Next, a sensing matrix comprised of coil sensitivity maps, an FFT and a randomized undersampling mask (drawn independently from the input k-space measurements) is applied to the generated image to simulate the imaging process. The discriminator takes simulated and observed measurements as inputs and tries to differentiate between them.

As shown in Figure 13, the unsupervised method achieves superior SSIM, PSNR, and NRMSE on a knee dataset compared to compressed sensing reconstruction. However, the supervised GAN is superior in comparison to the proposed unsupervised GAN, which is expected because the supervised GAN has access to fully-sampled data, giving the network a stronger prior. Of note, the superior performance of the unsupervised GAN compared to compressed sensing is fairly negligible for lower accelerations, but becomes more significant at accelerations of 4-6 (Figure 14), highlighting the potential of achieving imaging speed ups greater than can be obtained conventionally. Representative results from a DCE dataset are shown in Figure 15. The generator greatly improves input image quality by recovering sharpness and adding more structure to input images. Additionally, the proposed method produces a sharper reconstruction compared to CS. In the first row, the anatomical right kidney of the unsupervised GAN is visibly much sharper than input and CS.

Figure 13.

Knee application representative results, showing, from left to right: the input undersampled complex image to the generator, the output of the unsupervised generator, the output of the supervised generator, and the fully-sampled image. The acceleration factors of the input image are 6.5, 9.9, and 15.6, from top to bottom. The quantitative metrics that are plotted next to the images are for the slice that is shown. In all rows, the unsupervised GAN has superior PSNR, NRMSE, and SSIM compared to CS. In the first row, the unsupervised GAN has metrics that are notably worse than the supervised GAN. In the middle row and last rows, the unsupervised GAN has metrics that come close to the performance of the supervised GAN.

Figure 14.

The results of the reconstruction performance on the set of knee scans of the unsupervised GAN as a function of the acceleration factor of the training datasets. The y-axis represents PSNR, NRMSE, or SSIM, depending on the plot. The x-axis represents the acceleration factor of the datasets. The gap between PSNR of CS and the unsupervised model is fairly negligible over the range of accelerations. The gap between NRMSE of CS and the unsupervised model is fairly negligible at first, for low accelerations, but becomes more significant at an acceleration of 6 and beyond. The gap between SSIM of CS and the unsupervised model is fairly negligible at first, for an acceleration of 2, but becomes more significant at an acceleration of 4 and beyond.

Figure 15.

2D DCE application representative results, where the left slice is the magnitude of one input undersampled complex image to the generator, the middle slice is the output of the generator and the right slice is a compressed sensing L1-wavelet regularization. The generator greatly improves the input image quality by recovering sharpness and adding more structure to the input images. Additionally, the proposed method produces a sharper reconstruction compared to CS. In the first row, the kidneys of the unsupervised GAN are visibly much sharper than that of the input and CS.

The main advantage of this method over existing DL reconstruction methods is eliminating the need for fully-sampled data. Another benefit is other additional datasets are not needed to use as ground-truth, as in some other works on semi-supervised training.⁹⁰ Additionally, the method produces better quality reconstruction compared to baseline compressed sensing methods.

One significant challenge in the implementation of unsupervised methods is ensuring adequate clinical validation. In supervised methods, a ground truth image can be obtained for comparison. In that circumstance, comparison can be by quantitative image quality metrics, such as SNR. However, an open question in the field is whether conventional image quality metrics are adequate to ensure diagnostic performance. Thus, clinical assessment by expert radiologists is critical. The challenge of clinical validation in unsupervised methods is often harder, since the big applications are for cases where a gold standard is impossible to obtain, such as high spatiotemporal resolution DCE or perfusion MRI. In this case, validation would require comparison against either another modality, such as CT, or by assessing clinical outcomes.

Summary

The field of upstream AI for medical imaging is broad, covering the steps in the medical imaging process from selection of an imaging modality to image processing. Potential benefits include a more streamlined clinical practice, faster and more accurate exam scheduling and protocoling, faster exams, and improved image quality. In this article, we have highlighted only a few representative applications, giving the reader insight into directions in this rapidly evolving field.

KEY POINTS.

• This article reviews multiple applications of artificial intelligence (AI) in which it has been used to vastly improve upstream components of the medical imaging pipeline.

• All medical imaging examinations can be accelerated by using AI to help manage exam schedules and imaging protocols.

• In ultrasound, AI has been used to form high-quality images with enhanced robustness to speckle noise.

• In computed tomography, AI-based image reconstruction approaches have been used to reconstruct high-quality images from low-dose acquisitions.

• In magnetic resonance imaging, AI has been used to design optimal data sampling schemes, and reconstruction approaches to accelerate scan time

SYNOPSIS.

Artificial intelligence (AI) has the potential to drastically improve workflows in radiology at multiple stages of the imaging pipeline. However, most of the attention has been garnered by applications focused on improving the end of the pipeline – image interpretation. This article reviews how AI can be applied to improve upstream aspects of the imaging pipeline as well including modality selection for an exam, hardware design, protocol selection for an exam, data acquisition, image reconstruction and image processing. A breadth of applications and their potential for impact is shown across multiple imaging modalities including ultrasound, computed tomography, and magnetic resonance imaging.

Clinical Care Points.

• As of today, there has been little research on validating upstream AI approaches in large clinical cohorts for deep learning-based image reconstruction approaches.

• Although these methods demonstrate great potential for accelerating clinical workflows in radiology, further studies are necessary to understand and determine the robustness of such approaches across a variety of patient populations.

• Future studies should not only validate upstream AI approaches based on standard image quality metrics, but also based on quantitative measurements which serve as clinical endpoints to ensure that the algorithms are truly improving the standard-of-care.