Radiology Alchemy: GAN We Do It?

See also article by Sveinsson et al in this issue.

Paul H. Yi, MD, was a musculoskeletal radiology fellow at Johns Hopkins Hospital and is affiliate faculty at the Malone Center for Engineering in Healthcare. His research focuses on application and limitations of deep learning in radiology, including the potential for algorithmic bias. He serves on the RSNA Machine Learning Steering Subcommittee and the trainee editorial board of Radiology: Artificial Intelligence and is the journal's podcast co-host. In July 2021, Dr Yi joined the radiology faculty at the University of Maryland and serves as director of the University of Maryland Intelligent Medical Imaging Center.

Jan Fritz, MD, is a full-time musculoskeletal radiologist, associate professor, and the division chief of musculoskeletal radiology at the NYU Grossman School of Medicine in New York City. His research and practice focus on the development and clinical integration of novel and rapid musculoskeletal MRI techniques, metal artifact reduction MRI, MR neurography, interventional MRI, and machine learning techniques. He has authored over 200 peer-reviewed scientific articles, reviews, and book chapters and lectured at many national and international meetings. He serves on the editorial boards of Skeletal Radiology, Investigative Radiology, Current Radiology Reports, and PlosONE.

In medieval times, practitioners of alchemy sought to convert one material into another more valuable one, such as changing base metals into gold. Although sometimes successful, alchemy relied more on random trial and error of various materials and conditions rather than rigorous scientific experimentation. Recently, artificial intelligence (AI) and deep learning have come under fire by AI luminaries as a form of modern alchemy, relying heavily on random trial and error without understanding why one method is better than another (1). Nevertheless, the effectiveness of deep learning to perform a multitude of tasks has been demonstrated across nearly every facet of our lives, ranging from automated face recognition to self-driving cars and even to diagnosis of disease on medical imaging.

Recently, a different form of “alchemy” has emerged in the application of deep learning to radiology: the conversion of one type of medical image into another using generative adversarial networks (GANs), a well-described neural network architecture. For example, deep learning methods have been described to generate synthetic CT scans from MRI scans of the pelvis to evaluate for sacroiliac disease, which could obviate the need for a separate CT scan and its associated radiation (2,3). GANs can convert one MRI sequence of the knee into another (4). Like converting a cheap metal like copper into a valuable one like gold, GANs have demonstrated the potential to convert one type of medical image into another to obtain more “value” from that single image. Only in this case, the value is in the form of enhanced diagnostic capabilities, reduced radiation, or reduced scan time.

One potential use case for using GANs in medical imaging is to convert conventional anatomic MRI sequences into advanced quantitative sequences, such as T2 maps of knee cartilage (5), which may not be routinely obtained in clinical practice. Such advanced quantitative techniques can help us detect and monitor early stages of cartilage degeneration before they become apparent on standard morphologic MR images (6). However, the additional acquisition of dedicated pulse sequences to create quantitative maps can be time-consuming (7). Thus, the synthesis of quantitative data from standard MRI sequences could add value by providing comprehensive patient care independent of access to advanced imaging centers.

In this issue of Radiology: Artificial Intelligence, Sveinsson et al report a GAN-based system to produce synthetic T2 maps of femoral condyle cartilage from conventional anatomic MR images of the knee (8). The authors used a dataset of 4621 right knee MRI anatomic sequences from the Osteoarthritis Initiative to develop and test a type of GAN called a conditional GAN (cGAN, for short) to generate synthetic T2 maps (“CNN T2”). These CNN T2 maps were then evaluated qualitatively through a reader study by two musculoskeletal radiologists and quantitatively through correlation of CNN T2 values to T2 values from corresponding “real” T2 maps obtained from multiecho spin-echo (MESE) scans. The investigators found that the radiologists evaluated the CNN T2 maps as comparable to the real T2 maps and that these CNN T2 maps were free of artifacts present in the original images. Furthermore, the synthetic T2 values had significant correlation with the real T2 map values of femoral condyle cartilage, both in specific subregions and as a whole. Altogether, these results suggest clinical utility of these synthetic CNN T2 maps, both from a qualitative and quantitative standpoint.

Although GANs can generate synthetic images from one modality or sequence type to another, it is important that these synthetic images are visually similar to their “real” counterparts, lest they create images that depict anatomy and/or disease inaccurately. In this study, two board-certified musculoskeletal radiologists, each with more than 20 years of experience reading musculoskeletal MRI, evaluated 30 paired CNN T2 and real T2 maps and were asked to identify which image was which, as well as to grade technical quality of the images, including signal-to-noise ratio, sharpness, and presence or absence of artifacts. Only one reader was able to reliably distinguish between CNN T2 and real T2 maps, and even then, this reader did not grade one image higher quality than the other. These findings suggest that the cGAN was able to accurately reproduce the visual features of the original T2 map, despite only seeing standard anatomic MRI sequences, and are consistent with a recent work showing visual similarities between GAN-generated and real mammograms as determined by radiologists (9). One additional interesting finding was that the CNN T2 maps sometimes removed artifacts present in the real T2 maps, suggesting that these synthetic images might provide an improvement over the original images in some cases.

Although qualitative realism of images is important, for advanced images such as T2 maps, evaluating quantitative “realism” is perhaps even more important because the purpose of these T2 maps is to provide information beyond visual perception. Specifically for T2 maps, these quantitative values provide measures of T2 relaxation at each image voxel, which help assess cartilage matrix integrity, thereby potentially diagnosing osteoarthritis earlier than currently possible (8). In this study, the CNN T2 values were significantly correlated to the real T2 map values both in the femoral condyle cartilage as a whole and in specific subregions. Although not a perfect correlation, these findings suggest that the relative differences in CNN T2 values are maintained between different cartilage regions, and thus, could be used to indicate relative areas of cartilage degeneration, which could then be targeted for early intervention to prevent progression of disease into advanced osteoarthritis.

Osteoarthritis affects 11% of U.S. adults aged 55 and older and results in considerable morbidity and mortality related to disability and fragility fractures (10). As musculoskeletal radiologists, we are excited by the potential to obtain T2 maps on any patient who undergoes a standard knee MRI, to identify cartilage disease early, and to prevent progression to osteoarthritis. As physician-scientists, we are excited for the broader applications of GANs to “squeeze” more information out of each imaging study obtained across all specialties. Although alchemy has been associated with negative connotations, we are optimistic about the potential for deep learning and GANs to perform MRI “alchemy” to generate advanced MRI sequences from standard anatomic images, ultimately to the improvement of human health.