Synthetic Images Are Here to Stay

See also the article by Longuefosse et al in this issue.

Dr Schiebler is a professor of cardiothoracic imaging in the Department of Radiology at the University of Wisconsin–Madison. His research interests are in the diseases of the small airways and coronary arteries and how imaging biomarkers and computational fluid dynamics can be used for more cost-effective patient outcomes. He currently serves as a senior consultant to the editor of Radiology.

Dr Glide-Hurst is a tenured professor and director of radiation oncology physics in the Department of Human Oncology at the University of Wisconsin–Madison. Her research interests focus on applying artificial intelligence to MRI-based tasks in radiation oncology and in the emerging field of cardio-oncology. Dr Glide-Hurst is a fellow of AAPM and principal investigator for an NIH grant applying advanced artificial intelligence technologies for cardiotoxicity evaluation in patients with cancer.

Image postprocessing is familiar to us all. The most used photograph editing software is Adobe Photoshop. And the term “photoshopping” has come to mean reworking a digital image to show items not in the original photograph. On the other hand, a synthetic image is an image that has been partially or fully created using computer-generated graphics, rather than taken by a camera or from an imaging data set used for radiologic interpretation.

Synthetic images in medicine have been widely implemented in a variety of settings. For example, in radiation therapy, to eliminate multimodality image registration uncertainty and improve clinical efficiency, MRI-only treatment planning has become more readily available in the clinic. However, MRI does not provide the electron density information required for accurate dose calculation; thus, efforts to generate synthetic CT images from MRI data have emerged for many disease sites. In 2013, Hsu et al (1) reported one of the first studies of this kind on head and neck cancer. Predating artificial intelligence (AI), their fuzzy c-means clustering with spatial constraining method supported the mixing of tissue types within a voxel. This method also used the composition of tissue in the immediate vicinity of each voxel to harmonize the assignment of its mean Hounsfield unit density (1). In 2018, Emami et al (2) reported one of the first uses of generative adversarial networks (GANs) and convolutional neural networks to create synthetic CT images from cranial MRI examinations for the purpose of radiation treatment planning. The authors evaluated their GAN synthetic CT images versus real treatment planning CT images (the reference standard). They found that their GAN synthetic CT images outperformed the convolutional neural network output across several quantitative metrics, with the GAN better representing the bone-air interfaces while retaining finer image features (2).

In this issue of Radiology, Longuefosse et al (3) describe their experience using a GAN to transform ultrashort echo time (UTE) MRI scans (4) in patients with cystic fibrosis (CF) into synthetic CT images. Historically, UTE MRI quality has been inferior to that of CT, often due to texture, contrast, and the presence of artifacts. Thus, generating a superior synthetic CT image is advantageous. The results presented are intriguing and suggest that by using MRI only, the use of ionizing radiation can be completely avoided, particularly in the pediatric population, to study the progression of bronchiectasis and mucus plugging. The authors trained their GAN using contemporaneous chest CT and UTE pulmonary MRI examinations (approximately 33 000 image pairs) from 82 patients with CF, an internal test set of 28 patients, and an external test set (5) (five institutions contributing paired CT and 1.5- and 3.0-T MRI scans from 46 patients with CF). Two readers evaluated CF-related structural abnormalities by using the end point of the Bhalla score determined independently by each reader for UTE MRI, synthetic CT, and the reference CT. Interestingly, in the external test set, there was no evidence of statistically significant differences in the signal-to-noise ratio of the synthetic CT from the reference CT and UTE MRI, whereas the synthetic CT had a higher contrast-to-noise ratio than UTE MRI and had lower overall noise than the reference chest CT (Table). Also, the multireader study showed that when comparing synthetic CT and the reference CT, the visibility and sharpness scores were nearly identical, while synthetic CT had improved artifact scores. Taking these quantitative and qualitative data together supports the notion that the marriage of UTE MRI to GAN synthetic CT is at least equivalent or better than the reference standard chest CT in the setting of CF. This remarkable result suggests strong potential for using this method in the future, particularly when it takes less than 30 seconds to generate a synthetic CT image from MRI data. Further, the excellent performance of the external test set across different platforms and institutions suggests promising generalizability of the technique.

Quantitative Assessment of Image Quality for UTE MRI, Synthetic CT, and Real CT

While AI offers incredible opportunities to the medical imaging community, perhaps the biggest risk in using these technologies is the potential for missed or delayed diagnosis, which may cause patient harm. While the generation of synthetic CT images by the GAN in the study by Longuefosse et al (3) did not introduce additional false-positive findings in the internal or external test data sets, their GAN did remove motion artifacts. A plausible concern may be that GAN processing could potentially eliminate key image features needed for accurate diagnoses. Nevertheless, the authors highlighted the higher signal-to-noise ratio in the synthetic CT data sets but did not provide their interpretation. Impressively, their GAN interpolated low-resolution voxel objects with poor signal-to-noise ratio from MRI and successfully transformed them into higher-resolution images with a much better signal-to-noise ratio. This method of “making something from nothing” is quite remarkable. It suggests that further acceleration of the UTE MRI data may be a real opportunity, as synthetic CT has the strength of a high signal-to-noise ratio. Halving the time of a UTE MRI examination (reduction of approximately 5 minutes) also has a meaningful clinical benefit.

At this juncture, a thought experiment to help highlight the potential problems with using UTE lung MRI data as the basis for conversion to synthetic CT images will be useful for the reader. The synthetic CT images are only as good as the training set used to generate the algorithm. In a hypothetical patient with multiple small, calcified granulomas (smaller than a voxel), there would be no MRI signal intensity from the granulomas. These would be “hidden” in the background noise of that image. It is likely that the GAN used to convert these UTE MRI scans to synthetic CT images would not show these granulomas, and they would be calculated to be normal lung on the corresponding synthetic CT image. While one could make the argument that calcified granulomas are not important enough to constitute a “miss” at UTE MRI, this is not the case with pulmonary metastases from osteosarcoma.

There are limitations to every study. In the study by Longuefosse et al (3), registration to the reference CT used for training was performed by warping the images to fit the average position of the lungs from the free-breathing UTE MRI in patients with CF. The CT examination was performed at a breath hold in full inspiration, and typically there will be a 1-to 4-cm difference in the position of the diaphragm between full inspiration and the average resting state of residual volume. All CT Hounsfield unit metrics of lung density are thus compressed, and a new calculated version of the CT image and its various lung densities can then be used to train the AI. Thus, assumptions are being made about lung density that were not verified in this study sample. A second limitation is fiducial accuracy between the imaging methods under review. When two imaging acquisitions are being compared, both distance and overlap indexes (ie, mean distance to agreement and the Dice coefficient) can be used to evaluate the reference standard for image fidelity (2). The authors could have considered adding this extra step to further prove their hypothesis that the synthetic chest CT images were equivalent to the reference CT images for the examined lung structure and bronchial pathologies. One expects that a GAN trained to find mucus plugs and bronchiectasis would perform well in generating synthetic CT chest examinations from UTE MRI studies performed the same day and compared with a contemporaneous chest CT image. Pushing the boundaries of the GAN may be further explored by evaluating patients without CF. Would the GAN mistake a normal bronchus for bronchiectasis or a wandering pulmonary vein for a mucus plug, or how does it perform with an unexpected finding?

The “blue sky” version of this AI method promises to have a bright future for medical imaging, surgical planning, opportunistic screening, and radiation therapy. Most applications to date have been in the use of GANs to take MRI from PET/MRI studies and derive synthetic CT examinations to calculate the attenuation correction coefficient for the PET photons from different parts of the body. Synthetic CT images are currently being implemented to support MRI-only radiation treatment planning to ensure a robust dose calculation (6).

How far can this technology take us? Perhaps, one acquisition will be the genesis for multiple types of images using different imaging hardware. For example, MRI may be used as the original data for calculation and estimation of synthetic images from multiple modalities, such as routine chest CT, dual-energy CT, contrast-enhanced CT angiography, photon-counting CT, and fluorodeoxyglucose PET/CT. Also of interest is how well coronary artery atherosclerotic plaques (calcified and lipid) are shown on a synthetic CT angiograph derived from a three-dimensional cardiac MRI examination. Suffice it to say, possible applications for this novel image postprocessing method are nearly limitless. However, rigorous efficacy and effectiveness trials are needed to show the safety of this technological advance and where its weaknesses lie in the clinical realm.

In summary, we applaud the authors on this outstanding contribution to the literature that uses AI to create synthetic chest CT images from pulmonary UTE MRI data, offering the potential to overcome the challenges of imaging CF without the need for medical radiation. Synthetic images are here to stay. Recent publications in Radiology (7–10) suggest these images may help with reducing the need for contrast material, increasing spatial resolution, and decreasing acquisition and postprocessing time for cardiac, breast, and musculoskeletal imaging and neuroimaging.