Application of Artificial Intelligence to Cardiovascular Computed Tomography

Abstract

Cardiovascular computed tomography (CT) is among the most active fields with ongoing technical innovation related to image acquisition and analysis. Artificial intelligence can be incorporated into various clinical applications of cardiovascular CT, including imaging of the heart valves and coronary arteries, as well as imaging to evaluate myocardial function and congenital heart disease. This review summarizes the latest research on the application of deep learning to cardiovascular CT. The areas covered range from image quality improvement to automatic analysis of CT images, including methods such as calcium scoring, image segmentation, and coronary artery evaluation.

INTRODUCTION

Since artificial intelligence (AI) surpassed humans in the computerized version of the traditional board game Go, deep learning technology has become indispensable to technological innovation in medicine [1,2]. In cardiovascular imaging, where various technological innovations are rapidly applied, several papers on the applications of deep learning technology have been published [3]. Among cardiovascular imaging modalities, computed tomography (CT) is one of the most active fields with technical innovation. Cardiac CT has been used to evaluate coronary stenosis, identify hemodynamically significant stenosis, elucidate the pathology in structural heart disease, and measure cardiac function [4,5,6,7]. Some excellent reviews on the application of AI in cardiovascular imaging have been published recently [3,8,9,10,11]. Although these studies included cardiac CT, the subject of discussion was multimodality imaging, including echocardiography, nuclear imaging, and cardiac magnetic resonance imaging (MRI). These studies also addressed how clinicians could apply AI in the clinical workflow with multimodality imaging, such as patient screening, decision support, prognostication, and follow-up [3,9,11]. Although these discussions are worthwhile, a more focused review of the CT imaging workflow is also meaningful from the perspective of the radiologist. Litjens et al. [3] summarized deep learning research based on PubMed search results, including publications from inception until January 2019. In the present review, the results of the search using ‘deep learning’ OR ‘machine learning’ AND ‘Cardiac CT’ as keywords, with publication dates specified between January 2019 and August 2020, were added. AI can be applied to various tasks related to cardiovascular CT, such as the improvement of CT image quality, segmentation, and coronary stenosis evaluation. In this review, the latest studies have been divided into the following categories by topic: image quality improvement, segmentation of anatomic structures, automatic coronary calcium score, and coronary stenosis/plaque evaluation. The AI technologies that are currently available and those that require further research have also been discussed.

DEEP LEARNING ALGORITHMS

A detailed description of the deep learning networks is provided in the previous literature [1,3,12]. Radiologists need to understand that applied networks should be different depending on the deep learning task. The following four algorithm types have been used in the deep learning applications for cardiac CT: convolutional neural network (CNN), fully convolutional neural network (FCN), recurrent neural network (RNN), and generative adversarial network (GAN). Brief descriptions and application examples of these networks are summarized in Table 1. CNNs are the most widely used architectures, and they consist of a convolutional layer, pooling layer, and fully connected layers [3]. The convolutional layer extracts various features from an image and generates multiple feature maps. To apply CNNs to a large field of view, feature maps are progressively and spatially reduced by pooling the pixels together [1]. The pooling layer helps CNNs increase the receptive field through downsampling to facilitate an understanding of the contextual information. In the context of cardiovascular CT, CNNs can perform calcium scoring through the classification of specific voxels [13,14] or play a role in classification or slice selection as part of a segmentation algorithm [15,16]. FCNs are a modified form of CNNs that are specialized for image segmentation [3]. U-net is a type of FCN that is most widely used in organ segmentation studies [4,17,18]. After the downsampling path, which is similar to that of the CNNs, the FCNs contain an upsampling path in the architecture that produces an output image with the same resolution as the input images [1]. RNNs feed their output back as input through feedback loops, which is suitable for sequence data analysis [3]. Use cases of RNNs include electrocardiography (ECG), text, tracking of vessel centerline, cine MRI, and automatic labeling of anatomic structures [19]. GANs consist of two networks [3]: a generator and a discriminator. If the GAN network is sufficiently trained so that the discriminator cannot distinguish the image produced by the generator, a realistic image can be created [3]. Recently, GANs have become popular for image quality improvement [20] and the generation of virtual images [21].

Table 1. Deep Learning Algorithms for Cardiac CT.

Method	Description	Example Applications
CNN	- The most common architecture in image analysis - CNN is composed of convolutional and pooling layers - Convolutional layer is to detect distinctive local motif-like edges and other visual elements. This operation mimics the extraction of visual features, such as edges and colors - Pooling layer downsamples the data, which helps CNNs incorporate more contextual information	- Calcium scoring: CNN for voxel classification [13,14] - Segmentation of abdominal aortic thrombi in CT angiography [15] - Adipose tissue segmentation in a non-contrast CT: subsequent CNNs for slice selection and image segmentation [16]
FCN/U-net	- Adapted CNN to perform image segmentation - FCN takes an image an input and directly predict an image-sized segmentation - The most common FCN in cardiovascular imaging is U-net	- Segmentation of cardiac structures in CT angiography [4,17,18]
RNN	- RNNs feed their own output back as input, which is suitable for sequence data analysis, such as text, electrocardiography, or cine-MRI	- Automated anatomical labeling of the coronary a rtery tree [19]
GAN	- GANs consist of two networks: generator and discriminator - GANs are used for image noise reduction or generation (e.g., conversion of MRI to CT)	- Image noise and artifact reduction [20,23] - Generation of virtual images [21]

CNN = convolutional neural network, CT = computed tomography, FCN = fully convolutional neural network, GAN = generative adversarial network, MRI = magnetic resonance imaging, RNN = recurrent neural network

IMAGE QUALITY IMPROVEMENT

Iterative reconstruction (IR) is a classical method that has been widely used for image denoising [22]. However, IR requires huge computing resources, and IR performance highly depends on hyperparameters, which are often difficult to tune. AI-based CT denoising is rapidly emerging as an alternative to conventional IR [20,23,24]. For the AI-based method, denoised image reconstruction can be performed almost in real-time when the network has been trained, without concern about hyperparameter tuning. A ‘matched dataset’ of low-dose and standard-dose CT images is required for training a denoising network in supervised learning. Obtaining a matched dataset based on two different CT scans is a complex task, even in research settings. In contrast-enhanced CT scans, it is necessary to increase the contrast agent twice. Therefore, previous researchers have obtained low-dose datasets using anthropomorphic phantoms [23] or mathematical noise additions [24] from standard-dose images. Using paired datasets for training, AI networks learn noise patterns to predict noise maps. Denoised images can be obtained by subtracting the predicted noise maps from the low-dose images. In supervised learning, CNNs or FCNs have been used (Table 2) [24,25,26,27]. Green et al. [24] showed the efficiency of the FCN-based technique for reducing image noise in low-dose coronary computed tomography angiography (CCTA). Lossau et al. [25] applied a CNN technique to motion artifact detection and quantification in coronary arteries on CT, which is potentially useful for reducing motion artifacts. GANs are architectures for unsupervised learning [3,20]. Wolterink et al. [23] applied GANs to noise reduction of low-dose CT images obtained from an anthropomorphic phantom and nonenhanced cardiac CT. One of the important limitations of GANs for CT denoising is that potential mode-collapsing behavior means that some imaging features that are not present in the input images can be generated [28]. Kang et al. [20] introduced a cycle GAN as an alternative unsupervised learning method for denoising CCTA. They used a low-dose (20% of standard dose) image in early systole and late diastole and generated a retrospective ECG-gated scan with dose modulation. The standard-dose image was a mid-diastole (70–80% of the R-R interval) image. Using cycle GANs, they successfully obtained a denoised CT image with better performance than the state-of-the-art model-based IR technique [20]. Figure 1 and Supplementary Movie 1 show an example of denoised imaging using a cycle GAN-based image quality improvement algorithm with a very low-dose (4% of the standard dose in retrospective ECG-gated scan mode).

Table 2. Application of Deep Learning to Reduce Image Noise or Artifacts.

Study	Year	CT Scans for Test (n)	Low Dose Data	Network	Summary
Wolterink et al. [23]	2017	28	Phantom	GAN	Noise reduction in low-dose CCTA; 3D GAN
Green et al. [24]	2018	45	Synthetic data	FCN	Noise reduction in low-dose CCTA; FCN for per voxel prediction
Lossau et al. [25]	2019	4	Synthetic data	CNN	Motion artifact recognition and quantification; CNN
Tatsugami et al. [26]	2019	30	Synthetic data	CNN	Deep learning–based imaging restoration; lower image noise and better CNR compared with hybrid IR images
Kang et al. [20]	2019	50	Patients data with different cardiac phase	GAN	Noise reduction in low-dose CCTA (multiple cardiac phase data); 2D cycle-consistent GAN (CycleGAN)
Benz et al. [52]	2020	43	Synthetic data	NA	Comparison between model-based IR and deep-learning image reconstruction in CCTA
Hong et al. [49]	2020	82	Synthetic data	FCN	Applying a deep learning–based denoising technique to CCTA along with IR for additional noise reduction

CCTA = coronary computed tomography angiography, CNN = convolutional neural network, CNR = contrast-to-noise ratio, D = dimensional, FCN = fully convolutional neural network, GAN = generative adversarial network, IR = iterative reconstruction, NA = not available

Fig. 1. Image noise reduction by artificial intelligence in multiphase cardiac computed tomography obtained by retrospective electrocardiography-gated scanning.

SEGMENTATION OF ANATOMIC STRUCTURES

In cardiovascular CT research and quantitative reporting [6,17,29,30,31], image segmentation is the starting point and the most time-consuming step. Even with advanced cardiovascular CT applications, such as CT-derived fractional flow reserve (FFR) [5] and three-dimensional printing [7,32,33], accurate image segmentation is one of the most critical steps. If a segmentation algorithm with an accuracy similar to that of an expert is developed, radiological reporting will change, and advanced applications, such as CT-based simulation and printing, will be more widely used. Several researchers have published various studies that have applied AI to image segmentation and verified such techniques on this background (Table 3).

Table 3. Application of Deep Learning for Image Segmentation in Cardiovascular CT.

Study	Year	Development Dataset	CT Scans for Test (n)	Performance, Dice Coefficient	Network	Target of Segmentation/Modality for Prediction
Trullo et al. [34]	2017	Nonenhanced chest CT	30	0.91	FCN	Heart, thoracic aorta/nonenhanced chest CT
Commandeur et al. [16]	2018	Nonenhanced cardiac CT	250	0.82	Multi-task CNN (ConvNet)	Epicardial adipose tissue, paracardial adipose tissue, thoracic adipose tissue/ nonenhanced cardiac CT
Jin et al. [18]	2017	CCTA	150	0.94	FCN	Left atrial appendage/CCTA
López-Linares et al. [15]	2018	Aorta CT angiography	13	0.82	FCN	Postoperative thrombus after EVAR/aorta CT angiography
Cao et al. [38]	2019	Aorta CT angiography	30	0.93	FCN (3D U-Net)	Whole aorta, true lumen, false lumen/aorta CT angiography
Morris et al. [35]	2020	Paired CT/MRI data	11	0.88	FCN (3D U-Net)	Cardiac chambers, great vessels, coronary artery/nonenhanced CT for simulation
Baskaran et al. [4]	2020	CCTA	17	0.92	FCN (3D U-Net)	Cardiac chambers, LV myocardium/CCTA
Bruns et al. [36]	2020	CCTA (dual-energy set)	290	0.89	3D CNN	Cardiac chambers, LV myocardium/nonenhanced cardiac CT
Chen et al. [31]	2020	CCTA	518	-	CNN	Left atrium/CCTA
Koo et al. [17]	2020	CCTA	1000	0.88	FCN	LV myocardium/CCTA

CCTA = coronary computed tomography angiography, CNN = convolutional neural network, CT = computed tomography, D = dimensional, DC = dice coefficient, EVAR = endovascular aneurysm repair, FCN = fully convolutional neural network, LV = left ventricle

The most popular and essential targets for image segmentation are the cardiac chambers [4,31,34,35,36,37] (Fig. 2). Researchers have reported AI-based automatic segmentation of the four cardiac chambers using cardiac CT [4,17]. Koo et al. [17] reported the accuracy of AI-based segmentation of the left ventricular (LV) myocardium and contrast-filled LV chamber in 1000 CT scans. The sensitivity and specificity of the automatic segmentation for each LV segment (1–16 segments) were high (85.5–100.0%) [17]. Some researchers have studied specific areas, other than cardiac chambers, that are suitable for clinical purposes [18]. Commandeur et al. [16] reported the results of segmentation of epicardial and paracardial fat tissue on nonenhanced CT. Automatic quantification of adipose tissue showed good agreement with manual segmentation in epicardial locations (median Dice score coefficient 0.82) [16]. Cao et al. [38] split false and true lumen automatically in 276 patients with type B aortic dissection (Fig. 3). Although they showed good agreement, as defined by the Dice score coefficient (whole aorta, 0.93; true lumen, 0.93; false lumen, 0.91), the patients were relatively few (training set n = 246; testing set n = 30). Automatic quantification of a peri-graft thrombus after endovascular repair of an aortic aneurysm can be performed automatically using an AI-empowered algorithm [15]. For segmentation tasks, most researchers have applied FCNs or U-Nets (Table 3).

Fig. 2. Example of ventricular segmentation by artificial intelligence.

Fig. 3. Fully automated segmentation of true and false lumina in a patient with aortic dissection.

When developing an AI segmentation algorithm, preparing and labeling the dataset for development is a fundamental and critical step [1,3]. Most studies use the same imaging technique for labeling and prediction (e.g., left atrial labeling on CCTA for AI training [labeling] and automatic segmentation of the left atrium on CCTA [prediction]) [4,17,31,38]. Some researchers have developed an AI algorithm that divides the four cardiac chambers into nonenhanced cardiac CT images [35,36]. With nonenhanced CT, it is challenging, even for experts, to prepare a labeled dataset for AI training. Researchers have solved this problem in a quite creative way. Morris et al. [35] used a paired CT/MRI dataset to label nonenhanced CT images. Bruns et al. [36] prepared enhanced CT and virtual nonenhanced CT datasets using dual-energy cardiac CT and developed a segmentation algorithm that works for nonenhanced CT images. Lee et al. [39] developed an algorithm that works for nonenhanced CT images using matched CCTA and nonenhanced CT training datasets, which were used for automatic coronary calcium (CAC) scoring. The use of these unmatched datasets to solve technically challenging issues is an excellent example of how radiologists can contribute to AI algorithm development.

CORONARY CALCIUM SCORING

The CAC scoring workflow can be described as follows: 1) in a nonenhanced scan, determine whether calcium over 130 Hounsfield unit is present in the coronary artery, and 2) if calcium is in a coronary artery, identify the coronary artery (e.g., left main, right coronary artery, left anterior descending artery). Several researchers attempted automatic CAC scoring using feature extraction or multi-atlas methods even in the pre-AI era, but the results were unsatisfactory based on the high false-positive lesions and long calculation durations [40,41,42]. Recently, automatic CAC scoring using AI has been reported, showing promising results in clinical practice (Table 4). Lessmann et al. [13] employed a CNN to classify calcium candidate objects (sensitivity of 91.2% per lesion). Martin et al. [14] presented a multi-step deep learning model and tested it in 511 patients. The first step was used to identify and segment the regions, such as the coronary artery, aorta, aortic valve, and mitral valve. The second step classified the voxels as coronary calcium. They achieved a good sensitivity of 93.2% and an intraclass correlation coefficient of 0.985. Recently, Lee et al. [39] proposed an atlas-based fully automated CAC scoring system that uses AI (Fig. 4). The novelty of the system is that it can precisely detect coronary artery regions using a deep learning model based on semantic segmentation in a single step. This method can also provide regional information about the coronary artery and surrounding structures, such as the aorta, ventricular chambers, and myocardium. Therefore, this method can be easily extended to the segmentation of the aortic and mitral valves. This atlas-based automatic algorithm showed good agreement with manual segmentation (sensitivity, 93.3%; intraclass correlation coefficient, 0.99) as well as a low false-positive rate (0.11 calcium lesion per CT scan).

Table 4. Application of Automatic Coronary Calcium Scoring.

Study	Year	Sensitivity (%), per Lesion	False Positives per CT Scan	ICC	CT Scans for Test (n)	Protocol of CT Scan	Method of Detection
Wolterink et al. [53]	2016	79	0.2	0.96	530	ECG-gated CT	2.5D patch-based CNN (15 or 25 sizes from axial, coronal, sagittal planes)
Lessmann et al. [13]	2018	91.2	40.7 mm3/scan	NA	506	Non-ECG-gated chest CT	Cascaded two 2.5D CNNs (CNN1 with large receptive field and CNN2 with smaller receptive field)
Cano-Espinosa et al. [54]	2018	NA	NA	0.93*	1000	Non-ECG-gated chest CT	CNN based regression model, which directly predicts CAC score
Martin et al. [14]	2020	93.2†	NA	0.985	511	ECG-gated CT	Two-fold deep-learning models, first one to exclude aorta, aortic valve, mitral valve regions; second one to classify coronary calcium voxels
van Velzen et al. [55]	2020	93	4 mm3/scan	0.99	529	ECG-gated CT Non-EGC-gated chest CT (n = 3811)	Same as Lessmann et al. 2018 [13]
Lee et al. [39]	2021	93.3	0.11	0.99	2985	ECG-gated CT	3D patch-based CNN for semantic segmentation

^*Pearson correlation coefficient, ^†Per-patient sensitivity. CAC = coronary artery calcium score, CNN = convolutional neural network, CT = computed tomography, D = dimensional, ECG = electrocardiography, ICC = intraclass correlation coefficient, NA = not applicable

Fig. 4. Fully automated coronary artery calcium scoring software.

CORONARY STENOSIS AND PLAQUE EVALUATION

Among the techniques for predicting hemodynamically significant coronary stenosis with CCTA, CT-FFR has attracted the most attention [29,43]. However, this technique requires extensive coronary lumen segmentation and complex simulations of computational fluid dynamics [29]. Recently, machine learning-based CT-derived FFR technology was reported and is expected to increase efficiency [44]. This machine learning application was trained using 12000 synthetic coronary trees with various degrees of coronary stenosis, for which the CT-FFR values were computed using the computational fluid dynamics method [4]. It should be noted that machine learning shortens the computational simulation time and not the time required for coronary lumen segmentation [44]. The diameters of the coronary artery and plaque are relatively small, and research to accurately distinguish them is still in the early stages (Table 5). Other alternatives for predicting FFR using CCTA have also been reported [45,46,47]. van Hamersvelt et al. [46] applied feature extraction from the LV myocardium to classify patients with hemodynamically significant stenosis. The deep learning algorithm that characterized the LV myocardium improved the diagnosis of hemodynamically significant coronary stenosis (area under the receiver operating characteristic curve [AUC] = 0.76) as compared with the diameter stenosis (AUC = 0.68) [4]. Zreik et al. [45] successfully identified patients requiring invasive angiography in the stretched multiplanar reformatted image of CCTA using an autoencoder and a support vector machine. Using invasive FFR as a reference standard, the AUC for detecting coronary stenosis requiring invasive evaluation was 0.81 at the per-vessel level and 0.87 at the per-patient level [45]. Deep learning applications for relatively basic technologies, such as centerline extraction of the coronary tree [48] and annotation of coronary segments [19], have also been recently reported. The tree labeling network, a type of RNN, can annotate main coronary branches and major side branches with high accuracy (main branch 97%, side branch 90%) [19]. This automated anatomical labeling technology can streamline the diagnostic workflow in daily practice for radiologists and radiology technologists. Hong et al. [27] reported deep learning-based coronary stenosis quantification using CCTA. All quantitative measurements showed a good correlation between an expert reader and a deep learning algorithm (minimal lumen area r = 0.984, diameter stenosis r = 0.975, and percent contrast density difference r = 0.975) [27]. To date, one study reported by Kumamaru et al. [21] predicted FFR using only CT images without any human input. They overcame the complex process of coronary lumen segmentation using GAN. The deep-learning CT-FFR model achieved 76% accuracy in detecting abnormal FFR [21]. Coronary stenosis and plaque analysis require continuous technological development in the future for clinical applications.

Table 5. Application of Coronary Stenosis and Plaque.

Study	Year	Role of Artificial Intelligence	Dataset for Test	Performance	Algorithm
Coenen et al. [44]	2018	Prediction of FFR based on synthetic data (manual segmentation of coronary tree) in CCTA	351	0.78 accuracy	Machine learning using synthetic stenosis model and computational fluid dynamics results
Zreik et al. [47]	2019	Automatic plaque and stenosis characterization in stretched MPR image of CCTA	65	0.77 accuracy	3D recurrent CNN
van Hamersvelt et al. [46]	2019	Identification of patients with functionally significant coronary stenosis in CCTA	101	0.76 AUC	CNN for LV myocardial segmentation; SVM for patient classification
Wolterink et al. [48]	2019	Coronary centerline extraction in CCTA	24	93.7% overlap	3D CNN for prediction of vessel orientation and radius to guide iterative tracker
Wu et al. [19]	2019	Coronary artery tree segment labeling in CCTA	436	0.87 F1	Bidirectional LSTM in three graph representation
Hong et al. [27]	2019	Quantification of coronary stenosis automatically in CCTA	156	0.95 correlation coefficient	U-Net
Zreik et al. [45]	2020	Identification of a patient requiring invasive coronary angiography in stretched MPR image of CCTA	137	0.81 AUC	Autoencoder and SVM
Kumamaru et al. [21]	2020	Fully automatic estimation of minimum FFR from CCTA (i.e., free from human input)	131	0.76 accuracy	Lumen extraction block using GAN; residual extraction block; prediction block for minimum FFR estimation

AUC = area under the receiver operating characteristic curve, CCTA = coronary computed tomography angiography, CNN = convolutional neural network, CT = computed tomography, D = dimensional, ECG = electrocardiography, FFR = fractional flow reserve, GAN = generative adversarial network, LSTM = long short-term memory, LV = left ventricle, MPR = multiplanar reformatted, SVM = support vector machine

FUTURE PERSPECTIVES

The application of AI in cardiovascular CT is being developed in various ways, from image improvement to quantitative diagnosis. Some technologies are ready to be used in routine workflows or are already being used (Fig. 5); for example, low-dose CT denoising [49] or automatic quantification of relatively easy to distinguish structures (e.g., coronary calcification, cardiac chambers) [17]. Some technologies have promising results [27], but they require more time to be applied in routine practice. Typical examples include the improvement of ultra-low-dose CT and the segmentation of very small or delicate structures (e.g., coronary plaques and valves) [21,27]. Recently, electronic medical records with large data have been prepared for various AI research [50]. Cardiovascular CT powered by AI or radiomic analysis [51] can be combined with other imaging modalities or clinical information (e.g., ECG and blood laboratory tests) to guide decision-making or prognostication.

Fig. 5. Current status and future perspectives of artificial intelligence in cardiovascular CT.

CT = computed tomography, GAN = generative adversarial network

Cardiovascular CT requires quantitative reporting for several scenarios, such as CAC scoring, ventricular function assessment, and coronary stenosis evaluation [29,33,43]. For quantitative reporting, a process that lasts for a shorter duration is essential, and AI can significantly help shorten the duration of the quantification process [17,39]. To date, one of the best techniques for shortening the actual workflow is automatic CAC scoring [14,39]. At Asan Medical Center, Seoul, Korea, from October 2020, manual primary analysis of calcium scoring by an experienced radiological technologist was replaced by AI software-guided analysis. After primary analysis by AI, the CAC results are automatically transferred to a picture archiving system, and the radiologist carries out the final confirmation of CAC. The time saved by AI will be used for higher-level image analysis processes (e.g., ventricular function analysis) to improve the quality of CT reporting. Cardiac CT can be applied to the evaluation of the coronary arteries and other indications, such as the evaluation of the cardiac valves, myocardium, and congenital heart disease. Therefore, it is necessary to develop specific AI applications to promote research and quantitative reporting in related fields.

CONCLUSION

AI can be implemented and applied in various ways to cardiovascular CT-from image reconstruction to quantitative analysis. The interpretation room environment of radiologists changes as AI technology develops. As imaging specialists, radiologists should be actively involved in the entire process of AI development: from conception to clinical validation.

Supplement

The Supplement is available with this article at https://doi.org/10.3348/kjr.2020.1314.

Movie 1

Image noise reduction by AI in multiphase cardiac CT obtained by retrospective ECG-gated scanning.