Artificial intelligence: improving the efficiency of cardiovascular imaging

Abstract

Introduction

Artificial intelligence (AI) describes the use of computational techniques to mimic human intelligence. In healthcare, this typically involves large medical datasets being used to predict a diagnosis, identify new disease genotypes or phenotypes, or guide treatment strategies. Noninvasive imaging remains a cornerstone for the diagnosis, risk stratification, and management of patients with cardiovascular disease. AI can facilitate every stage of the imaging process, from acquisition and reconstruction, to segmentation, measurement, interpretation, and subsequent clinical pathways.

Areas covered

In this paper, we review state-of-the-art AI techniques and their current applications in cardiac imaging, and discuss the future role of AI as a precision medicine tool.

Expert opinion

Cardiovascular medicine is primed for scalable AI applications which can interpret vast amounts of clinical and imaging data in greater depth than ever before. AI-augmented medical systems have the potential to improve workflow and provide reproducible and objective quantitative results which can inform clinical decisions. In the foreseeable future, AI may work in the background of cardiac image analysis software and routine clinical reporting, automatically collecting data and enabling real-time diagnosis and risk stratification.

Artificial intelligence (AI) is rapidly gaining importance in healthcare and in cardiology. The clinical introduction of data-rich technologies such as the “omics” fields (such as genomics, metabolomics, proteomics, lipidomics), electronic health records (EHRs), and wearable device biometrics provides cardiologists with access to vast amounts of information from many different fields of biomedicine. The big data generated by cardiovascular imaging is heterogeneous and quickly changing, requiring increasingly sophisticated and efficient computational models for the interpretation of these datasets. AI has the potential to facilitate every stage of the imaging process, from acquisition and reconstruction, to measurement, interpretation, and risk stratification (Figure 1). In doing so, AI may assist physicians in delivering more efficient and personalized patient care with improved accuracy, reduced analysis time and associated cost. Precision medicine refers to the tailoring of diagnostic, prognostic, and therapeutic strategies based on specifics derived from an individual’s omics data, clinical features, biomarkers, imaging results, behavior, and environmental factors. Noninvasive imaging has a fundamental role in the diagnosis, risk stratification, and management of patients with cardiovascular disease. It is important for clinicians and scientists working in cardiac imaging to understand the impact of AI in this field. In this paper, we review state-of-the-art AI techniques and their current applications in improving the efficiency of cardiovascular imaging, and discuss the future role of AI as a precision medicine tool.

Figure 1. Applications of artificial intelligence in precision cardiovascular imaging.

Anti-clockwise from the 12-o’clock position: integration of biometric and clinical data from wearable devices and electronic health records; automated image acquisition and reconstruction; automated image segmentation and quantification, including radiomic analysis; real-time prediction of diagnosis; and risk stratification for adverse cardiovascular events.

1. INTRODUCTION TO AI

AI describes the use of computational techniques to perform tasks that normally require human intelligence, such as visual perception, understanding language, pattern recognition, planning, problem solving, and decision making (1). In healthcare, this typically involves complex medical data being used to predict a diagnosis, identify new disease genotypes or phenotypes, or guide treatment strategies.

In 1950, Alan Turing, one of the founders of modern computers and AI, developed the “Turing test” to determine whether a computer can demonstrate intelligent behavior equivalent to that of humans (2). Passing the test would require a computer to hold natural language conversations with a human interrogator, without the interrogator being able to distinguish the computer’s responses from those of a human. Whilst this conceptualized artificial general intelligence, many researchers since have focused on training computers to perform specific cognition related tasks. The 1980s and 1990s saw a surge in AI interest, with approaches such as fuzzy set systems, Bayesian networks, and artificial neural networks applied to various clinical settings in medicine. In 2016, AI research in healthcare attracted more funding than AI projects in any other sector (3).

The term big data refers to extremely large datasets which cannot be analyzed, interpreted, or stored efficiently using conventional data-processing techniques (4). In healthcare, this includes “omic” data (genomics, proteomics, metabolomics, lipidomics), biometrics from streaming mobile devices, clinical data from EHRs, and image-derived information. Biobanks are structured collections of big data containing biological samples and associated medical data that are stored specifically for research purposes (5). Atlases are collections of big data from different domains, either individualized or population-based, which integrate vast amounts of otherwise disconnected information to identify the relationships and variations within patients or groups (6). While big data by themselves have no clinical value, the processing of big data using AI enables the discovery of patterns within datasets, as well as the training and refinement of diagnostic or prognostic computational algorithms.

1.1. Machine learning

Machine learning (ML) is a subset of AI which gives computers the ability to automatically learn to perform a task and improve from experience by being exposed to large amounts of data, without being explicitly programmed in decision making (7). Such techniques can be used to analyze big data, learn rules, and identify interaction patterns among variables. A feature is an individual measurable property or characteristic within a dataset that can be used as inputs to ML computational techniques (8). In imaging, this could be a voxel intensity or brightness, the spatial relation of voxels, a vector of motion, or any other measurement from images. Feature selection involves choosing a subset of relevant variables while eliminating redundant ones, and is used to reduce the dimensionality of data and prevent overfitting of the model.

ML can be categorized into three learning types: supervised; unsupervised; and reinforcement (Table 1). Supervised learning algorithms use a labeled dataset to predict a desired outcome. This involves the iterative selection, processing, and weighting of individual features to learn the underlying patterns within the data that best fit the given outcome (7). Supervised learning can be sub-categorized into regression and classification algorithms, which use continuous and categorical output variables, respectively. Regression is a well-known and widely applied supervised ML algorithm, whereby stepwise models are effectively learned. Classification methods test data using support vector machines, random forests, and artificial neural networks to identify the best fit differentiation (7). Limitations of supervised learning include the need for large labeled training datasets and validation datasets. Also, it often requires manual labeling of large amounts of data which can be time-consuming. Furthermore, algorithms can only predict known outcomes.

Table 1.

Machine learning techniques

Supervised learning
Regression analysis	Widely used form of machine learning that generates an algorithm to define a relationship between multiple predictor variables and an outcome of interest. Stepwise models iteratively add or remove predictors based on the strength of their association with the outcome in order to find the best performing model.
Support vector machines	This algorithm defines linear or higher dimension “hyperplanes” that separate training data into classes to predict outcomes. When presented with new data, these models can predict the class to which it belongs.
Random forest	Consists of a multitude of individual decision trees that operate as an “ensemble”; an item is classified according to the most common output from all of the decision trees.
Artificial neural networks (ANN)	Modeled on human neural processing, these nodal networks are comprised of interconnected layers (input layer, multiple hidden layers, output layer) which analyze and classify the input data. A mathematical model is used to pass on the results to successive layers. ANNs can learn which connections are most useful for classifying data and weight these accordingly.
Unsupervised learning
Principal component analysis (PCA)	Data is summarized by combining existing features to create a new set of independent variables, or principal components. The process of focusing in on only a few variables is called dimensionality reduction. PCA allows the features that account for the most variation in the dataset to be identified.
Clustering techniques	The algorithm identifies inherent groupings, or clusters, in the data, and uses these clusters to classify new data.
Hierarchical clustering	A hierarchical decomposition of the data is created by sequentially merging similar clusters (an agglomerative approach), or beginning with a single cluster which is split into smaller clusters at each step (a divisive approach).
K-means clustering	A distance-based algorithm, in which similarity is defined by the proximity to the “centroid” (mean, median, or medoid) of the cluster. The ‘K’ in K-means denotes the number of clusters.
Model-based clustering	Based on the assumption that data originates from a distribution (normal or Gaussian) that is a mixture of two or more clusters, and that each data point has a probability of belonging to each cluster.
Density-based spatial clustering	The algorithm focuses on the proximity and density of data to find arbitrarily shaped clusters and outliers (noise). For each point of a cluster, the “neighborhood” must contain at least a minimum number of points.
Reinforcement learning	The machine learns how to interact with its environment through trial and error, in order to maximize the total reward.
Deep learning (DL)	A class of ANN algorithms in which more hidden layers are used than in traditional neural networks; “deep” simply describes a multilayer separation approach. Most DL models use convolutional neural networks, in which the hidden layers contain one or more convolutional layers that apply a filter to an input to create a feature map. As with machine learning, DL can be classified as supervised, unsupervised, or reinforcement learning.

In unsupervised learning, unlabeled data are used to predict unknown outcomes, and the algorithm must discover hidden patterns within the dataset. Such techniques include principal component analysis and the wide array of clustering algorithms which cluster data into groups based on similarity (7) (Table 1). The major challenge in unsupervised learning is difficulty in identifying the initial cluster pattern (how many clusters there are, and their respective boundaries), which may lead to overfitting of the model to the dataset. Hence, these models require validation in multiple cohorts. Reinforcement learning is learning by interacting with an environment. The algorithm learns, through sequential trial and error, so that the actions that maximize the cumulative reward over time are enforced, while others are forgotten (9). This approach features strongly in gaming, however is infrequently applied to medicine.

1.2. Deep learning

Deep learning (DL), a subset of ML, mimics human cognition using multiple interconnected layers of artificial neural networks. The most distinct characteristics of DL networks is that they learn from the data thereby circumventing the need for expert-design of features used to perform a task. The most commonly used DL networks are convolutional neural networks (CNNs), which contain one or more convolutional layers that apply a filter to an input to create a feature map summarizing the presence of detected features in the input (10). The growth of DL has been accelerated by hardware advances such as graphic processing units and vast labeled datasets. DL has been implemented in daily technologies such as self-driving vehicles, robots, and speech- and image-recognition software. In 2016, DeepMind’s AlphaGo Zero neural network defeated the human world champion in the ancient Chinese game of Go. The earliest commercial applications of DL were for computer vision, or computational techniques to ‘see’, identify and process digital images and videos in the same way as human vision (10). Accordingly, initial healthcare applications of DL were in image analysis, most notably training CNNs to detect diabetic retinopathy (11) and malignant melanomas (12).

2. CURRENT APPLICATIONS OF AI IN CARDIOVASCULAR IMAGING

2.1. Image acquisition and reconstruction

Echocardiography remains the most widely used cardiac imaging modality. The increasing uptake of hand-held ultrasound devices and focused scanning protocols have enabled the rapid, point-of-care evaluation of cardiac structure, function and hemodynamics (13). However, performance of echocardiographic examinations is operator-dependent and intensive training is required for proficiency. The automation of image acquisition may eliminate some of these training and skill maintenance requirements. Commercial vendors have developed DL-based real-time guidance software to guide untrained providers in acquiring standard echocardiographic views, whereby the operator optimizes an accuracy signal that indicates how close they are to the desired view (14). These DL models can automatically analyze the acquired image dataset to identify and label heart anatomy, then slice standard views for presentation. This process reduces inter-operator variability, as the optimal views are selected based on thousands of deep learned examinations representing the spectrum of anatomical variations. Recently, a CNN used to identify 8 standard 3D echocardiographic views achieved an accuracy of 92% compared to ground truth training data labeled by clinicians (15). Such automation of image acquisition may enable non-cardiologists to perform echocardiography as a first-line diagnostic tool in acute settings, and also facilitate the training of cardiac sonographers.

In cardiac MRI (CMR), high-quality imaging requires careful patient positioning and planning of image acquisition planes by experienced operators. Vendors have developed ML-based automated software for anatomical localization of the heart and view-planning which have shown high agreement with manual methods (16,17). AI has also been applied to real-time detection and suppression of imaging artefact using random forest (18) and CNN (19) algorithms. One of the major limitations of standard cine CMR is the slow acquisition of images, which are often complicated by cardiac and respiratory motion. This requires patient cooperation with breath-holding and long scan times, while increasing the cost and reducing the accessibility of CMR. Compressed sensing reconstruction was developed to accelerate CMR image acquisition and improve patient compliance, at the same time achieving high spatiotemporal resolution by undersampling k-space (or temporary image space) data. Certain compressed sensing methods (20,21) apply ML algorithms which exploit the sparsity (or compressibility) of CMR images to sample k-space and recover only data with significant image information. Recently, CNNs trained to learn the optimal sparse representation from image data have demonstrated very fast reconstruction speeds (<10 s) (22), enabling their potential integration into clinical workflow.

2.2. Image segmentation

Segmentation is the process of partitioning a medical image into anatomically meaningful regions, based on which quantitative measures can be extracted. Traditional ML techniques have been shown to perform well in cardiac image segmentation (23), however require significant feature selection or prior knowledge to achieve satisfactory accuracy. By contrast, DL algorithms can automatically discover features from data for object detection and segmentation.

In echocardiography, the LV has been the focus of most AI segmentation studies. Fully automated ML algorithms using random forest classifiers have demonstrated good accuracy for delineating the LV endocardial boundary on 2D echocardiography compared to expert readers (24,25). Recently, DL using encoder-decoder CNNs was applied to a labeled dataset of 2D echocardiographic images from 500 patients, outperforming state-of-the-art non-DL methods in segmenting LV endocardial and epicardial boundaries. This fully automated analysis enabled the accurate estimation of LV volumes and ejection fraction compared to manual annotation by cardiologists (26). LV segmentation on 3D echocardiography is challenging due to the lower contrast, higher noise and data dimensionality, and limited annotation of datasets. To address these limitations, investigators have combined DL with atlas prior knowledge (27) and deformable models (28) to achieve accurate and efficient automated LV segmentation. For the evaluation of myocardial motion, LV segmentation is usually combined with a temporal tracking algorithm that follows the myocardial border across frames (29).

More AI-based segmentation algorithms have been developed for CMR than for any other cardiac imaging modality, partly due to the increase in publicly available data for CMR segmentation tasks in recent years. The majority of work has focused on segmenting cardiac chambers, with relatively fewer studies on segmenting abnormal myocardial tissue regions such as LV scar and atrial fibrosis. Delineating the borders of the chambers and myocardium is an integral step in CMR post-processing, however manual contouring is time-consuming and subject to reader bias (30). AI algorithms can automate this task while improving accuracy and reproducibility (31). A common DL technique for CMR segmentation is the use of a fully convolutional neural network (FCN), a special type of CNN without any fully connected layers (32). The addition of spatial or temporal contexts (33) or anatomical constraints (taking into account contour and shape information) (34) to guide 2D FCNs can improve their accuracy. In a large dataset of 4,875 subjects and 93,500 images from the UK Biobank, an FCN-based method performed as well as clinical experts in the segmentation of all four cardiac chambers (35). Similarly, an anatomically-constrained CNN was shown to accurately segment the LV and capture anatomical shape variations that are linked to cardiac pathologies (36). Nevertheless, current DL segmentation models lack reliability, often providing little agreement with expert readers or producing anatomically implausible results (31). To address this limitation, Sander et al (37) developed an automated segmentation model based on dilated CNNs which generated segmentation masks and spatial uncertainty maps for each input image. The investigators trained this model to segment the LV and RV in 100 CMR scans, and demonstrated that image areas of “high uncertainty” accurately corresponded to segmentation errors; such uncertainty maps thus have the potential to increase segmentation performance.

One of the first applications of DL in cardiac CT was an algorithm which used Marginal Space Learning – which incrementally learns classifiers in marginal spaces of lower dimensions – to model and automatically segment all four heart chambers from 3D volumes (38). Several CNN-based cardiac chamber segmentation methods have been applied to non-contrast cardiac CT (39,40) and CT angiography (41,42); most have used U-Net, a FCN designed specifically for medical image segmentation. Beyond cardiac chambers, coronary artery segmentation from CT angiography enables luminal stenosis grading, plaque assessment, and blood flow simulation. The two major steps in this process, coronary centerline extraction and lumen segmentation, can now be performed using CNNs either end-to-end (43) or as pre- or post-processing steps for traditional methods (44,45). Segmentation of the heart valves has been of particular interest due to the surge in transcatheter valve replacement procedures, which require valve modeling and precise sizing of the implants. Marginal Space Learning can provide patient-specific modeling and segmentation of all 4 heart valves from cardiac CT; this framework incorporates a constrained multilinear shape method to represent the complex spatio-temporal variations of the valves (46).

2.3. Automated measurements

AI has been increasingly applied to the automated quantification of 2D and 3D echocardiograms. Early software used traditional ML to automatically perform rapid and accurate calculation of LV ejection fraction based on 2D LV volumes (47). ML and DL algorithms have since been utilized to quantify a variety of other 2D parameters such as LV hypertrophy (48), wall motion abnormalities (49), myocardial velocity (50), and longitudinal strain (51). Doppler echocardiography, which measures blood flow, provides a velocity-time image for assessment of valvular function. A ML framework using hierarchical clustering has provided automated Doppler measurements of mitral inflow and aortic and tricuspid regurgitation (52). Further, a support vector machine was able to accurately quantify the severity of mitral regurgitation based on textural features on 2D images (53).

Current chamber quantification guidelines recommend the use of 3D over 2D echocardiography due to its superior accuracy and reproducibility (54). However, 3D echocardiography requires specialized training and is time-consuming, limiting its widespread adoption into routine clinical practice. ML techniques can provide rapid and reproducible automated measurements of LV and left atrial volumes throughout the cardiac cycle, which are comparable to CMR reference values (55,56) and perform well in different centers (57) and unselected patient populations (58). Further refinements to these ML algorithms have enabled the determination of LV mass (59) and right ventricular (RV) volume and ejection fraction (60). Utilizing AI for 3D echocardiography may improve workflow as more cardiologists incorporate this technology into their clinical laboratories.

DL methods can also be used to quantify a number of functional parameters on CMR. LV segmentation algorithms which automatically detect myocardial boundaries now enable fully automated measurements of LV volume, ejection fraction, and mass from cine imaging which have high agreement with expert readers (61–63). CMR is the preferred modality for clinical RV assessment, however manual measurements of the RV are time-consuming and subject to variability due to the complex anatomy of this chamber. Recent studies have applied DL algorithms to automatically segment the RV endocardium, producing accurate measurements of RV volumes and ejection fraction compared to manual annotation by experts (64,65).

AI has also been applied to a variety of CMR tissue characterization tasks. Scar volume quantification on late gadolinium enhancement (LGE) CMR is a valuable imaging biomarker for risk stratification and procedural planning, and DL can potentially replace the subjective and labor-intensive manual measurements of routine clinical practice. CNN-based algorithms are able separate LV scar from normal myocardium in the feature space and accurately quantify scar volume in patients with ischemic (66) and hypertrophic (67) cardiomyopathies. Similarly, fully automated techniques exist for measuring the extent of left atrial scarring in patients with chronic atrial fibrillation which could enhance selection of the appropriate patient and strategy for catheter ablation (68,69). Native T1 mapping on CMR enables the quantification of interstitial diffuse fibrosis, and a deep FCN has been used to automate global and regional myocardial T1 measurements, demonstrating high correlation with manual analysis (70).

The presence of coronary artery calcium (CAC) assessed via CT is a direct marker of coronary atherosclerosis and strong predictor of cardiovascular events (71). Manual measurement of the CAC score can be tedious, especially when image artefacts or numerous calcifications are present. ML and DL algorithms applied to large datasets of ungated, non-contrast chest CTs acquired for lung cancer screening have been shown to accurately and automatically calculate the Agatston CAC score compared to the manual reference standard (72,73). More recent ML and DL approaches have enabled reliable and rapid automated Agatston score measurements from dedicated non-contrast calcium scoring CT scans (74,75) and coronary CTA (76).

The detection and quantification of coronary stenosis is perhaps most important clinical application of coronary CTA, with current reporting based on subjective visual assessment by clinicians. Kelm et al. (77) used a ML algorithm to automatically identify, grade and classify coronary stenosis caused by both calcified and non-calcified plaques. Following centerline extraction and lumen segmentation, their random forest model trained on 229 CTA volumes performed accurate stenosis identification and lumen cross-sectional area estimation, with an average processing time of 1.8 s per case. Similarly, Zreik et al. (78) trained a recurrent CNN to accurately detect coronary plaque, determine its composition, and classify the coronary stenosis as obstructive or non-obstructive. The hemodynamic importance of a given stenosis can also be estimated through AI techniques. A novel DL algorithm to calculate noninvasive fractional flow reserve (FFR), trained on a large dataset of synthetically generated coronary models, demonstrated similar accuracy and faster execution times (∼2.4 s per case) compared to computationally expensive 3D flow simulations, with invasive FFR reference (79). This DL model was further validated against invasive FFR in a multicenter registry of 351 patients and 525 vessels (80), and has the potential to deliver real-time, on-site FFR computations with high predictive power. Very recently, an unsupervised DL technique to analyze a complete coronary artery from a CCTA-derived 3D volume and extract lower-dimensional encoding was shown to reliably detect functionally-significant stenosis as defined by invasive FFR (81).

Epicardial adipose tissue (EAT) is a metabolically active fat depot within the visceral pericardium which directly encases the coronary arteries. There is evidence that EAT exerts a local pathological effect on the coronary vasculature (82) and associates with adverse cardiovascular events (83). DL has been used to perform rapid (< 2 s), fully automated quantification of EAT volume from non-contrast CT datasets, with high accuracy compared to an expert reader (R = 0.979, p<0.001) (84). This DL model was validated across different patient cohorts, CT scanners, and protocols (85). Automated measurements of CAC and EAT have the potential for integration into routine clinical reporting of calcium scoring CT, and may improve workflow and accuracy in cardiovascular risk assessment.

2.4. Image interpretation

Automated cardiac image interpretation using computer vision has shown great promise in assisting clinicians with image-based diagnosis in radiology, pathology and dermatology, for example, in detecting pneumothorax on chest X-rays (86), identifying cancer on biopsy specimens (87), and classifying photographs of benign versus malignant skin lesions (12). Echocardiography presents its own challenges, with a typical study consisting of over 70 video clips collected from many unlabeled acquisition angles, subject to intrinsic beat-to-beat variability in cardiac performance. Given the sheer amount of multidimensional data in each study that often goes unused, several studies have applied automated learning approaches to assist in human interpretation of echocardiography. Madani et al. (88) used a CNN to simultaneously classify 15 standard views with more accuracy than experienced echocardiographers. Zhang et al. (89) trained a CNN on 14,035 echocardiograms to accurately identify 23 viewpoints and flag partially obscured cardiac chambers. The same investigators then applied disease classification models to accurately detect one of three states (hypertrophic cardiomyopathy, cardiac amyloidosis, and pulmonary hypertension) based on median calculated probability. Narula et al. (90) showed a ML model incorporating 20 different echocardiographic variables in the systolic phase of the cardiac cycle to outperform conventional diastolic and strain parameters in discriminating physiological versus pathological patterns of LV hypertrophy. Finally, an unsupervised ML algorithm using hierarchical cluster analysis of diastolic echocardiographic parameters reliably classified the presence or absence of diastolic dysfunction compared to guideline-based diagnoses. This unique clustering of patients provided significant survival discrimination for cardiac mortality and rehospitalization, whereas guideline-based strata did not (91).

The inclusion of CMR biomarkers into ML models also has the potential to augment traditional risk prediction scores. Ambale-Venkatesh et al. (92) applied a random forest technique to 6,814 subjects from the MESA (Multi-Ethnic Study of Atherosclerosis), demonstrating that among 725 variables, CMR-derived LV structure and function were among the top predictors for incident heart failure. The random forest risk prediction performed better than the Framingham, ASCVD, and MESA risk scores for prediction of adverse cardiovascular events. ML has also been used for risk stratification of patients with established cardiovascular disease. Kotu et al. (93) applied supervised ML to predict the occurrence of cardiac arrhythmias in survivors of myocardial infarction. The investigators found that CMR-derived scar textural features had discriminative power comparable to current criteria such as LV ejection fraction and scar size. Moreover, Bello et al. (94) trained a FCN using CMR-derived time-resolved 3D models of RV motion to predict outcomes in patients with pulmonary hypertension; this DL model outperformed conventional Cox models containing RV volume and strain parameters.

AI has also been used to integrate clinical and coronary CTA data for ischemia and outcome prediction. A substudy of the NXT (Analysis of Coronary Blood Flow Using CT Angiography: Next Steps) trial showed a ML risk score combining clinical variables and quantitative plaque metrics on CTA to improve the prediction of lesion-specific ischemia by invasive FFR, over pre-test likelihood of CAD or individual plaque meaures (95). Further, in an analysis of 10,030 patients from the CONFIRM (Coronary CT Angiography EvaluatioN For Clinical Outcomes: An InteRnational Multicenter) registry, a ML model combining 25 clinical and 44 visually assessed coronary CTA parameters predicted 5-year all-cause mortality significantly better than existing clinical or CTA metrics alone (96) (Figure 3). Finally, Commandeur et al. (97) used a gradient-boosted ML algorithm (XGBoost) for prognostication in 1,912 asymptomatic subjects from the EISNER (Early Identification of Subclinical Atherosclerosis by Noninvasive Imaging Research) trial who underwent CAC scoring with non-contrast CT scans and 14-year follow-up. The final ML risk score integrating clinical variables, CAC score, and automated EAT measurements outperformed the atherosclerotic cardiovascular disease (ASCVD) risk score and CAC score alone for long-term prediction of cardiac events.

Figure 3. ML prediction of all-cause mortality.

Receiver-operating characteristics curves for prediction of 5-year all-cause mortality: a ML risk score integrating clinical and coronary CTA data had a higher AUC than the Framingham Risk Score (FRS) and CTA severity scores (Segment Stenosis Score [SSS], Segment Involvement Score [SIS], Duke Index [DI]) (p<0.001†). AUC: area under the curve; CTA: computed tomography angiography; ML: machine learning. Reproduced with permission from Motwani et al. (96).

The field of nuclear cardiology has seen an increasingly high level of AI-based automation for image interpretation compared to other modalities. SPECT myocardial perfusion imaging (MPI) is widely used for the diagnosis of coronary artery disease (CAD), and the total perfusion deficit (TPD) is a computer-derived measure of the extent and severity of ischemia. Fully automated quantification of TPD using ML algorithms has shown equivalent accuracy to visual assessment by expert readers (98) for the detection of obstructive CAD on invasive angiography; and superior accuracy to expert analysis (area under the curve [AUC] 0.94 vs 0.89, p<0.001) when clinical data is included in the ML model (99). The first application of DL in cardiology to obtain a direct diagnosis from the entire 3D patient image was recently reported by Betancur et al. (100). Using 1,638 fast SPECT MPIs from REFINE SPECT (Registry of Fast Myocardial Perfusion Imaging with NExt generation SPECT), the investigators trained and validated a CNN to directly analyze polar map image data and detect obstructive CAD in each vascular territory (Figure 2). This model outperformed standard TPD assessment (AUC 0.80 vs 0.78, p<0.01), with a computation time of <0.5 s per patient. Recently, a boosted ensemble ML algorithm (LogitBoost) integrating clinical and quantitative SPECT MPI parameters predicted early revascularization (101) and 3-year risk of major adverse cardiac events (102) with higher accuracy than visual or TPD assessments alone.

Figure 2. Deep learning prediction of obstructive CAD from myocardial perfusion imaging.

SPECT polar map images are directly connected to the CNN, and patterns of perfusion defects are identified via feature extraction (left). These image features then pass through 3 fully connected layers (FC) in a deep learning process (centre) which predicts the probability of obstructive CAD in each vascular territory (right). CAD: coronary artery disease; CNN: convolutional neural network; LAD: left anterior descending; LCx: left circumflex; RCA: right coronary artery; ReLU: rectified linear unit (linear function mapping input to output values); QPS: quantitative perfusion SPECT. Reproduced with permission from Betancur et al. (100).

2.5. Radiomics

The ability of ML techniques to handle high-dimensional data has facilitated the growth of a new field in medical imaging known as radiomics. This refers to the process of extracting a large number of quantitative imaging features from a given region of interest to create big data in which each abnormality is characterized by hundreds of parameters extending far beyond those that can be characterized by the human eye (103). The quantitative features are calculated using dedicated software, which accepts the image datasets as inputs. Following feature extraction, datamining and ML approaches are used to find new, meaningful patterns between the different parameters, with the aim of identifying novel imaging biomarkers which may reflect the underlying pathophysiology of a tissue. Radiomic features can be typically be classified as shape-based, intensity-based, or texture-based (103). Shape features describe the geometric properties of the region of interest, while intensity-based metrics are calculated from the absolute voxel values themselves. Texture analysis is the modeling of the spatial distribution of voxel gray-level intensities, using second- or higher-order statistics to provide a measure of heterogeneity within the region of interest (104).

Given that many of the extracted radiomic features are redundant, unsupervised ML techniques are applied for feature selection prior to model building; the two most common techniques being cluster analysis and principal component analysis (105). All selected features considered reproducible, informative, and non-redundant are then used for association analysis. In addition to traditional statistical methods, ML approaches such as regression, random forests and artificial neural networks are utilized to build a multivariable model to predict an outcome (105).

The first applications of radiomics and texture analysis in CMR were for segmentation of LV scar in areas of myocardial infarction, enabling determination of scar size and a precise description of scar characteristics (106,107). More recent studies used texture analysis for distinguishing between acute and chronic infarction (108) and performing myocardial viability assessment purely from non-enhanced cine imaging (109). Several investigators have also applied texture analysis to distinguish between etiologies of LV hypertrophy (110) and to detect fibrosis in patients with hypertrophic cardiomyopathy (111). Further, textural features may have prognostic importance in hypertrophic cardiomyopathy, with a recent study demonstrating increased fibrosis heterogeneity on LGE imaging to predict adverse cardiovascular events (112). Beyond structural heart disease, radiomics and texture analysis may enable more accurate quantification of myocardial inflammation. High intramyocardial inhomogeneity on T1 and T2 maps has been shown to perform better than the average T1 and T2 values for diagnosing acute myocarditis using either clinical criteria (113) or endomyocardial biopsy (114) as the reference standard.

Coronary atherosclerotic plaques consist of distinct histological components with different attenuation values on CTA; each voxel is a separate measurement of the amount of radiation absolved in the given volume. This has enabled the assessment of plaque morphology in vivo, and several qualitative (115) and quantitative (116) imaging biomarkers are known to associate with adverse cardiovascular events. However, visual assessment of plaque features is prone to inter-reader variability, and current plaque quantification is based on the absolute voxel attenuation values without considering the relationship between voxels. Radiomics can provide an objective, mathematical or AI-based description of plaque which incorporates the spatial distribution of voxels (Figure 4). This approach has been shown to outperform conventional qualitative and quantitative parameters for the identification of high-risk plaque (117,118). Moreover, in an ex vivo study, a radiomics-based ML model improved the discriminatory power of coronary CTA in identifying histologically-determined advanced atherosclerotic lesions (119).

Figure 4. Radiomic analysis of plaque on coronary CTA.

(A) Curved multiplanar reconstruction of the left anterior descending artery showing mixed plaque in the midsegment. (B) For radiomic analysis, all voxels containing plaque are extracted from the region of interest. (C) Large numbers (typically >1,000) of intensity-, texture-, or shaped-based features are quantified to create big data, which can be analyzed using machine learning. CTA: computed tomography angiography. Reproduced with permission from Nicol et al. (120).

3. PITFALLS AND BARRIERS TO AI IMPLEMENTATION

While the AI developments described are exciting, there are several foreseeable obstacles to their translation into everyday clinical practice. Firstly, large, well-structured and labeled datasets are needed for the training and validation of new techniques. In healthcare, legal barriers often hamper data sharing, and thus far only a limited number of datasets have been made publicly available. Encouragingly, biobanks and international consortia managing medical imaging databases, such as the UK Biobank (5) and the Cardiac Atlas Project (6), have overcome many of the ethical, legal, and organizational barriers associated with data sharing. Besides data availability, the quality of data is crucial for development of accurate AI systems. Hence, standardization of clinical data recording and image acquisition protocols is required before data from different centers can be used as input by a standard AI model. Although the addition of clinical and other non-image data enhances the diagnostic and prognostic value of images, seamless integration of the EHR with image datasets is needed. The ImageGuide registry was established to collect images with linked clinical and outcome data from echocardiography and nuclear cardiology laboratories across the United States; such systems can potentially be used to feed AI models.

Secondly, AI algorithms trained and tested with single institution data may not be generalizable to different cohorts. There is the need for external validation of these models at multiple centers before their widespread clinical adoption. Further, as the input of patient characteristics and disease patterns change over time, AI models may benefit from continuous update, and thus large and dynamic datasets are desired. Thirdly, the emergence of high-level image analysis and interpretation may be met with some level of physician resistance. AI is often seen as a “black box”, achieving analytic, diagnostic and predictive accuracy without providing an explanation of its decision making. It is therefore important to demystify ML by introducing, wherever possible, “explainable AI”. Figure 5 shows an example of such explainable AI, whereby ML integration of clinical data and imaging measures was able to provide individualized risk prediction for adverse cardiac events in two patients from the EISNER trial (97).

Figure 5. Explainable AI through individualized ML risk prediction.

ML model for the prediction of myocardial infarction (MI) and cardiac death with subject-specific feature importance for a 62-year-old man with no event (A) and a 74-year-old woman with MI (B). The X-axis corresponds to the ML risk score. The arrows represent the influence of each covariate on the overall prediction; either decreasing (blue) or increasing (red) the risk of future events. The combination of all covariates’ influence provides the final ML risk score. Subject (A) has a low ML risk score (0.0105), while subject (B) has a high ML risk score (0.2206). The blue and red colors provide the separation between low and high ML risk as determined by Youden’s index. The gray dashed lines represent the overall prevalence of events in the population (~4%). Reproduced with permission from Commandeur et al. (97).

Doubts may also exist among medical professionals over the potential replacement of their roles by increasing automation. However, AI should be a change that is embraced by clinicians, viewed as a companion physician aid which works in an interactive manner to streamline healthcare and facilitate precision medicine. The concept of “AI augmentation” is one in which cognitive technology enhances human intelligence rather than replacing it. It will be important to train the new generation of cardiac imaging specialists regarding the concepts and applicability of AI, and how to function efficiently alongside this technology.

Moreover, although the overall statistical performance of AI may be higher than that of physicians, there will always be specific, complex, or rare individual patient cases where an expert clinician could provide a better answer. Thus, the concept of using AI in medicine in should be as a decision support system, with the final action being from physicians.

5. EXPERT OPINION

Cotemporary progress and studies in AI and cardiovascular imaging have been driven by several forces. First, rapid advances in imaging technology, EHR, and mobile solutions have made big data encompassing the heterogeneity of patient and disease features available. Second is the development of new and sophisticated AI algorithms and exploratory methods to analyze high-dimensional data. Third is hardware advances such as graphics processing units and other digital accelerators. Fourth, the emergence of cloud-based computing and open-source AI software packages have provided low-cost and portable solutions to complicated data processing. Cardiovascular medicine is primed for scalable AI applications which can interpret more imaging data in greater depth than ever before, via automated algorithms and advanced data analysis techniques. This will enable novel insights into disease genotypes and phenotypes and enhancing outcome prediction.

As detailed in this paper, applications of AI can greatly improve the efficiency of noninvasive cardiac imaging modalities, by facilitating image acquisition, automating the processes of image segmentation and quantification, and assisting physician interpretation. The integration of clinical and imaging data using AI models has been shown to improve risk stratification over conventional methods. Further, radiomic analysis is an exciting new field which quantifies spatial and textural features from images that are difficult to discern with the human eye, and utilizes ML approaches to identify novel imaging biomarkers from big data.

Despite the potential pitfalls, the incorporation of AI into cardiology is not a change that clinicians should fear, but rather, one that should be embraced. Many technology companies, such as IBM, Google, and Apple, are investing heavily in health care analytics. Furthermore, both the American Heart Association and American College of Cardiology have established precision medicine and digital innovation platforms to develop and implement AI technologies to improve cardiovascular care. AI-augmented medical systems will serve to streamline workflow and provide more reproducible and objective quantitative results which can inform clinical decisions. Physicians will be able to focus on tasks best performed by human intelligence without being distracted by the tedium of many tasks (such as manual measurement of imaging parameters and interacting with EHRs) which can be fully automated. This may lead to higher physician job satisfaction by avoiding data fatigue; lower healthcare costs through the reduction in response time to critical data and avoiding unnecessary hospital admissions; and improved quality of care by providing prompt, personalized treatment.

4.1. FIVE-YEAR VIEW

In the near future, one can envisage AI working in the background of cardiac image analysis software and routine clinical reporting, automatically collecting data and enabling real-time diagnosis prediction and risk score computation. Over the next 5 years, AI algorithms and the level of automation for cardiac image acquisition and analysis will advance significantly. Greater standardization of image acquisition protocols and EHR documentation will enhance the quality of data used for AI input. Increasing collaboration between research institutions, industries, and healthcare providers may facilitate improved data sharing. While AI advances rely on the availability of standardized imaging, they can also provide guidance on the pathway to standardization. Further, AI may play a role in generating hypotheses and intelligent clinical trial design. This relates to the next frontier of AI applications in healthcare, in which AI may move beyond pattern detection and towards discovery of the underlying causal mechanisms.

Risk stratification will shift from the paradigm of simplified, population-based scoring systems to objective and personalized AI-based metrics incorporating a vast number of clinical and imaging variables. Physiological and disease marker data from wearable devices and remote digital sensors could also be processed and analyzed in real-time, together with image data, to provide immediately actionable insights or flag patients at risk of impending adverse health events. Ultimately, the most important AI application in cardiovascular medicine may be for the selection of patient therapy, whereby the algorithms utilize all available data to objectively quantify the potential benefit to an individual. Such end-to-end ML applications would truly enable personalized delivery of precision medicine.

ARTICLE HIGHLIGHTS.

• Artificial intelligence is being increasingly used to analyze “big data” generated by cardiovascular imaging

• Machine learning and deep learning algorithms can automate image acquisition and reconstruction, segmentation, measurement, and interpretation

• Integration of clinical and imaging data using machine learning risk scores provides personalized outcome prediction, enabling the practice of precision cardiovascular medicine

• Radiomics is an exciting field of image quantification which enables characterization of features beyond those visible to the human eye

• In the near future, artificial intelligence may function as a precision medicine tool which aids physicians in clinical decision making