Artificial Intelligence and Machine Learning in Cardiovascular Imaging

Abstract

Cardiovascular disease is the leading cause of mortality in Western countries and leads to a spectrum of complications that can complicate patient management. The emergence of artificial intelligence (AI) has garnered significant interest in many industries, and the field of cardiovascular imaging is no exception. Machine learning (ML) especially is showing significant promise in various diagnostic imaging modalities. As conventional statistics are reaching their apex in computational capabilities, ML can explore new possibilities and unravel hidden relationships. This will have a positive impact on diagnosis and prognosis for cardiovascular imaging. In this in-depth review, we highlight the role of AI and ML for various cardiovascular imaging modalities.

INTRODUCTION

Revolutionary progress in cloud infrastructures, processing capabilities in computers, and broadband technology has opened new frontiers in information technology.1 As a result, many of these technological advances have trickled down into cardiovascular imaging. Single photon emission computed tomography, cardiac magnetic resonance imaging, and other technologies have allowed us to capture more images with each respective scan, leading to exponential growth in the complexity and sheer size of data.2,3 While this may appear to be a significant step forward in medicine, it actually creates a paradox: A surplus of information at a physician's disposal can potentially be daunting, counterproductive, and lead to profound ramifications in patient management.4 This is where artificial intelligence (AI) comes in.

From self-driving cars to voice recognition software such as Siri or Alexa, AI has propelled significant developments in technology and commercial industries.5 Machine learning (ML), a component of AI, will play a paramount role in cardiovascular imaging in the years to come.6 The algorithms produced in ML can analyze and comprehend a vast expanse of imaging data and lead to data-driven discoveries.7 Furthermore, they can efficiently automate a number of tasks and provide additional insight to physicians.8 By properly harnessing ML in cardiovascular imaging, it can reduce the cost and improve the quality of care. In this review, we discuss how AI can increase the diagnostic and prognostic capabilities of cardiovascular imaging and its potential to enhance patient care.

RELEVANCE OF MACHINE LEARNING

Conventional statistics currently play a paramount role in clinical trials and studies. However, new data is arising from multiple sources—including wearable devices, smartphone apps, and electronic medical records.1 As data evolves in complexity, magnitude, and dimension, it will exceed the threshold of analysis by conventional statistics. Conversely, due to its data-driven capabilities, the diagnostic performance of ML algorithms will increase significantly with more data.9 Unlike traditional statistical approaches, ML can unravel hidden relationships within the data.10 In addition, certain ML approaches can operate independently and provide further insight regarding the nature of the data.

One example of this kind of ML-inspired insight is in cardiovascular imaging of coronary artery disease (CAD), the leading cause of global mortality. Because CAD can trigger a number of complications, there has been a progressive increase in cardiovascular imaging to predict the occurrence of underlying CAD.11 Nevertheless, many of these conditions are heterogeneous in nature and can have varied presentations depending on the type of imaging modality used.12 A number of factors could be responsible for these varied observations, and prediction is not a strong suite of conventional statistics.7 In contrast, ML algorithms can identify unique patterns or groups within large and heterogeneous data.3 By identifying these unique subtypes, it can lead to more effective therapies and medical management.

TYPES OF MACHINE LEARNING

ML is an umbrella term encompassing a wide variety of algorithms that can automatically learn and improve with experience. Each of these algorithms has unique properties and features (Figure 1),2 and the investigator must determine which one is most appropriate for any initiative. ML algorithms can be broadly subdivided into supervised, unsupervised, semi-supervised, and reinforcement learning. Among these, supervised and unsupervised learning are frequently used.

Figure 1.

Progression of machine learning.

Supervised learning requires the dataset to have labels and clearly defined outcomes for the algorithm to “learn,” whereas unsupervised learning can operate independently and identify relationships with minimal guidance and without predefined datasets.2,6 Semi-supervised learning is a mix of supervised and unsupervised learning and works with both labelled and unlabeled datasets.3 Reinforcement learning is less commonly used and has yet to gain a significant foothold in cardiovascular imaging.3 This approach draws comparisons to human psychology because it uses rewards criteria for the algorithm to execute a desired function in a dataset.

RISE OF DEEP LEARNING

Deep learning (DL) is a type of ML that is programmed with large artificial neural networks that mimic the workings of the human brain.5 From voice recognition software to self-driving cars, it is gaining significant prominence in various sectors of commercial industry and information technology.13 The architecture of DL is arranged in a series of layers, with information passed from preceding and subsequent layers in a dynamic manner. The algorithm learns by processing both unstructured and unlabeled data and creating patterns to use in decision making.4 Convolutional neural networks (CNN) are commonly used in DL for cardiovascular imaging and research.14 Simply speaking, CNN algorithms broadly encompass a convolutional part, which can extract features and recognize images, and a fully connected part, which can classify images.4

Transition of Machine Learning to Deep Learning

From an evolutionary standpoint, DL can be seen as the next transition point for contemporary algorithms. Compared to other ML algorithms, DL improves significantly with larger datasets, giving it boundless potential for application in cardiovascular imaging.13 Deep learning can analyze and learn from the data and then make appropriate decisions,15 whereas traditional ML still requires extensive guidance, and engineers may need to step in at various points to help the algorithm function properly. As stated earlier, DL processes information in layers to create a neural network that is independently capable of executing decisions.16 An excellent example is Google's AlphaGo DL program, which was able to process information and defeat renowned players in chess.17 Increasing integration of DL in cardiovascular imaging can create clinical pipelines that can aid in clinical diagnosis and medical management.17

ROLE OF AI IN ECHOCARDIOGRAPHY

Echocardiography (echo) frequently serves as the first line of diagnostic imaging and is indispensable in clinical management.18 It is an inexpensive test that provides abundant information regarding various pathologies. With the advent of speckle tracking, echo provides an array of information regarding myocardial function beyond conventional metrics such as ejection fraction.19 This vast plethora of information allows ML algorithms additional opportunities to analyze various clinical entities.

Samad et al. used a random forest algorithm to predict all-cause mortality in 171,510 patients. A random forest algorithm uses a large number of uncorrelated decision trees that operate together to form a prediction. Using echocardiographic and clinical parameters in over 300,000 patients,20 this ML algorithm was able to demonstrate a superior prediction model (all AUC > 0.82) compared to clinical risk scores (AUC = 0.69 to 0.79) and logistic regression models for all survival intervals (P < .001). Khamis et al. used a supervised learning ML algorithm to enable spatial-temporal extraction and showed that apical two-chamber, four-chamber, and long-axis images could achieve accuracies of 97%, 91%, and 97%, respectively.21 In another study, Knackstedst et al. demonstrated that ML algorithms could accurately compute ejection strain and longitudinal strain.22 In addition to the superior speed, ML algorithms were able to provide accurate values to visual estimation and manual tracing. Narula et al. examined the role of an ensemble algorithm for distinguishing hypertrophic cardiomyopathy from an athlete's heart. Both volume and mid-left ventricular segment were found to be the best predictors for separating hypertrophic cardiomyopathy and athlete's heart.23 Sengupta et al. used an associate memory classifier ML algorithm for separating constrictive pericarditis from restrictive cardiomyopathy. The diagnostic area under the curve (AUC) was 89.2% and 96.2% with echocardiographic parameters and 63.7% for left ventricular strain.24

Automation of echocardiographic measurements can alter and greatly improve the clinical pipelines in various echo labs, saving time, reducing costs, and enabling rapid reporting and greater accuracy.25 Medvedofsky et al. utilized an automated algorithm to measure left atrial and ventricular volumes and ejection fraction in 180 patients for 3-dimensional (3D) echo. Strong correlations between manual and automated calculations were observed (left ventricular end diastolic volume = 0.97, left atrial volume = 0.96, and ejection fraction = 0.88).26 Similarly, Tsang et al. assessed the role of an automated algorithm for 3D transthoracic echo evaluation of left ventricular, left atrial, and ejection fraction27 and found solid correlations between automated and manual (r = 0.89 to 0.96) and CMR (r = 0.84 to 0.95).

A number of disease subtypes can be deciphered by various ML architectures by examining different patterns within pathologies.3 In unsupervised ML, clustering is frequently used for this purpose. In addition, a number of novel algorithms can be used for this purpose. Casaclang-Verzosa et al. used a topological data analysis (TDA) to discern patient similarity for precise phenotypic recognition of left ventricular responses during the natural course of aortic stenosis (AS).28 The TDA algorithm created a loop that uniquely grouped patients with mild and severe AS (P < .0001) on the right and left sides. Both components were connected by moderate AS, with the upper arm showing patients with reduced ejection fractions and the lower arm showing patients with preserved ejection fractions (P < .001); these findings were corroborated in mice (P < .001). Similarly, Todoki et al. used an unsupervised learning approach by integrating echocardiographic properties of left ventricular function and structure to predict major adverse cardiac events (MACE) in 866 patients. A loop was created that subdivided patients into four groups, and Kaplan Meier curves demonstrated significant differences in MACE-related complications and death (both P < .001). With the addition of network information to clinical risk predictors, there were substantial improvements in net regression and median risk scores for predicting MACE (P < .05).29

ROLE OF AI IN COMPUTED TOMOGRAPHY

Computed tomography (CT) is a primary diagnostic modality for a number of clinical entities.6 It provides a comprehensive description of entire coronary artery anatomy, from the presence of plaques to stenosis.30 As CT angiography (CTA) is becoming increasingly integrated into many diagnostic algorithms, this further emphasizes the importance of ML algorithms. By analyzing these vast troves of CTA data, ML can provide additional insight that can aid in clinical practice.

Motwani et al. compared the role of an ML algorithm to predict 5-year mortality versus traditional cardiac metrics in CT for 10,030 patients with supposed CAD.31 Interestingly, the ML algorithm showed a substantially higher AUC than CT severity scores for 5-year all-cause mortality prediction. Santini et al. used a CNN algorithm for classifying and segmenting lesions in cardiac CT imaging and demonstrated a Pearson correlation of 0.983 after adequate training with various CT volumes.32

Baskaran et al. used an ML algorithm for automatic segmentation of cardiac structures on computed tomography angiography.33 The overall Dice score was 0.932 and was consistent across structures. In addition, the automatic segmentation took an average of 440 seconds, far quicker than manual or semi-automated segmentation. In addition, Baskaran et al. used a DL algorithm for assessing cardiovascular structures from CTA in 166 patients.34 The combined Dice score was 0.9246, and the ML algorithm verified with manual annotation for left ventricular volume (r = 0.98), right ventricular volume (r = 0.97), left atrial volume (r = 0.78), and right atrial volume (r = 0.97) with substantial statistical significance (P < .05).

Han et al. explored the role of an ML algorithm for predicting rapid coronary plaque progression in 1,083 patients with CTA from the Progression of Atherosclerotic Plaque Determined by Computed Tomographic Angiography Imaging (PARADIGM) registry.35 The ML framework exhibited superior performance for identifying patients with rapid coronary plaque progression compared with conventional metrics and statistical models (area under the receiver operating characteristic curve [ROC] in ML model 3 was 0.83 [95% CI, 0.78–0.89] versus 0.60 [0.52–0.67] for atherosclerotic cardiovascular disease risk score and 0.74 [0.68–0.79] for the Duke coronary artery disease score). This study, in particular, demonstrates the ability of ML algorithms to offer new insights that are currently not possible with clinical tools.

ROLE OF AI IN NUCLEAR CARDIOLOGY

Single photon emission computed tomography (SPECT) myocardial perfusion imaging (MPI) enables risk stratification in cardiac imaging and is the hallmark test in nuclear cardiology.7 It encompasses a broad role in cardiovascular imaging by providing vital information regarding ventricular function and perfusion defects.36 ML algorithms can open new pathways in nuclear cardiology for their ability to predict coronary artery disease and cardiovascular complications.

Betancur et al. used a DL algorithm to predict CAD events with MPI.37 The algorithm clearly showed a higher area under the receiver-operating curve than total perfusion deficit (TPD) for predicting CAD (per patient: 0.80 vs 0.78; per vessel: 0.76 vs 0.73: P < .01). The researchers also used a DL algorithm to predict CAD occurrence with both semi-upright and supine stress MPI relative to TPD.38 The area under the receiver-operating curve for predicting disease per patient and per vessel with an ML algorithm was superior (per patient: 0.81 vs 0.78; per vessel 0.77 vs 0.73; P < .001). Arsajani integrated SPECT imaging with clinical information to predict CAD.39 The ROC curve for the ML approach was substantially superior to TPD and two readers with considerable significance (P < .001). Alonso et al. used a supervised ML algorithm to predict the risk of cardiac death with adenosine myocardial perfusion SPECT and clinical characteristics in 8,321 patients and 551 cases of cardiac death.40 The ML framework was substantially better than logistic regression (AUC = 0.76; 14 features), and demonstrated a higher discriminatory capacity (AUC = 0.83; P < .0001; 49 features). Notably, many ML studies to date have leveraged the data from databases while multivariable logistic regression models—to which ML algorithms are often compared—have more limited numbers of variables (eg, 49 vs 14 features in the Alonso study). Thus, future studies should evaluate both approaches equally, using the same type and number of features embedded in an ML framework as those used in a logistic regression model.

ROLE OF AI IN CARDIAC MAGNETIC RESONANCE IMAGING

Cardiac magnetic resonance (CMR) imaging is heralded as the benchmark for noninvasive depiction of ejection fraction and ventricular volume.19 Furthermore, CMR facilitates tissue characterization and provides excellent temporal and spatial resolution.19 As a result, CMR has become instrumental for evaluating a number of pathological entities in cardiology. With the implementation of ML algorithms, it can expand the existing capabilities to greater heights.

Winther et al. found that DL for automatic segmentation of the right and left ventricular endocardium and epicardium to evaluate cardiac mass and function parameters achieved outcomes similar to human counterparts.41 In a similar fashion, Tan et al. used DL for automatic segmentation of the left ventricle in all short-axis slices in a publicly available datasets.42 Surprisingly, they achieved a Jaccard index—which measures the intersection and union of observed versus predicted segmentation—of 0.77 in the left ventricular segmentation challenge dataset and demonstrated a continuous ranked probability score of 0.0124 with the Kaggle second annual data science bowel. Leng et al. demonstrated superior DL-based left ventricular contour identification with exceptional agreement (r = .975) and fractional area changes (r =.959 to .971) with manual tracing.43

POTENTIAL OF AI IN BIG DATA

Big data continues to be a valuable means of providing additional clinical insight and is frequently used to improve patient care or aid in clinical guidelines (Table 1).44 Although it plays an important role, many valuable findings may go unrecognized due to the immense scope of captured information and the uncertainty of how to handle it. AI can have a profound impact in this regard since it can decipher a number of key relationships and findings within troves of information present. Zhang et al. demonstrated the potential of a DL framework for automatic interpretation in 14,035 echocardiograms over a large time span.45 The algorithm identified 96% parasternal long-axis views and facilitated cardiac chamber segmentation. In a number of aspects, such as the correlation of left atrial and ventricular volumes, the automatic measurements outperformed manual measurements. Hu et al. examined an ML algorithm for predicting coronary revascularization following SPECT MPI in 1,980 patients. Per vessel, the AUC for discriminating future coronary revascularization by ML framework (AUC 0.79, 95% CI, 0.77–0.80) was higher than regional stress TPD (0.71) or combined-view stress TPD (AUC 0.71, 95% CI, 0.69–0.72; P < .001). Similarly, for each patient, the AUC was superior to stress or ischemic TPD (P < .001).38

Table 1.

Description of an array of studies that have leveraged machine learning applications in echocardiography, computed tomography angiography, nuclear imaging, and cardiac magnetic resonance imaging.20–22,28,29,31–34,37,38,41,43 CTA: computed tomography angiography; CAD: coronary artery disease; TPD: total perfusion deficit; MACE: major adverse cardiac events

STUDY	MACHINE LEARNING ALGORITHM	TYPE OF IMAGING	BRIEF STUDY DESCRIPTION
Samad et al. 20	Supervised learning	Echocardiography	Utilizing clinical and echocardiographic variables to predict complications
Khamis et al. 21	Supervised learning	Echocardiography	To automatically detect correct views on echocardiography
Knackstedt et al. 22	Machine learning	Echocardiography	To automatically calculate ejection fraction and longitudinal strain
Casaclang-Verzosa et al. 28	Unsupervised learning	Echocardiography	To detect unique phenotypes of aortic stenosis
Todoki et al. 29	Unsupervised learning	Echocardiography	Utilizing echocardiographic variables to predict MACE complications
Motwani et al. 31	Multiple machine learning algorithms	Computed Tomography	To estimate 5 year mortality in patients with CAD
Santini et al. 32	Deep learning algorithm	Computed Tomography	To classify and segment lesions on CTA
Baskaran et al. 34	Machine learning algorithm	Computed Tomography	To perform automatic segmentation of structures on CTA
Baskaran et al. 33	Deep learning algorithm	Computed Tomography	To identify CTA cardiovascular structures
Arasajani et al.	Supervised learning algorithm	Nuclear Cardiology	Integrated echocardiographic and clinical characteristics to predict CAD.
Betancur et al. 38	Deep learning	Nuclear Cardiology	To predict CAD and compare to TPD
Betancur et al. 37	Supervised learning	Nuclear Cardiology	To estimate the occurrence of MACE events
Winter et al. 41	Deep learning	Magnetic Resonance Imaging	To automatically calculate right and left ventricular mass and volumes
Leng et al. 43	Deep learning	Magnetic Resonance Imaging	To identify ventricular contours and compare with manual tracing

Al'Aref et al. used an ML model incorporating clinical characteristics with calcium score to predict coronary artery events in 35,281 patients from the CONFIRM registry.46 The AUC for ML and coronary calcium was superior to Ml alone (0.881 vs 0.773, P < .005), coronary calcium (0.866), and updated Diamond-Forrester score (.682). Han et al. evaluated ML-derived prediction of all-cause mortality in 86,155 patients and found the AUC (0.82) to be superior to the Framingham risk score, coronary artery calcium score (0.74), and atherosclerotic cardiovascular disease and coronary calcium score (0.72, P < .05).47 In addition, the ML algorithm performed better reclassification in low-to-intermediate risk individuals (P < .001 for all).

LIMITATIONS OF MACHINE LEARNING

Although ML has a tremendous impact on clinical prediction, it is not without pitfalls.5 A common misconception is that AI is “all knowing” or “ready from the get go,” but this is not the case. For any ML architecture to thrive, a number of criteria must be achieved. Prior to analysis, all ML algorithms must first undergo some form of training and a series of iterations prior to functioning effectively. Secondly, ML algorithms require large datasets for training. It can be a tedious task for smaller academic centers to obtain and/or organize these datasets,7 and there is significant cost associated in obtaining and training ML algorithms. Furthermore, the findings of ML must be taken with caution if applied to smaller datasets, as biases in any given population upon which the ML algorithms were trained may propagate unpredictably in de novo populations that have not been seen by the ML.

The “black box” nature of AI—that is, the fact that AI computing systems are not transparent to the user—is another important aspect that needs to be considered.3 Even with extensive and careful programming, a number of unintentional biases can be entered into any model, and this can lead to significant ramifications in clinical care. As a result, clinical teams must be actively involved in all stages of ML development to ensure accurate and safe algorithms for medical care.

Finally, the role of AI in cardiovascular imaging must be actionable. Limiting ML applications to segmentation of structures may improve efficiency but does not leverage the vast potential of ML to identify relationships that are as-yet unknown using conventional approaches. Linking ML segmentation of cardiovascular imaging with patient-centric outcomes will likely maximize the potential for the integration of AI into cardiovascular medicine.

THE PROMISE OF MACHINE LEARNING

In this current era of medicine, physicians are facing exorbitant workplace demands and rigorous time constraints since there are exceedingly high patient volumes in a number of clinical institutions.48 In parallel, imaging modalities are becoming increasingly complex and offering an excess of information. This constant need to multitask and process multidimensional data can potentially lead to inadvertent errors in medical judgement.8 In this regard, ML algorithms can serve as a valuable clinical decision support tool for physicians by automating a number of tedious or tiring tasks and performing calculations.49 It can also provide a number of suitable diagnostic options for physicians while allowing them to focus more on the patient and medical management (Table 2).45–47,50

Table 2.

Summary of studies that have evaluated machine learning algorithms for prediction or automated interpretation.^45–47,50 SPECT: single photon emission computed tomography; MPI: myocardial perfusion imaging; CAD: coronary artery disease

STUDY	SAMPLE SIZE	DESCRIPTION
Zhang et al.45	14,035	To perform automatic interpretation from echocardiography
Al’ Aref et al.46	35,821	To estimate the occurrence of CAD by using calcium score and clinical factors
Han et al.47	86,155	To predict all-cause mortality with a number of variables and compare with other metrics
LH et al.50	1,980	To predict coronary revascularization from SPECT MPI

Some may fear that AI and ML algorithms may replace physician judgment altogether, but this is not likely in the foreseeable future. In any ML output, the physician is still responsible for evaluating the clinical relevance of the findings. In each patient encounter, ML algorithms can assist physicians by integrating information arising from multiple sources and analyzing this data in real time.2 This will allow them to deliver tailored medical regimens designed according to each patient's specific needs. In the near future, these algorithms will pave the way for precision medicine, which assesses an individual's unique disease characteristics rather than surrogate markers of population-based risk.

Randomized controlled trials (RCTs) are the benchmark in clinical research and frequently dictate guidelines,50 yet a number of RCTs are never completed due to deficient power or incomplete follow-up. In addition, RCTs may not appropriately enroll the proper patient population for the study. These weaknesses can hamper the overall findings of related RCTs and their relevance to the general community. The integration of AI can potentially improve the performance of RCTs by analyzing results of a trial and providing investigators a glimpse of the outcomes.50 In turn, this information could help investigators restructure their trials to become more effective or to measure feasibility. These algorithms can also potentially enhance the randomization process by incorporating additional characteristics. The integration of AI in clinical trials will be pivotal in patient care.

Clinical research is often rigid in nature and guided by hypothesis-driven objectives. While this approach is the sine qua non of the scientific process, it can overwhelm even the most competent clinical investigator due to the sheer amount and dimension of collected data (eg, physical exam, laboratory testing, imaging, patient-reported outcomes, etc.). Given our limited ability to account for the depth and breadth of data in large-scale studies that alter a single variable, AI can help recognize and interpret critical patterns emerging from these various data points.51 To maximize this potential, AI must be introduced at earlier stages in a physician's career, such as during medical school or residency training,4 and precision medicine must be simplified so it can be truly appreciated by the medical community. Collaboration between medical societies, organizations, and teaching institutions can result in task forces and/or guidelines to achieve this purpose, thus allowing physicians to tap the potential of AI and direct it towards clinical care.

CONCLUSION

AI and ML algorithms will soon represent the critical lynchpin connecting patient care and technology. They can open new pathways in clinical care by extrapolating hidden relationships and performing beyond conventional statistics. AI has the potential to greatly expand the diagnostic and prognostic capabilities in cardiovascular imaging and augment patient care. But as with any new scientific development, numerous financial, medical, and social hurdles must be overcome to achieve widespread adoption of AI into daily clinical care.

KEY POINTS

• Machine learning is a branch of artificial intelligence that can be accomplished through supervised learning, unsupervised learning, and semi-supervised learning.

• Applications of machine learning in cardiovascular imaging include automated segmentation, diagnosis, and prognostic risk stratification.

• Machine learning outputs must be carefully considered for potential biases and non-generalizability, and machine learning algorithms applied to clinical care must be actively integrated into our traditional approaches to improving patient outcomes.