Advancing cardiovascular medicine with machine learning: Progress, potential, and perspective

Summary

Recent advances in machine learning (ML) have made it possible to analyze high-dimensional and complex data—such as free text, images, waveforms, videos, and sound—in an automated manner by successfully learning complex associations within these data. Cardiovascular medicine is particularly well poised to take advantage of these ML advances, due to the widespread digitization of medical data and the large number of diagnostic tests used to evaluate cardiovascular disease. Various ML approaches have successfully been applied to cardiovascular tests and diseases to automate interpretation, accurately perform measurements, and, in some cases, predict novel diagnoses from less invasive tests, effectively expanding the utility of more widely accessible diagnostic tests. Here, we present examples of some impactful advances in cardiovascular medicine using ML across a variety of modalities, with a focus on deep learning applications.

Highlights

• •Cardiovascular medicine is very well suited to benefit from machine learning.
• •Machine learning has been used for various cardiovascular modalities and diseases
• •Rigorous clinical evaluation is critical for machine learning algorithms
• •There are increasing numbers of clinical trials and FDA applications in the field

Cardiology is uniquely well positioned to benefit from machine learning (ML). ML can automatically perform clinically relevant tasks and expand diagnostic utility of tests, ranging from electrocardiograms to cardiovascular imaging. Clinical adoption will likely require rigorous prospective evaluation. Updated FDA guidance and multi-institutional data will be key for long-term success.

Cardiovascular medicine is ripe for machine learning

Across many industries, there has been significant interest in machine learning (ML) in recent years, and biomedical science and medicine are no exception.^1,2 Broadly speaking, ML describes a variety of computational techniques that have expanded the traditional analytical toolkit beyond linear statistical models and generally comprise what is often referred to as “artificial intelligence”(AI). ML algorithms have made it possible to analyze higher dimensional data, and they have shown an ability to learn complex associations within these data without human-provided instructions. To do this, ML algorithms learn from many examples of data that have been labeled for a specific task, a process referred to as “training” or “learning.” This typically requires not only a large amount and variety of data but also diagnostic tests that capture information that are incrementally closer to the diagnostic ground-truth, to provide labels for the data. For several important reasons, cardiovascular medicine is ripe for advancement by ML.

Modern medical practice is awash in data of numerous types. In cardiovascular medicine, recent decades have seen an expansion in the variety and quality of diagnostic tests, such as noninvasive imaging like computed tomography (CT) angiography, physiologic testing such a fractional flow reserve or serum biomarkers. These tests provide physicians with greater amounts of complementary information upon which to make diagnostic and therapeutic decisions but present a broad range of accessibility, cost, and risk. This hierarchy of diagnostic tests is larger in cardiovascular medicine than other areas, with some tests providing rich labels upon which ML algorithms can be trained to use less costly/invasive tests to predict the results of more costly or invasive tests.

The widespread digitization of medical data over the past two decades makes medicine highly ripe for ML. With the move to electronic health record systems,³ nearly all presently recorded medical data from imaging to labs are stored and accessed in digital formats. The ability to access most modern medical data electronically unlocks the potential of data-intensive algorithms for application in biology and medicine. Even so, a substantial amount of data processing and harmonization may still be required before these data can be used to train ML algorithms. Much of the data collected in medical practice may now be digitized but remains unstructured. For data formats that are less easily computable, such as free-text notes, recent ML-driven advances in fields such as natural language processing^4,5,6 can make it possible to either obtain structure from such unstructured data or to work with unstructured data formats directly.

The high prevalence of cardiovascular disease overall also translates to large amounts of cardiovascular-relevant patient data. Electrocardiograms (ECGs), for example, are frequently obtained many times over the course of a patient’s care, often by non-cardiovascular specialists or clinics, which reflects the relevance of cardiovascular concerns to medicine more broadly. This provides greater amounts of data to train ML models and offers opportunities for ML to assist in a greater number of clinical workflows.

Like many chronic diseases, cardiovascular disease diagnosis and management depend heavily on trends across data, providing another opportunity for assistance from ML algorithms. For diagnostic tests such as ECGs or echocardiograms (echo), understanding a given patient’s prior study is critical to appropriately interpreting the current study. While physicians commonly compare a current study against the most recent prior study or several prior studies to identify changes, humans are generally less adept at evaluating subtle changes that arise over longer periods of time. Often it is simply not practical for a cardiologist to review more than a handful of prior studies prior to interpreting every current study. ML algorithms are positioned to assist with this in various ways, such as by providing quantification of previously subjectively determined binary diagnoses or by learning patterns of change in raw data that are associated with disease outcomes. ML can help to process large amounts of longitudinal data and discover patterns that may otherwise go unnoticed. Importantly, however, ML algorithms will not be the most appropriate analytical option for every clinical task. There are many tasks for which traditional statistical models or rule-based algorithms provide the best solution. Depending on the task of interest and the nature and quantity of the available input data and ground-truth labels, researchers should select the most appropriate analytic technique: traditional biostatistical models, ML algorithms, or a combination (Figure 1).

Figure 1.

Understand the problem and the data to select the most appropriate analytic technique

Clearly defining the clinical task of interest and understanding the nature of the available data and training labels is essential to selecting the most appropriate analytic technique. Created with BioRender.com. Link to BioRender illustration: https://app.biorender.com/illustrations/62e46a207b029dd42f0dcfb5 .

We will review some fundamentals of ML algorithms, then discuss some examples of specific applications to cardiovascular medicine.

Fundamentals of ML and AI

ML refers to a collection of computational techniques that learn patterns within data to accomplish certain tasks for which they are trained. The most common approach used to train ML algorithms in most settings, including in medicine, is called supervised learning. Training an algorithm with supervised learning involves presenting to the algorithm many examples of input data that have been labeled (often by a human expert) for the task of interest. The algorithm makes predictions each time, and learns patterns within the data by minimizing error when comparing its predictions against the labeled data. This error is calculated mathematically using a customizable equation called a loss function. The parameters of the algorithm are adjusted on each round to minimize this loss, causing the algorithm to learn patterns in the data. For example, in order to distinguish a normal sinus rhythm ECG from an atrial fibrillation ECG, an ML algorithm would learn from previously categorized raw 12-lead ECG waveforms what ECG features differentiate atrial fibrillation from normal sinus rhythm. These features might be irregular R-R intervals or absence of a P-wave or other similar ECG features, but the features identified and the relative importance of each feature (and the interactions between features) are learned by the algorithm during the training process.

To examine the ability of the algorithm to generalize to unseen data, the algorithm is continually tested during training on a separate subset of the data called the development, or validation, dataset. The algorithm’s parameters are typically adjusted by investigators based on the development dataset results to maximize generalization performance and prevent overfitting. Overfitting describes the situation in which the model learns the training dataset perfectly, memorizing training dataset-specific features at the expense of features that can allow the model to perform well on unseen data. Final algorithm performance is then reported on a third held-out dataset, the test dataset, that had no role in the training of the algorithm. In medical applications, it is also increasingly important to report performance in one or more additional test datasets from external institutions. This provides additional evaluation of the robustness of the algorithm, helping to ensure that the algorithm performance is not dependent on specific characteristics of the training population or dataset.

Typical tasks of medical ML models include classification (e.g., to predict “disease” versus “no disease”) or regression (e.g., to predict the continuous value of a diagnostic test) although many other relevant tasks can be performed such as drawing a bounding box around an item of interest in an image (e.g., localizing a pulmonary nodule on an X-ray). Based on the task, the relevant metrics of performance may vary. Relevant metrics can include accuracy, specificity, and sensitivity, much like any other diagnostic test, or the F1 score, which is the harmonic mean of precision (positive predictive value) and recall (sensitivity). ML reports will often also include the area under the receiver operating characteristic curve (AUROC), which is a metric of discrimination performance across all possible thresholds of its output score.

Traditional ML algorithms, such as support vector machines or tree-based models, are best suited for structured data resembling numbers on a spreadsheet, also called “tabular data.” For example, these data might contain a series of lab values, vital signs, or measurements derived from more complex data modalities, such as cardiac chamber dimensions from a cardiac MRI. In the case of derived measurements, humans are usually required to interpret the raw data before they can be input into the ML model in a process called feature engineering (Figure 2). In general, these traditional ML algorithms can capture complex relationships and have shown promise in medical applications. However, although these algorithms can process higher dimensional input data compared with traditional statistical models, they are still limited in the types of data that they can reasonably accept. For high-dimensional data modalities such as images or videos, these traditional ML algorithms are not able to handle raw formats and require humans to first derive summary measurements, which are subject to bias in the selection of which measurements to derive and human error during measurement.

Figure 2.

Example data types for traditional ML versus DNN algorithms

Traditional ML algorithms such as regression models, tree-based methods (e.g., random forest or gradient boosted models), or support vector machines typically require tabular input data formats. These tabular data can be extracted from structured data such as electronic health records, or they can be derived from more complex data types through manual extraction of human-defined features such as manual measurement of chamber volumes from an echo. DNNs allow for more high-dimensional and complex input data types.

Perhaps the most medically relevant ML advances in recent years have been driven by algorithm architectures that can accept raw data modalities such as free text, images, and videos directly as inputs, led primarily by a category of algorithms called deep neural networks (DNNs).¹ DNNs are highly flexible and customizable algorithms composed of many layers of model parameters or weights that can be added or removed depending on the complexity of the input data and the nature of the task of interest. Augmented by additional data-transforming filters (such as convolutional filters), which help further distil the data, the multi-layered architecture of a DNN is thus able to learn very complex relationships within the input data to accomplish the target task, without requiring (or being limited by) pre-specified human-defined features (Figure 2). This combination of a DNN’s ability to learn complex features and the ability to accept high-dimensional, even multi-modal, raw input data makes it possible to accomplish novel tasks and potentially enable data-driven discovery of new physiologic associations. These advances fundamentally changed the types of applications in which ML could provide value to medicine, making it possible to algorithmically analyze complex, raw medical data without requiring physician review.

Recent advances and applications of ML in cardiovascular medicine

ML algorithms have been applied to data modalities across the spectrum of the cardiovascular workflow. Various groups have trained DNNs to support fully automated analysis of ECGs, echos, radiologic studies, angiograms, and others. In addition to performing standard medical tasks with a specific type of data, ML algorithms have demonstrated the ability to perform novel tasks not previously associated with specific diagnostic tests, effectively expanding the utility of existing tools (Figure 3). Here, we will review some recent advances in ML applications in cardiovascular medicine.

Figure 3.

Using ML to extract more diagnostic information at lower cost

Ideally, a physician aided by ML algorithms can obtain more information per test, at a lower cost. This would increase the sensitivity of less invasive and more accessible diagnostic tests, possibly decreasing the need for more specific testing or time to diagnosis. The diagnostic tests shown are representative examples of cardiovascular tests that can provide incrementally more information for certain diagnoses. The threshold for diagnosis (red line) will vary based upon clinical disease or application. ECG, electrocardiogram; RHC, right heart catheterization.

Electrocardiography

The ECG is the most common diagnostic test in cardiology, providing widely accessible information about cardiac electrical function and structure. While traditional ML algorithms have long been applied to derived measurements from the ECG, more recently DNNs have demonstrated the capability to analyze raw ECG waveforms to perform a variety of tasks. Automated ML algorithms have replicated standard ECG diagnoses, performed novel tasks not typically performed by cardiologists using ECG alone, and assisted in analyzing longitudinal trends in disease status and response to drug therapy.

Most commercial ECG algorithms apply what are called rule-based algorithms to analyze an ECG; for example, to detect atrial fibrillation or to measure the QT interval. These rule-based models rely on previously defined diagnostic criteria or measurements typically used by human readers and have enabled the automated ECG diagnosis algorithms that have assisted clinicians for decades. ML algorithms provide complementary strengths to rule-based algorithms since they do not depend on previously defined criteria, tend to improve performance as available training dataset sizes increase, and are capable of learning patterns for themselves from the raw data, some of which may have been previously unknown to humans.

Hannun et al.⁷ was among the first to train an end-to-end DNN using raw ECG waveforms to detect 12 arrhythmia diagnoses. They demonstrated performance similar to cardiologists for set of 12 arrhythmia diagnoses, providing proof of concept that DNNs can be trained to analyze raw ECG data. The average F1 score was similar to or higher than the average F1 score of a committee of cardiologists. When matching specificity to the specificity of the cardiologists, the model showed similar or higher sensitivity for all 12 arrhythmia diagnoses.

Attia et al.⁸ demonstrated that a clinically relevant novel task could be performed through ML analysis of an ECG: prediction of asymptomatic left ventricular (LV) dysfunction. Using paired ECGs and echoes, they trained a DNN to identify LV ejection fraction ≤35% from standard 12-lead ECG data. Others have similarly used DNN-based approaches to accomplish a variety of novel tasks using ECGs. Examples include detection of hypertrophic cardiomyopathy (HCM),^9,10 pulmonary hyptertension,^9,11 amyloid,^9,12 mitral valve prolapse,⁹ mitral and aortic regurgitation,¹³ aortic valve stenosis,^13,14 hyperkalemia,¹⁵ and mortality risk estimation.^16,17

Recent work by Hughes et al.¹⁸ trained a DNN to detect 38 different ECG diagnoses across a range of diagnostic classes, including arrhythmia, conduction, infarct, and chamber enlargement. In addition to comparing the DNN against a committee of electrophysiologists, they also compared against automated diagnoses from the commercially available GE MUSE system. The DNN outperformed the rule-based MUSE diagnoses for all classes except supraventricular tachycardia. Compared with clinical cardiologist reads, the DNN demonstrated comparable or better performance on 34 out of 38 classes. Human readers outperformed the MUSE system for 30 out of 38 classes. Hughes et al. also applied a technique, broadly called AI explainability, that helps illuminate what portions of the ECG waveform the DNN learned as being the most important for each diagnosis it makes. In addition to illuminating well-understood ECG regions associated with diagnoses, such as the delta-wave for Wolff-Parkinson-White syndrome, such techniques provide the possibility to discover new physiologic associations in a data-driven manner using ML. Such techniques may be even more important when ML algorithms successfully make novel diagnoses based on a given data modality, providing an opportunity for us to learn from the algorithms what patterns in the data—and their physiologic underpinnings—are consistently associated with the novel diagnosis.

More recently, DNN analysis of ECGs has been used to track longitudinal disease status and response to drug therapy, potentially providing a new paradigm for longitudinal monitoring. In a cross-institutional/industry collaboration, Tison et al.¹⁹ deployed two DNNs separately trained to detect HCM to clinical trial patients receiving a new drug for HCM, who also underwent serial ECG, echo, and lab measurements. Both DNNs’ predictions of HCM risk not only decreased over time in HCM patients receiving drug therapy but also tracked decreases in cardiac hemodynamics (measured by serial echos) and serum lab values, providing the first demonstration of ECGs not only capturing this type of information but also tracking changes in these measurements over time.

Other demonstrations further illustrate the use of ML to make predictions beyond what can typically be done by human experts using standard ECG data. Tison et al.⁹ demonstrated that ECG data alone can predict continuous cardiac structural and functional metrics as quantified by echoes such as LV mass (in g/m²), LA volume (in mL/m²), and mitral annulus (e′). While ECG-based rules exist for clinicians to predict some of these metrics in a binary yes/no manner from ECGs, the possibility of estimating the continuous value of a measure such as LV mass in gm/m² from an ECG alone demonstrates the broader range of possibilities ML can offer. Ulloa Cerna et al. developed an ML model incorporating an ECG DNN score to predict valvular disease, reduced EF, and increased interventricular septal thickness.²⁰ Attia et al.²¹ trained a DNN to identify patients with atrial fibrillation (Afib) using normal sinus rhythm ECG, and Raghunath et al. trained a DNN to predict new-onset Afib within 1 year after the ECG in patients with no prior history of Afib.²² Together, these results suggest that sinus ECGs contain signals to predict Afib that are not currently appreciated by human experts. Kwon et al. developed a DNN to detect anemia from the ECG,²³ whereas Bos et al.²⁴ trained a DNN to diagnose concealed long QT syndrome (LQTS), a genetically defined disorder in which 40% of patients present with a normal QT at rest. The DNN was not only able to identify patients with this concealed LQTS but it was also able to distinguish between the three main genotypic subgroups using ECG alone.

In all cases, the AI/ML algorithms exhibit some degree of error. Therefore, prior to clinical deployment of ML algorithms, the next critical step in evaluation must be a real-world prospective assessment of the clinical benefit of deploying the algorithm and what degree of error is acceptable, which is specific to each disease and clinical context. Properly designed randomized controlled clinical trials (RCTs) provide the highest-level of evidence in this regard, and RCTs examining ML algorithm deployment will certainly be forthcoming in the near term. One important example is the EAGLE trial, one of the first RCTs to evaluate DNN analysis of 12-lead ECGs to predict low ejection fraction.²⁵ In the EAGLE trial, 120 primary care teams were cluster randomized to access to the DNN’s analysis, and the primary outcome was new diagnosis of low ejection fraction (≤50%) within 90 days. The intervention arm exhibited increased diagnosis of low ejection fraction, suggesting that application of the DNN algorithm for ECG analysis can increase the earlier diagnosis of low EF in a primary care setting.

Echo

Echo is a central diagnostic test in cardiology, providing invaluable information about cardiac anatomy and real-time hemodynamics that are essential for the diagnosis and management of most cardiac diseases. Although it is noninvasive, does not use ionizing radiation, and has lower cost, echo does require greater specialized training to both obtain and interpret compared with other tests such as the ECG. ML algorithms can provide value across the echo workflow, from acquisition to pre-processing, anatomic measurement, and interpretation. As reviewed below, the bulk of AI applications in echos have focused on one or more of three main goals: image pre-processing, automating echo measurements, and automating disease diagnosis.

Echo studies consist of multiple two-dimensional videos of the heart from various standard views. The first task a cardiologist in training must learn is to recognize each of the standard echo views, before progressing to learn how to interpret each view. In a similar way, since echo views are typically not currently recorded in echo video metadata, the first step in automated AI echo analysis is view classification. Early work showed some success using traditional ML approaches relying on features hand-derived from echo images^26,27,28,29; however, in the past 5 years, DNNs have achieved reliable echo view classification directly using the raw data. Gao et al.³⁰ was among the first to apply DNNs in this manner, showing that DNNs could be trained to analyze raw image frames directly and outperforming hand-engineered ML approaches. Subsequent work expanded the number of views that could be identified,³¹ while Zhang et al.³² additionally averaged a DNN’s predictions across multiple video frames to achieve high accuracy in classifying 23 distinct views.

Once echo views have been identified, the next step in the clinical echo work flow is performing standard measurements to quantify size and volume of cardiac structures. This is usually performed manually by either the sonographer or the interpreting cardiologist. AI-based automation may increase accuracy, reproducibility, and speed of performing these measurements. Various efforts have developed ML-based approaches to replicate specific echo measurements, such LV size and volume,^33,34,35 or leaflet dimensions.³⁶ Zhang et al.³² was the first to employ a specific DNN architecture called a U-Net to segment multiple cardiac chambers simultaneously, similar to an echocardiographer tracing cardiac chambers, and then use these to calculate standard measurements of various chambers such as volume, mass, ejection fraction and longitudinal strain. In contrast, Ghorbani et al.³⁷ showed that a DNN could be trained to directly predict chamber volumes, such as end-systolic LV volume or end-diastolic LV volume, and even ejection fraction from raw images without first segmenting the chamber. In subsequent work, Ouyang et al.³⁸ trained a video-based DNN to perform segmentation of the LV and estimation of ejection fraction, and showed that this could be done across multiple cardiac cycles captured during the echo video. The more cardiac cycles examined, the lower the error in the DNN-estimated ejection fraction. When measurements are performed manually, only several frames (often from a single cardiac cycle) are chosen and measured by the sonographer on account of the time required to perform the manual measurements. While this is the standard of care, it does raise the possibility that the frames selected may be not representative or that measurement error may more greatly influence the final assessment. The work by Ouyang et al. thus demonstrates an important manner in which ML can be used to increase reproducibility and decrease measurement error within the clinical workflow, since there is minimal to no incremental cost in deploying the AI algorithms across a large number of cardiac cycles or frames. In a similar manner, strategically trained AI algorithms could decrease variability and increase reproducibility at various other points in the echo work flow from echo acquisition, measurement of other echo parameters, and disease interpretation.

ML algorithms have been trained to detect various cardiovascular diseases from raw echo images and from echo-derived parameters. Examples include pulmonary arterial hypertension,³² HCM,^32,39 cardiac amyloidosis,^12,32 coronary artery disease,^40,41 LV hypertrophy,⁴² regional wall motion abnormalities,⁴³ and pediatric congenital heart disease.⁴⁴ Several studies have shown the ability to detect heart failure defined by low ejection fraction,^32,37,38 as well as heart failure with preserved ejection fraction,^45,46,47,48 and to differentiate stress cardiomyopathy from acute myocardial infarction.⁴⁹ Ulloa Cerna et al. trained an algorithm that demonstrated the ability to predict all-cause mortality from echo videos, a task not typically performed by cardiologists interpreting echos.⁵⁰ Similarly, Hughes et al. developed a DNN capable of estimating lab values such as brain natriuretic peptide (BNP), troponin I, blood urea nitrogen (BUN), and others from the echo video alone, a task clearly outside the normal range of diagnostic ability of the echo.⁵¹ Clearly, this is far from the comprehensive set of echo diagnoses that would be needed to support fully automated echo interpretation. These studies, however, provide a valuable proof of concept that ML algorithms can be trained across a range of clinically relevant diagnoses. Future work will be needed to continue to expand the list of diagnoses for which accurate reproducible algorithms exist.

ML algorithms have also been applied to assist with the acquisition of echo images that is typically performed by a specially trained echo sonographer. Becoming a proficient echo sonographer is a years-long process. ML algorithms could assist less experienced operators to obtain basic echo images. Narang et al.⁵² conducted a prospective study with such a DNN-based system to test how well it could guide untrained operators to obtain echo images. In this study, the ML system provided guidance to nurses who had not previously performed echo regarding how to acquire and improve specific echo views in real time. They demonstrated that the system could guide the novice sonographers to obtain diagnostic-quality echo images. This shows how ML algorithms can be deployed to extend the ability to obtain echos to those without prior experience, potentially expanding the accessibility of this specialized test. Future integration of DNNs into the clinical work flow in similar ways may offer clinicians and health systems an opportunity to reconsider what can be achieved with standard medical tests and who can help accomplish it.

Other data types in cardiovascular medicine

Other modalities in cardiovascular medicine also provide fertile ground for future ML efforts. ML may offer the greatest potential for data types that have traditionally been more difficult to analyze in an automated manner, such as diagnostic tests recorded in video formats such as coronary angiography, high-dimensional genetic data, and free-text notes from electronic health records.

Coronary angiography is the definitive diagnostic test for the evaluation of coronary artery disease, providing the basis for recommending both medical and procedural therapy (including percutaneous coronary intervention [also known as stents], and coronary bypass surgery). Coronary angiograms are acquired and stored as video data consisting of fluoroscopic (X-ray) clips obtained during the injection of radiocontrast dye into the various coronary arteries for visualization of the coronary artery lumen and potential stenoses. Prior research has suggested that there is substantial variability and possibility for operator bias in the process of manual interpretation of angiograms.^53,54,55 Until recently, the video-based nature of raw angiogram data had made it difficult to develop automated analysis approaches. Recent work from Avram et al.⁵⁶ and Zhou et al.⁵⁷ demonstrated that DNNs can be trained to accurately perform various steps required to analyze coronary angiograms, such as viewpoint classification or coronary artery localization, providing the basis for automated angiogram interpretation. Avram et al.⁵⁶ deployed multiple DNN’s together in a pipeline to achieve accurate automated estimation of coronary artery stenosis severity. Future AI-driven progress with coronary angiography may eventually lead to assisted or automated interpretation, which could help to reduce human error and increase the reproducibility of measurements, and possibly even expand the range of diagnoses that are possible to make through angiography, similar to what has been achieved with ML analysis of ECG and echo.

Several deep learning (DL) applications have been developed for cardiovascular CT applications. Recent ML work has expanded the information that can be derived from various types of CT scans; for example, enabling automatic estimation of coronary artery calcium score from CT angiography⁵⁸ or low-dose chest CT scans,⁵⁹ or quantifying epicardial and thoracic adipose tissue from non-contrast CT scans.⁶⁰ DL algorithms have accurately estimated coronary artery stenosis⁶¹ and fractional flow reserve⁶² using coronary CT angiography. These efforts demonstrate the potential for DL algorithms to broaden the scope of current clinical tools such as CT and potentially reduce the need for more invasive procedures such as coronary angiography.

Cardiac magnetic resonance imaging (CMR) provides high-resolution images of high diagnostic value, but is relatively expensive, slow, and labor intensive. Many DL applications in the CMR domain focus on alleviating these pain points by automating manual processes and improving efficiency. Schlemper et al.⁶³ developed a DL-based 3D reconstruction pipeline that can interpolate slices with long inter-slice distance, increasing the speed of acquisition and providing fast real-time reconstruction. Blanser et al.⁶⁴ present a DL system for automated prescription of imaging planes by using a U-Net to localize anatomic landmarks. A series of recent contributions have demonstrated automated segmentation of anatomical structures in CMR images,^65,66,67 with future similar disease-specific work sure to follow. The automated cardiac diagnosis challenge (ACDC) dataset is a publicly available dataset of 150 CMR studies with manual segmentation labels as well as five diagnostic labels. The availability of these dataset has enabled multiple efforts that have produced DL-based methods for automated diagnosis of these five diagnostic categories: normal, HF with infarction, dilated cardiomyopathy, HCM, and abnormal right ventricle.^68,69,70,71 This highlights the importance of labeled public datasets for the rapid development of algorithms.

ML algorithms have been developed for other modalities of large-scale data relevant for cardiovascular medicine, including genetic data and free-text data from the electronic health record. Free-text data present a substantial analytic challenge due to the complexity of natural language. The analytical field known as natural language processing (NLP) has developed over decades, which aims to accurately analyze and process free-text data. In the past 5–10 years, DNN-related architectures have revolutionized the NLP field by providing exponential progress compared with prior techniques, mirroring advances made in the image processing and computer vision fields. For example, the DNN architecture class known as the transformer is widely applied in NLP due to its ability to learn very long-range associations across large bodies of text.^4,5,6 As AI-based text-analysis capabilities continue to advance outside of medicine, there will be significant opportunity to extrapolate these advances to better analyze free-text medical data, such as from the electronic health record, diagnostic test reports, and possibly patient-reported symptoms and outcomes. Moon et al.⁷² recently demonstrated the extraction of sudden cardiac death risk factors from free-text notes, and Blecker et al. identified heart failure patients at large scale from the electronic health record.⁷³ As the field matures, medical implementations of transformers⁶ and future such developments in free-text analysis will undoubtedly be applied to advance cardiovascular applications.

Analysis of genetic data has long benefitted from the robust bioinformatics analysis community and library of techniques that have driven the countless advances in modern genetics. Cardiovascular genetics has recently exhibited increasing examples of ML-driven analysis, including for the investigation of coronary artery calcium,⁷⁴ pulmonary hypertension,⁷⁵ and multiple clinically relevant variant detection^76,77 from next-generation sequencing or proteomic data. The greatest potential for AI in genetic analysis may lie in the analysis of polygenic disorders, since algorithms such as DNNs have enormous capacity to examine high-level and very complex interactions within the genetic data, if provided adequate quantities of training data.

The critical need for prospective clinical evaluation of AI in cardiology

As with any diagnostic tool, development and validation of AI algorithms are only the first step toward clinical application. The next critical phase of evaluation requires prospective deployment of an AI-based algorithm into a clinical work flow and assessment that relevant clinical outcomes or work flow metrics are substantially improved. The goals of such evaluation are similar to those for a novel medical device. Such prospective clinical evaluation may be even more critical for ML software algorithms since, at least initially, they will likely still rely upon interaction with human clinicians to make a final diagnosis or plan of therapy. Methods that can help clinicians understand how the algorithm functions and makes its decisions, such as AI explainability or interpretability techniques, may not only help to build trust among clinicians who must integrate an algorithm’s conclusions into their decision making but also provide context to clinicians as to when an algorithm may be making unreasonable predictions.

There has been a recent increase in AI-related cardiovascular clinical trials registered on clinicaltrials.gov, the US government-sponsored Web site on which clinical trials are required to be reported. At the time of writing, we identified 17 cardiology-related trials reported on clinicaltrials.gov: three used 12-lead ECGs, four used single-lead smartwatch ECGs, five used echos, three used angiograms, one used a proprietary take-home stethoscope, and one used free text from the electronic health record. Seven were reported as being completed; however, only one²⁵ had published results. As discussed above, the EAGLE trial²⁵ demonstrated that a DNN-enabled ECG analysis system for probability of low ejection fraction increased early detection of low ejection fraction compared with standard of care in primary care clinics. Many future clinical trials examining the deployment of AI algorithms within clinical workflows and designed to evaluate clinically relevant questions—such as accuracy of diagnosis, speed to therapy or increased rate of early detection—will be needed to provide the evidence to support clinical adoption of AI algorithms. In some cases, it is expected that ML-based algorithms may either not outperform existing standard of care or may not provide the clinical outcomes to support adoption.

Governmental regulatory agencies, such the United States Food and Drug Administration (FDA), govern the level of scrutiny required for an ML algorithm to obtain approval for clinical adoption. While clinical trials are not necessarily required for approval of an algorithm, the level of evidence required varies by governmental agency and generally correlates with the potential risk an algorithm poses to patients and the existence of previously approved similar algorithms.⁷⁸ FDA approval of ML algorithms has increased in recent years, with 130 ML-based algorithms approved between 1997 and 2020, most being after 2016.⁷⁹ Under current FDA rules, once these algorithms are approved, they must be locked, meaning that the algorithms should not be updated in real time as more data is obtained. While this may help to ensure algorithm consistency, it shortchanges one of the inherent benefits of ML algorithms, which is an ability to improve performance as more training data become available. Similar approaches can also be applied to allow AI algorithms to adapt their predictions as populations and data change over time, also known as input distribution drift.^80,81,82,83 In 2019, the FDA acknowledged this limitation and proposed a plan for implementing a control plan that would allow algorithms to be updated based on a predetermined process.^84,85 To date, this plan has not yet been codified but is a welcome sign that the inherent strengths of AI algorithms can be appropriately leveraged for medical applications. The coming years will likely mark a turning point in the adoption of AI algorithms in medicine, driven by algorithm validation in RCTs, increased algorithm approvals by the FDA, and improved FDA guidelines for retraining and monitoring deployed AI algorithms.

Limitations and risks of ML in cardiovascular medicine

The nature of ML systems presents several notable challenges to implementation in medical practice. These include opacity of algorithm decision making and potential for bias, which are related to the complexity of the algorithms and biases present in the training data. There is also potential for degradation of algorithm performance over time due to data distribution drift. Significant work is ongoing to address these issues not only in medicine but in the broader ML field.

A common criticism of ML algorithms in general, and DNNs in particular, is the relative difficulty of interpreting how the algorithms make their decisions. These algorithms are extremely complex and derive their predictive power by allowing very high levels of interaction among the input data, which accordingly makes it nearly impossible to understand how the algorithms make a certain prediction simply by looking at the model parameters. In comparison, a simple generalized linear model (such as linear regression) with a handful of predictors can be interpreted by looking at the magnitude and directionality of the trained weight associated with each predictor. Interpretability of algorithms carries particular value in medicine because the decisions they influence involve human health and the operational efficiency of health care systems. Therefore, understanding why and how an algorithm makes its prediction can provide valuable information that physicians can use to either incorporate the prediction into clinical care and/or learn what features the algorithm found important in the data, providing data-driven physiologic insights. In medicine, interpretability can help increase trust by physicians in the algorithm. AI explainability techniques have been developed, such as variable importance analysis, local interpretable model-agnostic explanations (LIME),⁸⁶ or gradient-weighted class activation mapping (Grad-CAM),⁸⁷ providing some insight into aspects of the input data that a given trained algorithm finds important to predict the target task. However, each of these methods provides only a limited view into algorithmic decisions, and none provide a comprehensive view. As shown in Ulloa Cerna et al.,⁵⁰ clinicians can have a difficult time interpreting the output of these methods on a case-by-case basis, due in part to the variability between cases and because some highlighted associations may represent artifact or noise. The ML field is actively working on developing novel approaches to improve ML interpretability.

It is important to recognize the potential for bias in any predictive model, and this phenomenon can be enhanced in some circumstances with ML. In part because ML models can so powerfully learn from the data on which they are trained, they are also positioned to learn biases present in these datasets.⁸⁸ Biases may be present in the proxies used to define the target task or labels used to train the model.⁸⁹ They may also arise if certain patient demographics are poorly represented in the training data,⁹⁰ or when training data are restricted to a limited geographic area.⁹¹ The use of external validation datasets, and observing degradation of model performance therein, may help to illuminate presence of some biases, although this would only be true if the same biases do not also exist in the external validation dataset. Ideally algorithms should be trained and tested in demographically diverse target populations similar to the target population of interest.⁹² Efforts should be made to identify potential sources of bias at every step of the research process from problem formulation to clinical validation.

Real-world clinical implementation of DL algorithms will require addressing data distribution shifts. Data distribution shift, or distribution drift, refers to changes that occur in either the input data, the output labels, or the relationship between them. For example, a model is trained to detect HCM. If HCM guidelines undergo revision and the criteria for HCM diagnosis change, then this previously trained algorithm may perform worse to detect HCM as defined by the new guidelines. Other examples can be more subtle. For example, if patient populations presenting to a given healthcare institution change, this may affect the disease prevalence in the population or other associated characteristics, which may degrade algorithm performance. Companies outside of medicine relying on ML models in production often retrain these models either at fixed intervals or have established monitoring systems to detect performance degradation to trigger model retraining. Updated FDA guidance will help to inform how healthcare institutions should perform this kind of retraining while maintaining sufficient safety, oversight, and compliance.

Conclusions

The field of modern AI, which can be defined as beginning with the development of DNNs in the past decade, has driven substantial progress in medical algorithms thus far. But in almost every respect this progress represents only the tip of the iceberg of the depth of impact ML could ultimately have on medicine. Much of the important work discussed above represents the early proof of concept showing how ML algorithms can be deployed in various ways related to cardiovascular disease. In many cases, similar ML approaches can be replicated for countless other diseases using similar algorithm architectures and data types. For every reported study in the literature there are likely dozens of related diseases, medical tasks, or applications for which similar ML algorithms can be trained. In many cases, access to sufficient multi-institutional labeled training data is the limiting factor in expanding the number of medical applications for AI approaches, highlighting an important area of future development for medical AI. Additional methodologic innovation in ML may expand the types of medical tasks that can be performed. The bulk of the effort, time, and capital that will be required to truly drive the clinical adoption of ML algorithms in medicine will almost certainly lie in the long-tailed process of iterative algorithm refinement and process improvements, including developing the infrastructure and methods that enable algorithms to be improved in real time based on human expert feedback, patient outcomes or increases in labeled training data. Therefore, the work still required by the broader medical community to achieve widespread clinical adoption of ML is also just at the tip of the iceberg.

The enormous potential of data-driven ML in medicine lies in the ability to employ algorithms that can help clinicians provide better care and enable physiologic discovery from large amounts of medical and biologic data. Although most modern medical data are accessible in digitized formats, they remain largely siloed within individual healthcare systems, walled off by administrative barriers that limit the large-scale data sharing that would likely drive the greatest ML-based medical advances. For all of the advances in AI technology over the past 10 years, perhaps the greatest medical AI innovations remain ahead of us: a combination of administrative and technical privacy-preserving approaches to allow ML algorithms to derive insights from diverse cross-institutional data, to more completely realize the fullest potential of AI in medicine.