Deep learning for cardiovascular medicine: a practical primer

Abstract

Deep learning (DL) is a branch of machine learning (ML) showing increasing promise in medicine, to assist in data classification, novel disease phenotyping and complex decision making. Deep learning is a form of ML typically implemented via multi-layered neural networks. Deep learning has accelerated by recent advances in computer hardware and algorithms and is increasingly applied in e-commerce, finance, and voice and image recognition to learn and classify complex datasets. The current medical literature shows both strengths and limitations of DL. Strengths of DL include its ability to automate medical image interpretation, enhance clinical decision-making, identify novel phenotypes, and select better treatment pathways in complex diseases. Deep learning may be well-suited to cardiovascular medicine in which haemodynamic and electrophysiological indices are increasingly captured on a continuous basis by wearable devices as well as image segmentation in cardiac imaging. However, DL also has significant weaknesses including difficulties in interpreting its models (the ‘black-box’ criticism), its need for extensive adjudicated (‘labelled’) data in training, lack of standardization in design, lack of data-efficiency in training, limited applicability to clinical trials, and other factors. Thus, the optimal clinical application of DL requires careful formulation of solvable problems, selection of most appropriate DL algorithms and data, and balanced interpretation of results. This review synthesizes the current state of DL for cardiovascular clinicians and investigators, and provides technical context to appreciate the promise, pitfalls, near-term challenges, and opportunities for this exciting new area.

Introduction

The practice of cardiovascular medicine routinely requires management of conditions as complex as heart failure with reduced (HFrEF) or preserved (HFpEF) ejection fraction, multivessel coronary disease, complex arrhythmias, sudden cardiac arrest, cardiovascular diseases (CVDs) during pregnancy, or congenital heart disease. Despite advances in each of these areas, significant clinical challenges remain. Many challenges relate to the complexity of integrating data from multiple modalities, making actionable predictions and distilling these solutions to individual patients with heterogeneous phenotypes.^1–3 Deep learning (DL) is a branch of artificial intelligence (AI) that combines computer science, statistics and decision theory to find patterns in complex and often voluminous data. In general, DL is a type of machine learning (ML) that typically utilizes multi-layered neural networks (Figure 1). Deep learning has been already shown to outperform experts and other ML strategies in areas as diverse as voice recognition, image classification, commerce and game playing,^4–7 and there is anticipation in that DL could similarly disrupt clinical decision-making by integrating complex data streams, making ‘intelligent inferences’ and ultimately personalizing therapy.

Figure 1.

Relationship of deep learning to clinical and translational medicine. Venn diagrams show deep learning as one type of machine learning, within the scope of artificial intelligence. Statistical methods are applied across clinical and translational science, and the form known as statistical learning theory has overlap with machine learning. Automated decision making is often used in clinical practice. Deep learning may extend statistical approaches in some key areas by analysing large multivariate datasets, which often show complex interactions, in which simple hypotheses are difficult to formulate. Deep learning has been successful in medical image recognition (e.g. electrocardiogram, echocardiogram, and magnetic resonance imaging) and holds the promise of enhancing clinic decision making.

There are several parallels, as well as divergences, between traditional statistical methods, ML and DL. Statistical approaches typically test hypotheses or estimate parameters and emphasize inference based upon statistical sampling of a population. In these cases, traditional statistics can be as effective as DL even in large ‘big data’ applications; for instance in genome-wide association studies (GWAS), a single loci meeting a Bonferroni-adjusted P-value (e.g. P < 5 × 10⁻⁸) can identify important traits. However, if simple hypotheses are less readily formulated due to complex interactions, for instance if GWAS yields multiple concurrent ‘hits’, ML may be better suited because it does not require specific hypotheses, and can analyse varied data elements with complex interactions. Deep learning strategies generally attempt to use as much information as is available in a dataset (e.g. every pixel in echocardiography images) in order to generate novel features to be used for downstream analysis. Statistical methods and DL are both influenced by aberrations in sample data and may suffer from overfitting. While statistical methods include approaches to evaluate the possibility of overfitting, a limitation of DL is that it relies to a much greater extent on empirical validation.

To date, cardiovascular applications of DL have been promising^8–10 although many challenges remain. In particular, it is critical to select the right tools for each specific problem and dataset in CVD. This practical review is designed to enable the reader to understand and evaluate applications of DL to cardiovascular medicine or research. We discuss the historical development of DL, definitions, review the current literature to recognize optimal applications, summarize the design and interpretation of DL studies, discuss current challenges and pitfalls, and future directions.

History and definitions

Artificial intelligence is the field of computer science broadly focused on teaching computers to learn complex tasks and make predictions. Early AI applications focused on hand-developing complex decision rules for computers to follow but, due to the complexity of human decision making, this was by-and-large not successful. Instead ML, and particularly DL, have emerged as more promising. Machine learning analyses data in ways that automate the construction of analytical models and decision rules, developing systems that learn from data and can identify patterns and make decisions. Ideally, this can happen with minimal human intervention. Machine learning can be further subdivided into supervised, unsupervised, or reinforcement learning. Figure 2 summarizes categories of ML and DL in the context of emerging applications in cardiovascular science. Supervised learning identifies patterns in large amounts of data that are typically annotated (‘labelled’) by humans, such as the presence or absence of reduced systolic function on an echocardiogram or atrial fibrillation (AF) on an electrocardiogram (ECG). Supervised learning may be implemented using neural networks, long-used for medical pattern recognition in cardiology,^11,12 neuroscience,^13–15 and other fields yet still limited in clinical use. Unsupervised learning, on the other hand, analyses large amounts of unlabelled data to identify hidden patterns or natural structure in data,¹⁶ which greatly increases the volume of data that can be analysed (e.g. from large electronic medical records) at the potential cost of data quality and interpretability. Reinforcement learning trains software to make decisions that maximize a ‘reward’ function,¹⁷ which may address a clinical problem (e.g. improve ST-segment elevation myocardial infarction outcomes, or reduce error in ECG diagnosis).

Figure 2.

Types of machine learning in cardiovascular science. (A) Supervised learning uses inputs (e.g. electrocardiograms) each with a label (‘ground truth’, and a diagnosis of atrial fibrillation or not atrial fibrillation). Machines are iteratively ‘trained’, using direct feedback for multiple inputs, until their output matches the ground truth. Trained machines can then classify unknown (test) electrocardiograms. One misclassification is shown. (B) Unsupervised learning uses unlabelled data, ideally in large quantities, to identify novel patterns. In this example, QRS indices identified novel phenotypes (‘clusters’) for hypertrophic cardiomyopathy with distinct outcomes (Ref: Lyon et al.²⁶). (C) Reinforcement learning uses models developed from psychological training applied to gaming, but infrequently to medicine. An agent, e.g. a clinical decision-making tool, performs an action A_t (e.g. which therapy for non-ST-segment elevation myocardial infarction best reduces mortality? (1) non-invasive, (2) early invasive, and (3) mixed) that alters the environment (e.g. biomarker response or patient outcomes). A R_t reward is then given (e.g. higher survival rate) that alters the state S_t. This process is iterated with the intention of moving State S_t+1 closer to the desired goal (i.e. improved outcomes).

Deep learning is a specific type of ML inspired by the way that the human brain processes data, and enabled by hardware advances such as graphics processing units (GPUs),^16,18,19 vast catalogues of labelled data, and advances in computer science theory. To date, most implementations in medicine have used convolutional neural networks (CNNs). Following the ‘AI winter’ from the 1980s, when early rule-based and neural network applications were limited by hardware and algorithmic constraints, DL has accelerated supervised, unsupervised, and reinforcement learning. In 2016, DeepMind's AlphaGo Zero^4,5 beat the world champion in the ancient Chinese game of Go, and deep Q-learning⁶ proved as accurate as a professional human player in 49 interactive video games. In 2018, DL applications by DeepMind rivalled humans in the 3D multiplayer videogame Quake III Arena.⁷Figure 3 provides a schematic of how neural networks, a common basic architecture for DL, could be used to classify an ECG.

Figure 3.

Neural network design to classify atrial fibrillation from the electrocardiogram. Continuous electrocardiogram voltage points (red dots, arrows) are fed to ‘input neurons’ (x₀, x₁, x₂, …, x_m), which are coded as software objects. Hidden neurons within this three-layer network (h₀, h₁, h₂, …, h_n) connect input and output layer neurons (here, two neurons) by numerical weights (w). Deep learning typically uses multiple hidden layers, as shown here. The output indicates atrial fibrillation (y₁; correct, red) or non-atrial fibrillation (y₀). If the output is correct for that electrocardiogram input, weights are strengthened; else they are reduced. This process is iterated during training on multiple input electrocardiograms. The trained network can then be tested on new (unseen) electrocardiograms. Other designs could accept categorical variables (age, gender) or mixed data types.

Implementing deep learning in cardiovascular applications

Hardware and software considerations

Historically, ML was computationally expensive and performed by scientists using supercomputers or high-end workstations with multi-core processors. However, due to the highly parallelizable nature of DL algorithms, GPUs designed for gaming have enabled DL to be performed on desktop machines. Although professional-quality GPUs are still relatively expensive, DL can be performed using cloud services by services such as Amazon AWS or Google Cloud. Deep learning software packages are almost uniformly open-source, which means they are freely available with few constraints for academic research. Furthermore, many DL pre-trained models can be downloaded and repurposed for new tasks, including models such as AlexNet,²⁰ VGG Network,²¹ InceptionNet,²² and ResNet,²³ and will be further described in following sections. In fact, downloading pre-trained models and repurposing them for new datasets avoids much of the time consuming and computationally expensive steps of DL. Table 1 demonstrates step-by-step an example DL process in cardiovascular medicine.

Table 1.

A guide to approach of the deep learning applications in cardiovascular medicine research and clinical practice

Step	How to
Identify research questions and outcomes	A supervised learning problem (e.g. predict labelled outcomes) vs. unsupervised learning problem (e.g. identify or classify new phenotypes of HFpEF or new genotypes of PAD) If supervised, is it a regression task (e.g. prediction of the cost of care for a PCSK9 inhibitor treatment) or classification (e.g. predict if a given patient has a disease or not)
Data selection	Public databases vs. EHR databases vs. Registry Identify limitations of databases and attempt to replace with rational variables (i.e. no lab values, no vital signs between admission and discharge, no medications, no specific ICD codes for MitraClip) Identify appropriate methodologies for database (e.g. do we realistically have enough data to attempt DL? If not, are we better off selecting an alternate approach to our problem?)
Hardware selection	Computer cluster vs. workstation with GPUs vs. cloud computing services Can we build or buy a computer cluster or workstation with GPUs? In the long run, this will be much cheaper but with much greater expense upfront. Should we instead use cloud computing services (i.e. Google Cloud or Amazon Web Services), which have little-to-no upfront cost but can cost more in the long run
Data preparation	De-identify data if needed Quality control of data—assessment of missingness and verification that our dataset contains what it should. Identify mechanism of missing data and then data imputations for non-ignorable missing data Denoising of images/video/textures or variant calling in NGS data Exploratory analysis (summarization, visualization, identify structure of data/relationship between variables)
Feature selection	In general, DL requires little a priori feature selection. If the input dataset is highly multidimensional, strategies such as vector embedding may be required first in order to pass features to other DL models
Data splitting	Design and justify the proportion of training, validation and testing in the dataset (i.e. 70/10/20 or 80/10/10 or 60/20/20)
Modelling selection	One should always see if the task can be accomplished with a simple model or standard statistical approaches (can we simply apply linear/logistic regression or polynomial regression to our dataset? If so, does it perform adequately?) If simpler algorithms do not work, more complex strategies such as DL may be needed. In general, reuse of pre-trained models using transfer learning is preferred since many pre-trained models are well validated in a large database and their performance characteristics and limitations are known If no pre-trained models are available for specific type of data or research questions, develop new algorithms based on highest achievement model for specific type of data
Technical details for model	Know some DL technical terms to communicate with data scientists or programmers and understand the process (learning rate selection, tuning hyperparameter, batch dropout and normalization, regularization strategies, loss function selection, and network optimization)
Evaluation of model discrimination	Report ROC curve, C-statistics, NPV, PPV, sensitivity, and specificity
Evaluation of model calibration	Compare with standard statistical approaches (i.e. multivariable regression), goodness-of-fit, calibration plots, or the decision-curve analysis
Ground truth	Compare with human experts (cardiologists, electrophysiologist, primary care physicians)
Publication and transparency	Share codes with journal (i.e. online supplements) or public space (i.e. Githut, bioRxiv). DL methodologies should be clearly explained in details. Consider strategies for computational anonymization
Generalization and replication results	Test with different datasets in a different population
Clinical trials	To minimize risks or errors, testing prediction models in clinical trials is recommendated
Meta-analysis	Meta-analysis of DL is needed to assess publication bias and heterogeneity
AI Guidelines	With clinical trial and meta-analytic results, professional societies have to develop guidelines to regulate DL in clinical decision making or predictive analytics in clinical practice
Implement in clinical practice	Start implementing in clinical practice and monitor the results closely

AI, artificial Intelligence; DL, deep learning; EHR, electronic health record; GPU, graphics processing unit; HFpEF, heart failure with preserved ejection fraction; NGS, next-generation sequencing; NPV, negative predictive value; PAD, peripheral arterial disease; PCSK9, proprotein convertase subtilisin/kexin type 9; PPV, positive predictive value; ROC curve, receiver-operating characteristic curve.

Selecting a deep learning software package and modelling strategy

The first practical step for DL is to choose an appropriate software package to work with such as Keras, Tensorflow, or others. Keras is often used as a starting point, as it can be used in a relatively straightforward fashion with high-level programming languages, most commonly Python. Supplementary material online, Table S1 summarizes some platforms and related programming languages for DL. Second, one should choose a DL model appropriate for the problem under consideration, which may be pre-existing (pre-trained models) or require development of a custom model (novel models). In general, due to time restrictions, and computational requirement, pre-trained models are mostly used first, and then tailored to investigators' problems/datasets by modifying outcomes related layers [i.e. last few layers or final layer (softmax layer)] to optimize their results. This concept is called transfer learning.²⁴ For example, if a pre-trained model was trained to accurately predict diastolic dysfunction using global longitudinal strain in HFrEF patients in a large database, we might be able to use resultant knowledge to conduct the prediction of diastolic dysfunction in HFpEF patients. There are several pre-trained models such as AlexNet,²⁰ which recognizes visual patterns directly from pixel images with minimal preprocessing, the VGG Network²¹ that performed well in the ImageNet Large Scale Visual Recognition Challenge in 2014, or other models (e.g. InceptionNet²²). When validated and pre-trained models are unavailable for specific applications, which may be common in cardiovascular medicine at this early stage, custom models may be required. Each will require design (e.g. the number of layers, nodes, learning rate, and so on in Figure 3), validation and tuning.^4,25 Examples are provided in Supplementary material online, Table S2. However, transfer learning is challenging in medicine as there are differences in datasets, i.e. sources of data (non-medical videos vs. echocardiographic videos), data quality, vendor or softwares. There is no standardized guideline for when it is appropriate to use transfer learning between datasets, and this often requires empirical trial-and-error approaches.

Table 2.

Deep learning applications in cardiovascular medicine

First author	Disease application	Images/slides (N)	Type of images	Results	Algorithms
Arrhythmias
Bumgarner et al.60	AF detection	169 records	A rhythm strip from Kardia mobile phone	93% sensitivity and 84% specificity for AF detection	Kardia Band automated algorithm
Pyakillya et al.50	ECG classification	8528 records	ECG records	86% accuracy for classification results	Deep CNN
Hannun et al.49	Arrhythmias detection (i.e. VT, Mobitz I, Mobitz II)	64 121 ECG records	ECG records	The model outperforms the average cardiologist performance on most arrhythmias (measured by sequence level accuracy and set level accuracy)	Deep CNN
Tison et al.8	AF detection	9000 ECGs	ECG from smartphone	On recumbent ECG, c-statistic 0.97; on ambulatory ECG, c-statistic 0.72	Deep CNN
Xia et al.48	AF detection	123 848 samples	ECG signal (STFT and SWT)	98.29% accuracy for STFT and 98.63% accuracy for SWT	Deep CNN
Cardiac MRI
Avendi et al.63	Image segmentation (LV shape detection)	45 MRI datasets	Cardiac MRI images	90% accuracy (in terms of the Dice metric)	Deep CNN and stacked autoencoders
Bai et al.62	Image segmentation (LV and RV)	93 500 images	Cardiac MRI images	Reported as Dice metric, mean surface distance, and hausdorff surface distance	Deep CNN (a fully CNN)
Luo et al.64	Image segmentation (LV volumes)	1140 subjects	Cardiac MRI images	High accuracy in LV volumes prediction (measured accuracy by root mean squared error)	Deep CNN
Oktay et al.65	Image quality (i.e. LV segmentation, motion tracking)	1233 healthy adult subjects	Cardiac MRI images	Directly compare image quality and segmentation results (i.e. LV cavity volume differences or surface-to-surface distances)	Deep CNN (super resolution approaches)
Oktay et al.47	Image segmentation (cardiac pathologies)	1200 images	Cardiac MRI images	Reported as Dice metric, mean surface distance, and hausdorff surface distance	Deep CNN
Echocardiography
Dong et al.66	LV segmentation	60 subjects	Echocardiography images	Reported as Dice metric, mean surface distance, and hausdorff surface distance	Deep GAN (VoxelAtlasGAN)
Gao et al.67	8 echo views classification	432 video images	Echocardiography images	92.1% accuracy for classification results	Deep CNN
Knackstedt et al.68	Assessment of LV volumes and EF	432 video images	Echocardiography images	92.1% accuracy for classification results	Deep CNN
Luong et al.43	Assessment of echo image quality feedback	6916 images	Echocardiography images	The average absolute error of the model compared with manual scoring was 0.68 ± 0.58	Deep CNN
Madani et al.44	15 echo views recognition	223 787 images	Echocardiography images	91.7% accuracy for classification results (F-score 0.904 ± SD 0.058)	Deep CNN
Heart failure
Nirschl et al.69	Clinical heart failure classification	209 patients	H&E stained whole-slide images	> 93% accuracy for both training and testing datasets vs. 75% accuracy for pathologists	Deep CNN
Seah et al.70	Prediction of CHF	103 489 images	Chest X-rays	Model achieved an AUC of 0.82 (At a cut-off BNP of 100 ng/L)	Deep GAN
Islam et al.105	Pulmonary oedema detection	7284 images	Chest X-rays	The same architecture does not perform well across all abnormalities	Deep CNN
Myocardial perfusion imaging
Betancur et al.56	Prediction of obstructive CAD	1638 patients	Fast SPECT MPI	AUC for disease prediction by deep learning was higher than for total perfusion deficit	Deep CNN

AF, atrial fibrillation; AUC, the area under the receiver operating characteristic curve; CAD, coronary artery disease; CHF, congestive heart failure; CNN, convolutional neural network; ECG, electrocardiogram; EF, ejection fraction; GAN, generative adversarial network; HCM, hypertrophic cardiomyopathy; LV, left ventricle; MPI, myocardial perfusion imaging; MRI, magnetic resonance imaging; RV, right ventricle; SPECT, single-photon emission computerized tomography; STFT, short-term Fourier transform; SWT, stationary wavelet transform.

Learning rate is an important concept that determines how the network adjusts weights during training based on correct or incorrect decisions. Figure 4 shows appropriate learning rates in a well-trained network, and how inappropriate learning rates (too high) may impede network training. Figure 5 shows how complex cardiovascular data that are not readily separated by a simple cut point or threshold (i.e. cannot be partitioned by a linear classifier) can be transformed and now readily separated.

Figure 4.

Impact of deep learning design on learning: effect of learning rate. (A) Efficient learning. Cost function (network error) gradually descends (‘Gradient descent’) to achieve the optimal point (called local minimum) as a function of weight. (B) Learning rate is too high, so that the cost function overshoots the minimum and oscillates. This network design may not be trained effectively for this problem. (C) Gradient descent examining two variables on cost function simultaneously.

Figure 5.

Classifying complex data. (A) Transforming data to enable linear separation of non-linearly separable raw data. Raw non-linear data are transformed by mapping functions that may include time, frequency, or other operations. This projects them into higher-dimensional parameters space in which they are now linearly separable. One example is classifying patients with heart failure with preserved ejection fraction whose response to beta-blockers may vary due to obesity, atrial fibrillation, left ventricular hypertrophy, diabetes, or other factors. Data transformation to a higher-dimensional space now enables a simple partitioning process. (B) Bias–variance tradeoff. Model with high bias (straight line), when a straight line could not classify appropriately (here, between atrial fibrillation and normal sinus rhythm) in both training dataset (5.B.a) and testing dataset (5.B.b). This leads to prediction errors on other datasets (low variance − frequent errors). In contrast, model with low bias (i.e. due to overtraining) when data is fitted well in training set (5.B.c), but not in testing set (5.B.d), leading to reduced generalization (high variability due to difference between training and validation sets).

Finally, a major strength of DL is its ability to infer and classify data outside its original training set (i.e. generalization). To generalize well, DL strategies must avoid overfitting to the training data. Unlike statistical methods, for which strategies exist to avoid overfitting, in DL, this is often empirical and trial-and-error. The first and best approach is usually to increase power by adding data. Another common technique in DL is data augmentation, i.e. increasing the number of training samples using the same raw data. In image analysis, for example, it is common to rotate, invert, or skew an image if it does not distort data or change its output class (e.g. rotation would not alter hypokinesis on echocardiography or the location of a space-occupying lesion), but may be unacceptable in traditional biostatistics. Other strategies include cross-validation of training data, and transformation to reduce data complexity. In complex DL architectures (e.g. several hidden layers in a neural network, Figure 3), reducing the numbers of neurons or layers may be effective. The ultimate test of generalization is prediction in a new dataset.

Data preparation

Data preparation must be tailored to each specific DL architecture to optimize performance and requires feature selection and imputation of missing data.

Feature selection

This involves transforming raw data to features that reduce dimensionality of the data, but that still represent the relevant clinical or physiological question, and may reduce the risk of overfitting. In Figure 3, input data were digitized ECG voltage-time series, while in Figure 2B features were used instead such as QRS duration and the intrinsicoid deflection.²⁶ Since a key advantage of DL is to learn complex features, overly complex preprocessing of data into features may also impede performance. One general goal of DL may be summarized as letting the algorithm automatically perform feature selection for the user, instead of the user attempting to manually engineer features.

Missing data imputation

Similar to statistical tasks, the performance of DL can be highly sensitive to missing data. The cut-off to discard features where entries are missing remains debated. Decisions on how to treat missing data can be made by evaluating if the presence or absence of specific elements correlate with desired outcomes or predictors. Those data that are correlated are ‘non-ignorable’, those that are not correlated may be ‘ignorable’.²⁷ Ignorable missing data includes missing completely at random (no relationship to any variables) and missing at random (a systematic relationship between missing values and observed data). Identifying these types of missing data is a crucial before data imputation.²⁸ Several strategies exist for imputation. One approach to account for systematic data omission is to insert a label for those elements. For instance, since individuals too sick (e.g. with elevated serum creatinine) to receive a cardiac computed tomography (CT) may have worse outcome than those who do undergo cardiac CT, a label that indicates ‘too sick’ would likely still be prognostic for outcome in a model missing CT data in sick patients. Here, CT data is non-ignorable, but training can proceed with this label. In other cases, training can proceed with insertion of an imputed value, multiple imputations (i.e. MICE or missForest R packages), or expected values from the literature. One class of DL algorithms called autoencoders have been shown to produce best-in-class results for missing data imputation.²⁹

Training and expected results

Datasets should be separated into distinct partitions for training, testing, and validation. One area of future improvement in DL is in considering data containing repeated measures or correlations (time series, or where observations cluster by patient). However, this is an active area of research within the DL community.^30,31 More information on sampling by patient for correlated data (i.e. repeated measures) should also be included. The specific training, testing, and validation proportions used in DL often vary depending upon the task at hand, and in general selecting one strategy is more of an ‘art’ than ‘science’. 70/10/20 or 80/10/10 or 60/20/20 splits are common, but no standard methodologies exist to determine optimum proportions although each should come from similar data distributions to prevent mismatched proportions between sites. Training strategies are thus largely empirical with no standardized approaches.

Novel deep learning techniques

Anticipated advances in DL may mitigate some concerns of empiricism and lack of a theoretical framework. Capsule-based neural network,^32,33 meta-learning (learning how the network learns),^34–36 DeepMath (learn mathematical proofs),³⁷ or self-play (two agents learn by win and loss)³⁴ are promising innovations. Studies to optimize learning rate, network architectures, and activation functions may improve the learning efficiency of DL. There are efforts to adapt several non-neural network-based methods for DL (Supplementary material online, Table S2). Support vector machines, for instance, are effective for high-dimensional data^38,39 and could be useful in cardiovascular DL with diverse datasets.^40,41 Enriching the theoretical framework for DL may improve our ability to interpret their conclusions.

Deep learning applications in cardiovascular medicine

Table 2 summarizes early DL applications in cardiovascular medicine,^42–51 while Supplementary material online, Table S3 lists DL studies in other disciplines.

Imaging for ischaemic and structural heart disease

Deep learning has been used to classify images in many medical specialties,^52–54 recently extended to cardiac imaging (Table 2). By interpreting cardiac images rapidly and consistently, DL may circumvent clinical limitations of fatigue or distraction, variable inter- and intra-observer interpretation, and time-consuming interpretation of large datasets.

Zreik et al.⁵⁵ applied DL to automatically identify significant coronary artery stenosis in rest coronary CT angiograms in 166 patients. Compared with matched invasive fractional flow reserve (FFR) measurements, the network produced a c-statistic of 0.74 ± 0.02 with specificities of 77%, 71%, and 59% at sensitivities of 60%, 70%, and 80%, respectively, providing a possible alternative to invasive FFR.⁵⁵ Betancur et al.⁵⁶ applied DL to single-photon emission computed tomography myocardial perfusion imaging in 2619 consecutive patients at exercise (38%) or pharmacological stress. After 3.2 ± 0.6 years follow-up, DL better predicted major adverse cardiac events (MACE; 9.1% of patients overall) for DL using imaging with stress test data than imaging data alone (area under the receiver-operating characteristic curve: 0.81 vs. 0.78; P < 0.01), and both were superior to existing assessments. Motwani et al.⁵⁷ used DL on CT angiography in 10 300 patients with suspected coronary disease to improve prediction of 5 year all-cause mortality over existing CT or clinical indices as supported by others.⁵⁸ In a novel study applying DL to retinal fundus photographs trained on 284 335 patients, DL predicted MACE with c-statistic 0.70 in two validation populations (12 026 and 999 patients).⁵⁹ Several reports have used DL to define structural heart disease from echocardiography and MR imaging (Table 2). Deep learning can diagnose structural disease from limited echocardiographic views.⁴⁴ DeepVentricle,⁷¹ a DL application for cardiac MRI images, received FDA clearance for clinical use.⁷² Bai et al.,⁶² applied DL to automatically segment 93 500 labelled MR images in 4875 subjects from the UK Biobank with similar accuracy to experts for segmenting left and right ventricles on short-axis images and left and right atrium on long-axis images.

Deep learning for outcomes prediction in heart failure

Several studies have applied ML to predict outcomes in heart failure (HF).⁴¹,^73,74 Choi et al.⁷⁵ applied neural networks to detect new onset HF from electronic health records in 3884 patients who developed incident HF and 28 903 who did not, linking time-stamped events (disease diagnosis, medication and procedure orders). Networks provided a c-statistic for incident HF of 0.78 (12-month observation) and 0.88 (18-month observation), both significantly higher than the best baseline method. Medved et al.⁷⁶ compared the International Heart Transplantation Survival Algorithm developed using DL training, with the Index for Mortality Prediction After Cardiac Transplantation (IMPACT), in transplant recipients from 1997 to 2011 from the UNOS registry. In 27 705 patients, using those before 2009 for training and those from 2009 for validation, DL provided a c-statistic for 1-year survival of 0.654 for IHTSA, which reduced error by 1 compared with the IMPACT model (c-statistic 0.608). These results, while modest, show promise for DL beyond current clinical indices. Future studies may apply DL to multi-variable data (e.g. histopathology, echo, ECG, labs, multi-omics, wearable technology) to study HF outcomes. The recent BIOSTAT-CHF trial, a large registry for risk prediction for HF in 11 European countries, has multi-level data that could be used to reclassify HF patients.⁷⁷

Deep learning for arrhythmia detection and phenotyping

Several studies have used DL to diagnose AF from the ECG. Tison et al.⁸ trained a DL network on 9750 ambulatory smartwatch ECGs, then applied it to 12-lead ECGs. The network performed well in 51 recumbent patients before cardioversion (c-statistic 0.97 vs. 0.91 for current ECG algorithms), but less well in a cohort with ambulatory ECGs (c-statistic 0.72, sensitivity 67.7% and specificity 67.6%). A separate study used AI to diagnose AF from electrical ECG sensors in a smartphone case (Kardia), or a watch-strap which communicates via Bluetooth to a smartphone (Kardia Band).⁶⁰ In 100 patients with 169 simultaneous wearable and traditional ECGs, 57 recordings were uninterpretable. For interpretable ECGs, the device identified AF with a K coefficient of 0.77 (sensitivity 93%, specificity 84%) although physician interpretation improved results further.⁶⁰ Thus, while these data are promising, further advances in analytic algorithms and sensor technology are needed for automatic use. Emerging sensors beyond optical sensors (photoplesthysmography) in the iWatch⁸ include changes in facial reflectance to visible light,⁷⁸ bioimpedance in weighing scales⁷⁹ and others. The accuracy of each sensor needs validation since, in recent comparisons against gold standards, wearable sensors had acceptable accuracy for resting heart rate yet not for ambulatory exercise heart rates nor energy expenditure.⁸⁰ Current ESC guidelines provide a Class I recommendation for the opportunistic screening for silent AF in patients >65 years of age by pulse or ECG rhythm strip, based on evidence on cost effectiveness.⁸¹ Therefore, integrating DL into wearable technology for intermittent screening for silent AF may be cost effective by preventing sequelae such as stroke.

Finally, ML shows promise in identifying novel arrhythmia phenotypes, using unsupervised learning (since labelled data do not exist, by definition). For AF, Inohara et al.⁸² analysed the large clinical ORBIT-AF database to identify clusters labelled atherosclerotic-comorbid, tachy-brady device, low comorbidity, and younger behavioural disorder. The cluster approach did not improve CHADS2VASc, ORBIT, and ATRIA scores for endpoints of stroke or bleeding; however, there was a slight improvement in combination (c-statistics 0.67–0.72). In hypertrophic cardiomyopathy, Lyon et al.²⁶ identified four clusters of high risk of sudden cardiac arrest from ECG (primary T wave inversion) and echocardiographic (septal and apical hypertrophy) features. The clinical utility of novel machine learned phenotypes should be validated in independent populations compared with traditional clinical classification.

Challenges for, and limitations of, deep learning

To date, cardiovascular results of DL are promising but still modest, and several challenges must be overcome. First, and most importantly, DL is often criticized in the clinical context as a black box which cannot easily be explained. Interpretability may be enabled by capsule based networks, or strategies that systematically censor inputs to define those that most affect classification. Meta-analyses of several DL algorithms applied to the same data may increase confidence in results. A number of techniques may enable ‘model-agnostic’ metrics for interpretability of complex models.¹⁸ Marblestone et al.⁸³ hypothesized analogies between DL and human cognitive functioning, proposing that integrating heterogeneous ‘cost functions’ over time may simplify learning. Thus, speculatively, insights into human cognition may ultimately provide insights to interpret DL models.

Second, all ML may suffer from overfitting (Figure 5) if data is limited and/or algorithms complex. Indeed, in some clinical studies DL provided similar results to statistical models (e.g. logistic regression).⁸⁴ This may simply mean that different analyses are better suited to different types of data. Future studies may integrate DL with statistical classification.

Third, DL faces recent ethical criticisms if biased or poor-quality data lead to biased predictions or, worse, facilitate manipulation of results. Adversarial examples, cases in which slight modifications to input data cause a major change in DL classification, are a significant concern for DL with potentially serious medical sequelae.^85–87 Some methodologies have been proposed to prevent adversarial examples (i.e. reactive strategies), but they remain ineffective.⁸⁸

Fourth, DL studies must enable replication by other groups since differences in algorithms, initiating conditions, or parameter tuning may alter results.⁸⁹ One replication study,⁹⁰ for example, demonstrated different results from another⁹¹ using the same algorithms. Thus, a standardized approach to perform and validate AI-related clinical studies is needed. One initial step would be to require investigators to deposit their data and a link to the code for their DL model. Medicine lags behind computer science in this respect.

Fifth, DL in cardiovascular medicine have thus far compared c-statistics (area under the receiver-operating characteristic curve), which has several limitations. Problems are that the c-index is only a measure of discrimination, ignores calibration indices and has no single universally accepted c-statistic cut-off or range of acceptable c-statistics.^92,93 This and other limitations can be addressed by calibration: dividing underlying continuous variables representing the diagnostic into partitions (e.g. deciles) and testing diagnostic ability in each. It is important to use multiple metrics of accuracy since, for example, c-statistics discriminate true outcomes (e.g. high risk from low risk patients), but are insensitive to systematic errors and do not identify whether the model is anchored at the right level of absolute risk across the spectrum of observable risks. This should be applied to machine learned models, but has yet to be done. In addition, receiver-operating characteristic curve is an ordinal technique which assumes that the underlying biological process is monotonic, yet this is often not true. For instance, blood pressure at extreme high or low levels are of disproportionate importance. Machine learning has the advantage that it is not constrained by a monotonic assumption. Reporting a comparison between DL with traditional statistical results (CNN vs. logistic regression), the Brier score, the goodness of fit,⁹⁴ calibration plots,⁹⁵ standardized checklist/strobe diagrams of prediction models⁹⁶ or the decision-curve analysis⁹⁷ would be helpful.

Sixth, there is likely a positive publication bias in medical studies of ML and DL. However, since the number of such studies is currently limited, a meta-analysis of such studies may not be worthwhile, so that funnel plots or other indices of publication bias or heterogeneity (I²) would be difficult to quantify. As discussed in several of the specific examples, negative results are indeed discussed. These studies should be used to refine DL methods, which then need to be tested via external independent multicentre validation.

Seventh, DL and standard statistical methods may often be used for the same problems, although in addition to parallels between the methods, divergences exist. Both are influenced by aberrations in sample data, and can suffer from overfitting. This may be more predictable for statistical methods than for ML which relies on empirical validation. Cardiovascular problems which can be stated as a clear hypothesis may be equally addressed by traditional statistics or DL. Conversely, if simple hypotheses are less readily formulated due to complex interactions, DL may be advantageous. Thus, both techniques are complementary tools.

Finally, a theoretical framework to guide big-data and DL designs is urgently needed. Rather than ask simply if data quantity is sufficient for study, it is pertinent to ask if data quality and diversity are sufficient to span the parameter space necessary to address the question. Adding data dimensions increases the chance of alpha error, i.e. finding chance associations in traditional biostatistics, yet it may enrich training for DL. Are data reliable (‘garbage in, garbage out’)? In a large big-data arrhythmia study comparing genotype with phenotype in 2022 patients with long QT and Brugada syndromes,⁹⁸ variability in genetic testing compromised its results. A final consideration is how to organize data given that most data structures are generic and there are few which are standardized for cardiovascular medicines.

Training programmes in deep learning

Educational programmes should incorporate classes on DL, given its already ubiquitous presence. Such classes should cover the rationale, solutions, technical, and ethical challenges it poses in medicine. At the undergraduate and graduate level, such training may focus on its complementary role to biostatistics, and on detailed software programming and hardware aspects. In medical education, implementation of a broad AI curriculum is likely to enrich understanding of many conditions in cardiovascular medicine with heterogeneous aetiologies and/or phenotypes such as HFpEF, AF, and hypertension.^99,100 The medical curriculum should also discuss ethical and legal challenges, and their potential to shape medical practice. A significant barrier to implement DL more broadly is the need for at least some programming familiarity. This may be less problematic for newer generations of trainees. Deep learning is an excellent vehicle to foster interdisciplinary teams of engineers, physicians, businessmen, legal, and ethical teams.¹⁰⁰ It may be helpful to borrow engineering approaches such as ‘hackathons’, e.g. the Data Science Bowl, PhysioNet/Computing in Cardiology Challenges or Kaggle Competition. Scientists including biomedical trainees could compete at these or traditional medical conferences to analyse cardiovascular sample datasets (imaging, ECGs) using DL.

Funding opportunities for DL outside medicine are increasing, but funding from ESC/AHA/ACC/NIH are increasingly needed.¹⁰¹ Crowdfunding has been an alternative for DL funding outside medicine and, although rare thus far in cardiovascular research, the potential of crowdfunding for cardiovascular DL research is intriguing.¹⁰²

Guidelines must be developed to standardize broad applications of AI in medicine. This will require complex discussions between multiple stakeholders including regulatory agencies in Europe and Asia, the U.S. Food and Drug Administration (FDA) and others, patient-advocacy and privacy groups, professional societies in medicine and computer science, other healthcare organizations and technology companies.

Future direction of deep learning in cardiovascular medicine

Deep learning promises to better integrate medical data sources, address the heterogeneity in patient disease types, bridge the gap between omics research and bedside phenotypes^103,104 and ultimately enable personalized medicine. This may require advances in the science to overcome current limitations including a limited theoretical foundation for design and testing, limited interpretability, and strategies to resolve overfitting.⁶¹ Cardiovascular medicine, in particular, is well suited to benefit from smart analysis of continuous and massive data streams in this new era of wearable sensors, to integrate traditional health data with lifestyle indices (Take home figure). If data privacy and security concerns were satisfied, this integration would form the basis of a medical internet of things between medical devices and analytic systems. Seamless integration of diverse sources of data could enable continuous disease monitoring, risk stratification and early warnings of potential decompensation.

Take home figure.

Deep learning process flow for cardiovascular medicine. Deep learning has the ability to produce actionable clinical information from diverse datasets. Such data may span (i) comprehensive, traditional clinical data; (ii) non-traditional ‘real-world’ data such as near-continuous streams from wearable devices but also questionnaires or online forms. The deep learning process flow commences with designing the most appropriate model. Deep learning is usually implemented by deep neural networks with convolutional layers defined by specific parameters (e.g. max pooling, activation function, and learning rate). Several algorithms traditionally applied in machine learning (i.e. SVM, RF, KNN, RNN, AE, GAN) can be combined to address complex problems. Data is selected and pre-processed (curated), and missing elements are imputed. Training proceeds until the deep learning machine converges at acceptable accuracy. The deep learning machine is then ready to be applied to unseen test data.

Conclusion

Deep learning is a rapidly developing field in the computer sciences with great promise for cardiovascular medicine. Using DL for big data analysis may not only identify hidden information in complex, heterogeneous datasets, but also may bridge the gap between disease pathogenesis, genotypes, phenotypes to enable personalized medicine. However, to transform cardiovascular care, DL will have to address challenges in obtaining extensive labelled data, in improving interpretability and robustness, and in developing standardized approaches for validation and testing. Deep learning is one of the most exciting areas of innovation in cardiovascular medicine that holds the possibility to provide more efficient care with improvement in outcomes.

Funding

S.M.N. is supported, in part, by grants from the National Institutes of Health (NIH R01 HL 83359; K24 HL103800).

Conflict of interest: S.M.N. is author of patents owned by Stanford University and the University of California Regents, and is a consultant to Beyond Limits.ai. R.S.R. reports research grants to his institution from Akcea, Amgen, AstraZeneca, Medicines Company and Regneron; consulting from Akcea, Amgen, C5, CVS Caremark; non-speaker bureau lectures from Amgen and Kowa; and stock ownership in MediMergent. J.K.M. discloses the following relationships - medical advisory board: GE Healthcare, Arineta; equity interest: Cleerly. All other authors declared no conflict of interest.