Deep Learning–Enabled Assessment of Left Heart Structure and Function Predicts Cardiovascular Outcomes

Abstract

BACKGROUND

Deep learning interpretation of echocardiographic images may facilitate automated assessment of cardiac structure and function.

OBJECTIVES

We developed a deep learning model to interpret echocardiograms and examined the association of deep learning–derived echocardiographic measures with incident outcomes.

METHODS

We trained and validated a 3-dimensional convolutional neural network model for echocardiographic view classification and quantification of left atrial dimension, left ventricular wall thickness, chamber diameter, and ejection fraction. The training sample comprised 64,028 echocardiograms (n = 27,135) from a retrospective multi-institutional ambulatory cardiology electronic health record sample. Validation was performed in a separate longitudinal primary care sample and an external health care system data set. Cox models evaluated the association of model-derived left heart measures with incident outcomes.

RESULTS

Deep learning discriminated echocardiographic views (area under the receiver operating curve >0.97 for parasternal long axis, apical 4-chamber, and apical 2-chamber views vs human expert annotation) and quantified standard left heart measures (R² range = 0.53 to 0.91 vs study report values). Model performance was similar in 2 external validation samples. Model-derived left heart measures predicted incident heart failure, atrial fibrillation, myocardial infarction, and death. A 1-SD lower model-left ventricular ejection fraction was associated with 43% greater risk of heart failure (HR: 1.43; 95% CI: 1.23-1.66) and 17% greater risk of death (HR: 1.17; 95% CI: 1.06-1.30). Similar results were observed for other model-derived left heart measures.

CONCLUSIONS

Deep learning echocardiographic interpretation accurately quantified standard measures of left heart structure and function, which in turn were associated with future clinical outcomes. Deep learning may enable automated echocardiogram interpretation and disease prediction at scale. (J Am Coll Cardiol 2023;82:1936-1948) © 2023 by the American College of Cardiology Foundation.

Echocardiography is the most widely available noninvasive imaging modality for the assessment of cardiac structure and function.¹ It enables the diagnosis of a wide range of cardiovascular disease (CVD) conditions and guides the management of patients with existing CVD. Echocardiographic measures have been demonstrated to predict clinical outcomes in both healthy community-based samples and among patients with CVD.^2,3 Although rapid image acquisition, relative low cost, and lack of ionizing radiation make echocardiography a preferred noninvasive modality, the acquisition and interpretation of echocardiograms requires significant time, resources, and expertise.^4,5 Moreover, assessment of cardiac structure and function from echocardiography is limited by substantial inter- and intraobserver variability.⁶ Automated deep learning approaches have the potential to enable reproducible echocardiogram capture and interpretation at scale.

Previous studies have demonstrated the feasibility of deep learning interpretation of echocardiograms. This prior work has focused on view classification, measurement of cardiac structure and function, detection of specific myocardial diseases, and prediction of systemic cardiovascular risk factors.^7-11 These investigations have not examined incident outcomes, highlighting an important unmet need, as the clinical application of deep learning echocardiographic interpretation has been limited in part due to uncertain clinical relevance.

We developed Dimensional Reconstruction of Imaging Data (DROID), a deep learning echocardiogram interpretation model, to automate standard measurements of left atrial (LA) and left ventricular (LV) structure and function and evaluated the association of these measurements with incident CV outcomes. We leveraged 2 longitudinal multi-institutional electronic health record (EHR)-based cohorts of >500,000 ambulatory patients with echocardiographic studies and linked EHR data for the robust assessment of clinical outcomes.

METHODS

We have followed the PRIME (Proposed Requirements for Cardiovascular Imaging-Related Machine Learning Evaluation) checklist for consistent reporting for cardiovascular imaging-related machine learning investigations (Supplemental Methods).

STUDY COHORTS.

Training sample.

The Enterprise Warehouse of Cardiology (EWOC) is a retrospective community-based ambulatory cardiology sample of 99,253 adults aged 18 years or older receiving longitudinal cardiology care within Massachusetts General Brigham (MGB). We included participants in the EWOC cohort who had at least 1 transthoracic echocardiogram (TTE) performed at the Massachusetts General Hospital (MGH).

External validation samples.

Wevalidated model performance in 2 external samples. 1) The MGB Community Care Cohort Project (C3PO) is an EHR cohort of 523,445 individuals receiving longitudinal primary care within MGB. We included participants enrolled in the C3PO cohort with at least 1 TTE performed at MGH within 3 years before the start of followup. Individuals who belonged to both EWOC and C3PO were included in the C3PO cohort only. C3PO and EWOC study protocols were approved by the MGB Institutional Review Board and adhere to the principles of the Declaration of Helsinki. Cohort construction is detailed further in the Supplemental Methods.¹² 2) The EchoNet-Dynamic and EchoNet-left ventricular hypertrophy (LVH) data sets are publicly available research data sets. EchoNet-Dynamic comprises 10,030 apical-4 chamber (A4C) echocardiography videos and EchoNet-LVH comprises 12,000 parasternal long axis (PLAX) echocardiography videos from individuals who underwent TTE imaging between 2016 and 2018 as part of clinical care at Stanford University Hospital.⁸ Use of the EchoNet data sets complied with the Stanford University School of Medicine Research Use Agreement and Terms of Use.

ECHOCARDIOGRAM DATA SET AND PREPROCESSING.

Individuals included in EWOC and C3PO who had undergone TTE were identified and cross-referenced using the MGH Echocardiography Laboratory Database (Echo-Lab), a well-curated data set of all clinically indicated echocardiograms performed at MGH from 1980 to 2020. The EchoLab data set for the study period of 2001 to 2020 comprises raw echocardiographic images, clinical data, and physician reports for a total of 234,137 echocardiograms representing 91,680 patients. A custom data processing pipeline was implemented to remove identifying information, unintended human labels, and information beyond the scanning sector (Figure 1A).

FIGURE 1. Dimensional Reconstruction of Imaging Data Deep Learning Workflow.

(A) The workflow sequence for encoding each echocardiographic movie. Clinical echocardiograms are resized into standardized clips of 16 × 224 × 224 pixels. Batches of clips are then input to a 3-dimensional (3D) convolutional neural network (CNN). The weights of the network are progressively optimized to minimize mean square error between model predictions and gold standard EchoLab measurements. (B) The model refinement workflow. The 640-element encodings of up to 40 movies captured from the same echocardiogram study are collected from the embedding layer of the 3D CNN. Encodings are then input into a multi-instance attention model that generates study-level predictions that account for the weighted contributions from the different encodings. LA AP = left atrial anteroposterior dimension; LV = left ventricular; LVEF = left ventricular ejection fraction.

MODEL TRAINING, DEVELOPMENT, AND VALIDATION.

For model development, we randomly split the TTEs performed on EWOC cohort patients into independent training (39,658 echocardiograms/n = 12,895), test (9,848 echocardiograms/n = 3,215), and validation (internal validation: 5,301 echocardiograms/n = 1,777) sets. Multiple TTE studies were used per patient during model training. We validated models in a separate set of patients from the C3PO cohort who were not used for model training (C3PO external validation, 9,256 echocardiograms/patients) and in an independent sample from Stanford University Hospital (EchoNet-Dynamic, 10,030 echocardiograms/patients). A single TTE study was used per patient in each of the external validation data sets (Figure 2).

FIGURE 2. Consolidated Standards of Reporting Trials Diagram.

Selection of training, validation, and incident disease samples. C3PO = Community Care Cohort Project; CVD = cardiovascular disease; EWOC = Enterprise Warehouse of Cardiology; MGH = Massachusetts General Hospital.

CLASSIFICATION OF ECHOCARDIOGRAPHIC IMAGES (VIEW, DOPPLER, QUALITY, AND AXIS).

We developed a model to perform 4 classification tasks: 1) detection of image type (2-dimensional B mode vs Doppler vs 3-dimensional echocardiographic images); 2) assessment of image quality (good vs poor quality); 3) determination of axis (on- vs off-axis images); and 4) view classification. Further details are outlined in the Supplemental Methods.

PREDICTING STANDARD MEASURES OF LEFT HEART STRUCTURE AND FUNCTION.

We tested several model architectures to predict standard measures of left heart structure and function from TTE videos. LV and LA measures of interest included left ventricular ejection fraction (LVEF), left ventricular end diastolic dimension (LVEDD), left ventricular end systolic dimension (LVESD), interventricular septal wall thickness (IVS), posterior wall thickness (PWT), and left atrial anteroposterior dimension (LA AP). Further details of model development are summarized in the Supplemental Methods, Supplemental Figure 1, and Supplemental Tables 1 to 4. The final model architecture, hereafter named DROID, comprises: 1) an encoder backbone based on the Movinet-A2 convolutional neural network to extract 640-element representations of echo videos via multitask training (Figure 1A)¹³; and 2) multi-instance attention heads that translate sequences of encoded representations from the same echocardiogram into study-based predictions (Figure 1B). The 2-part model architecture of DROID is a manual segmentation-free approach that mimics the clinical echocardiographer interpretation workflow by first encoding important information from all available video clips (via 3-dimensional convolutional neural networks), and then by using attention heads to incorporate information from the most relevant videos to predict the target measurements (Supplemental Figure 2). We trained 2 specific DROID models to predict standard LA and LV measures (DROID-LA and DROID-LV, respectively).

COMPARISON WITH EXISTING MODELS.

We evaluated the performance of EchoNet-Dynamic on the C3PO external validation cohort. Model application to the C3PO data set is detailed in the Supplemental Methods.

INCIDENT CARDIOVASCULAR OUTCOMES.

For our incident disease analysis, we included all individuals from the C3PO with an MGH echocardiogram performed within 3 years before start of follow-up. Participants with prevalent heart failure (HF) (n = 1,063), atrial fibrillation (AF) (n = 1,483), myocardial infarction (MI) (n = 703), missing clinical data (n = 4), and nonphysiologic EchoLab measures (n = 25) were excluded, yielding a final sample of N = 5,970 individuals.

Outcomes of interest included incident HF, AF, MI, and all-cause death. Outcome definitions are detailed in the Supplemental Methods and Supplemental Table 5.

STATISTICAL ANALYSIS.

The performance of classification tasks (image type, quality, axis, and view) was evaluated by comparing one-vs-rest area under the receiver operating characteristic curve (AUROC) and confusion matrices. The performance of the measurement prediction task by the DROID models was evaluated by calculating the mean absolute errors and R² and assessing agreement using Bland-Altman plots between the model-predicted and measured label from the EchoLab data set.

For disease association analyses, Kaplan-Meier curves were used to display cumulative disease incidence stratified by LVEF <50% vs ≥50% and presence vs absence of LV hypertrophy (LVH) (PWT and/or IVS >11 mm) and left atrial enlargement (LAE) (LA AP >40 mm), ascertained by DROID predictions. We examined the association of DROID-derived LA (LA AP) and LV measures (LVEF, LVEDD, LVESD, IVS, PWT) with incident outcomes using Cox proportional hazards models adjusted for age and sex, and then further adjusted for body mass index, diabetes mellitus, systolic blood pressure, and hypertension treatment. For incident HF, AF, and MI analyses, participants were censored either at date of death or date of last clinical encounter. For mortality analyses, participants were censored at time of death or at time of last clinical encounter (for individuals who survived). Inspection of Schoenfeld residuals did not suggest substantive deviations from proportional hazards in the Cox regressions. Separate models were performed for each outcome of interest. We also evaluated model discrimination by calculating c-indices. All tests performed were 2-sided and P values of <0.05 were considered significant. All analyses were performed using python version 3.6.9 with packages pandas version 1.1.5, lifelines version 0.26.4, and scikit-learn version 0.24.2.

RESULTS

DROID was trained, developed, and validated on 64,028 echocardiograms from 27,135 patients and externally validated on 9,248 patients and echocardiograms from the C3PO external validation set and 10,030 patients and echocardiograms from the EchoNet-Dynamic sample (Figure 1, Table 1).

TABLE 1.

Baseline Characteristics of Training, Test, and Validation Sets

	EWOC				C3PO
	Total Sample (N = 27,135)	Training (n = 12,895)	Test (n = 3,215)	Internal Validation (n = 1,777)	External Validation (n = 9,248)	Incident Disease Sample (n = 5,970)
No. of echocardiograms	64,028	39,659	9,833	5,288	9,248	5,970
Clinical characteristics
Age, y	61 ± 17	63 ± 17	63 ± 17	63 ± 17	56 ± 18	51 ± 18
Men	15,486 (57)	7,289 (62)	1,992 (63)	1,094 (62)	4,645 (50)	2559 (43)
Race/ethnicity
White	23,031 (85)	11,281 (87)	2,818 (87)	1,567 (88)	7365 (80)	4656 (78)
Black	1,018 (4)	297 (2)	71 (2)	41 (2)	609 (7)	413 (7)
Hispanic or Latino	544 (2)	223 (2)	54 (2)	28 (2)	239 (2)	175 (3)
Asian or Pacific Islander	953 (3)	362 (3)	82 (3)	47 (3)	462 (5)	335 (5)
Other	1,589 (6)	732 (6)	190 (6)	94 (5)	573 (6)	391 (7)
Body mass index, kg/m2	28.4 ± 6.2	28.3 ± 6.1	28.5 ± 6.2	28.3 ± 5.9	28.6 ± 6.9	28.2 ± 6.8
Systolic BP, mm Hg	126 ± 19	125 ± 19	125 ± 18	129 ± 20	129 ± 20	128 ± 19
Diastolic BP, mm Hg	73 ± 11	73 ± 11	73 ± 11	73 ± 11	75 ± 12	76 ± 11
HTN treatment	49,126 (77)	31,806 (81)	7,896 (81)	4,213 (81)	5,211 (57)	2,646 (44)
Diabetes	12,880 (20)	8,410 (21)	2,046 (21)	1,169 (22)	1,255 (14)	599 (10)
Echocardiographic measures
LV ejection fraction, %	60 ± 15	59 ± 16	59 ± 16	59 ± 16	65 ± 11	67 ± 8
LV end diastolic diameter, mm	48 ± 8	48 ± 8	48 ± 8	48 ± 8	46 ± 6	45 ± 6
LV end systolic diameter, mm	33 ± 9	34 ± 10	34 ± 10	33 ± 10	30 ± 7	29 ± 5
IVS wall thickness, mm	11 ± 3	11 ± 3	11 ± 3	11 ± 3	10 ± 2	10 ± 2
Posterior wall thickness, mm	10 ± 2	10 ± 2	10 ± 2	10 ± 2	10 ± 2	9 ± 2
LA dimension, mm	41 ± 7	41 ± 7	41 ± 7	41 ± 7	38 ± 7	36 ± 6

Values are n, n (%), or mean ± SD.

BP = blood pressure; C3PO = Community Care Cohort Project; EWOC = Enterprise Warehouse of Cardiology; HTN = hypertension; IVS = interventricular septal; LA = left atrial; LV = left ventricular.

PREDICTING STUDY TYPE, QUALITY, AND VIEWS.

Our deep learning models accurately distinguished between 2-dimensional B mode vs Doppler vs 3-dimensional echocardiographic images (AUROC 2-dimensional: 1.0, Doppler: 1.0, 3-dimensional 1.0) (Supplemental Figure 3). Deep learning also demonstrated good discrimination of good- vs poor-quality studies (AUROC 0.87) (Supplemental Figure 4) and classified on-axis vs off-axis images (AUROC 0.87) (Supplemental Figure 5). Finally, our deep learning model accurately discriminated each of the standard echocardiographic views (PLAX: AUROC 0.98, accuracy 0.96, A4C: AUROC 0.98, accuracy 0.96, apical 2-chamber: AUROC 0.98, accuracy 0.97, Other: AUROC 0.96, accuracy 0.90) (Figure 3, full classification of 15 standard echocardiographic views displayed in Supplemental Figure 6). Among a total of 4,734,140 videos, videos that were classified as Doppler (1,675,419 videos, 35.4%), 3-dimensional (18,194 videos, 0.4%), poor-quality (218,565 videos, 4.6%), and off-axis (111,276 videos, 2.4%) were not used for training.

FIGURE 3. Deep Learning Accurately Classifies Standard Echocardiographic Views.

(A) An area under the receiver operator characteristic curve (AUROC) curve for the classification of parasternal long axis (PLAX), apical 4-chamber (A4C), and apical 2-chamber (A2C) echocardiographic views in the Enterprise Warehouse of Cardiology (EWOC) internal validation set. (B) The confusion matrix displaying accurate and inaccurate view classifications within the EWOC internal validation set. Diagonal entries represent accurate classifications, and off-diagonal entries represent misclassifications.

PREDICTING STANDARD MEASURES OF LV STRUCTURE AND FUNCTION.

In the internal validation set, DROID-LA accurately predicted LA AP (mean absolute error [MAE] = 2.87 mm, R² = 0.74). DROID-LV also accurately predicted LV dimensions (MAE for LVEDD = 2.33 mm, LVESD = 2.19 mm; R² for LVEDD = 0.86, LVESD = 0.90), LV wall thickness, (MAE for IVS = 1.11 mm, PWT = 1.04 mm; R² for IVS = 0.62, PWT = 0.49), and LVEF (MAE = 4.39% points, R² = 0.86) (Supplemental Figure 7). Bland-Altman plots demonstrated conservative estimation errors for LVEF, IVS, and PWT (95% limits of agreement −10.9% to 11.2% for LVEF, −3.5 mm to 3.3 mm for IVS, and −3.4 mm to 3.3 mm for PWT). Bland-Altman plots did not suggest systematic bias for LV dimensions and LA AP. DROID-LV accurately predicted preserved vs reduced LVEF (AUROC = 0.99) (Supplemental Figure 8) and LVH (AUROC = 0.90) (Supplemental Figure 9). DROID-LA accurately predicted LAE (AUROC = 0.93) (Supplemental Figure 10).

MODEL PERFORMANCE IN EXTERNAL VALIDATION SETS.

DROID-LA and DROID-LV accurately predicted LA and LV linear measures (MAE for LA AP = 2.52 mm, LVEDD = 2.09 mm, LVESD = 2.04 mm, IVS = 0.99 mm, PWT = 0.93 mm, R² for LVEDD = 0.81, LVESD = 0.82, IVS = 0.71, PWT = 0.59, LA AP = 0.76) (Figure 4), and LVEF (MAE = 4.23% points, R² = 0.74) in the C3PO external validation set. Bland-Altman plots again demonstrated conservative estimation errors for LVEF, IVS, and PWT (95% limits of agreement for LVEF = −10.3% to 10.9%, IVS = −2.6 mm to 2.5 mm, and PWT = −2.5 mm to 2.3 mm) and no systematic bias for LV dimensions and LA AP. In external validation, DROID-LV predicted preserved vs reduced LVEF (AUROC = 0.99) (Supplemental Figure 8) and presence of LVH (AUROC = 0.92) (Supplemental Figure 9). DROID-LA accurately predicted LAE (AUROC = 0.93) (Supplemental Figure 10).

FIGURE 4. DROID Model Performance in the External Validation Sets.

Correlation plots comparing Dimensional Reconstruction of Imaging Data (DROID)-derived vs gold standard EchoLab left heart measures for the Community Care Cohort Project (C3PO) External Validation set (A) and EchoNet-Dynamic (C). Bland-Altman plots of agreement between DROID-derived vs gold standard EchoLab left heart measures for the C3PO external validation set (B) and EchoNet-Dynamic (D). For each plot, the x-axis depicts the mean of paired values for an individual, and the y-axis plots the difference (ie, DROID-derived minus EchoLab). The solid horizontal line depicts the overall mean difference, and the hashed lines depict the estimated 95% limits of agreement, with actual values indicated below each line. IVS = interventricular septal wall thickness; LA A/P = left atrial anterior-posterior dimension; LVEDD = left ventricular end diastolic dimension; LVEF = left ventricular ejection fraction; LVESD = left ventricular end systolic dimension; MAE = mean absolute error; PWT = posterior wall thickness.

Model performance was slightly lower in EchoNet-Dynamic and EchoNet-LVH but accurately predicted LV measures (MAE for LVEDD = 3.90 mm, LVESD = 3.30 mm, IVS = 1.56 mm, PWT = 1.24 mm, R² for LVEDD = 0.63, LVESD = 0.77, IVS = 0.32, PWT = 0.40, LVEF = 0.69) (Figure 4). Bland-Altman plots again showed conservative estimation errors for IVS and PWT (95% limits of agreement for IVS = −4.31 mm to 2.18 mm and PWT = −3.45 mm to 2.68 mm) without systematic bias for LV dimensions and LVEF.

DEEP LEARNING-DERIVED LEFT HEART MEASURES ASSOCIATE WITH INCIDENT CARDIOVASCULAR OUTCOMES.

For disease association analyses, we included a total of 5,970 individuals in C3PO (mean age 51 ± 18 years, 57% women, Table 1). Mean DROID-derived LA AP was 36 ± 6 mm, LVEF was 67% ± 8%, LVEDD 45 ± 6 mm, LVESD 29 ± 5 mm, IVS 10 ± 2 mm, and PWT 9 ± 2 mm. During a median follow-up time of 5 years, 438 participants developed AF, 168 developed HF, 243 developed MI, and 198 died.

Deep learning-derived measures were consistently associated with incident outcomes. Compared with patients with DROID-LVEF ≥50%, individuals with DROID-LVEF<50% were at greater risk of new-onset HF, AF, MI, and all-cause death (Figure 5). A 1-SD lower DROID-derived LVEF was associated with a 42% greater risk of HF (age- and sex-adjusted HR: 1.42; 95% CI: 1.32-1.56; P < 0.001), 19% greater risk of AF (HR: 1.19; 95% CI: 1.10-1.27; P < 0.001), 22% greater risk of MI (HR: 1.22; 95% CI: 1.11-1.33; P < 0.001), and 18% greater risk of death (HR: 1.18; 95% CI: 1.05-1.30; P = 0.003). DROID-LA AP, LVEDD, LVESD, IVS, and PWT also predicted incident CV outcomes (Figure 6, Supplemental Table 6). DROID-predicted LVH was associated with 59% greater risk of AF (HR: 1.59; 95% CI: 1.28-1.96), 99% greater risk of HF (HR: 1.99; 95% CI: 1.41-2.82), and 55% greater risk of MI (HR: 1.55; 95% CI: 1.17-2.06). DROID-predicted LAE was associated with greater risk of incident outcomes (AF: HR: 2.29; 95% CI: 1.84-2.85, HF: HR: 3.57; 95% CI: 2.56-4.99, MI: HR: 1.85; 95% CI: 1.38-2.49, and death: HR: 2.22; 95% CI: 1.61-3.08) (Supplemental Figure 11). Findings persisted after further adjustment of key clinical covariates (Figure 6, Supplemental Table 7).

Figure 5. Deep Learning-Derived LVEF Predicts Clinical Outcomes.

Depicted are KapLan-Meier cumulative incidence curves for incident heart failure (A), atrial fibrillation (B), myocardial infarction (C), and death (D) stratified by DROID-LVEF ≥ 50% (red Line) vs DROID-LVEF <50% (blue Line). Abbreviations as in Figure 4.

FIGURE 6. Association of DROID-Derived Left Heart Measures With Incident Disease.

Displayed are Forest plots depicting the age- and sex-adjusted (circles) and multivariable-adjusted (triangles) associations of DROID-derived left heart measures with incident heart failure (A), atrial fibrillation (B), myocardial infarction (C), and death (D). Follow-up begins at time of echocardiogram capture. HR = hazard ratio per 1-SD increase in continuous echocardiographic measure. Abbreviations as in Figure 4.

ASSOCIATION OF ECHOCARDIOGRAPHIC MEASURES WITH CARDIOVASCULAR OUTCOMES IS GREATER FOR DEEP LEARNING-DERIVED VS STUDY REPORT MEASURES.

The associations of LA and LV linear measures with incident cardiovascular outcomes were greater in magnitude for deep learning-derived echocardiographic measures compared with gold standard EchoLab study report values, with c-indices that were comparable or higher (Supplemental Tables 8 and 9). A 1-SD increase in DROID-derived LVEDD was associated with a 44% greater risk of incident HF (DROID-LVEDD: HR: 1.44; 95% CI: 1.22-1.69) compared with 39% greater risk of HF for a 1-SD increase in human annotated LVEDD (EchoLab-LVEDD: HR: 1.39; 95% CI: 1.19-1.64). The associations of DROID-LVEF and EchoLab-LVEF with incident outcomes were similar.

COMPARISON WITH EXISTING MODELS.

The application of EchoNet-Dynamic on the C3PO external validation cohort demonstrated good correlation with the gold standard EchoLab-LVEF measurement (r = 0.69), but overall poor performance (R² = 0.18, Supplemental Figure 12). The Bland-Altman plot showed conservative LVEF estimation errors with EchoNet-Dynamic (mean bias 4.79%; 95% CI: 4.61%-4.97%).

Notably, EchoNet-Dynamic-derived LVEF was consistently associated with incident outcomes, and the associations with incident events were again greater in magnitude when compared with gold standard report values. A 1-SD lower EchoNet-Dynamic LVEF was associated with a 46% greater risk of HF (age- and sex-adjusted HR: 1.46; 95% CI: 1.33-1.61), 17% greater risk of AF (HR: 1.17; 95% CI: 1.09-1.27), 22% greater risk of MI (HR: 1.22; 95% CI: 1.11-1.35), and 17% greater risk of death (HR: 1.18; 95% CI: 1.05-1.31) (Supplemental Table 10).

DISCUSSION

We developed a deep learning model (DROID) for the automated measurement of left heart structure and function from echocardiographic images and evaluated the association of these measurements with incident cardiovascular outcomes in a large multi-institutional EHR cohort of >500,000 individuals receiving longitudinal care with more than a decade of follow-up. Our model accurately classified views and quantified standard measures of LA dimension, LV wall thickness, chamber diameter, and ejection fraction as compared with gold standard clinical report values. DROID-derived LA and LV measures were in turn independently associated with important cardiovascular outcomes including HF, AF, MI, and all-cause death. Finally, the association between echocardiographic measures with incident outcomes was greater in magnitude for deep learning-derived echocardiographic measures compared with gold standard study report measures. Taken together, this novel approach to integrate automated interpretation of echocardiograms within an EHR-based cohort with well-curated clinical data and longitudinal outcomes highlights the direct clinical relevance of deep learning models for echocardiogram interpretation (Central Illustration).

CENTRAL ILLUSTRATION. Dimensional Reconstruction of Imaging Data Deep Learning Model.

We developed a 3-dimensional (3-D) convolutional neural network (CNN) deep learning model for echocardiographic interpretation to predict standard measures of left heart structure and function. Model-derived echocardiographic measures were associated with incidentcardiovascular outcomes. C3PO = Community Care Cohort Project; LVEF = left ventricular ejection fraction.

Foundational work has demonstrated feasibility of deep learning of echocardiographic images for view classification, measurement of cardiac structure and function, detection of specific myocardial diseases, and prediction of cardiovascular risk factors. Refinement of these deep learning models has enabled increasingly reliable and precise automated classification of standard views, segmentation of cardiac chambers, and quantification of cardiac structure and function.^7,8,14 For example, Zhang et al⁷ showed that a deep learning model (trained and validated on 277 and 8,666 echocardiograms, respectively) accurately classified the PLAX view with 96% accuracy and predicted standard cardiac structural measures including LV mass, LV diastolic volume, and LA volume with mean absolute deviations of 15% to 17%. Another model by Ouyang et al⁸ (trained and validated on 7,465 and 1,288 echocardiograms, respectively) accurately predicted LVEF (R² 0.81 and MAE 4.1% vs human annotation) and reliably predicted HF with reduced ejection fraction (AUROC 0.97). These previous models were trained on modest numbers of echocardiographic videos (1,000-7,000 echocardiograms) and relied heavily on cardiac segmentation to guide downstream tasks like estimation of standard measures of cardiac structure and function.^7,8,14

To address this current limitation, we leveraged the large size of our sample (>40,000 echocardiograms) to develop a deep learning model to accurately predict cardiac structure and function without the need for resource-intensive quality control steps such as manual segmentation. Moreover, our model uses a unique architecture that was designed to simulate the echocardiographic interpretation clinical workflow. Our model comprises a multitask encoder combined with a multi-instance attention head that enables the model to integrate information across numerous echocardiographic views (eg, A4C, apical 2-chamber, and PLAX) to provide the most accurate prediction of standard measures of cardiac structure and function in a single model without the need for separate models for individual predictions or manual selection of specific videos for segmentation. Our deep learning approach represents a significant advancement in deep learning echocardiographic interpretation, as fully automated end-to-end models may be translated to practice at scale with superior performance to existing models.

An underexplored application of deep learning echocardiographic interpretation is its role in disease prediction and prognostication. Standard echocardiographic measures of cardiac structure and function, including LVEF, dimensions, volumes, and mass, as well as LA volume, have all been shown to be associated with incident disease in healthy participants and clinical outcomes (including all-cause mortality, cardiovascular death, and risk of hospitalization) in disease states like HF.^15-18 To date, machine learning echocardiographic interpretation has been limited to identifying local cardiac structures, diagnosing specific conditions, differentiating causes of LVH, and predicting clinical characteristics, and has not been evaluated with respect to outcomes.^7,9,19 We now show that our deep learning model-derived LA and LV measures are strongly and consistently associated with incident outcomes, including HF, AF, MI, and all-cause death, highlighting the ability of deep learning echocardiographic models to identify clinically important disease.

The association of deep learning-derived measures of cardiac structure and function with cardiovascular outcomes was greater than the associations seen with gold standard human annotated study report measures. This finding may be explained by the greater reproducibility of deep learning over human echocardiographic measurements as demonstrated in previous studies.^14,20 Deep learning may also integrate latent information from the echocardiogram that may inform disease risk. For example, DROID-LA may incorporate information from other cardiac chambers to generate a prediction for LA size, which may provide additional predictive value for disease risk.

Finally, an important novelty of our study is the integration of deep learning echocardiographic interpretation into a well-curated EHR cohort with rigorously defined longitudinal outcomes. EHR data sets are powerful tools with the potential to enable large-scale discovery and risk prediction but are limited because of susceptibility to ascertainment bias and data missingness.²¹ C3PO was designed to closely mirror an epidemiologic study to overcome these limitations through sampling of only patients receiving longitudinal primary care in MGB and recovery of missing data via natural language processing.¹² Leveraging the rigorous and intentional design of the C3PO EHR cohort, we were able to show that deep learning-derived measures of left heart structure and function predict incident cardiovascular outcomes among individuals without existing CVD, consistent with previous epidemiologic studies.¹²

STUDY LIMITATIONS.

First, echocardiographic videos selected for training and validation of our deep learning model were obtained from a retrospective cohort of patients receiving tertiary subspecialty cardiovascular care, which may introduce bias in patient selection and confounding by indication, potentially limiting generalizability of the model to patient populations with different characteristics. However, our model performed well when externally validated in both a primary care population and an external research cohort. Second, our model was trained on videos that were obtained by sonographers from an academic medical center with significant expertise in image acquisition, and videos deemed to be of inadequate quality or off-axis were not included for training. Whether model performance is significantly diminished when applied to echocardiograms of variable quality is not known and requires further exploration. Finally, although C3PO was designed to limit bias and maximize generalizability, residual indication bias and missingness may persist and generalizability to diverse ethnic/racial populations is limited, as C3PO is a predominantly White population.

CONCLUSIONS

We developed a deep learning approach for automated echocardiographic interpretation applied to an EHR cohort of >500,000 individuals receiving longitudinal care across a multi-institutional health care network. Our fully automated end-to-end model accurately quantified standard measures of left heart structure and function, which in turn were associated with future clinical outcomes including HF, AF, MI, and all-cause death. Machine learning of echocardiograms has the potential to scale echocardiogram interpretation and enable disease prediction and further biologic discovery.

PERSPECTIVES.

COMPETENCY IN PATIENT CARE AND PROCEDURAL SKILLS:

A deep learning model applied to echocardiographic data can be used to interpret measures of left heart structure and function associated with cardiovascular outcomes including HF, AF, MI, and death.

TRANSLATIONAL OUTLOOK:

Large-scale integration of machine Learning echocardiographic interpretation appLied to data from EHRs may provide insights into currentLy unrecognized aspects of cardiac structure and function.

ABBREVIATIONS AND ACRONYMS

A4C

apical 4-chamber

AF

atrial fibrillation

AUROC

area under the receiver operating characteristic curve

C3PO

Community Care Cohort Project

CVD

cardiovascular disease

DROID

Dimensional Reconstruction of Imaging Data

EHR

electronic health record

EWOC

Enterprise Warehouse of Cardiology

HF

heart failure

IVS

interventricular septal wall thickness

LA

left atrial

LA AP

left atrial anteroposterior dimension

LAE

left atrial enlargement

LV

left ventricular

LVEDD

left ventricular end-diastolic dimension

LVESD

left ventricular end-systolic dimension

LVEF

left ventricular ejection fraction

LVH

left ventricular hypertrophy

MAE

mean absolute error

MGB

Massachusetts General Brigham

MGH

Massachusetts General Hospital

MI

myocardial infarction

PLAX

parasternal long axis

PWT

posterior wall thickness

TTE

transthoracic echocardiogram