Deep learning from atrioventricular plane displacement in patients with Takotsubo syndrome: lighting up the black-box

Abstract

Aims

The spatiotemporal deep convolutional neural network (DCNN) helps reduce echocardiographic readers’ erroneous ‘judgement calls’ on Takotsubo syndrome (TTS). The aim of this study was to improve the interpretability of the spatiotemporal DCNN to discover latent imaging features associated with causative TTS pathophysiology.

Methods and results

We applied gradient-weighted class activation mapping analysis to visualize an established spatiotemporal DCNN based on the echocardiographic videos to differentiate TTS (150 patients) from anterior wall ST-segment elevation myocardial infarction (STEMI, 150 patients). Forty-eight human expert readers interpreted the same echocardiographic videos and prioritized the regions of interest on myocardium for the differentiation. Based on visualization results, we completed optical flow measurement, myocardial strain, and Doppler/tissue Doppler echocardiography studies to investigate regional myocardial temporal dynamics and diastology. While human readers’ visualization predominantly focused on the apex of the heart in TTS patients, the DCNN temporal arm’s saliency visualization was attentive on the base of the heart, particularly at the atrioventricular (AV) plane. Compared with STEMI patients, TTS patients consistently showed weaker peak longitudinal displacement (in pixels) in the basal inferoseptal (systolic: 2.15 ± 1.41 vs. 3.10 ± 1.66, P < 0.001; diastolic: 2.36 ± 1.71 vs. 2.97 ± 1.69, P = 0.004) and basal anterolateral (systolic: 2.70 ± 1.96 vs. 3.44 ± 2.13, P = 0.003; diastolic: 2.73 ± 1.70 vs. 3.45 ± 2.20, P = 0.002) segments, and worse longitudinal myocardial strain in the basal inferoseptal (−8.5 ± 3.8% vs. −9.9 ± 4.1%, P = 0.013) and basal anterolateral (−8.6 ± 4.2% vs. −10.4 ± 4.1%, P = 0.006) segments. Meanwhile, TTS patients showed worse diastolic mechanics than STEMI patients (Eʹ/septal: 5.1 ± 1.2 cm/s vs. 6.3 ± 1.5 cm/s, P < 0.001; Sʹ/septal: 5.8 ± 1.3 cm/s vs. 6.8 ± 1.4 cm/s, P < 0.001; Eʹ/lateral: 6.0 ± 1.4 cm/s vs. 7.9 ± 1.6 cm/s, P < 0.001; Sʹ/lateral: 6.3 ± 1.4 cm/s vs. 7.3 ± 1.5 cm/s, P < 0.001; E/Eʹ: 15.5 ± 5.6 vs. 12.5 ± 3.5, P < 0.001).

Conclusion

The spatiotemporal DCNN saliency visualization helps identify the pattern of myocardial temporal dynamics and navigates the quantification of regional myocardial mechanics. Reduced AV plane displacement in TTS patients likely correlates with impaired diastolic mechanics.

Graphical Abstract

Graphical Abstract.

Introduction

It is difficult to differentiate Takotsubo syndrome (TTS) from acute myocardial infarction (AMI), particularly ST-segment elevation myocardial infarction (STEMI), based solely on clinical characteristics, biomarkers, or electrocardiograms (ECGs).^1,2 Because a substantial portion of TTS cases are actually triggered by bleeding disorders (particularly from the central nervous system), frontline clinicians often face a dilemma to use anticoagulation (for cardiac catheterization) or thrombolysis in those patients with ‘STEMI’ ECGs.^3,4 In real-world practice, definitive diagnosis of TTS often becomes chronophagous and retrospective.^2,5 Nonetheless, misdiagnosing TTS as AMI in real time can lead to inappropriate pharmacological or device-based treatment and acute complications.^6,7 Improving diagnostic accuracy of TTS would help real-time patient triage and streamline clinical care pathways and give added impetus to prospective therapeutic trial design.^2,5,6

Myocardial pathology can alter contractile and relaxing patterns during cardiac cycles. The information within or between consecutive static images from echocardiographic videos improves the perception of temporal features and applicability of deep learning (DL) neural models in recognizing latent myocardial function changes.^8–10 We recently showed that effective spatiotemporal modelling in deep convolutional neural networks (DCNNs) reduced echocardiography readers’ erroneous ‘judgement calls’ on TTS.^11,12 The DCNN is usually regarded as a ‘black-box’ method, due to lack of explanations about how final decisions are made. Improving interpretability of an established DCNN appraises model competency and helps understand the rationales that have contributed to its specific decision-making and correct predictions, to discover latent imaging features associated to previously unknown pathophysiology.¹³ In the present study, we applied saliency visualization on a spatiotemporal DCNN to navigate quantitative echocardiography evaluations on myocardial mechanics and identify the pattern of regional myocardial temporal dynamics, so as to investigate and validate causative TTS pathophysiology.

Methods

An overview of the study design is illustrated in Figure 1.

Figure 1.

Overall study design and workflow. TTS echocardiography videos were presented to 48 human readers and a trained spatiotemporal DCNN classifier to classify the videos as either TTS or STEMI. The human readers highlighted the key areas they used for differentiation while a heatmap was generated for the most important pixels by the DCNN for disease classification. The myocardial mechanics and diastology were subsequently assessed through optical flow, longitudinal strain, and Doppler/tissue Doppler studies. DCNN, deep convolutional neural network; TTS, Takotsubo syndrome; STEMI, ST-segment elevation myocardial infarction.

Clinical and imaging data

Detailed methods, including DCNN Model construction, data training, and validation have been described in our previous publication.¹¹ Briefly, we obtained clinical, laboratory, ECG, coronary angiograms (CAG), and echocardiographic data of studied patients at the University of Iowa (UI) and eight other medical centres in the USA.¹¹ The differentiation between anterior wall STEMI and TTS was proven by CAG, following the updated diagnostic criteria for STEMI¹⁴ and TTS.¹ ST-segment elevation myocardial infarction patients had CAG-proven culprit stenosis (>70%) of the left anterior descending artery (LAD) with ventricular dilation/wall motion abnormalities consistent with the myocardial territories supplied by culprit coronary arteries in left ventriculography. The transthoracic echocardiograms (TTEs) were performed within 24 h after onset of symptoms in STEMI and TTS patients. Patients were excluded if they had primary valvular disorders, significant pulmonary hypertension, atrial fibrillation, anomalous LAD origin, or normalized wall motion abnormality. Based on anatomic features, Gensini score, culprit artery location, and dominant/side-branch circulations, segments were divided into culprit or non-culprit artery-supplied areas in a standard 17-segment left ventricle (LV) model.¹⁵ The TTE was performed with the Philips Epiq 7 machine (Philips North America Corporation, Cambridge, MA) and 3.5 MHz phased-array transducer for standard comprehensive two-dimensional (2D), Doppler and speckle tracking echocardiography. The TTE was performed using standard techniques following the updated guidelines of the American Society of Echocardiography.¹⁵ All images were stored digitally for playback and offline analysis. The 2D greyscale images were acquired in the standard apical views, and the standard apical four-chamber ventricular focused view images and videos were used for subsequent studies. Pixel data from the picture archiving and communication systems were pre-processed into numeric arrays, and the data were stored at a resolution of 800 × 600 pixels. If necessary, they were rescaled through bilinear interpolation. The echo videos from a control group of patients with matched age and risk factors for coronary artery disease were also included. The control patients were initially referred for echocardiography for suspected acute coronary syndrome in the absence of prior history of structural heart disease, and subsequent workup ruled out cardiac valvular/function (systolic) abnormalities.¹⁵

Human data visualization

Based on Qualtrics® software, the echocardiographic video image surveys to classify TTS and STEMI were completed in our previous study.¹¹ In the present study, a total of 48 human readers used Qualtrics® to record what each reader marked the important myocardial regions that had influenced their decision-making. All human readers have identified three regions of interest (ROIs) among the LV segments in apical four-chamber views of echocardiographic videos that were perceived as important in their differentiation of TTS from STEMI. The prioritized ROIs from human readers were transformed to the percentages of readers’ choices, and a weighted average of all results equipped with an averaging filter was made to generate a human heatmap.

Data visualization with the gradient-weighted class activation mapping method

Feature extraction and visualization

The DCNN model consists of nine convolution layers each equipped with a 3 × 3 × 3 kernel, as well as four max-pooling layers each with a 2 × 2 × 2 kernel. The last convolution layer is flattened and connected with a decision layer comprised of two neurons for each class prediction. The final convolution layer has 256 kernels and each of them contain a high-level abstract feature for disease classification. To visualize the feature maps from the last convolution layer, we implemented the gradient-weighted class activation mapping (GradCAM) with positive gradients, which is a variation of GradCAM,¹⁶ that used the gradients of the classification score for TTS (resp., STEMI) backpropagation into the last convolutional layer to produce a coarse heatmap $L_{TTS}$ (resp., $L_{STEMI}$) highlighting the important regions in the image for predicting TTS (resp., STEMI). Details on generating the heatmap are available in the Supplementary material online, Method.

Optical flow

We implemented the Lucas–Kanade (LK) optical flow measurement algorithm for quantifying pixel displacements among the echocardiogram frames.^17–20 Details on the LK optical flow algorithm are available in the Supplementary material online, Method. Briefly, The LK method is a two-frame differential method for motion estimation when the motion in a local neighbourhood is similar. There was no abrupt change for ventricular motion in a small region. The speckle noises in echocardiograms might yet generate randomness during calculation of motion. We thus first selected a random number of pixels from the location of interest and use optical flow to calculate the frame-to-frame pixel displacements. Then the average displacement for all pixels in each frame was computed as the estimate of the motion for the specific location of interest to reduce the randomness. To determine the transition frame between the systolic and diastolic cycles, we divided the total cardiac cycle into three phases with an equal number of frames. Intuitively, at the end of the systolic cycle, the velocity was close to minimum. We selected the frame with least velocity in the middle phase as the transition frame. Any frame after the transition frame was considered in the diastolic cycle and the displacement was defined as a negative one. Finally, we obtained location-specific smooth velocity curve by using bicubic interpolation after suppressing the negligible displacements in the systolic and diastolic cycles. Details on measuring the frame-by-frame pixel displacement are available in the Supplementary material online, Method.

Speckle tracking echocardiography

The 2D greyscale images were acquired in the standard apical views. Only images with frame rates > 40 frames/s were selected for analysis. All images were stored digitally for subsequent offline analysis. Speckle-tracking analysis automatically tracked myocardial motion throughout the cardiac cycle to quickly generate regional myocardial strain curves. We adjusted the width of ventricular walls to include the entire myocardium. Built-in software automatically accepted only those segments with good tracking quality and rejected segments with poor tracking quality. Finally, automated function imaging based on speckle-tracking analysis integrated quantitative data of ventricular peak systolic longitudinal strain into a standard, 17-segment model with a ‘bulls-eye’ figure, helping to detect both regional and global ventricular dysfunction. The average magnitudinal and longitudinal motion of a whole cardiac cycle related to basal-inferoseptum, mid-inferoseptum, apical-inferoseptum, basal-anterolateral, mid-anterolateral, apical-anterolateral, and apex of the heart for TTS, STEMI, and control patients were computed. Two echocardiographic board-certified cardiologists blinded to patients’ clinical and biochemical data independently performed strain analyses. When there was a disagreement, a consensus was made by the third echocardiographic board-certified reader.

Tissue Doppler/Doppler studies

Early blood flow velocity across the mitral valve (E) and mitral annulus (both septal and lateral) diastolic myocardial motion velocity (Eʹ) were measured during early diastole with pulse wave Doppler and tissue Doppler study, respectively. Mitral annulus (both septal and lateral) systolic myocardial motion velocity (Sʹ) was measured during systole with tissue Doppler study. Tricuspid annular plane systolic excursion (TAPSE) was measured with 2D guided M-mode study following the updated guidelines of the American Society of Echocardiography.¹⁵

Statistical analysis

All statistical analysis was performed using the open source software Python 3.7.4 with the package Scipy. Statistical significance was defined as P value of <0.05. Quantitative data were expressed as mean ± standard deviation (SD), compared using unpaired Student’s t-test and the Mann–Whitney U test (SPSS Statistics, IBM, Armonk, NY). Otherwise, the median (interquartile range) was used and compared using the Mann–Whitney U test. Categorical data were presented as absolute values and percentages and compared using Fisher’s exact test.

Results

Baseline characteristics

The demographic and clinical data of control, STEMI, and TTS patients are summarized in Table 1.

Table 1.

Demographic and clinical characteristics

	Control	AMI (AS-STEMI)	TTS	AMI vs. control	TTS vs. control	AMI vs. TTS
Patient number (n)	150	150	150	t-Test	t-Test	t-Test
Age, years (SD)	62.3 (14.2)	61.3 (13.4)	63.2 (13.1)	P = 0.51	0.61	0.21
Male, n (%)	67 (44.7)	104 (69.3)	26 (17.3)	P < 0.01	<0.001	<0.001
Previous CAD n (%)	13 (8.7)	25 (16.7)	17 (11.3)	0.05	0.47	0.22
HLD, n (%)	70 (46.7)	73 (49.0)	51 (34.0)	0.8	0.08	0.05
Systolic BP (SD)	128.7 (21.6)	124.6 (24.3)	118.5 (21.5)	0.16	<0.001	0.02
Diastolic BP (SD)	74.1 (13.1)	79.0 (18.1)	73.5 (16.0)	0.01	0.73	0.007
Heart rate (SD)	81.4 (18.6)	89.7 (22.0)	99.0 (25.1)	0.001	<0.001	0.001
DM, n (%)	29 (19.3)	37 (24.7)	37 (24.7)	0.32	0.32	1
COPD, n (%)	10 (6.7)	8 (5.3)	28 (18.7)	0.64	0.03	<0.001
CKD, n (%)	17 (11.3)	18 (12.0)	24 (16.0)	0.87	0.27	0.35
Hyperthyroid, n (%)	3 (2.0)	1 (0.67)	4 (2.7)	0.32	0.71	0.18
Smoking, n (%)	76 (50.7)	101 (68.2)	95 (66.0)	0.05	0.09	0.81
ACEi/ARB, n (%)	54 (36.0)	48 (32.0)	48 (32.0)	0.55	0.55	1
BB, n (%)	52 (34.7)	43 (28.7)	55 (36.7)	0.36	0.77	0.23
Chest pain, n (%)	n/a	115 (76.7)	41 (27.3)	n/a	n/a	<0.001
Peak CTnT, ng/mL (SD)	n/a	5.72 (8.2)	0.60 (0.81)	n/a	n/a	<0.001

AMI, acute myocardial infarction; AS-STEMI, anterior wall ST-segment elevation myocardial infarction; TTS, Takotsubo syndrome; CAD, coronary artery disease; PCI, percutaneous coronary intervention; HLD, hyperlipidaemia; BP, blood pressure; DM, diabetes mellitus; COPD, chronic obstructive pulmonary disease; CKD, chronic kidney disease; ACEi, angiotensin-converting enzyme inhibitor; ARB, angiotensin receptor blocker; BB, beta blocker; CTnT, cardiac troponin; LVH, left ventricular hypertrophy; EF, ejection fraction; LA, left atrium.

Human and saliency visualization

A total of 48 board-certified human readers performed human-side classification and data visualization on the same echocardiography dataset developing DCNN. They included 30 cardiologists (8 interventional board-certified cardiologists and 22 National Board of Echocardiography board-certified general cardiologists), 11 senior the American Registry for Diagnostic Medical Sonography board-certified cardiology sonographers, and 7 frontline care (emergency and critical care) physicians with more than three years’ experience of POCUS training (Acknowledgements). We retrospectively analysed the imaging phenotype of each TTS case that human readers had missed in our previous study,¹¹ and classified diagnostic difficulties based on percentage of human readers who made erroneous diagnoses: we defined ‘difficult’ cases as those in which >70% of human readers made erroneous diagnoses, ‘moderately difficult’ cases as those in which 40% to 70% of human readers made erroneous diagnoses, and ‘easy’ cases as those in which <40% human readers made erroneous diagnoses. Compared to ‘easy’ and ‘moderately difficult’ cases, most ‘difficult’ cases showed atypical TTS imaging phenotype, with smaller ‘apical ballooning (akinesis or dyskinesis)’ size (likely in different TTS evolving stages). The improvement in diagnostic accuracy from DCNNs mainly occurred in those TTS patients with evolving/atypical imaging phenotypes: DCNNs helped improve the diagnostic accuracy in 70% of ‘difficult’ TTS cases, in 19% of ‘moderate difficult’ cases, and in 0% of ‘easy’ cases (see Supplementary material online, Figure S1). Of note, while human readers’ visualization predominantly focused on the apex of the heart, the DCNN temporal arm’s saliency visualization was attentive on the base of the heart, with both typical and atypical imaging phenotypes. Compared with relatively extensive activation in patients with typical imaging phenotypes, the activation in patients with evolving/atypical imaging phenotypes appeared to be more active on the base of the heart, the AV plane (Figure 2).

Figure 2.

Discrepant human and spatiotemporal DCNN visualization. The gradient-weighted class activation mapping (GradCAM) interpretability method are used to unfold the activations of the network layers in a DCNN. In the heatmap, a brighter point indicates that the corresponding pixel in the input image plays a more important role in class prediction. The colour range of each of the heatmaps is from dark blue to dark red, where dark blue marks the least important and dark red marks the most important pixels for model prediction. (A) This is a heatmap of the prioritized regions of interest from human readers used to differentiate TTS and anterior wall STEMI among left ventricular segments in the apical four-chamber view echocardiography (the percentages of readers’ choices are depicted using the digital colour scale in panel (D). (B) This is a heatmap of accumulated DCNN classification on echocardiographic videos of patients with typical TTS imaging phenotypes (DCCNs outperformed human readers at the least in diagnostic accuracy¹¹). (C) This is a heatmap of accumulated DCNN classification on echocardiographic videos of patients with atypical (evolving) TTS imaging phenotypes (DCCNs outperformed human readers at the most in diagnostic accuracy¹¹). (D) Normalized colourmap scale used for heatmaps (A, B, C). 0.0: the lowest-priority regions; 1.0: the highest-priority regions. DCNN, deep convolutional neural network; STEMI, ST-segment elevation myocardial infarction; TTS, Takotsubo syndrome.

The assessment of ventricular temporal dynamics and myocardial mechanics

The results on the assessment of LV myocardial mechanics in control, TTS, and STEMI patients are shown in the Supplementary material online, Table S1. Optical flow assessment showed an apex to base gradient with regard to temporal dynamics of myocardium in both TTS and STEMI patients (Figure 3). The quantification assessments with optical flow showed consistently weaker peak longitudinal displacement (in pixels) in LV basal inferoseptal (systolic: 2.15 ± 1.41 vs. 3.10 ± 1.66, P < 0.001; diastolic: 2.36 ± 1.71 vs. 2.97 ± 1.69, P = 0.004) and basal anterolateral (systolic: 2.70 ± 1.96 vs. 3.44 ± 2.13, P = 0.003; diastolic: 2.73 ± 1.70 vs. 3.45 ± 2.20, P = 0.002) segments in TTS patients compared to STEMI patients (Figure 4 and see Supplementary material online, Table S1). Compared to STEMI patients, TTS patients also showed impaired longitudinal displacement in RV free wall basal segments (systolic: 3.86 ± 2.54 vs. 5.13 ± 3.43, P = 0.002; diastolic: 4.13 ± 3.29 vs. 5.32 ± 4.62, P = 0.027) and TAPSE (16.8 ± 3.3 mm vs. 18.9 ± 3.6 mm, P = 0.004) (see Supplementary material online, Table S1). Speckle tracking echocardiography showed worse longitudinal myocardial strain in basal inferoseptal (−8.5 ± 3.8% vs. −9.9 ± 4.1%, P = 0.013) and basal anterolateral (−8.6 ± 4.2% vs. −10.4 ± 4.1%, P = 0.006) segments in TTS patients compared to STEMI patients (Table 2). Meanwhile, tissue Doppler/Doppler studies showed worse diastolic mechanics/function (Eʹ/septal: 5.1 ± 1.2 cm/s vs. 6.3 ± 1.5 cm/s, P < 0.001; Sʹ/septal: 5.8 ± 1.3 cm/s vs. 6.8 ± 1.4 cm/s, P < 0.001; Eʹ/lateral: 6.0 ± 1.4 cm/s vs. 7.9 ± 1.6 cm/s, P < 0.001; Sʹ/lateral: 6.3 ± 1.4 cm/s vs. 7.3 ± 1.5 cm/s, P < 0.001; E/Eʹ: 15.5 ± 5.6 vs. 12.5 ± 3.5, P < 0.001) in TTS patients compared to STEMI patients (Table 2).

Figure 3.

Myocardial optical flow assessment in control, STEMI, and TTS patients. Red, blue, magenta, green, yellow, orange, and purple pixels mark the basal-inferoseptum, mid-inferoseptum, apical-inferoseptum, apex, apical-anterolateral, mid-anterolateral, and basal-anterolateral sections, respectively, of an apical four-chamber view in an echocardiogram. (A) Motion trajectory for a systolic cycle. The lines corresponding to each colour mark the motion trajectory for each pixel. (B) Motion trajectory for a diastolic cycle. (C) Motion trajectory for a full cardiac (systolic + diastolic) cycle. STEMI, ST-segment elevation myocardial infarction; TTS, Takotsubo syndrome.

Figure 4.

Impaired AV plane displacement and diastolic mechanics in TTS patients. Optical flow measurement: X-axis: frame change during a cardiac cycle; Y-axis: pixel displacement during a cardiac cycle. (A) Compared with STEMI patients, TTS patients showed significantly impaired amplitudinal displacement in LV basal inferoseptal (systolic: 2.72 ± 1.43 vs. 3.72 ± 1.92, P < 0.001; diastolic: 2.93 ± 1.85 vs. 3.49 ± 1.86, P = 0.15) and basal anterolateral (systolic: 4.23 ± 2.32 vs. 5.17 ± 2.55, P = 0.002; diastolic: 4.52 ± 2.69 vs. 5.08 ± 2.69, P = 0.091). (B) Compared with STEMI patients, TTS patients showed significantly impaired longitudinal displacement in LV basal inferoseptal (systolic: 2.15 ± 1.41 vs. 3.10 ± 1.66, P < 0.001; diastolic: 2.36 ± 1.71 vs. 2.97 ± 1.69, P = 0.004) and basal anterolateral (systolic: 2.70 ± 1.96 vs. 3.44 ± 2.13, P = 0.003; diastolic: 2.73 ± 1.70 vs. 3.45 ± 2.20, P = 0.002). AV, atrioventricular; LV, left ventricle; STEMI, ST-segment elevation myocardial infarction; TTS, Takotsubo syndrome.

Table 2.

The assessment of AV plane displacement and diastolic mechanics

	Control	AMI (AS-STEMI)	TTS	AMI vs. control	TTS vs. control	TTS vs. AMI
Longitudinal strain
Apex	−18.0 ± 4.0	−7.0 ± 3.3	−6.3 ± 3.0	P < 0.001	P < 0.001	P = 0.228
Apical septal segment	−19.2 ± 4.0	−7.2 ± 3.6	−6.6 ± 4.6	P < 0.001	P < 0.001	P = 0.401
Apical lateral segment	−16.9 ± 4.6	−8.8 ± 3.9	−7.7 ± 4.3	P < 0.001	P < 0.001	P = 0.098
Mid-inferoseptal segment	−18.6 ± 3.2	−8.9 ± 3.3	−8.0 ± 3.8	P < 0.001	P < 0.001	P = 0.115
Mid-anterolateral segment	−16.5 ± 3.8	−9.8 ± 3.9	−9.0 ± 3.4	P < 0.001	P < 0.001	P = 0.125
Basal inferoseptal segment (%)	−16.0 ± 3.2	−9.9 ± 3.4	−8.5 ± 3.8	P < 0.001	P < 0.001	P = 0.013
Basal anterolateral segment (%)	−17.1 ± 3.7	−10.4 ± 4.1	−8.6 ± 4.2	P < 0.001	P < 0.001	P = 0.006
TAPSE	22.2 ± 3.9	18.9 ± 3.6	16.8 ± 3.3	P < 0.001	P < 0.001	P = 0.004
Tissue Doppler/Doppler
Eʹ (septal, cm/s)	8.5 ± 3.5	6.3 ± 1.5	5.1 ± 1.2	P < 0.001	<0.001	P < 0.001
Sʹ (septal, cm/s)	8.5 ± 2.3	6.8 ± 1.3	5.8 ± 1.3	P < 0.001	<0.001	P < 0.001
Eʹ (lateral, cm/s)	10.5 ± 3.5	7.9 ± 1.6	6.0 ± 1.4	P < 0.001	<0.001	P < 0.001
Sʹ (lateral, cm/s)	9.7 ± 2.8	7.3 ± 1.5	6.3 ± 1.4	P = 0.77	<0.001	P < 0.001
E (cm/s)	83.5 ± 24.4	77.2 ± 15.3	75.4 ± 14.1	P = 0.016	P = 0.002	P = 0.349
E/Eʹ	8.7 ± 4.1	12.5 ± 3.5	15.5 ± 5.6	P < 0.001	P < 0.001	P < 0.001

Control: 130 patients; STEMI: 129 patients; TTS: 121 patients.

TTS, Takotsubo syndrome; AS-STEMI, anterior wall ST-segment elevation myocardial infarction; TAPSE, tricuspid annular plane systolic excursion.

Discussion

We investigated a new application of explainable DL in extracting latent spatiotemporal imaging features in echocardiography videos that correlate with underlying pathophysiology. Deep convolutional neural network saliency visualization characterized the regional myocardial temporal dynamics and navigated quantitative studies of myocardial mechanics in TTS patients. Impaired AV plane displacement likely correlates with TTS associated diastolic dysfunction.

In a multi-tasking human mind, the visual cortex may not be able to process all the diverse information that impinges on vision at the same time. The human biological visual system tends to ignore ‘irrelevant’ visual stimuli and focus on the most ‘important’ regions to quickly parse complex scenes in real time.²¹ Although an echocardiographic video holds comprehensive spatiotemporal information that allows wide-ranging examinations of structure and function of cardiac chambers, human brains are wired for a bias towards selective myocardial areas or features to identify pathology—‘the eye sees what it expects to see.’²² The present study shows the potential of spatiotemporal DL neural models to help human readers avoid ‘inattentional blindness’ and navigate parametric imaging assessment to extract inherent spatiotemporal imaging features that correlate with underlying pathophysiology.²³

The ventricular contractile patterns can change at different time points due to the dynamic and evolving nature of TTS. Imaging diagnosis is often challenging based on one-time echocardiographic study, which compromises readers’ confidence and accuracy, particularly in diagnosing TTS with evolving imaging phenotypes. We previously demonstrated that DL neural networks helped reduce erroneous human ‘judgement calls’ in distinguishing TTS from anterior wall STEMI based on echocardiographic videos.^11,12 Further analysis showed that the improvement in diagnostic accuracy from the DCNNs mainly occurs in those TTS patients with evolving (therefore atypical) imaging phenotypes. These results provided the evidence that integrating DCNNs into echocardiography interpretation could improve readers’ diagnostic accuracy to support urgently needed triage and frontline management decisions during cardiovascular emergencies.

In the present study, we used GradCAM, a technique for making DCNN-based prediction models more transparent by visualizing the regions of input that are ‘important’ for predictions from these models—that is, to determine which parts of supporting features are more important than the others among the raw imaging dataset.²⁴ Meanwhile, we requested 48 human expert readers interpreted the same echocardiography dataset and prioritized the important myocardial regions of interest to differentiate TTS and anterior wall STEMI. Interestingly, while humans’ basic knowledge on this ‘apical ballooning syndrome’ might subconsciously draw much of their attention to the apex of the heart, the trained spatiotemporal DCNNs focused predominantly on the base of the hearts for differentiation, particularly in those patients with evolving/atypical TTS imaging phenotypes (Figure 2). Subsequent myocardial function assessments with optical flow (on raw images) and clinical speckle tracking echocardiography both showed worse AV plane displacement in patients with TTS compared to patients with anterior wall STEMI. Although apical LV dysfunction is a well-known crucial imaging feature in both TTS and anterior wall STEMI patients, the former shows worse temporal dynamics in the base of the heart. This finding challenges the traditional eyeball appearance of ‘basal hypercontractility’ in TTS patients (possibly related to the contrasting markedly akinetic middle and apical segments with a ‘ballooning’ appearance from human visualization). The present study demonstrates the ability of spatiotemporal DCNNs to track and compute coherent changes rapidly throughout a cumulative evaluation of the continuous and multi-dimensional movements of different parts of the heart. This robust and meticulous analytic process allows the identification of ‘truly’ distinctive myocardial mechanics caused by different disorders that are ‘invisible’ to naked human eyes.

Due to a paucity of automated resources for processing raw imaging data and lack of consistent reporting of data quality measures, large-scale and standardized training imaging databases are often unavailable for rare cardiovascular disorders, such as TTS.^25,26 As a consequence, echocardiography readers often must rely on instinct or limited personal experience to make urgently needed diagnosis and management decisions in clinical practice. Inter-observer variations and errors may occur due to inherent subjectivity. A major benefit of direct image interpretation with DCNNs is that predictions can be generated automatically from raw imaging data alone to avoid subjectivity.^27,28 The DCNN data visualization can assist readers by pointing out the contributing regions of interest, extracting latent temporal imaging features not obvious to the human eye, and supporting the development of new quantitative parametric reference values.^29,30 This functionality has the potential to help ‘coach’ trainees with various backgrounds and experience in order to facilitate and standardize imaging training workflows and pathways, particularly for rare cardiovascular disorders in future.²⁶

The unique motion and displacement patterns of the base of the heart have been previously found to play vital roles in ventricular filling and emptying,³¹ and they were considered as the ‘overall’ functional expression of a series of haemodynamic events.³² Particularly, characteristic longitudinal AV plane movement correlated well with intrinsic diastolic mechanic performance of the heart.³³ In our TTS patients, while the real-time myocardial perfusion (see Supplementary material online, Figure S2) and metabolism³⁴ are well preserved in the base of the heart, the motions of basal segments still appeared to be significantly worse than patients with anterior wall STEMI (Figure 4 and see Supplementary material online, Figure S3). By integrating DL navigation into the traditional echocardiography assessment approaches, our study demonstrated global myocardial stiffness and advanced diastolic dysfunction in TTS patients, which may be important contributors to their adverse long-term outcomes.^35–37 These results suggest that extracting latent spatiotemporal features reflecting underlying cardiac structural/functional abnormalities have the potential to help TTS phenomapping and prognostication. Further classifying and quantifying pre-existing myocardial pathology in TTS patients help advance our knowledge about the pathophysiology and support the development of personalized treatments.

Limitations

The present study has several limitations: (1) due to retrospective nature of this study, we could not exclude patients with overlapped STEMI and TTS phenotypes. For example, the comparable contractility patterns with typical apical TTS phenotype (‘Takotsubo effect’) have been increasingly recognized in patients with anterior wall STEMI,^38–40 but they have not been excluded from our echocardiography training database. Including new training datasets with more delicate phenotyping of STEMI may help refine the prediction models and develop distinctive parametric values between these two disease groups; (2) in clinical practice, readers always interpret imaging results in the clinical context, and multitudes of imaging and other clinical features are evaluated simultaneously to derive meaningful clinical decisions. Meanwhile, the different haemodynamic features of TTS and STEMI patients can also become confounding variables to potentially affect imaging clarification results.^41,42 The present study only aimed to determine the possible added value of DL neural networks to assist human imaging readers for urgently needed disease triage and management decisions, and a hybrid model combining clinical features and physician interpretation informed by DL needs to be further evaluated in the future; and (3) a larger scale of STEMI and TTS patients with diversity in ethnicity, gender, and age are needed to build widely applicable prediction models to help pathophysiology investigation, and we are collaborating with other international TTS registries for this research purpose.

Conclusion

The spatiotemporal DCNN saliency visualization helps identify the pattern of myocardial temporal dynamics and navigates the quantification of regional myocardial mechanics. Reduced AV plane displacement in TTS patients likely correlates with impaired diastolic mechanics.

Supplementary material

Supplementary material is available at European Heart Journal – Digital Health.

Funding

This work was funded by an Obermann Center for Advanced Studies Interdisciplinary Research Grant and an Institute for Clinical and Translational Science Grant from The University of Iowa to K.L. and X.W.

Data availability

Additional information on methods, research results, extended data, and statements of data availability are available in our submitted supplemental methods and results, and are also available online (UI SharePoint/OneDrive shared folder) with all original echocardiographic images and videos.

Authors’ contributions

F.Z., N.I., A.C., K.L., and X.W. have full access to the study data. Concept and design: K.L., X.W. Literature review and data collection: F.Z., N.I., K.L., A.C., A.A., M.A. Deep learning model development and performance evaluation: F.Z., X.W. Data analysis and validation: F.Z., N.I., K.L., A.K., A.C. Statistical analyses: F.Z., Y.G.W., A.A. Manuscript drafting: K.L., X.W., F.Z., N.I., A.C., A.K., M.M., M.A. Supervision of study: K.L., X.W., M.M. All authors participated in critical revision and approval of the manuscript.