Utility of machine learning algorithms in assessing patients with a systemic right ventricle

Abstract

Aims

To investigate the utility of novel deep learning (DL) algorithms in recognizing transposition of the great arteries (TGA) after atrial switch procedure or congenitally corrected TGA (ccTGA) based on routine transthoracic echocardiograms. In addition, the ability of DL algorithms for delineation and segmentation of the systemic ventricle was evaluated.

Methods and results

In total, 132 patients (92 TGA and atrial switch and 40 with ccTGA; 60% male, age 38.3 ± 12.1 years) and 67 normal controls (57% male, age 48.5 ± 17.9 years) with routine transthoracic examinations were included. Convolutional neural networks were trained to classify patients by underlying diagnosis and a U-Net design was used to automatically segment the systemic ventricle. Convolutional networks were build based on over 100 000 frames of an apical four-chamber or parasternal short-axis view to detect underlying diagnoses. The DL algorithm had an overall accuracy of 98.0% in detecting the correct diagnosis. The U-Net architecture model correctly identified the systemic ventricle in all individuals and achieved a high performance in segmenting the systemic right or left ventricle (Dice metric between 0.79 and 0.88 depending on diagnosis) when compared with human experts.

Conclusion

Our study demonstrates the potential of machine learning algorithms, trained on routine echocardiographic datasets to detect underlying diagnosis in complex congenital heart disease. Automated delineation of the ventricular area was also feasible. These methods may in future allow for the longitudinal, objective, and automated assessment of ventricular function.

Introduction

Patients with congenital heart disease represent a growing cohort with lifelong chronic heart disease. These patients require longitudinal follow-up at specialized centres.¹ Over time, especially patients with a systemic right ventricle (RV) are at risk of developing progressive RV dysfunction^2–4 and heart failure⁵ with associated morbidity and mortality.^6,7 As a consequence, early detection of myocardial dysfunction and objective assessment of RV function are paramount. Regular transthoracic echocardiographic investigations remain the main imaging modality for detecting complications and guiding therapy in this patient population.⁸ In addition, undiagnosed patients with congenitally corrected transposition of the great arteries (ccTGA) are occasionally identified as part of a routine echocardiographic investigation in adulthood.⁹ Even experienced echocardiographers without major exposure to adult congenital heart disease may occasionally struggle to make the correct diagnosis due the complex anatomy involved in patients with a systemic RV. Given the dramatic progress in the field of machine learning, especially involving deep learning (DL)/neuronal networks over the last few years,^10–13 we aimed to test the utility of this technology in complex adult patients with a systemic RV. Specifically, we hypothesized that DL networks would be able to discriminate between patients with transposition of the great arteries (TGA) after atrial switch operation, patients with ccTGA and normal controls. Expanding on this, we also aimed to develop methods for automatic recognition of the systemic ventricle and area segmentation, which may be useful for objective longitudinal assessment of ventricular function in the long-term follow-up of these patients.

Methods

We identified patients with a systemic RV who attended the Adult Congenital Heart Disease Programme, Royal Brompton Hospital, London and the Adult Congenital and Valvular Heart Disease Centre at the University of Muenster, Germany and had undergone a transthoracic echocardiogram between 2005 and 2018. In addition, subjects with structurally normal hearts were identified and served as normal controls. We excluded patients who lacked adequate echocardiographic sequences. Echocardiographic recordings, performed according to current recommendations,^14,15 of the apical four-chamber and parasternal short-axis view stored in DICOM format were retrieved. The cine loops were split into individual frames and converted to a bitmap format using a dedicated MatLab (Version R2018a) programme. Patient identifying information and metadata had been removed before analysis. Retrospective analysis of the de-identified echocardiographic recordings has been approved by the local ethics committee.

Model architecture and training for diagnostic classification

Image frames were categorized by diagnosis and the dataset was split into a training/validation (80%) and test group (20%). Frames from test group patients were not used for training to ensure that the external validity of the model—when exposed to new patients—could be quantified. The image resolution was sampled down to 150×150 pixels with 256-shaded greyscale pixel values. Image augmentation with random application of rotations (±10°), width and height shifts (10% and 5%, respectively) as well as shears and zoom (up to 10% and 5%, respectively) were applied to the frames at run-time. The underlying neuronal network is illustrated in Figure 1. It is based on a series of convolution layers (3×3 size), max pooling layers (2×), and fully connected layers at the end. In addition, drop-out layers were utilized to avoid overfitting. Rectified linear units were used as activation functions, except for the last layer being a softmax classifier layer.

Figure 1.

Architecture of the convolutional neural network used for image classification. Frames down-sampled to a resolution of 150×150 pixels were used as input data. ccTGA, congenitally corrected transposition of the great arteries; Conv, convolutional layer; FC, fully connected layer; TGA, transposition of the great arteries.

The model was trained over 40 epochs with RMSprop optimization and a batch size of 32. Hyperparameters were adjusted to ensure maximal accuracy while avoiding overfitting based on inspection of the convergence plots for training and validation accuracy/loss. Accuracy was assessed by quantifying the percentage of correctly classified frames.¹⁶ To illustrate learning representations, we extracted and displayed individual activation of intermediate convolutional layers. Networks were implemented based on Tensorflow using the keras package for R.¹⁷ Training and testing was performed on an Intel i7 platform with GPU support (Nvidia GX 1070). Analyses were performed using RStudio Version 1.1.456/R-package version 3.5.1. Training of the network described above required approximately 30–45 min.

Model architecture and training for image segmentation

Image segmentation was performed based on the U-Net architecture implemented in R/keras. The details of the U-Net have been described previously.¹⁸ In brief, it consists of convolutional layers with rectified linear and max pooling units forming a contracting path with the addition of a symmetrical, expanding path consisting of up-convolutional layers and concatenations of features from the contracting path. The model was previously specifically developed for biomedical image analysis. For full details see https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/. A random subsample of approximately 1200 ccTGA, TGA, and normal subject frames were all manually annotated (by G.P.D. and S.O.), providing a mask for the systemic ventricle. The mask delineated the endocardial border during all stages of the heart cycle in an apical four-chamber view. The dataset was split into a training/validation (80%) and test group (20%). Frames were down-sampled to a resolution of 128×128 pixels before analysis. As above, we ensured that no frames from the test group were used for training or validation of the model. Before being presented to the model, frames were subjected to image augmentation as described above and intensity of the image frames was normalized. It took about 180 min to train the network on the Nvidia GX 1070 setup. Method evaluation was performed by assessing the Dice metric as well as relative area difference calculated as the ratio of the area difference between predicted ventricular area and actual ventricular area divided by the actual area (ground-truth area). The Dice metric measures the overlap between automated and manual (ground truth) segmentation. The metric’s value ranges between 0 and 1, with 0 denoting no overlap and 1 corresponding to perfect agreement. A higher Dice value is indicative of better agreement.¹⁹ Relative area difference is conceptually similar to the metrics used for assessing the agreement between two imaging modalities. The two models (classification and segmentation) were run independently. Combining a classification model with an appropriate segmentation model is equally feasible; we refrained from doing so to reduce computational resource use.

Statistical analysis

Descriptive data are presented as mean ± standard deviation or median and interquartile range for continuous variables, while categorical variables are presented as number (%). Comparisons between groups were made using a two-tailed Student’s t-test or non-parametric methods depending on data distribution. Comparison between groups, a two-sided P-value of <0.05 was considered indicative of statistical significance.

Results

Overall 132 patients with a systemic RV (92 with TGA after atrial switch and 40 ccTGA) and 67 normal controls formed the basis of this study. Patients’ demographics and clinical information are presented in Table 1. After splitting the apical four-chamber cine loops, 73 425 TGA frames, 33 394 ccTGA frames, and 24 354 frames from normal controls were included in the analysis. In addition, 8892 ccTGA frames, 17 053 normal frames, and 37 272 d-TGA frames from a parasternal short-axis view were available.

Table 1.

Patient demographics

Parameter	Atrial switch TGA	ccTGA	Normals	P-value
N (% male)	92 (55)	40 (70)	67 (57)	0.27
Age (years)	35.4 ± 8.7	45.6 ± 15.9	48.5 ± 17.9	<0.001
NYHA baseline	I: 75% II: 23% III: 2%	I: 45% II: 36% III: 18%	I: 100%	<0.001
Syst. EF (%)	51.2 ± 10.0%	47.5 ± 18.0%	64.0 ± 6.2%	<0.001
Subpulm. EF (%)	59.5 ± 10.8%	55.8 ± 12.4%		0.66
Severe TR	2%	9%		0.08
Pre-existing PM (%)	26%	34%		0.56

Ejection fraction was measured on magnetic resonance imaging in patients with a systemic right ventricle; ejection fraction in the normal controls is based on echocardiographic assessment (Simpson method).

EF, ejection fraction; NYHA, New York Heart Association; PM, pacemaker; Subpulm, subpulmonary ventricle; Syst., systemic ventricle; TR, tricuspid regurgitation.

Classification into ccTGA, TGA, or normal cardiac anatomy

The model for the four-chamber apical view achieved an accuracy of 95.5% in the training set and 94.4% in the test set (based entirely on frames from new patients, not used for model training) on an on frame basis. The accuracy to detect the correct diagnosis was 86.2% for individual frames of ccTGA subjects, 91.6% for frames of normal hearts, and 97.5% for frames of TGA patients. When combing all cine loop frames of one particular patient, 100% of the TGA, 100% of the normal hearts, and 93.8% of the ccTGA hearts were correctly identified.

For the parasternal short-axis two-chamber view, the model achieved an accuracy in the training set of 96.7% but only 76.8% in the test set. Especially, categorizing ccTGA frames correctly proved to be challenging (accuracy 49.2% in the test set), while detecting normal hearts and TGA subjects had better accuracy rates (85.6% and 86.8%, respectively) in the test set.

When combining apical four-chamber and parasternal short-axis views the model achieved an accuracy of 98.0% in the test set. On an on frame basis, it correctly classified 97.0% of ccTGA frames, 94.0% of normal frames, and 98.2% of TGA frames. Requiring a class probability of at least 90%, 75.7% of ccTGA, 83.8% of normal heart frames 94.8% of TGA frames were correctly identified. When combining all frames from one cine loop, the model correctly categorized diagnosis in all patients from the test set.

Visualization of the network activation

To illustrate model stimulation for different patient cohorts, we investigated the activation of intermediate convolutional layers for ccTGA, normal, and TGA echocardiographic frames, respectively. Figure 2 illustrates the activation pattern for the last 2D convolutional layer of the DL model, showing the dissimilar layer response depending on underlying diagnosis. Especially, the systemic ventricle was found to display prominently on the layer activation map.

Figure 2.

Activation maps of the intermediate network layers, illustrating the predominant activation of the convolutional network based on underlying diagnosis especially in the area of the systemic ventricle. Orange, yellow, and white indicate increasing local activation levels.

Ventricular segmentation

Models for ventricular segmentation were trained individually for ccTGA, TGA, and normal hearts based on a training set of approximately 1000 random frames each (consisting of a pair of original frame and manually produced endocardial mask) for the systemic ventricle as illustrated in Figure 3. The model achieved a good recognition rate, and in fact, the systemic ventricle was correctly identified in all frames from the test set, irrespective of diagnosis. When assessing accuracy of area detection, the best results were obtained in normal hearts (Dice score 0.881 ± 0.062), followed by TGA (Dice score 0.867 ± 0.067), and ccTGA hearts (Dice score 0.794 ± 0.083). The mean relative area difference between predicted and actual mask was −2.25% (±8.4%) for normal hearts, −5.87% (±7.00%) for TGA hearts, and 1.88% (±17.20%) for ccTGA hearts, respectively (Figure 4).

Figure 3.

Illustration of the training process of the U-Net model based on pairs of raw image and manually provided ground-truth mask. The model is trained to provide a segmentation mask based on this information.

Figure 4.

Representative examples of systemic ventricular segmentation in subjects with a structurally normal heart, TGA, and ccTGA in the external test cohort (upper panel). The red line represents the manually provided ground-truth line, the green line is the automatically generated segmentation line by the deep learning algorithm. The lower panel illustrates the area variability between automatic segmentation and manually provided ground-truth mask. The values indicate mean ± standard deviation.

Comparison to human interobserver variability

Two experienced echocardiographers (G.P.D. and S.O.) each performed ventricular segmentation independently in a blinded fashion on a random subsample of 50 ccTGA, TGA, and normal hearts. When applying the same metrics as for the automatic detection algorithm to these two datasets, the following results were obtained: Dice scores were highest for normal hearts (0.878 ± 0.037), followed by TGA (0.868 ± 0.040), and ccTGA (0.817 ± 0.049). The mean relative area difference was −6.62% (±7.75%) for normal hearts, −11.29% (±8.55%) for TGA hearts, and −15.71% (±13.76%) for ccTGA hearts, respectively.

Discussion

The present study demonstrates that automated detection of the underlying anatomy is feasible based on 2D transthoracic echocardiographic frames acquired as part of routine care in patients with a systemic RV. The algorithm correctly identified over 95% of frames in the test cohort of patients. When including all frames acquired in a single patient no individual was misclassified by the combined apical and short-axis model. The automatic segmentation model also identified the correct systemic ventricle in 100% of frames with a relative error comparable to human experts.

Progressive systemic right ventricular dysfunction is a recognized complication of TGA patients after atrial switch operation and ccTGA patients.^4,8,20,21 Therefore, lifelong regular echocardiographic assessment is recommended and part of routine follow-up in this population. While, alternative methods such as tricuspid annular plane systolic excursion, RV strain, or strain rate have been proposed,^22–26 traditional RV fractional area change remains an established and widely used echocardiographic marker of systolic RV function.²⁷ The current study illustrates how appropriate DL models can be trained to recognize the systemic ventricle even in patients with complex cardiac anatomy and delineate the endocardial border in this setting. We contend that this technology has the potential to assist non-congenital echocardiographers arriving to the correct diagnosis and may allow in the future for a semi-automated or fully automated longitudinal assessment of RV size and function. Recently, Madani et al.¹⁶ have described a DL framework for automatic view classification of routine echocardiograms based on convolutional networks. This methodology could well be combined with our network to automatically split the echocardiographic recordings into individual views and subsequently use the apical four-chamber and parasternal two-chamber short-axis view for diagnostic classification and area quantification. Recently, networks assessing the quality of echocardiographic investigations have been described and could also be incorporated in the general workflow, thus avoiding the quantification of inadequate sequences.²⁸ As clinicians are often faced with the question of longitudinal change in systemic RV function, these tools could significantly enhance workflow and promise important efficiency gains in the follow-up of congenital heart patients. Furthermore, recent research highlighted the scientific value of longitudinal data assessment in patients with a systemic RV. This suggests that such information should be incorporated in future risk stratification models, potentially yielding superior prognostic power compared with conventional cross sectional analyses.²⁹

The present study is—to our knowledge—the first to assess the utility of DL algorithms in the imaging of complex congenital heart disease. While we focused specifically on patients with systemic RV, the methods presented here could easily be extended to other complex congenital cohorts such as Ebstein anomaly or univentricular hearts. Within a multicentre approach with appropriate funding, comprehensive DL models for various underlying diagnoses could be established covering different imaging modalities across the entire spectrum of congenital heart disease and age groups. Applying the methods described here to cardiac magnetic resonance imaging (MRI) is a logical next step and should yield superior detection rates and segmentation accuracy thanks to the better image quality of the latter. In fact, recently Bai et al.¹⁹ have described the results of automatic image segmentation using fully convolutional network models applied to cardiac MRI cine-images from normal controls. This technology is very similar to the U-Net utilized in the current study. Although, neuronal networks are commonly regarded as black boxes, our study illustrates that by interrogating the activation response of intermediate layers to various samples presented, important clues on the relevant features for recognition can be obtained. Interestingly, we found that our model activates especially on the myocardial mass of the systemic ventricle, an appropriate choice based on the underlying anatomic differences.

Patients with a systemic RV present with heterogenous anatomy and are inherently difficult to assess echocardiographically.¹⁵ Therefore, we contend that demonstrating the performance of DL algorithms in this challenging population using echocardiographic frames of varying quality highlights the potential of this technique for other groups of patients with congenital heart disease and structural cardiac defects. We can only speculate why model performance was consistently lower in ccTGA patients compared with the TGA cohort. This may be related to the fact that the anatomic variability is greater in ccTGA compared with other forms of congenital heart disease and especially the short axis in ccTGA hearts can be variable in appearance, bearing similarities to normal anatomy as illustrated in the Supplementary data online, Appendix, especially if image quality is suboptimal.

Limitations

Our study was based on retrospective imaging data from two tertiary care centres for adult congenital heart disease. Therefore, we cannot exclude the possibility that the echocardiographic image quality may be superior to that obtainable in the community. In addition, the performance of the model may be worse when applied to a different patient population. The model, therefore, requires external validation. However, the study covered a period of >10 years and image quality varied between patients and across the study period. The current study used single 2D frames for identification of underlying diagnosis and ventricular segmentation. With increasing experience and more powerful computer technology, 3D image analysis and networks such as 3D convolutional networks, working directly with cine loops should be addressed in future studies and may further improve classification results. The published segmenting performance results—based on MRI frames in normal subjects¹⁹—are slightly superior to the current results. However, the echocardiographic image quality available to us was limited compared with cardiac magnetic resonance data and Dice scores as well as the area difference obtained in the current study were comparable to those obtained when assessing interobserver difference between human investigators employing our echocardiographic data. Further studies are required to assess the performance of the model when applied to different congenital heart defects not used for model training. This reinforces the need for external validation based on a large sample of congenital echocardiographic frames spanning the entire spectrum of congenital heart disease. For practical use, however, it is possible to set the model a probability threshold. This would require a certain prediction probability to be reached before making a certain diagnosis, thus, avoiding that samples with an unknown diagnosis are merely assigned to the category with the highest probability.

Conclusions

The current study demonstrates for the first time the utility of machine learning algorithms, trained on routine echocardiographic datasets, to detect underlying diagnosis in patients with complex congenital heart disease. Automatic segmentation of the ventricular area is also feasible based on appropriate DL algorithms. These methods may allow for longitudinal, objective, and automated assessment of ventricular function in future and guide management.

Funding

This study was supported by a research grant from the EMAH Stiftung Karla Voellm, Krefeld, Germany. The Adult Congenital Heart Centre and Centre for Pulmonary Hypertension, Royal Brompton Hospital, London, UK have received support from Actelion, UK; Pfizer, UK; GSK, UK; the British Heart Foundation; and the NIHR Cardiovascular and Respiratory Biomedical Research Units.

Conflict of interest: none declared.