Echocardiography-based intelligent diagnosis and risk stratification management for tetralogy of Fallot

Summary

Background

Early and accurate detection of Tetralogy of Fallot (TOF), along with proper risk stratification management, is critical for improving patient survival and prognosis. We developed an end-to-end automated framework for TOF, aimed at supporting decisions from preoperative diagnosis through postoperative risk prediction.

Methods

A total of 1986 filtered participants (1018 healthy controls, 480 TOF mimics, and 488 patients with TOF) from four centres were recruited for the development and validation of DynaTOF, an integrated diagnostic and predictive system. The DynaTOF system comprises: (1) an echocardiographic view classification module built on ResNet-18; (2) a key diameter localisation and calculation module constructed with HRNet and a custom composite loss combining heatmap loss with geometric constraint loss; (3) a multimodal TOF diagnostic module that integrates a ResNet-LSTM-based video feature extractor for echocardiographic videos and a Transformer-based feature extractor for key diameters; (4) a time-aware postoperative prediction module, implemented with a GNN (Graph Neural Network), which estimates postoperative abnormal score dynamics based on preoperative video data, key diameters, surgical type, and specific postoperative time; and (5) a risk-stratification module that employs a Random Forest classifier to differentiate between high- and low-risk patients using the predicted abnormal score series.

Findings

The view classification module achieved AUC values of 0.999, 0.999, 0.998, and 0.998 for classifying the Apical Four-Chamber (A4C), Apical Five-Chamber (A5C), Parasternal Short-Axis (PSAX), and Parasternal Long-Axis (PLAX) views, respectively. The key diameter localisation and calculation module demonstrated R² values of 0.98, 0.76, and 0.97 for the prediction of LVD (left ventricular diameter), RVD (right ventricular diameter), and MPAD (main pulmonary artery diameter). The multimodal diagnostic module exhibited excellent performance in identifying TOF, with an accuracy of 0.910 (95% CI 0.881–0.938), an AUC of 0.989 (95% CI 0.977–0.992), a precision of 0.893 (95% CI 0.860–0.927), and a recall of 0.892 (95% CI 0.856–0.927), surpassing all single-modality approaches. The time-aware prediction module showed a high correlation (R² = 0.852) between predicted and observed postoperative abnormal scores. Finally, the risk stratification module achieved an AUC of 0.904 for distinguishing between high-risk and low-risk patients.

Interpretation

DynaTOF enables efficient diagnosis of TOF and provides personalised abnormal dynamics after operation, facilitating early screening and longitudinal monitoring. This system holds promise for improving comprehensive clinical care for infants with TOF.

Funding

This work was supported by Shanghai Municipal Education Commission (No. 2024AIYB010), Fundamental Research Funds for the Central Universities (YG2025LC03), Shanghai Special Fund for Promoting High-Quality Industrial Development - Pilot Industry Innovation Development (AI Special Topic) Project (No. 2025-GZL-RGZN-02078), National Key Research and Development Program of China (2025YFC2511603), Shenzhen Medical Research Special Project clinical multi-center study (No. C2405001), the Science and Technology Commission of Shanghai Municipality (STCSM) (Grant No. 23JS1400700; 24JS2840200; 25JS2850100), the Innovative Research Team of High-Level Local Universities in Shanghai, and the Sanya Science and Technology Special Fund (No. 2022KJCX41).

Research in context.

Evidence before this study

Before undertaking this study, we surveyed the existing landscape of TOF diagnosis and management worldwide. We observed that most existing approaches focus on isolated tasks—such as preoperative diagnosis or the prediction of specific postoperative complications—while lacking a unified framework capable of addressing the full continuum of care. In addition, to the best of our knowledge, prior studies have not explored the preoperative prediction of dynamic postoperative trajectories for patients with TOF.

Added value of this study

This study introduces DynaTOF, a comprehensive echocardiography-based solution that integrates view classification, key diameter quantification, automated TOF diagnosis, time-aware postoperative prediction and risk stratification grounded in preoperative features. By leveraging a multimodal fusion strategy and dynamic time modelling, we effectively identify TOF and postoperative high-risk infants earlier, bridging critical gaps between preoperative diagnosis and postoperative monitoring in paediatric cardiology.

Implications of all the available evidence

Our findings demonstrate the feasibility and high accuracy of an AI-driven, full-cycle care system for patients with TOF. Implementing DynaTOF could further advance clinical practice by delivering automated and individualised solutions from initial diagnosis to postoperative follow-up, supporting improved cardiac care for neonates.

Introduction

Tetralogy of Fallot (TOF) occurs in approximately 3–6 per 10,000 live births, accounting for 5–10% of all congenital heart diseases, and represents the most common cyanotic congenital heart defect.^1,2 The condition is characterised by four classic anatomical abnormalities: right ventricular outflow tract obstruction (RVOTO), ventricular septal defect (VSD), overriding aorta, and right ventricular hypertrophy.^1,3,4 Accurate and timely diagnosis is essential, as it critically influences surgical planning, clinical management, and long-term outcomes.⁵ Although echocardiography serves as the primary imaging modality for diagnosis, its interpretation is operator-dependent and subject to substantial inter-observer variability.⁶ Automated diagnostic systems can enhance reproducibility, accuracy, and efficiency in TOF identification by reducing subjectivity and workload in image analysis.⁶

Furthermore, while surgical repair has markedly improved survival, lifelong management remains crucial due to the risk of late complications, including pulmonary regurgitation, right ventricular dysfunction, arrhythmias, reduced exercise capacity, and sudden cardiac death.⁷ Additionally, the postoperative clinical course and the pattern of late sequelae are strongly influenced by the type of initial surgical repair—particularly whether a transannular patch was used for right ventricular outflow tract reconstruction.⁸ Ongoing follow-up and risk stratification are necessary to optimise long-term survival and quality of life in this patient population.⁹

Recent advances in echocardiography-based artificial intelligence (AI) techniques have significantly advanced the automation of diagnosis and prognosis in cardiovascular diseases.10, 11, 12 However, existing AI research on TOF has primarily focused on isolated aspects of the clinical workflow—either on preoperative index prediction or postoperative management.13, 14, 15, 16, 17, 18 For instance, studies have explored automated image segmentation and quantification of key cardiac structures,^18,19 or the prediction of major adverse cardiovascular events (MACE) and mortality based on postoperative imaging and clinical features.^13,14 No existing work has proposed a complete automated system for TOF that spans from preoperative diagnosis to postoperative prediction.

Establishing a connection between preoperative and postoperative conditions holds significant value.^18,20 For example, predicting specific postoperative outcomes or adverse events based on preoperative imaging features such as echocardiography can enable early risk warnings.^20,21 Nevertheless, no studies have addressed this issue in the context of TOF. Moreover, existing methods lack the ability to dynamically model postoperative changes over time—that is, they are unable to predict patient-specific outcomes at multiple postoperative time points.

In this study, we propose DynaTOF, an echocardiography-based end-to-end system for patients with TOF that spans intelligent diagnosis and time-aware postoperative outcome prediction. This system includes several key modules: echocardiographic view classification and selection, inner diameter localisation and calculation, TOF diagnosis, time-aware postoperative prediction, and risk stratification. External test results demonstrate DynaTOF is capable of accurately diagnosing TOF, dynamically predicting a patient's postoperative abnormal scores at any given time point based on preoperative features, and providing reasonable risk stratification management. Developing such a comprehensive system paves the way for integrating AI into the clinical workflow of TOF. Furthermore, dynamic postoperative prediction enables personalised postoperative management, risk stratification, and early warning for patients with TOF.

Methods

Overview of DynaTOF

The core objective of the DynaTOF system is to provide end-to-end decision spanning from preoperative diagnosis to postoperative prediction based on echocardiographic data. Fig. 1 illustrates the overall construction workflow of DynaTOF.

Fig. 1.

Overall construction process of DynaTOF. a. Data acquisition and processing. The study included participants with healthy controls with no cardiac abnormalities, those mimicking TOF, and those diagnosed with TOF. Following cohort filtering (see Fig. 2), multi-view echocardiographic videos were obtained for each participant. A View Classification and Selection module was developed to automatically identify and extract four standard echocardiographic views: Apical 4-Chamber (A4C), Apical 5-Chamber (A5C), Parasternal Short Axis (PSAX), and Parasternal Long Axis (PLAX). Subsequently, clinicians annotated the left and right ventricular inner diameters at end-systole and end-diastole on the A4C view, and the main pulmonary artery diameter on the PSAX view. b. Core modules of DynaTOF. All A4C and PSAX views were processed through the Key Inner Diameter Localisation and Calculation module (see Fig. 3a), resulting in the extraction of five key diameters: LVDs (left ventricular diameter at end-systole), LVDd (left ventricular diameter at end-diastole), RVDs (right ventricular diameter at end-systole), RVDd (right ventricular diameter at end-diastole), and MPAD (main pulmonary artery diameter). These five key diameters from healthy controls, TOF mimics, and preoperative patients with TOF were used to construct the TOF Diagnosis module (see Fig. 3b). Additionally, echocardiographic videos of each view were processed through a Video Feature Extraction module, and the resulting features were also incorporated into the TOF Diagnosis module (see Fig. 3b). Furthermore, for patients with TOF, video features and the key diameters at both preoperative and postoperative time points, were used to develop the Time-aware Postoperative Prediction module (see Fig. 3d). The predicted postoperative outcomes at various time points are intended to support risk stratification for patients undergoing TOF repair.

First, we performed cohort filtering on the participants. The participants in this study were recruited from Shanghai Children's Medical Center (Hospital 1), Ruijin Hospital (Hospital 2), Xinhua Hospital (Hospital 3), and Xuzhou City Central Hospital (Hospital 4). Data were collected from January 2021 to April 2024. For patients who underwent TOF repair, postoperative follow-up echocardiography was performed from postoperative day 1–3 years. As shown in Fig. 2, the inclusion criteria for this study were: undergoing an echocardiography examination, being aged between 1 and 12 months, and either having no cardiac abnormalities confirmed by the examination or being diagnosed with TOF or other congenital anomalies that can anatomically and clinically mimic TOF (TOF mimics), such as isolated pulmonary stenosis, pulmonary atresia with VSD, and double-outlet right ventricle. 3516 healthy controls, 497 TOF mimics, and 537 participants with TOF were enrolled from Hospital 1, while 1263 healthy controls, 139 TOF mimics and 132 TOF participants were enrolled from Hospitals 2, 3, and 4. The exclusion criteria were as follows: echocardiographic video resolution lower than 256 × 256 pixels, the duration shorter than one cardiac cycle, or absence of any of the four views: Apical Four-Chamber (A4C), Apical Five-Chamber (A5C), Parasternal Short-Axis (PSAX), and Parasternal Long-Axis (PLAX). After applying these criteria, 811 healthy controls, 368 TOF mimics and 395 TOF participants from Hospital 1, and 207 healthy controls, 112 TOF mimics and 93 TOF participants from Hospitals 2, 3, and 4 were retained. Among the TOF participants, 175 cases in Hospital 1 and 49 cases in Hospital 2, 3, and 4, had undergone surgical TOF repair and had received at least one postoperative echocardiographic examination that included the four views. The data in Hospital 1 were used for internal training and validation, and the data in Hospital 2, 3, and 4 were used for external testing. The baseline characteristics of the filtered study cohort are summarised in Supplementary Table S2.

Fig. 2.

Cohort filtering process. Participants aged between 1 and 12 months who underwent echocardiographic examinations and were either confirmed to have no cardiac abnormalities or diagnosed with TOF or TOF mimics were included in the study. A total of 3516 healthy controls, 497 TOF mimics and 537 TOF participants were recruited from Hospital 1, while 1263 healthy controls, 139 TOF mimics and 132 TOF participants were recruited across Hospitals 2, 3, and 4. Participants were excluded if their echocardiographic videos had a resolution lower than 256 × 256 pixels, contained less than one complete cardiac cycle, or lacked any of the four standard views: A4C, A5C, PSAX, or PLAX. After screening, 811 healthy controls, 368 TOF mimics and 395 TOF participants from Hospital 1, and 207 healthy controls, 112 TOF mimics and 93 TOF participants from Hospitals 2, 3, and 4 were retained, with all four qualified echocardiographic views available. Among the TOF participants, 175 cases from Hospital 1 and 49 individuals from Hospitals 2, 3, and 4 had undergone surgical repair for TOF and had received at least one postoperative echocardiographic follow-up that included all four qualified views.

The echocardiographic videos of the filtered cohorts were first used to train the View Classification and Selection module, enabling the system to automatically recognise the four standard views (A4C, A5C, PSAX, PLAX). Subsequently, clinicians annotated the left and right ventricular internal diameters at end-diastole and end-systole on the A4C view, as well as the main pulmonary artery diameter on the PSAX view. These diameters were chosen based on clinical experience as simple yet effective indicators to facilitate rapid screening of potential patients with TOF. Therefore, these measures were employed in this study to further investigate and quantify their clinical utility.

Fig. 1b depicts the construction of the core modules of DynaTOF. All A4C and PSAX views were processed by the Key Inner Diameter Localisation and Calculation module (see Fig. 3a) to extract five key diameters: LVDs (left ventricular diameter at end-systole), LVDd (left ventricular diameter at end-diastole), RVDs (right ventricular diameter at end-systole), RVDd (right ventricular diameter at end-diastole), and MPAD (main pulmonary artery diameter). These diameters, obtained from healthy controls, patients with TOF mimics, and patients with preoperative TOF, were used to develop the TOF diagnosis module.

Fig. 3.

The architecture of DynaTOF's core modules. a. Key Inner Diameter Localisation and Calculation module. This module employs a multi-layer keypoint detection architecture based on HRNet²² to progressively localise the start and end points of LVDd through multi-scale feature representations. These keypoints are then converted into diameter values using a built-in inner diameter calculation block. The same process is applied to compute the values of LVDs, RVDs, RVDd, and MPAD. b. TOF Diagnosis module. This module consists of two branches: a video branch and a diameter branch. In the video branch, echocardiographic videos of the four standard views are first passed through the frame feature extraction block to obtain spatial features for each frame. These features are then processed by a Long Short-Term Memory (LSTM) network to capture and integrate temporal dependencies, resulting in a comprehensive representation for each view. These representations are subsequently used to predict whether a patient has TOF. In the diameter branch, the five inner diameter values are first transformed via a Multilayer Perceptron (MLP) to expand the feature representation. The expanded features are then passed through a Transformer network to model their interactions. These learnt features are also used for TOF diagnosis. Additionally, features extracted from both the video and diameter branches can be fused in a multimodal manner to jointly predict the presence of TOF. c. Graph Neural Network (GNN) Structure and Output. A proxy node is connected to each preoperative diameter feature, each preoperative extracted video feature, the surgical type feature, and the postoperative time feature. The GNN aggregates information from these neighbours to produce a context-aware representation for the proxy node. d. Time-aware Postoperative Prediction module. Preoperative echocardiographic videos of all views, five inner diameter values, surgical type, and postoperative time points are each processed through corresponding feature extraction modules. These features are then input into a GNN to perform deep interaction and integration. The GNN outputs an enriched representation, which is passed through an MLP to predict the time-specific postoperative abnormal score.

Additionally, echocardiographic videos capturing the four standard views from healthy controls, patients with TOF mimics, and patients with preoperative TOF were input into a video feature extraction module. The resulting representative features were subsequently incorporated into the TOF diagnosis module. The diagnosis of TOF can be performed using either the diameter measurements or video features alone, or by combining both modalities into a multimodal diagnostic approach (see Fig. 3b).

For patients with TOF, video features from all views and the five key diameters, both from pre- and postoperative stages, were used to develop the Time-aware Postoperative Prediction module. This module integrates preoperative video features, diameter measurements, surgical type and postoperative days to predict the abnormal score at specific postoperative time points (see Fig. 3d). The predicted postoperative abnormal score series serve to characterise postoperative risk dynamics and to facilitate risk-stratified management in patients after TOF repair.

View Classification and Selection

For the construction of the echocardiographic View Classification and Selection module, 1000 videos were selected from each of the four standard views—A4C, A5C, PSAX, and PLAX—from the internal dataset, along with 2000 videos from other views, such as the Apical Two-Chamber (A2C) and Apical Three-Chamber (A3C) views. For each video, three frames were randomly extracted, resulting in 3000 frames for each of the A4C, A5C, PSAX, and PLAX views, and 6000 frames from other views. All frames were resized to a uniform resolution of 256 × 256 pixels for the construction of a five-class view classification model. For each class, the dataset was randomly split into internal training and validation sets at a ratio of 4:1. Additionally, 600 frames were randomly selected from each of the four standard views in the external dataset for testing. ResNet18²³ was used as the view classification model and the following focal loss was used as the loss function:

$$\begin{array}{c}{\mathcal{L}}_{\mathrm{F}}=-\frac{1}{N}\sum _{i=1}^N\sum _{c=1}^C{\alpha }_c{y}_{i,c}{\left(1-p_{i,c}\right)}^{\gamma }\log \left(p_{i,c}\right)\end{array}$$ (1)

where N denotes the batchsize, C represents the total number of classes (C = 5), and α_c is the weight assigned to class c, determined by the inverse class frequency. y_i,c is the ground truth label for class c of the i-th sample (either 0 or 1), and p_i,c is the predicted probability that the i-th sample belongs to class c. The focussing parameter γ, set to 1.5 in this study, is used to reduce the relative loss for well-classified examples and thus focus learning on difficult samples. Training was stopped early when the performance on the validation dataset ceased to improve. The trained view classification model was subsequently used to categorise incoming echocardiographic videos and select those corresponding to the A4C, A5C, PSAX, and PLAX views for downstream tasks.

Inner Diameter Localisation and Calculation

To automate the measurement of five key cardiac diameters—LVDs, LVDd, RVDs, RVDd, and MPAD, we need to identify the start and end points of these diameters in the echocardiographic images. This transforms the task into a keypoint localisation problem. To address this, we developed a keypoint detection model based on the HRNet²² architecture. First, physician-annotated key frames were extracted from A4C and PSAX view videos as input data, with random affine transformations applied to enhance robustness against probe angle variations. To accommodate the anatomical differences and task-specific requirements of each view, we adopted an independent training strategy. Specifically, separate keypoint localisation models were trained for the start and end points of the LVD in the A4C view, the RVD in the A4C view, and the MPAD in the PSAX view. For internal training and validation, a total of 2400, 2400, and 1200 frames with annotations for LVD, RVD, and MPAD, respectively, were used. These were randomly divided into training and validation sets at a 3:1 ratio. For external testing, 600, 600, and 300 frames with corresponding LVD, RVD, and MPAD annotations were utilised, respectively.

The model architecture follows the multi-resolution parallel structure of HRNet, as illustrated in Fig. 3a. The input images were resized to 256 × 384 pixels and passed through an initial convolutional layer to extract low-level features, producing a feature map at 1/4 the input resolution. The network then progressively expands to include multi-scale branches in three sequential blocks, adding sub-branches at 1/8, 1/16, and 1/32 resolutions. Each sub-branch undergoes hierarchical downsampling via convolutions and incorporates stacked residual modules composed of dual convolutional layers and batch normalisation to progressively enhance feature representation. To fuse multi-scale semantic information, high-resolution branches inject local detail features into adjacent lower-resolution branches via convolutional layers, while the low-resolution branches are upsampled using bilinear interpolation and concatenated with their high-resolution counterparts. This bidirectional cross–scale interaction enables the model to retain fine-grained cardiac texture details while incorporating global structural priors of the ventricles. Finally, all branch features are upsampled to 1/4 of the input resolution and concatenated along the channel dimension. A convolutional layer followed by a Sigmoid activation function is applied to generate keypoint heatmaps with the same spatial resolution as the input. To extract keypoint coordinates from the heatmaps while leveraging global contextual information, we adopt the Soft-Argmax method, which computes a smooth weighted average of all spatial locations. This approach enhances localisation accuracy and mitigates issues arising from ambiguous or shifted peak responses in the heatmaps.

To optimise the localisation of the start and end points of the cardiac diameters, we designed a composite loss function that combines heatmap loss with geometric constraint loss, reflecting the anatomical properties of the diameters. First, we used the heatmap loss to evaluate the pixel-level discrepancy between the predicted and ground truth heatmaps, which is defined as follows:

$$\begin{array}{c}{\mathcal{L}}_{\text{heatmap}}=\frac{1}{BHW}\sum _{b=1}^B\sum _{i=1}^H\sum _{j=1}^W{\left({\mathrm{h}}_{ij}^{\text{pred}}-{\mathrm{h}}_{ij}^{\text{true}}\right)}^2\end{array}$$ (2)

where H and W denote the height and width of the input image, respectively. ${\mathrm{h}}_{ij}^{\text{true}}$ represents the pixel value at position (i,j) in the ground truth heatmap, and ${\mathrm{h}}_{ij}^{\text{pred}}$ denotes the corresponding value in the predicted heatmap. To further constrain the spatial distribution of the diameter keypoints, we introduce a geometric loss function that incorporates constraints on length, angle, and midpoint consistency. Specifically, the length loss quantifies the deviation in predicted diameter length using relative error, and is formulated as:

$$\begin{array}{c}{\mathcal{L}}_{\text{length}}=\frac{1}{B}\sum _{b=1}^B\left|\frac{l_b^{\text{pred}}-l_b^{\text{true}}}{l_b^{\text{true}}}\right|\end{array}$$ (3)

$$\begin{array}{c}{l}^{\text{pred}}=\sqrt{{\left(x_2^{\text{pred}}-x_1^{\text{pred}}\right)}^2+{\left(y_2^{\text{pred}}-y_1^{\text{pred}}\right)}^2}\end{array}$$ (4)

$$\begin{array}{c}{l}^{\text{true}}=\sqrt{{\left(x_2^{\text{true}}-x_1^{\text{true}}\right)}^2+{\left(y_2^{\text{true}}-y_1^{\text{true}}\right)}^2}\end{array}$$ (5)

where l^pred and l^true denote the predicted and ground truth lengths of the diameter, respectively. The coordinates (x₁,y₁) and (x₂,y₂) represent the start and end points of the diameter. By computing the relative error in length, this formulation helps mitigate the imbalance caused by large variations in diameter lengths across samples, and more accurately captures the deviation in predicted length. The angle loss measures the minimum angular deviation between the predicted and ground truth direction vectors, and is defined as:

$$\begin{array}{c}{\mathcal{L}}_{\text{angle}}=\frac{1}{B}\sum _{b=1}^B\left|\Delta {\theta }_b\right|\end{array}$$ (6)

$$\begin{array}{c}\Delta {\theta }_b=\arctan \left(\frac{\sin \left({\theta }_b^{\text{pred}}-{\theta }_b^{\text{true}}\right)}{\cos \left({\theta }_b^{\text{pred}}-{\theta }_b^{\text{true}}\right)}\right)\end{array}$$ (7)

$$\begin{array}{c}{\theta }^{\text{pred}}=\arctan \left(\frac{y_2^{\text{pred}}-y_1^{\text{pred}}}{x_2^{\text{pred}}-x_1^{\text{pred}}}\right)\end{array}$$ (8)

$$\begin{array}{c}{\theta }^{\text{true}}=\arctan \left(\frac{y_2^{\text{true}}-y_1^{\text{true}}}{x_2^{\text{true}}-x_1^{\text{true}}}\right)\end{array}$$ (9)

where θ^pred denotes the predicted diameter angle, θ^true denotes the ground truth diameter angle and Δθ represents the angular difference between the two angles. Δθ is regularised to lie within the interval [-π,π] to prevent errors caused by angular periodicity, thus directly reflecting the minimal rotational deviation.

The midpoint loss quantifies the overall positional deviation by computing the Euclidean distance between the normalised coordinates of the predicted and true midpoints:

$$\begin{array}{c}{m}^{\text{pred}}=\left(\frac{x_1^{\text{pred}}+x_2^{\text{pred}}}{2H},\frac{y_1^{\text{pred}}+y_2^{\text{pred}}}{2W}\right)\end{array}$$ (10)

$$\begin{array}{c}{m}^{\text{true}}=\left(\frac{x_1^{\text{true}}+x_2^{\text{true}}}{2H},\frac{y_1^{\text{true}}+y_2^{\text{true}}}{2W}\right)\end{array}$$ (11)

$$\begin{array}{c}{\mathcal{L}}_{\text{midpoint}}=\frac{1}{B}\sum _{b=1}^B\parallel m_b^{\text{pred}}-m_b^{\text{true}}{\parallel }_2\end{array}$$ (12)

By normalising the midpoint coordinates to the [0,1] range, scale sensitivity caused by differences in image resolution is eliminated, allowing the model to focus on relative positional consistency. The final loss is defined as a weighted sum:

$$\begin{array}{c}{\mathcal{L}}_{\text{total}}=k_1·{\mathcal{L}}_{\text{heatmap}}+k_2·{\mathcal{L}}_{\text{length}}+k_3·{\mathcal{L}}_{\text{angle}}+k_4·{\mathcal{L}}_{\text{midpoint}}\end{array}$$ (13)

where k₁, k₂, k₃ and k₄ are weighting coefficients used to balance the contributions of the different loss components.

The predicted diameter length, determined from the localised start and end points, is further converted into the true physical length by the built-in inner diameter calculation block, which incorporates the original image dimensions and the pixel spacing information provided in the metadata.

TOF Diagnosis

The diagnosis of TOF is formulated as a three-class classification task to distinguish among healthy controls, TOF mimics, and TOF cases. We utilised both video data and inner diameter measurements from all 811 healthy controls, 378 TOF mimics, and 395 TOF patients in the internal dataset for model training. The dataset was randomly stratified and split into internal training and validation sets at a 4:1 ratio. For testing, video data and inner diameter measurements from all 207 healthy controls, 112 TOF mimics, and 93 TOF patients in the external dataset were used. As illustrated in Fig. 3b, the TOF Diagnosis module consists of two branches: a video branch and a diameter branch. Each branch can independently perform TOF classification, or a multimodal fusion strategy can be employed to make a joint prediction.

In the video branch, all echocardiographic videos are first processed through uniform frame sampling and resizing to produce sub-videos consisting of 16 frames with a resolution of 256 × 256 pixels. To fully leverage the anatomical characteristics unique to each echocardiographic view, four independent ResNet18 models are used as the frame feature extraction block—one for each of the A4C, A5C, PSAX, and PLAX views. This design ensures that the specific diagnostic cues present in each view are effectively captured. The frame feature extraction block extracts a 256-dimensional feature vector for each frame. The frame-level features for each view are subsequently fed into a single-layer bidirectional LSTM to capture temporal dynamics across frames. The hidden state dimension of the LSTM is set to 128, and the outputs from both directions (forward and backward) are concatenated to form a 256-dimensional feature vector. Finally, the LSTM outputs from all four views are concatenated and passed through a fully connected layer to generate the final classification output.

In the diameter branch, the inputs consist of the normalised physical measurements of five key inner diameters: LVDs, LVDd, RVDs, RVDd, and MPAD. Each diameter is first encoded into a 128-dimensional feature vector using an MLP. These encoded features are then passed through two Transformer encoder layers to enable cross–feature interaction and fusion. The Transformer encoder is configured with an embedding dimension of 128, 4 attention heads, and a feedforward network dimension of 256. The output features from the Transformer are concatenated and subsequently fed into a fully connected layer to produce the classification result.

In the multimodal fusion approach, the classification outputs from the video branch and the diameter branch are weighted separately and summed to produce the final classification output using a learnt weight.

Both unimodal and multimodal training of this module utilise the cross-entropy loss function, which is defined as follows:

$$\begin{array}{c}{\mathcal{L}}_{\text{CE}}=-\frac{1}{N}\sum _{i=1}^N\sum _{c=1}^C{y}_{i,c}\log \left(p_{i,c}\right)\end{array}$$ (14)

where C denotes the number of classes, which is 3 in this case. y_i,c represents the ground truth label indicating whether the i-th sample belongs to class C (with a value of 0 or 1), and p_i,c denotes the predicted probability that the i-th sample belongs to class C.

Time-aware Postoperative Prediction

To enable dynamic prediction of postoperative cardiac diameters at different time points for patients with TOF, we designed a multimodal regression framework. The inputs to the model include: four preoperative echocardiographic videos from different standard views (A4C, A5C, PSAX, PLAX), five preoperative diameter values (LVDs, LVDd, RVDs, RVDd, MPAD), the type of surgery (whether or not it involved transvalvular repair), and the postoperative follow-up time. The output consists of a predicted abnormal score at the specified postoperative time point, which is labelled as the sum of TOF and TOF mimics probability output by the TOF diagnosis module. In the internal dataset, a total of 651 paired preoperative and postoperative samples were obtained from 175 qualified TOF-repaired participants. These paired data were split into internal training and validation sets at a 4:1 ratio based on individual participants. In the external dataset, 71 paired preoperative and postoperative samples from 49 qualified TOF-repaired participants were used for external testing.

As illustrated in Fig. 3d, the four preoperative echocardiographic videos and five preoperative diameter values are first processed using the video and diameter branches from the TOF Diagnosis module to extract their respective feature embeddings. These yield four video embeddings and five diameter embeddings, all of which are projected to a uniform dimension of 256 via a linear layer. The surgical type is encoded into a 256-dimensional feature using an embedding layer. The postoperative time is first transformed using a logarithmic function to address the imbalance in follow-up intervals and is then encoded into a 256-dimensional vector via an MLP. All the resulting feature embeddings are then structured as inputs to a GNN. As shown in Fig. 3c, a proxy node (learnable embedding) is connected to each preoperative diameter embedding, the surgical type embedding, the follow-up time embedding, and all four preoperative video embeddings, forming a graph with rich multimodal interactions. The GNN consists of three GCN layers, which iteratively update the embeddings by aggregating information from their connected nodes. The final updated learnable embedding is passed through an MLP to predict the corresponding postoperative abnormal score at the given time point.

The model is optimised using the mean squared error loss, defined as:

$$\begin{array}{c}{\mathcal{L}}_{\text{MSE}}=\frac{1}{N}\sum _{i=1}^N{\left(s_i-{\hat{s}}_i\right)}^2\end{array}$$ (15)

where s_i and ${\hat{s}}_i$ represent the ground truth and predicted values, respectively.

Risk Stratification

In this study, the composite high-risk endpoint for post-TOF repair was defined as the occurrence of any of the following within one year: all-cause mortality, reintervention, recurrence of RVOTO, significant pulmonary regurgitation, or clinically significant arrhythmias. Clinical outcome events were assessed by investigators blinded to the predictor information.

The time-series outputs from the Time-aware Postoperative Prediction module, sampled at specified postoperative time points (days 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 90, 100, 110, 120, 180, 240, 300, and 360), were used as input features for a Random Forest binary classification model to predict the occurrence of the composite high-risk endpoint.

Statistics

All hypothesis tests were two-tailed, with a significance level of 0.05. The accuracy, precision, recall, and R² were used to measure the model performance. These metrics are defined as follows:

$$\begin{array}{c}\text{accuracy}=\frac{N_{TP}+N_{TN}}{N_{TP}+N_{FP}+N_{TN}+N_{FN}}\end{array}$$ (16)

$$\begin{array}{c}\text{precision}=\frac{N_{TP}}{N_{TP}+N_{FP}}\end{array}$$ (17)

$$\begin{array}{c}\text{recall}=\frac{N_{TP}}{N_{TP}+N_{FN}}\end{array}$$ (18)

$$\begin{array}{c}\overline{y}=\frac{1}{n}\sum _{i=1}^n{y}_i\end{array}$$ (19)

$$\begin{array}{c}{R}^2=1-\frac{\sum _{i=1}^n{\left(y_i-{\hat{y}}_i\right)}^2}{\sum _{i=1}^n{\left(y_i-\overline{y}\right)}^2}\end{array}$$ (20)

where N_TP, N_TN, N_FP, N_FN represent the number of true positive, true negative, false positive, false negative samples, respectively. The significance of differences in metrics between different methods was assessed using the Mann–Whitney U test on the results from 100 bootstrap iterations. The DeLong test was employed to compare our model with other models under the same modality.

The View Classification and Selection module is trained for 300 epochs, with an initial learning rate of 0.0001, the Adam optimiser, and the CosineAnnealingLR scheduler. The Inner Diameter Localisation and Calculation module is trained for 600 epochs, with an initial learning rate of 0.001, the AdamW optimiser, and the CosineAnnealingWarmRestarts scheduler, where k₁ = 1.0, k₂ = 0.5, k₃ = 0.2, and k₄ = 0.3. For the TOF Diagnosis module, under single-modal training, the video branch is trained for 300 epochs, with an initial learning rate of 0.0001, the Adam optimiser, and the CosineAnnealingLR scheduler; the inner diameter branch is trained for 50 epochs, with an initial learning rate of 0.005, the Adam optimiser, and the StepLR scheduler. Under multi-modal training, the module is trained for 500 epochs, with an initial learning rate of 0.0001, the Adam optimiser, and the CosineAnnealingLR scheduler. The Time-aware Postoperative Prediction module is trained for 600 epochs, with an initial learning rate of 0.001, the Adam optimiser, and the CosineAnnealingLR scheduler. In the Risk Stratification module, the Random Forest model was configured with n_estimators set to 50 and max_depth set to 3, while all other hyperparameters were retained at their default values.

Ethics

The study was approved by the IRB of Shanghai Children's Medical Center Affiliated to Shanghai Jiao Tong University School of Medicine (SCMCIRB-K2022141-1). All the guardians of the participating infants provided written informed consent, and all the images and data were anonymised. The study adhered to the ethical principles of the 2024 Declaration of Helsinki.

Role of funder

The funders played no role in the study design, analysis and collection of data, interpretation of results, or writing and submission of the paper.

Results

View Classification and Selection

Fig. 4a presents the Receiver Operating Characteristic (ROC) curves of the echocardiographic view classification module on the external dataset. As shown, the ROC curves for the four standard views—A4C, A5C, PSAX, and PLAX—are all positioned close to the top-left corner, indicating near-perfect classification performance. This is further supported by the quantitative results in Table 1, where the model consistently demonstrates high performance across all evaluation metrics, including accuracy, AUC, precision, recall, and F1 score. These results confirm the module's capability to accurately classify echocardiographic views, which is critical for downstream tasks such as video feature extraction and disease diagnosis, as it ensures reliable and high-quality input data for subsequent modules.

Fig. 4.

DynaTOF's results for view classification and diameter localisation. a. ROC curves for classifying the A4C, A5C, PSAX, and PLAX echocardiographic views. b. Results of key inner diameter localisation and calculation, along with comparisons to ground truth. From left to right, examples of LVD, RVD, and MPAD are shown. Green indicates the ground truth, and orange indicates the predicted result. c. PCK (Percentage of Correct Keypoints) curves for the localisation of key inner diameter endpoints. Each point on the curve represents the proportion (y-axis) of predicted keypoints that are considered correctly localised when their distance from the ground truth keypoints is less than a given threshold (x-axis). d. Scatter plots comparing predicted and actual inner diameter lengths for LVD, RVD, and MPAD.

Table 1.

The performance of the view classification model for classifying the four echocardiographic views—A4C, A5C, PSAX, and PLAX.

	Accuracy	AUC	Precision	Recall
A4C	0.986 ± 0.005	0.999 ± 0.001	0.982 ± 0.010	0.974 ± 0.011
A5C	0.990 ± 0.004	0.999 ± 0.001	0.970 ± 0.015	0.970 ± 0.015
PSAX	0.989 ± 0.004	0.998 ± 0.002	0.990 ± 0.009	0.958 ± 0.017
PLAX	0.987 ± 0.005	0.998 ± 0.001	0.971 ± 0.013	0.971 ± 0.013

Data are represented by mean ± sd.

Inner Diameter Localisation and Calculation

The Inner Diameter Localisation and Calculation module accurately measures key diameters in echocardiographic images through multi-layer feature fusion-based detection. Fig. 4c provides intuitive examples comparing the module's predictions with ground truth annotations, demonstrating that the predicted start and end points of the inner diameters closely align with expert labels, and the calculated lengths are also highly consistent with the true values. To further quantify the model's localisation performance for the start and end points of various diameters, Fig. 4b presents the Percentage of Correct Keypoints (PCK) curves. It can be observed that as the threshold increases, the PCK value rises rapidly, approaching 1.0 when the threshold exceeds 0.5, indicating high localisation accuracy even under relatively strict criteria. Notably, the PCK curves for LVD and RVD increase more rapidly at lower thresholds, suggesting that these diameters are easier for the model to localise. In contrast, the PCK curve for MPAD rises more slowly. This may be due to the motion artifacts in pulmonary artery imaging caused by respiration, which leads to unclear anatomical boundaries in some cases.

In Fig. 4d, the scatter plots show the comparison between predicted values and actual values for LVD, RVD, and MPAD in the external dataset. The points closely align with the identity line (y = x), indicating low prediction error across all diameters. Among the three measurements, the predicted values of LVD showed the highest agreement with the ground truth, achieving an R² of 0.98, a minimal systematic bias of only 0.01 mm, and 95% limits of agreement (LoA) of (−0.21 mm, 0.23 mm). The model also demonstrated excellent performance in predicting MPAD, with an R² of 0.97, a systematic bias of 0.01 mm, and the narrowest 95% LoA of (−0.16 mm, 0.18 mm), highlighting its capability in quantifying pulmonary artery narrowing. For the anatomically complex RVD, although the R² was relatively lower at 0.76, the systematic bias remained small at 0.06 mm, with a LoA of (−0.56 mm, 0.45 mm), still within the clinically acceptable range of ± 0.6 mm.

TOF Diagnosis

We first evaluated the performance of unimodal models using either the five key inner diameters or the echocardiogram videos of the four standard views. As shown in Fig. 5a, for the inner diameter modality, our custom Transformer²⁴-based model achieved 0.806 (95% CI 0.771–0.837) accuracy, 0.958 (95% CI 0.941–0.973) AUC, 0.772 (95% CI 0.730–0.812) precision, and 0.765 (95% CI 0.732–0.803) recall. A comparison between our proposed model and other mainstream models for inner diameter classification is summarised in Supplementary Table S1. For the video modality, our ResNet²³ + LSTM²⁵-based model achieved 0.857 (95% CI 0.824–0.893) accuracy, 0.978 (95% CI 0.960–0.984) AUC, 0.835 (95% CI 0.801–0.879) precision, and 0.828 (95% CI 0.791–0.875) recall. A comparison between our proposed model and other mainstream models for video classification is also summarised in Supplementary Table S1.

Fig. 5.

DynaTOF's results for TOF diagnosis and postoperative prediction. a. A comparison of the performance metrics for TOF diagnosis under different modalities. b. Scatter plot of predicted versus real risk values for patients with TOF across different postoperative time points. c. ROC curve for TOF diagnosis under different modalities; “Normal” indicates healthy controls. d. Temporal trend of TOF's postoperative abnormal scores. Patients are stratified into high-risk (red) and low-risk (blue) groups based on the defined endpoint events. e. ROC curve for classifying postoperative patients into high-risk and low-risk groups based on predicted postoperative abnormal score series.

We then further evaluated the performance of the multimodal model, which integrates both the four echocardiographic views and the five key inner diameters. The multimodal model achieved 0.910 (95% CI 0.881–0.938) accuracy, 0.989 (95% CI 0.977–0.992) AUC, 0.893 (95% CI 0.860–0.927) precision, and 0.892 (95% CI 0.856–0.927) recall. These results significantly outperformed the single-modality method across all evaluation metrics, with all differences being statistically significant (p-value <0.001). The findings suggest that multimodal fusion effectively leverages the complementary strengths of anatomical measurements and multi-view video data, enhancing both the diagnostic performance and stability.

We further investigated the model's performance for each category, as shown in Fig. 5c. The comparison revealed that all models achieved the highest AUC for the healthy controls category, followed by the TOF category, while the lowest AUC was observed for the TOF mimic category.

We also compared the performance of the three models across three independent hospitals for external testing. Supplementary Figs. S1–3 present the confusion matrices, metric distributions, and ROC curves, respectively. The results of each model on the data from the three hospitals are consistent with the conclusions drawn from the overall dataset, further demonstrating the stability of the models.

Time-aware Postoperative Prediction

The Time-aware Postoperative Prediction module leverages preoperative multi-view echocardiographic videos, key inner diameters, surgical type, and postoperative days to predict abnormal scores at any postoperative time point. As shown in Fig. 5b, the model demonstrates minimal discrepancies between predicted and actual values, with most scatter points closely aligned along the ideal y = x line, resulting in an R² value of 0.852.

Fig. 5d illustrates the temporal trends of the predicted postoperative abnormality scores, which can be utilised to construct a potential dynamic recovery trajectory for a patient during the preoperative phase.

Risk Stratification

To further evaluate the postoperative recovery trajectory and potential risks in patients with TOF, we introduced the Risk Stratification module. We employed a Random Forest model to perform binary classification of the postoperative abnormal score series into high- and low-risk groups. The results, as shown in Fig. 5e, demonstrate that the model achieved an AUC of 0.904 for predicting these risk categories, indicating its strong capability for risk stratification.

Discussion

In this study, we developed DynaTOF, an echocardiography-based intelligent system that integrates five key modules: View Classification and Selection, Inner Diameter Localisation and Calculation, TOF Diagnosis, Time-aware Postoperative Prediction, and Risk Stratification. This system overcomes the limitations of existing research, which typically separates preoperative diagnosis and postoperative outcome prediction, by enabling the full-cycle dynamic modelling of TOF. DynaTOF begins by accurately classifying echocardiographic views, ensuring the precise selection of clinically relevant perspectives for downstream analysis. It then automatically quantifies key inner diameters—clinically validated indicators for TOF diagnosis—directly from echocardiographic videos. These quantitative features are combined with video-derived visual representations from multiple views to construct a robust multimodal diagnostic model, achieving a high AUC of 0.989 for TOF prediction. For patients diagnosed with TOF, DynaTOF is designed to estimate postoperative abnormal score trajectories prior to surgery, which may help inform clinicians and families about potential postoperative trends. In addition, the model enables early risk stratification, potentially supporting closer postoperative monitoring for infants identified as higher risk.^18,26

Automated TOF diagnosis based on echocardiography can reduce manual interpretation time and minimise subjective errors.^26,27 However, existing AI approaches for TOF diagnosis primarily focus on cardiac structure segmentation or the prediction of specific clinical measurements, falling short of directly identifying TOF cases.^16,28 Our system addresses this gap by leveraging a multimodal fusion strategy that integrates multi-view echocardiographic videos with key inner diameter measurements. This approach leverages the complementary information from structural imaging and quantitative indicators to facilitate the differentiation among healthy controls, TOF mimics, and TOF. It has the potential to reduce diagnostic burden for clinicians and to alleviate inter-observer variability in interpretation.²⁶ The identification of each cardinal component of TOF—namely RVOTO, VSD, overriding aorta, and right ventricular hypertrophy—typically requires the integration of multiple echocardiographic views. Explicitly modelling and individually classifying each component would therefore be cumbersome. Instead, we adopt a direct modelling strategy that implicitly learns these abnormalities in a unified manner. To further prevent the model from merely capturing generic pathological features rather than TOF-specific representations, we additionally introduce a “TOF mimics” category. This design encourages the model to distinguish TOF from other conditions with overlapping phenotypic characteristics, thereby enabling it to learn a more holistic and discriminative representation arising from the joint presence of all four defining components.

Postoperative prognosis prediction for patients with TOF has long been a central focus in this field.²⁹ Previous studies have attempted to predict MACE or common postoperative complications closely associated with TOF surgery—such as pulmonary regurgitation and residual RVOTO—based on imaging acquired after the surgery.^14,21 However, these approaches are limited in that they cannot forecast postoperative outcomes before the surgery takes place. In our work, we address this limitation by collecting and analysing follow-up echocardiographic data from patients with TOF at multiple postoperative time points. We developed a model that uses preoperative imaging and surgical type to predict postoperative abnormal scores at any future time point, effectively bridging the gap between preoperative assessments and postoperative patterns. This forward-looking approach could potentially aid help families gain a clearer understanding of long-term prognoses, thereby reducing decision-related anxiety.^30,31

Risk stratification can help healthcare providers prioritise high-risk patients while avoiding excessive monitoring of low-risk cases, thereby improving the efficiency of resource utilisation.³² In our study, we grouped patients into high-risk and low-risk categories based on the trajectories of predicted postoperative abnormal scores, achieving an AUC of 0.904. This approach may help inform follow-up planning and support more rational allocation of clinical resources, serving as an adjunct to existing care pathways.³³

Once all modules of DynaTOF are trained, the full application pipeline can be constructed. As illustrated in Fig. 6, the DynaTOF system begins by collecting echocardiographic videos of multiple standard views. These videos are then passed into the View Classification and Selection module, which accurately selects videos corresponding to the A4C, A5C, PSAX, and PLAX views. The A4C and PSAX view videos are fed into the Key Inner Diameter Localisation and Calculation module to obtain values for five key internal diameters. Simultaneously, videos of the A4C, A5C, PSAX, and PLAX views are processed by the video feature extraction module to extract relevant visual features. The outputs from these two modules are then jointly input into the TOF Diagnosis module to determine whether the patient has TOF. For patients diagnosed with TOF, the extracted features are further passed into the Time-aware Postoperative Prediction module, which models the temporal trajectory of abnormal score before surgery, considering different surgical strategies. The predicted score series are subsequently analysed by the Risk Stratification module, which classifies patients into high-risk or low-risk groups. Infants identified as higher risk may merit closer clinical attention, such as consideration of alternative management strategies and more frequent postoperative follow-up.

Fig. 6.

The application pipeline of DynaTOF. First, echocardiographic videos were collected from infants who underwent echocardiography examinations. These videos were processed by the View Classification and Selection module, which selects A4C and PSAX views for input into the Key Inner Diameter Localisation and Calculation module, and A4C, A5C, PSAX, and PLAX views for input into the Video Feature Extraction module. Features extracted by these two modules were then fed into the TOF Diagnosis module to determine whether the patient has TOF. For patients diagnosed with TOF mimics, the detailed diagnosis and treatment are needed. For patients diagnosed with TOF, the Time-aware Postoperative Prediction module was used to predict postoperative risk trajectories prior to surgery. These predicted risk dynamics were further analysed by the Risk Stratification module, which classifies patients into high-risk and low-risk groups. Infants identified as high-risk were then placed under intensive surveillance for closer postoperative monitoring.

This study still has several limitations. First, the number of key inner diameters used for automatic quantification and postoperative prediction is relatively limited. In the future, the system's comprehensiveness could be further enhanced by annotating and incorporating additional clinically relevant cardiac parameters. Second, all training data in this study were obtained from a single medical centre, which may restrict the generalisability of the system. Although the overall sample size is sufficiently representative, the number of samples used for external testing—particularly the relatively small cohort for postoperative prediction—remains substantially lower than that used for training. This could potentially impact the broader applicability of the results. Further training and validation with larger, multicentric datasets are therefore necessary to enhance the system's robustness and applicability across diverse clinical settings.

In summary, we present a fully automated echocardiography-based intelligent diagnosis and time-aware postoperative prediction system for TOF. This system enables efficient diagnosis of TOF and provides personalised dynamic abnormal scores for postoperative outcomes, facilitating early screening and longitudinal monitoring. By supporting intelligent postoperative management, the deployment of this system holds the potential to impact clinical practice in congenital heart disease and contribute to the broader advancement of AI-driven cardiovascular care.

Contributors

Q.G., A.W., and Y.G. contributed equally to this work. H.L., Y.Z., L.C., and S.Y. supervised the study. Q.G., A.W., Y.G., H.L., Y.Z., L.C., and S.Y. conceived and designed the study. A.W., W.X., J.Y., M.Y., S.C., and L.Z. collected and curated the data. Q.G. and Y.G. developed the model and performed the statistical analysis. A.W., L.Z., M.Y., J.Y., and S.C. provided clinical interpretation and managed patient recruitment and follow-up. Q.G., Y.G., and A.W. drafted the manuscript. H.L., Y.Z., L.C., and S.Y. critically revised the manuscript. Q.G., Y.G., and A.W. verified the data. H.L., Y.Z., L.C., and S.Y. acquired funding for the study. All authors contributed to data interpretation, had full access to all the data, and reviewed and approved the final version of the manuscript.

Data sharing statement

Data are available upon reasonable request. The data that support the findings of our study are available upon request from the corresponding author through a data use agreement (DUA) process. The code for the DynaTOF framework is publicly available at: https://github.com/shuangaaa/dynatof_code.

Declaration of interests

All authors have declared that no conflict of interest exists.