Chamber Attention Network (CAN): Towards interpretable diagnosis of pulmonary artery hypertension using echocardiography

Graphical abstract

Highlights

• •We introduced a novel chamber attention network (CAN) that utilizes the Grad-CAM technique to assess the relevance of different chambers in identifying PAH.
• •The attention vector generated by the chamber attention module provides a quantitative measure of the chambers’ importance in PAH diagnosis, which aligns well with clinical knowledge and enhances the model’s interpretability.
• •Our CAN model achieved outstanding performance on a large training-validation dataset consisting of 13912 individual subjects.
• •Evaluation on an external test dataset from a different hospital demonstrated the superior generalization ability of our CAN model.

Abstract

Introduction:

Accurate identification of pulmonary arterial hypertension (PAH) in primary care and rural areas can be a challenging task. However, recent advancements in computer vision offer the potential for automated systems to detect PAH from echocardiography.

Objectives:

Our aim was to develop a precise and efficient diagnostic model for PAH tailored to the unique requirements of intelligent diagnosis, especially in challenging locales like high-altitude regions.

Methods:

We proposed the Chamber Attention Network (CAN) for PAH identification from echocardiographic images, trained on a dataset comprising 13,912 individual subjects. A convolutional neural network (CNN) for view classification was used to select the clinically relevant apical four chamber (A4C) and parasternal long axis (PLAX) views for PAH diagnosis. To assess the importance of different heart chambers in PAH diagnosis, we developed a novel Chamber Attention Module.

Results:

The experimental results demonstrated that: $1)$ The substantial correspondence between our obtained chamber attention vector and clinical expertise suggested that our model was highly interpretable, potentially uncovering diagnostic insights overlooked by the clinical community. $2)$ The proposed CAN model exhibited superior image-level accuracy and faster convergence on the internal validation dataset compared to the other four models. Furthermore, our CAN model outperformed the others on the external test dataset, with image-level accuracies of 82.53% and 83.32% for A4C and PLAX, respectively. $3)$ Implementation of the voting strategy notably enhanced the positive predictive value (PPV) and negative predictive value (NPV) of individual-level classification results, enhancing the reliability of our classification outcomes.

Conclusions:

These findings indicate that CAN is a feasible technique for AI-assisted PAH diagnosis, providing new insights into cardiac structural changes observed in echocardiography.

Introduction

The prompt and accurate diagnosis of PAH is crucial to ensure timely treatment and prevent further deterioration of the cardiovascular system.[1], [2]. While direct pressure measurement through right cardiac catheterization is currently considered as the ”gold standard” for PAH diagnosis[3], this invasive procedure carries the risk of minor physical trauma. To avoid such risks, echocardiography has been recommended as a noninvasive imaging modality that can effectively estimate pulmonary artery pressure (PAPm)[4]. Echocardiography is capable of detecting structural changes in the heart that can serve as a diagnostic ”lesion” for PAH [5], [6]. However, the interpretation of echocardiography results requires extensive expertise and training, posing challenges for its widespread use in primary care and rural settings. In recent years, there has been a growing focus on utilizing mathematical models to aid in medical diagnosis, such as characterizing organ conditions[7], [8] and enhancing medical images based on fractional calculus[9], [10]. Among these approaches, deep learning technology, due to its characteristics of automatic feature extraction, end-to-end learning, and large-scale parallel computing, has been widely applied in the recognition of medical images, including the automatic segmentation and detection of ”lesions” in echocardiograms, with the goal of identifying pulmonary arterial hypertension (PAH) [11], [12]. The application of AI algorithms to various cardiovascular imaging modalities, including chest X-rays [13], electrocardiography [14], and echocardiography [15], has gained traction in enhancing PAH diagnosis. Diverse AI algorithms, ranging from Random Forest (RF) and Support Vector Machines (SVM) [16] to Convolutional Neural Networks (CNN) [17], have exhibited potential in refining echocardiography interpretation, encompassing tasks such as viewpoint classification [18], cardiac chamber segmentation [19], cardiac function assessment [20], and disease identification [21]. Despite these remarkable advances, the current state-of-the-art methods for identifying PAH still grapple with challenges such as low accuracy, limited generalization capacity, modest reliability, and lack of interpretability.

In this study, we developed an analytic pipeline (CAN) for automated PAH diagnosis using echocardiograms, which requires no human intervention and can be deployed on a high-performance computing cluster or web application. Firstly, a view classification model distinguished A4C and PLAX from other views. Then, we created a Segmentation Module to separate the cardiac region and heart chambers from the original echocardiographic images. Segmenting the cardiac region eliminated irrelevant information, while segmenting the chambers facilitated the analysis of each chamber’s importance in PAH diagnosis. A Chamber Attention Module based on Grad-CAM [22] was then developed to measure the effect of different cardiac chambers on PAH identification. The Classification Module paid more attention to chambers with greater influence on PAH, while treating chambers with less effect as background. The training set was reconstructed using the spatial attention of chambers and then given to ResNet50 [23] for training the Classification Module. Image-level results from the Classification Module were combined to provide view-level results for A4C and PLAX, respectively, as a single subject may have multiple echocardiographic images. We employed a voting technique to increase the reliability of the final individual-level result, meeting high security requirements for medical image diagnosis. We demonstrated the effectiveness of our model by comparing its performance to other models. Our Chamber Attention Module based on Grad-CAM not only corroborated the clinical understanding that the right ventricle plays a pivotal role in PAH identification [1], but also uncovered additional insights. These insights extended beyond clinical expertise, revealing potential significance in PAH diagnosis attributed to the right atrium. Experimental results reinforced the notion that our attention mechanism substantially contributes to the accuracy of PAH diagnosis.

More precisely, the contributions of this study are as follows:

• 1.Quantitative interpretation with attention vector: The attention vector generated by the Chamber Attention Module offered a precise quantitative representation of the relative importance of each cardiac chamber in PAH diagnosis, which aligned well with clinical expertise. This enhanced the interpretability of the entire model and provided valuable insights into the underlying mechanisms of PAH identification.
• 2.State-of-the-art performance on training-validation dataset: Our proposed Chamber Attention Network (CAN) model attained a state-of-the-art performance benchmark on the training-validation dataset. This achievement underscored the efficacy of the model in successfully capturing the intricate patterns crucial for PAH diagnosis.
• 3.Robust generalization on external test dataset: Additionally, we evaluated the CAN model’s generalization capability on an external test dataset from a distinct medical institution. The outcomes from this assessment showed the model’s exceptional performance relative to other existing methods, highlighting its robustness and applicability across different clinical settings.

Materials and methods

Overall framework

The overall framework of the proposed CAN model can be divided into three main parts: $\left(1\right)$ The Segmentation Module, based on U-Net [24], separates the heart and chambers, enabling subsequent attention and classification process; $\left(2\right)$ The Chamber Attention Module measures the impact of different heart chambers on PAH diagnosis to determine the chamber attention vector; $\left(3\right)$ The Classification Module reconstructs the echocardiographic images based on the chamber attention vector and then classifies the reorganized echocardiograms using ResNet50. The overall framework is illustrated in Fig. 1.

Fig. 1.

The overall framework of our method consists of three modules: Segmentation Module, Attention Module, and Classification Module. The Segmentation Module is responsible for segmenting and positioning the cardiac chambers in the echocardiographic images. The Attention Module identified the key areas relevant to PAH diagnosis using Grad-CAM, which, combined with chamber locations, generates importance weights of each cardiac chamber. The Grad-CAM-based weights are then combined with expert scores to calculate the chamber attention vector. The Classification Module reconstructed the echocardiographic images based on the chamber attention vector and utilizes ResNet50 for PAH diagnosis. In the context, we denoted the $k$-th segmented cardiac image of a certain view as matrix ${\stackrel{\sim }{I}}_k$, with its complement matrix denoted as $I_k$. Additionally, $M_k^{\mathrm{RV}},M_k^{\mathrm{LV}},M_k^{\mathrm{RA}}$ and $M_k^{\mathrm{LA}}$ represent the mask matrices of the corresponding chambers. The gradient weights of the C-th channel of the last convolutional layer (layer L) of the convolutional network are denoted as $w_C^L$. The resized heatmap is denoted as $H_k$, and the reconstructed image is represented by $R_k$. The functions $f_1,f_2$, and $f_3$ correspond to Eqs. (1), (4), and the voting strategy, respectively.

View classification

During ultrasound examinations, sonographers often manipulate the ultrasound probe by rotating and adjusting the zoom level to focus on different substructures within an image. This variability leads to a range of views [25]. However, not all of these views contribute to diagnosing PAH. Clinical knowledge suggests that the A4C and PLAX viewpoints are particularly relevant for PAH identification [26]. Thus,it becomes essential to develop a model capable of distinguishing between various cardiac views, aiming to filter out irrelevant perspectives and retain A4C and PLAX views for subsequent automated CAN model processing. Given the absence of explicit view labels, we randomly selected a total of 6238 echocardiographic images from both the internal and external datasets. We then engaged three echocardiographers to manually assign a label from six categories (A3C, A4C, A5C, PLAX, PSAX, and others) to each image. The distribution of images across these labels was nearly uniform. Subsequently, we utilized these labeled images to train a 50-layer ResNet (ResNet50) for extracting image features. Additionally, a three-layer fully connected network was trained to perform image classification. For every processed image, the output of this view classification model constituted a 6-dimensional vector representing class confidence. We evaluated the model’s accuracy using 10-fold cross-validation. Moreover, to visually inspect the outcomes of our view classification network, we employed t-Distributed Stochastic Neighbor Embedding (t-SNE) [27], a technique implemented in the scikit-learn package, to cluster the output of the top layer.

Segmentation Module

To extract the regions of interest, which in this case are the cardiac structures, echocardiographic images undergo preprocessing to eliminate irrelevant interferences. The thresholding method based on Hounsfield unit (HU) values is widely used for chest image preprocessing, but its performance, especially in terms of accuracy, is often unsatisfactory. Precise segmentation of cardiac structures from echocardiograms is of paramount importance. Therefore, we trained cardiac segmentation models based on U-Net, a commonly used technique in biomedical image segmentation. Specifically, we input two distinct echocardiographic views (A4C and PLAX) into separate instances of the U-Net model. This strategy allowed us to construct cardiac segmentation models that were customized for each respective view. Subsequently, we employed the Chamber Attention Module to perform the segmentation of distinct heart chambers and investigate their diverse impacts on the diagnosis of PAH.

We trained a convolutional neural network using 1200 2D echocardiography images, containing approximately equal numbers of A4C and PLAX images randomly selected from the two datasets, to segment the cardiac area before segmenting the chambers. These images were downsampled to dimensions of $224\times 224$ pixels using the nearest-neighbor technique. We split the data into a training/validation set with 1000 images and a test set with 200 images. To generate masks for the echocardiographic images that contained the field of view, we employed a custom labeling tool that allowed users to select vertices to generate polygons. The labeling tool was developed as an interactive polygon editor using the matplotlib library [28]. During the training phase, the original images were paired with the manually created masks and fed into the neural network. The trained model processed the input raw grayscale image and output an image with pixel values of the same size falling within the (0,1) range. By extracting pixel values from the original image corresponding to coordinates with output pixel values exceeding 0.75, while setting other pixel values to 0, we created the segmented image.

After obtaining the segmented cardiac regions using the previously described cardiac segmentation models, we proceeded to segment different heart chambers. For this purpose, we used the same dataset as for the cardiac segmentation, and manually drew a mask around the epicardium of each chamber using the marking tools described earlier. These masks were then used to train a separate convolutional neural network to segment each chamber individually.

The U-Net architecture formed the basis of our segmentation model, as depicted in Fig. 1. However, we made some adaptations to fit the $224\times 224$ pixel resolution of the images. These modifications included altering the filter size, removing the convolution layer of 1024 filters, and introducing Dropout before the first upsampling convolution layer. We employed the Adam optimizer to train the model for 30 epochs, with an initial learning rate of 0.01 and a reduction factor of 0.1. We assessed the model’s performance by computing the pixel loss of the test set and used early stopping during training. To further verify the segmentation results, we visually examined the segmentation outputs of U-Net on the test set.

Chamber Attention Module

Grad-CAM (Gradient-weighted Class Activation Mapping)[22] is a technique used to generate heat maps that visualize the regions of an image that are important for a given classification decision made by a convolutional neural network. Grad-CAM uses the gradients of a specific class output with respect to the feature maps of the last convolutional layer to weight the importance of each feature map. This allows us to highlight the regions of an image that are most important for the network’s classification decision.

In this case, Grad-CAM is used to quantify the influence of different heart chambers on the diagnosis of PAH. The heat maps generated by Grad-CAM highlight the regions of the image that are most important for the PAH diagnosis, and the labeled region that each heart chamber occupies can be used to determine the influence of each chamber on the diagnosis.

By analyzing the heat maps of the echocardiographic images, it is possible to determine which heart chamber is most important for the PAH diagnosis. If the majority of heat maps converge on a particular heart chamber, it suggests that this chamber has a higher influence on the diagnosis of PAH.

Grad-CAM

We conducted Grad-CAM analysis on the segmented echocardiographic images using the pre-trained VGG-16 model (fine-tuned from Echocv[21]), allowing us to visualize the regions of interest.

Resizing the heat map

In our experiment, the original heat map is of size $14\times 14$ and the target size is $224\times 224$, indicating a scaling factor of $k=16$. To achieve this scaling, we employed bicubic interpolation to resize the generated heat map to match the dimensions of the segmented images.

Attention formula $f_1$

In order to quantitatively assess the impact of the four cardiac chambers on the diagnostic process of PAH, an attention mechanism denoted as $f_1$ is formulated. The methodology commences with a training dataset comprising n echocardiographic images. From this dataset, mask matrices $M_k^{\mathrm{RV}},M_k^{\mathrm{LV}},M_k^{\mathrm{RA}}$, and $M_k^{\mathrm{LA}}$ are derived through the utilization of a dedicated chamber segmentation model. These matrices correspond respectively to the right ventricle (RV), left ventricle (LV), right atrium (RA), and left atrium (LA). Each mask matrix assumes a binary form, possessing dimensions of $224\times 224$. Within each matrix, the element aligned with the spatial region corresponding to the specified chamber is set to the value 1, while all other elements are assigned a value of 0. Furthermore, the utilization of the Grad-CAM technique results in the generation of a resized heat map, denoted as $H_k$, corresponding to each individual echocardiographic image. The anticipated unnormalized attention vector ${\left[\begin{array}{cccc}\hfill {\stackrel{\sim }{\omega }}_{\mathrm{RV}}\hfill & \hfill {\stackrel{\sim }{\omega }}_{\mathrm{LV}}\hfill & \hfill {\stackrel{\sim }{\omega }}_{\mathrm{RA}}\hfill & \hfill {\stackrel{\sim }{\omega }}_{\mathrm{LA}}\hfill \end{array}\right]}^{\mathrm{T}}$ for RV, LV, RA and LA are:

$$\begin{array}{cc}\hfill & {\stackrel{\sim }{w}}_{\mathrm{RV}}=\frac{1}{n}{\displaystyle \sum _{k=1}^n}\frac{W_k^{\mathrm{RV}}}{W_k^{\mathrm{H}}},{\stackrel{\sim }{w}}_{\mathrm{LV}}=\frac{1}{n}{\displaystyle \sum _{k=1}^n}\frac{W_k^{\mathrm{LV}}}{W_k^{\mathrm{H}}},\hfill \\ \hfill & {\stackrel{\sim }{w}}_{\mathrm{RA}}=\frac{1}{n}{\displaystyle \sum _{k=1}^n}\frac{W_k^{\mathrm{RA}}}{W_k^{\mathrm{H}}},{\stackrel{\sim }{w}}_{\mathrm{LA}}=\frac{1}{n}{\displaystyle \sum _{k=1}^n}\frac{W_k^{\mathrm{LA}}}{W_k^{\mathrm{H}}}\hfill \end{array}$$ (1)

where $W_k^{\mathrm{H}},W_k^{\mathrm{RV}},W_k^{\mathrm{LV}},W_k^{\mathrm{RA}}$, and $W_k^{\mathrm{LA}}$ are computed as:

$$\begin{array}{cc}\hfill & W_k^{\mathrm{H}}={\displaystyle \sum _{i,j}}{\left(H_k\right)}_{i,j},W_k^{\mathrm{RV}}={\displaystyle \sum _{i,j}}{(M_k^{\mathrm{RV}}\odot H_k)}_{i,j},\hfill \\ \hfill & W_k^{\mathrm{LV}}={\displaystyle \sum _{i,j}}{(M_k^{\mathrm{LV}}\odot H_k)}_{i,j},W_k^{\mathrm{RA}}={\displaystyle \sum _{i,j}}{(M_k^{\mathrm{RA}}\odot H_k)}_{i,j},\hfill \\ \hfill & W_k^{\mathrm{LA}}={\displaystyle \sum _{i,j}}{(M_k^{\mathrm{LA}}\odot H_k)}_{i,j}\text{.}\hfill \end{array}$$ (2)

in which $\odot$ is the Hadamard element-wise product.

The provided formulas delineate the stepwise procedure for deriving attention weights for each of the four cardiac chambers, namely the RV, LV, RA, and LA, through the utilization of heat maps generated via convolutional neural networks. These heat maps serve to accentuate the salient regions within the image, as identified by the neural network, that are deemed significant in the context of diagnosing PAH. In the context of these expressions, $W_k^{\mathrm{H}}$ denotes the aggregate of all pixel values contained within the heat map corresponding to the k-th image, thus reflecting the collective attention allocated to the image. The quantities $W_k^{\mathrm{RV}},W_k^{\mathrm{LV}},W_k^{\mathrm{RA}}$, and $W_k^{\mathrm{LA}}$ represent the summation of pixel values within the heat maps associated with the RV, LV, RA, and LA, respectively, signifying the attention attributed to each individual chamber. Subsequently, the weighting for each chamber is ascertained by computing the proportion of its specific attention to the total attention, which is equivalent to dividing the chamber’s attention value by the total attention value: $\frac{W_k^{\mathrm{RV}}}{W_k^{\mathrm{H}}},\frac{W_k^{\mathrm{LV}}}{W_k^{\mathrm{H}}},\frac{W_k^{\mathrm{RA}}}{W_k^{\mathrm{H}}}$, and $\frac{W_k^{\mathrm{LA}}}{W_k^{\mathrm{H}}}$. The anticipated attention (${\stackrel{\sim }{w}}_{\mathrm{RV}},{\stackrel{\sim }{w}}_{\mathrm{LV}},{\stackrel{\sim }{w}}_{\mathrm{RA}},{\stackrel{\sim }{w}}_{\mathrm{LA}}$) assigned to each chamber is determined as the mean of these proportions across all images. Ultimately, the ${\ell }_1$ normalized attention vector for the four cardiac chambers is derived through the normalization of the expected attention vector, thereby ensuring that the summation of the weights attributed to all chambers equals 1.

$${\left[\begin{array}{cccc}\hfill {\omega }_{\mathrm{RV}}\hfill & \hfill {\omega }_{\mathrm{LV}}\hfill & \hfill {\omega }_{\mathrm{RA}}\hfill & \hfill {\omega }_{\mathrm{LA}}\hfill \end{array}\right]}^{\mathrm{T}}\text{.}$$ (3)

Classification Module

We proposed a Classification Module to diagnose PAH using cardiac images. For each view, we obtained the attention vector and segmented heart chambers. We hypothesized that heart chambers with higher attention would exhibit greater variation in pixel values compared to the surrounding regions. Conversely, distinguishing the chamber with less attention should be more challenging. To address this, we suggested a method to reconstruct the training dataset that preserved the contours of the heart chambers and enhanced structural information important for PAH diagnosis. We trained a ResNet50 model on the rebuilt training dataset to perform image-level classification for each view. Since each subject may have multiple cardiac images for A4C and PLAX views, we averaged their image-level results to obtain view-level findings. Finally, we employed a voting strategy to combine the view-level results and reach an individual-level classification decision for PAH diagnosis.

Reconstruction for each view

For the k-th segmented cardiac image from a specific perspective, it is denoted as the matrix ${\stackrel{\sim }{I}}_k$, where the pixel values range between 0 and 255, as echocardiograms are represented in grayscale. The matrix ${\stackrel{\sim }{I}}_k$ exhibits a distinct characteristic, whereby a significant portion of the cardiac chambers’ regions appear entirely black, corresponding to pixel values of 0, while the surrounding areas are predominantly white. It is well-established that the product of 0 and any number is 0. In light of this, we introduce the concept of a complement matrix $I_k$ derived from ${\stackrel{\sim }{I}}_k$. The rationale behind this transformation is to enable the influence of the cardiac chamber weights on the resulting cardiac image. Specifically, each element of the complement matrix $I_k$ is defined as ${\left(I_k\right)}_{\mathit{ij}}=255-{\left({\stackrel{\sim }{I}}_k\right)}_{\mathit{ij}}$. This complementary matrix approach effectively adjusts the pixel values of the segmented image to facilitate the impact of cardiac chamber weighting on the overall cardiac image representation. We express the reconstructed image $R_k$ as follows:

$$\begin{array}{cc}{R}_k=\hfill & f_2(I_k,M_k^{\mathrm{RV}},M_k^{\mathrm{LV}},M_k^{\mathrm{RA}},M_k^{\mathrm{LA}},{\omega }_{\mathrm{RV}},{\omega }_{\mathrm{LV}},{\omega }_{\mathrm{RA}},{\omega }_{\mathrm{LA}})\hfill \\ =\hfill & I_k-\left(\right(1-{\omega }_{\mathrm{RV}}\left)\right(M_k^{\mathrm{RV}}\odot I_k)+(1-{\omega }_{\mathrm{LV}}\left)\right(M_k^{\mathrm{LV}}\odot I_k)+\hfill \\ \hfill & (1-{\omega }_{\mathrm{RA}})(M_k^{\mathrm{RA}}\odot I_k)+(1-{\omega }_{\mathrm{LA}})(M_k^{\mathrm{LA}}\odot I_k))\hfill \\ =\hfill & I_k-\left(\right(1-{\omega }_{\mathrm{RV}})M_k^{\mathrm{RV}}+(1-{\omega }_{\mathrm{LV}})M_k^{\mathrm{LV}}+\hfill \\ \hfill & (1-{\omega }_{\mathrm{RA}})M_k^{\mathrm{RA}}+(1-{\omega }_{\mathrm{LA}})M_k^{\mathrm{LA}})\odot I_k\text{.}\hfill \end{array}$$ (4)

The formulation of $R_k$ is grounded in the principle that the assigned weight $\omega$ to a specific cardiac chamber should accurately reflect its significance in the context of PAH identification. Moreover, the reconstruction process of the image seeks to emphasize regions within the cardiac chambers that bear higher weights. This objective is realized by orchestrating the pixel values within the reconstructed image such that they exhibit a pronounced disparity from the pixel values characterizing the adjacent areas. Consequently, the design of $R_k$ entails a strategic manipulation of pixel values to accentuate the visual distinction between the targeted chamber and its proximate surroundings. This approach aligns with the overarching goal of magnifying the visual impact of chambers with greater diagnostic relevance, thereby enhancing the interpretability of the reconstructed image in the context of PAH identification.

Although the mask matrices $M_k^{\mathrm{RV}},M_k^{\mathrm{LV}},M_k^{\mathrm{RA}}$, and $M_k^{\mathrm{LA}}$ generated by the Segmentation Module might not achieve complete coverage of the respective cardiac chambers, the reconstruction process defined by Eq. (4) is designed to retain the fundamental contours of each individual chamber while preserving their structural attributes. Furthermore, it is noteworthy that the degree of variation between the pixel values specific to a particular cardiac chamber and those of its neighboring regions maintains a direct proportionality to the weight assigned to the corresponding chamber. This particular arrangement ensures that regions of elevated weight exhibit enhanced visual prominence within the reconstructed image. Consequently, this process facilitates a pronounced focus on chambers with comparatively higher weights, thereby providing a mechanism through which the subsequent classification model can allocate greater attention to those heart chambers that have a more substantial impact on the task of PAH identification.

Classification model for each view

We implemented our proposed method by training ResNet50 classifiers separately for A4C and PLAX using the reconstructed images as the training set. To evaluate the performance of our method, we used 5-fold cross-validation and calculated six metrics, including accuracy, recall, precision, specificity, F1-measure, and the area under the receiver operating characteristic curve (AUC).

We chose ResNet50 for two reasons: first, ResNet50 has a strong track record of high performance in image classification tasks, and second, by comparing the performance of our chamber attention mechanism with that of ResNet50 and other models such as Inceptionv3, Xception, and Densenet121, we can demonstrate the effectiveness of our attention mechanism.

To obtain the view-level results, we first calculated the image-level results for each reconstructed image, and then averaged the results for each subject across all the A4C or PLAX images. We compared the performance of our proposed chamber attention mechanism combined with ResNet50 (CAN) with the other four models using the aforementioned metrics to demonstrate the effectiveness of our method.

Merging view-level results to achieve individual-level result

The pipeline described above addresses the classification of each view, i.e. A4C and PLAX, respectively. We then need to integrate the view-level results to achieve individual-level classification and determine whether the subject suffers from PAH or not. Due to the high safety requirements of medical image diagnosis, we propose a voting strategy to enhance the reliability of the individual-level diagnostic result, thereby decreasing both false positive and false negative rates. The strategy is straightforward: we only accept the individual-level result when the two view-level results agree. If both A4C and PLAX yield positive (negative) results, then the individual-level result is accepted as positive (negative); If the two view-level results are inconsistent, the individual-level result is not acceptable and further clinical evaluation is required for diagnosis. We classified subjects exhibiting consistent view-level results as ’uncontroversial’ and those showing inconsistent view-level results as ’controversial’.

The objective of the voting strategy is to enhance the proportion of correctly classified positive (negative) samples among all the samples classified as positive (negative), even if this comes at the cost of overall accuracy. To evaluate the final classification performance of our voting strategy, we utilize metrics including ACC (accuracy), PPV (positive predictive value), and NPV (negative predictive value). Additionally, we introduce the UNC (uncertainty) metric to indicate the ’controversial’ results.

$$\begin{array}{cc}\mathrm{ACC}\hfill & =\frac{\mathrm{TPTP}+\mathrm{TNTN}}{\mathrm{Z}},\hfill \\ \mathrm{UNC}\hfill & =\frac{\mathrm{TPFN}+\mathrm{FNTP}+\mathrm{TNFP}+\mathrm{FPTN}}{\mathrm{Z}},\hfill \\ \mathrm{PPV}\hfill & =\frac{\mathrm{TPTP}}{\mathrm{TPTP}+\mathrm{FPFP}},\hfill \\ \mathrm{NPV}\hfill & =\frac{\mathrm{TNTN}}{\mathrm{TNTN}+\mathrm{FNFN}},\hfill \\ \mathrm{X}\hfill & =\mathrm{TPTP}+\mathrm{FNFN}+\mathrm{TNTN}+\mathrm{FPFP},\hfill \\ \mathrm{Y}\hfill & =\mathrm{TPFN}+\mathrm{FNTP}+\mathrm{TNFP}+\mathrm{FPTN},\hfill \\ \mathrm{Z}\hfill & =\mathrm{TPTP}+\mathrm{TPFN}+\mathrm{FNTP}+\mathrm{FNFN}\hfill \\ \hfill & +\mathrm{TNTN}+\mathrm{TNFP}+\mathrm{FPTN}+\mathrm{FPFP}\text{.}\hfill \end{array}$$ (5)

TPTP (True-Positive True-Positive) represents the number of subjects with true-positive results for both A4C and PLAX. TPFN (True-Positive False-Negative) denotes the number of subjects with true-positive results for A4C but false-negative results for PLAX. Similarly, other symbols have corresponding meanings.

The total subjects count, represented as Z, is derived from the aggregation of subjects distributed across distinct categories. TPTP, FNFN (False-Negative False-Negative), TNTN (True-Negative True-Negative), and FPFP (False-Positive False-Positive) collectively correspond to the number of ”uncontroversial” subjects, designated as X. Conversely, the number of ”controversial” subjects, denoted as Y, encompasses the summation of subjects labelled as TPFN, FNTP (False-Negative True-Positive), TNFP (True-Negative False-Positive), and FPTN (False-Positive True-Negative).

The overall ACC of the classification model is assessed. Additionally, we calculate the proportion of ”controversial” subjects (UNC) among all the subjects under consideration. PPV and NPV are also computed, representing the proportions of correctly predicted positive and negative subjects, respectively, among all the subjects classified as positive and negative. We have created a visual representation in the form of a table to help clarify the mathematical symbols discussed above (Figure S1).

Results

Materials

Data acquisition and preprocessing

We sourced the dataset for this study from two independent medical institutions, namely the Department of Information, Medical Support Center, The General Hospital of Western Theater Command, located in Chengdu, Sichuan, China, and the Department of Ultrasonic Diagnosis, Army 954 Hospital, situated in Shannan, Tibet, China. This dataset consisted of 2-dimensional (2D) echocardiographic images in DICOM format from subjects diagnosed with pulmonary arterial hypertension (PAH) or normal subjects. These DICOM files were converted and split into individual images using PyDicom (Version 2.3.0). Patient identifying information and metadata were meticulously removed to ensure privacy. The diagnosis of pulmonary hypertension was made through Doppler echocardiography assessment, following the relevant guidelines (normal morphology and opening of the tricuspid valve; reflux signal detected at closure of the tricuspid valve; mean pulmonary artery pressure $\mathrm{PAPm}⩾25\mathrm{mmHg}$ at rest estimated by tricuspid regurgitation velocity) in all subjects. The inclusion criteria for this study were as follows: $\left(1\right)$. Adults aged $>18$ years. $\left(2\right)$. Patients diagnosed as PAH or normal based on their ultrasonic diagnosis reports. $\left(3\right)$. Availability of A4C or PLAX echocardiographic images.

Our internal dataset consisted of echocardiographic images and clinical data from 2122 patients with pulmonary arterial hypertension and 11790 normal subjects. This dataset was utilized for training the Attention Module and Classification Module. All participants underwent a structured echocardiographic investigation at the General Hospital of Western Theater Command between January 2021 and June 2022. The dataset comprised a total of 150561 echocardiographic images, including 13624 A4C and 19679 PLAX images. These images were acquired using the S5 or S8 probes with a frequency of 2–5 MHz on PHILIPS IE33 and PHILIPS EPIC 7C ultrasonic diagnostic instruments. Initially, the images had a resolution of $768\times 576$ pixels, but we resized them to $224\times 224$ pixels grayscale format to enhance the training efficiency of the neural network. When dividing the dataset into training and validation sets, we ensured that images from a single subject were grouped together within the same subset, thus eliminating any overlap between the training and validation images. Considering the imbalance in the number of positive and negative subjects, we applied image augmentation techniques, such as rotations (±15°) and shifts (15%), exclusively to the images from positive subjects within the training sets. This approach aimed to mitigate biases in the dataset during the training process and promote more balanced learning.

Our external test dataset included 3150 subjects from the Army 954 Hospital registry, spanning from January 2015 to November 2020. This dataset comprised 7711 images from 242 PAH-diagnosed subjects and 2908 normal subjects. At least one A4C or PLAX image was available for each individual subject. The studies were carried out using GE Vivid e8 with the 3.3-MHz M5Sc cardiac probe and Siemens ACUSON S2000 with the 4.5-MHz 4P1 probe. The images were $720\times 576$ pixels and were resized to $224\times 224$ pixel greyscale images. This dataset was employed to evaluate the generalization ability of the Classification Module.

View classification

Utilizing a dataset of 6238 manually labeled echocardiographic images randomly selected from the two datasets, our model achieved a training accuracy of 94.4% and a test accuracy of 92.8%. The clustering results of the top layer features for different viewpoints are presented in Fig. 2, using the popular t-SNE. These results clearly demonstrate the distinct separation of A4C and PLAX views from other perspectives. Due to the challenges in identifying the aortic valve (AV) in some images, the distinction between A4C and A5C at the boundary is somewhat difficult, reducing the classification accuracy of A4C. However, the image-level classification result indicates that the view classification accuracy is decent and meets the requirements of the subsequent processes.

Fig. 2.

T-SNE visualization of view classification. Each dot in the figure represents an echocardiography image, which is the 2-dimensional space representation of the top-layer features of the convolutional neural network. Different colors represent different viewpoint categories. It illustrates successful grouping of test images corresponding to 6 different echocardiographic views. Abbreviations: A4C apical 4 chamber, PLAX parasternal long axis, PSAX parasternal short axis basal, A3C apical 3 chamber, and A5C apical 5 chamber.

Cardiac and chambers segmentation result

In the Segmentation Module experiment, we employed a widely used metric, the Dice Similarity Coefficient (DSC), to assess the segmentation performance.

$$\begin{array}{cc}\hfill & \mathrm{DSC}=\frac{2\Vert X_A\cap X_B\Vert }{\Vert X_A\Vert +\Vert X_B\Vert }=\frac{2N_{\mathrm{TP}}}{2N_{\mathrm{TP}}+N_{\mathrm{FP}}+N_{\mathrm{FN}}},\hfill \\ \hfill & \mathrm{Recall}=\frac{N_{\mathrm{TP}}}{N_{\mathrm{TP}}+N_{\mathrm{FN}}}\text{.}\hfill \end{array}$$ (6)

in which $X_A$ and $X_B$ represent the automatically segmented region and manually annotated one respectively; and N denotes the pixel number with subscripted T/F indicating the pixel is correctly/incorrectly predicted and subscripted P/N referring to whether the pixel is positive/negative.

The cardiac segmentation results achieved 99.5% DSC. A comparison between manual cardiac annotation and U-Net based cardiac segmentation is shown in Fig. 3, indicating the high degree of consistency between our cardiac segmentation and the ground truth. This robust agreement lends strong support to the accuracy of our subsequent analysis.

Fig. 3.

Manual and AI-based segmentation of cardiac regions and chambers in echocardiographic images of four subjects. The first two subjects are A4C view samples while the latter two are PLAX. The original echocardiographic images are showed in the first column. The second and third columns are the manual and AI segmented cardiac regions respectively. The fourth and fifth columns show the manual and AI segmented chambers.

For the segmentation of heart chambers, we attained DSC values of 90.34% for A4C and 95.28% for PLAX, as shown in Fig. 3. The displayed images in the figure include both A4C and PLAX views. Notably, in the PLAX view, the right atrium (RA) chamber is absent, indicated by the corresponding mask matrix $M^{\mathrm{RA}}=0$. Thus the chamber segmentation model provides the mask matrices $M_k^{\mathrm{RV}},M_k^{\mathrm{LV}},M_k^{\mathrm{RA}}$ and $M_k^{\mathrm{LA}}$ for RV, LV, RA and LA respectively. As is known in our method, the heart chamber segmentation was utilized to derive the attention vector of heart chambers on PAH identification. The chambers segmentation inaccuracy might lead to a great variation in the absolute weights $W_k^{\mathrm{RV}},W_k^{\mathrm{LV}},W_k^{\mathrm{RA}}$ and $W_k^{\mathrm{LA}}$ for the $k$-th image. But this inaccuracy was averaged for each chamber and thus their relative weights were less affected. More importantly, the relative weights of many images were counted and normalized as ${\stackrel{\sim }{w}}_{\mathrm{RV}},{\stackrel{\sim }{w}}_{\mathrm{LV}},{\stackrel{\sim }{w}}_{\mathrm{RA}}$ and ${\stackrel{\sim }{w}}_{\mathrm{LA}}$. This small segmentation inaccuracy finally has very little effect on the attention vector.

Attention vector of chambers

Fig. 4(a) visualizes the discriminative features, namely the target heat map, used for PAH classification. The original heat map generated by Grad-CAM was of dimensions $14\times 14$, and we resized it to $224\times 224$. We initially quantified the occurrences of intersections between each segmented heart chamber and the Grad-CAM heat map. The proportions of these intersections are shown in Fig. 4(a)(c) for A4C and PLAX views, respectively. For A4C, RV and LV contribute to 37% and 35% respectively, while PLAX primarily focused on RV, accounting for 88% of the proportion. Generally, RV and LV required more attention. Furthermore, the specific example in Fig. 4(a) shows although the Grad-CAM heat map intersects with RV, LV and RA, the extent of intersection and the significance of the corresponding pixels are not equal. As a result, they should not be treated uniformly. Our proposed Eq. (1) accounts for these differences in pixel values and provides a more accurate quantification of the influence of different chambers on PAH diagnosis. The experimental outcomes of the ${\ell }_1$-normalized attention vectors Eq. (3) for A4C and PLAX views were [0.468 0.401 0.127 0.004]^T and [0.937 0.052 0 0.011]^T, respectively. These findings indicated that RV and LV captured the majority of attention. In the case of A4C view, RV and LV collectively received 0.869 attention, while for PLAX view, they accumulated 0.989 attention. Fortunately, these experimental results aligned with clinical expertise, indicating that PAH leads to RV enlargement, subsequently impacting LV. Moreover, slight structural alterations occur in other chambers at distinct phases of PAH. For instance, the left ventricular septum may exhibit arching at the advanced stage of PAH. Therefore, our proposed chamber attention mechanism based on Grad-CAM gave a proper quantitative description of the influence of different chambers on the diagnosis of PAH, which might even reveal some potential diagnostic information that was clinically ignored.

Fig. 4.

(a) Visualization of the attention mechanism based on Grad-CAM to differentiate PAH from normal in cardiac regions. The highlighted areas indicated by red arrows are discriminative features for identification of PAH. The Grad-CAM of A4C and PLAX viewpoints are shown in the top and bottom of the figure. (b) (c) The proportion of the number of times the cardiac chamber was covered by the Grad-CAM heat map in A4C and PLAX viewpoints. Different colors in the figure represent different chambers.

Classification Module

Echocardiography reconstruction

The attention vectors obtained for the chambers played a pivotal role in reconstructing the training dataset for each view using Eq. (4). This process is illustrated in Fig. 5 through a specific example involving the reconstruction of A4C echocardiograms. In order to generate the reconstructed $R_k$, several components were needed: the mask matrices $M_k^{\mathrm{RV}},M_k^{\mathrm{LV}},M_k^{\mathrm{RA}}$, and $M_k^{\mathrm{LA}}$ from chamber segmentation, as well as the complement $I_k$ of the original segmented cardiac regions ${\stackrel{\sim }{I}}_k$. The results presented in Fig. 5 clearly demonstrate that the reconstructed $R_k$ significantly improved the recognition of RV and LV, with attention scores of 0.468 and 0.401 respectively. However, the minimal attention weight of 0.004 assigned to LA made it notably challenging to distinguish from its surroundings without careful examination. When these reconstructed images were used to train a ResNet50 classifier, the convolutional layers effectively extracted distinguishable features from RV’s and LV’s structures due to their distinct differentiation from the surrounding regions. In contrast, RA and especially LA faced challenges, as the convolutional layers were less likely to highlight their features given their merging with the surrounding context. Furthermore, Fig. 5 reveals that the reconstructed images generally retained the contours of the heart chambers, a guarantee provided by Eq. (4). The clear boundaries between RV, LV, and RA were maintained due to their distinct contours after reconstruction. However, the lower attention weight assigned to LA resulted in a less distinct contour in the reconstructed images, even though its presence was still preserved. Overall, the attention vectors and the reconstructed dataset enhanced the focus of the convolutional neural network on extracting features closely related to the identification of PAH.

Fig. 5.

Schematic of the process for obtaining the reconstituted $R_k$.

Internal and external datasets

We proceeded to train ResNet50 classifiers on the reconstructed A4C and PLAX echocardiographic images respectively. For the sake of comparison, we also employed four other deep learning models (ResNet50, Inceptionv3, Xception, DenseNet-121) without incorporating the attention mechanism, keeping the cardiac segmentation methodology consistent across all models. The image-level training and validation performances can be observed in Fig. 6(a)(b)(d)(e) respectively. Our CAN model demonstrated distinct advantages, converging more rapidly and achieving higher accuracy ($92.68\%$ and $94.28\%$ for A4C and PLAX respectively) compared with other models in both A4C and PLAX viewpoints. A more comprehensive analysis of the internal validation dataset is presented in Table 1, which includes metrics like Recall, F1-measure, and AUC. It is worth noting that ”ResNet50” specifically indicates the utilization of the ResNet50 network with the segmented cardiac images directly as input, while ”CAN” corresponds to the application of the ResNet50 network with the segmented cardiac images as input after they have undergone reconstruction via the Chamber Attention Module. The results of the ablation study demonstrated a notable enhancement in performance with the incorporation of the Chamber Attention Module, observed across both the training and testing datasets.

Fig. 6.

Training and testing performance of five models (ResNet-50, Inceptionv3, Xception, DenseNet-121, and the proposed CAN) for PAH identification in A4C and PLAX viewpoints separately. (a)(b)(d)(e) For two separate viewpoints, the training and validation accuracy of five models on the internal dataset is plotted vs number of epochs separately. Different models are represented by different colors. The CAN model converges faster and has higher accuracy than other models in both viewpoints. (c)(f) ROC curves of five models in two separate viewpoints on the external dataset.

Table 1.

Image-level performance of the five models for the PAH identification task on the validation set and the independent test set at two viewpoints.

Viewpoints	Models	Internal Validation				External Test
		Accuracy($\%$)	Recall($\%$)	F1-measure($\%$)	AUC	Accuracy($\%$)	Recall($\%$)	F1-measure($\%$)	AUC
A4C	ResNet50	86.51	88.54	88.76	0.942	61.48	10.37	18.48	0.665
	Inceptionv3	89.53	84.49	90.65	0.967	66.41	21.01	34.50	0.813
	Xception	89.67	93.32	91.57	0.964	63.38	16.22	27.17	0.784
	Densenet121	91.10	91.17	92.49	0.962	64.84	60.64	59.22	0.692
	CAN(Ours)	92.68	93.79	93.91	0.979	82.53	71.54	77.52	0.862
PLAX	ResNet50	83.12	86.05	83.09	0.909	50.98	62.39	47.50	0.547
	Inceptionv3	90.56	91.69	90.35	0.967	72.12	34.69	46.94	0.754
	Xception	91.27	91.69	91.02	0.967	69.43	27.70	39.18	0.674
	Densenet121	89.13	88.72	88.72	0.960	65.60	5.25	9.78	0.688
	CAN(Ours)	94.28	95.55	94.15	0.985	83.32	60.35	72.00	0.856

Given the potential variations introduced by different echocardiography equipment and parameters, we proceeded to conduct experiments on an external test dataset to validate the generalization capabilities of our model. For all subjects winthin the external dataset, we achieved image-level accuracies of $82.53\%$ and $83.32\%$ for A4C and PLAX respectively. These results significantly outperformed the other four models. Furthermore, our CAN model exhibited higher F1-measure and AUC values. Additional detailed results for our CAN model and the other four models on the external test dataset are presented in Table 1 and Fig. 6(c)(f). It is worth noting that compared with the plain ResNet without attention mechanism, our attention-based CAN model significantly enhanced the performance across both internal dataset and external dataset. This experimental validation underscored the efficacy of our proposed chamber attention mechanism in effectively capturing the diverse influences of different cardiac chambers on PAH diagnosis and accurately embedding these attention distinctions in the reconstructed images.

In cases where each subject may have multiple cardiac images for A4C and PLAX, we adopted a straightforward approach of calculating the average of their individual image-level results to derive the corresponding view-level outcomes. As highlighted in Table 2, the view-level accuracy surpassed the image-level accuracy ($86.9\%⩾82.53\%$ for A4C and $85.5\%⩾83.32\%$ for PLAX) for the external test dataset. Further analysis of our experimental outcomes at both image-level and view-level unveiled that diverse classification outcomes could emerge from multiple cardiac images of A4C or PLAX within a subject. However, the instances of correct outcomes slightly outweighed the incorrect ones, thus contributing to an enhanced accuracy at the view-level in comparison to the image-level.

Table 2.

View-level (first two lines) and individual-level (third line) performances on the internal validation set and the external test set, where ACC stands for accuracy, UNC represents uncertainty rate, PPV represents the true positive rate, and NPV represents is true negative rate.

Level	View	Internal Validation				External Test
		ACC	UNC	PPV	NPV	ACC	UNC	PPV	NPV
View-level	A4C	0.932	\	0.909	0.956	0.869	\	0.836	0.882
	PLAX	0.957	\	0.980	0.937	0.855	\	0.809	0.872
Individual-level	A4C&PLAX	0.895	0.099	1.00	0.987	0.793	0.138	0.976	0.903

We further merged the view-level results to obtain the conclusions at the individual level. Adhering to the prescribed voting strategy, we considered the individual-level outcomes acceptable only when the two view-level (i.e., A4C and PLAX) results agree. The view-level outcomes can be analogously characterized as decisions rendered by two distinct individual experts. In contrast the voting strategy combines the opinions of these two experts to obtain the individual-level conclusions. Intuitively, when both experts individually have elevated PPV values, the jointly determined result is expected to exhibit an augmented PPV. A similar rationale applies to NPV. The view-level and individual-level findings in Table 2 confirm this expectation. Of particular significance is the observation that within the external dataset, A4C yielded a PPV of $83.6\%$, PLAX exhibited $80.9\%$, while subsequent integration of the individual-level results yielded an impressive $97.6\%$ PPV, thereby notably surpassing the view-level outcomes. It’s essential to underscore that the UNC does not indicate an error rate in our results. Rather, it serves as an indicator for further diagnosis when the outcomes are deemed controversial. More individual-level results are presented in Table 2. In essence, our approach involved enhancing the reliability of the classification results by making a conscious trade-off with the overall accuracy. This strategic compromise was aimed at bolstering the process of echocardiography identification, especially in scenarios where high security requirements are of paramount importance.

Comparison to other PAH studies

To comprehensively evaluate the effectiveness of our approach, we conducted a thorough comparison with the Echocv model [21] for PAH classification. To ensure a fair and equitable assessment, we employed the same dataset that was utilized by our CAN model. For the purpose of facilitating a direct comparison between the two methods, we have presented the performance metrics of both the Echocv model and our CAN model for the A4C view in both the internal validation and external test sets. These results are summarized in Table 3. Through this comparison, the clear superiority of our proposed method becomes evident. It is noteworthy that across the internal validation set, our method consistently outperforms Echocv in terms of various performance metrics, including accuracy, recall, F1-score, and AUC. Furthermore, despite the slightly lower recall observed for our CAN model on the external test dataset, it still exhibits significantly higher accuracy, precision, and specificity values when compared to Echocv. This contrast signifies that our method is capable of achieving a lower false positive rate. In addition, the higher AUC value achieved by our model underscores its enhanced generalization capability.

Table 3.

Image-level performance of Echocv and CAN for the PAH identification task on the validation set and the independent test set at A4C viewpoint.

Set	Models	Accuracy($\%$)	Recall($\%$)	Precision($\%$)	Specificity($\%$)	F1-measure($\%$)	AUC
Internal Validation	Echocv	90.96	93.08	91.98	87.77	92.53	0.971
	CAN(Ours)	92.68	93.79	94.02	91.01	93.91	0.979
External Test	Echocv	58.01	90.96	50.07	34.04	64.59	0.795
	CAN(Ours)	82.53	71.54	84.59	90.52	77.52	0.862

Discussion

Echocardiography has been shown to be an effective diagnostic tool for PAH. However, interpreting echocardiography results requires extensive expertise and training, making it difficult for primary care and rural settings, especially those located in highland areas. To address this issue, this study proposed a fully automated PAH diagnostic pipeline, the CAN Model, based on echocardiography, which significantly reduced the need for continuous involvement of a specialized cardiologist or skilled sonographer at every stage of the diagnostic process. The CAN Model achieved a view-level accuracy of 93.2% and 95.7% for A4C and PLAX, respectively, on the internal validation datasets using fivefold cross-validation. Moreover, the model’s generalization ability was confirmed through evaluation on an external test dataset, with A4C view-level accuracy of 86.9% and 85.5% for PLAX. The experiment demonstrated that our model outperformed four other models lacking attention mechanism. Furthermore, we fused view-level results with a voting strategy to obtain individual-level classification results. While this strategy led to a slight reduction in overall accuracy, it substantially improved the PPV and NPV, indicating that our classification results were more reliable and robust.

Our dataset, comprising over 17,000 subjects, played a central role in model training, validation, and testing. The substantial scale of this dataset underscores the credibility of our models. However, like any real-world dataset, ours also has some certain limitations. Firstly, the selection bias might occur due to the nature of the clinical data collection process. Secondly, the data might be influenced by inherent noise and variability present in medical records. To address these concerns, we employed various data preprocessing techniques to mitigate the potential biases and limitations. These techniques included defining specific subject inclusion criteria and implementing data augmentation strategies. Besides, we took steps to ensure the reliability of our results by establishing a separate testing dataset, allowing us to assess effectively the generalization ability of our model.

To enhance the interpretability of our model’s predictions, we proposed a Chamber Attention Module based on Grad-CAM. This module enabled us to assess the impact of individual heart chambers on the identification of PAH. This is particularly significant as cardiologists usually focus on the right ventricle (RV) structure, which undergoes enlargement during the progression of PAH. With the incorporation of our Attention Module, we can offer a more precise and quantitative account of the contribution of each chamber to the final diagnosis, which can assist in the clinical decision-making process. Overall, our attention mechanism significantly elevates the model’s interpretability, delivering valuable insights to medical professionals.

The experimental results demonstrated that the attention vectors for A4C and PLAX were [0.468 0.401 0.127 0.004]^T and [0.937 0.052 0 0.011]^T, respectively. These values are in line with clinical expertise and enhance the interpretability of our entire model. By providing a quantitative description, we can obtain a clear understanding of the significance of different heart chambers in PAH diagnosis. Furthermore, it may uncover some diagnostic information that was typically overlooked clinically, such as the small weights (0.08 and 0.03) of the LA in A4C and PLAX, respectively. This ability to provide detailed insights may aid medical professionals in making more informed clinical decisions.

Furthermore, we proposed a reconstruction formula (4) to incorporate the attention vector obtained by the Chamber Attention Module into the reconstructed images. This formula enabled us to highlight the heart chamber with a large attention weight in the reconstructed images, thereby aiding the Classification Module in focusing more on the structure of that particular chamber. On the other hand, the chamber with a lower attention weight will be less pronounced in the reconstructed images, and its influence will be suppressed. Our experimental results demonstrated that our CAN model significantly outperformed the mere-ResNet model in all performance comparisons, which confirmed the effectiveness of our chamber attention mechanism in improving the performance of the Classification Module.

Notable, too, the attention module and quantitative analysis of the importance of different cardiac chambers could have broader applications beyond PAH. For other cardiac diseases or conditions, if clinical experience indicates that it results in visible organ-level alterations across multiple areas, with varying degrees of relevance to diagnostic outcomes, our approach using an attention module to highlight important features could potentially be explored and adapted. However, further research and validation would be necessary to ascertain the feasibility and efficacy of applying our model to these diagnostic tasks. We will consider exploring these broader applications in future studies.

We also introduced notations for the positive predictive value (PPV) and negative predictive value (NPV) in the voting strategy. According to the strategy, the outcome of the Classification Module has two states: the determined state and the uncertain state. If the classification outcome is uncertain, it means that our model fails to determine whether a subject suffers from PAH, and a further precise diagnosis is needed. This failure is attributed to the inaccuracy of the model. The determined state includes two sub-states, positive and negative. For positive outcomes, the proportion of subjects that are correctly predicted positive is used as the PPV in the policy. NPV has a similar definition. The experimental results show that the PPV and NPV of the uncontroversial results by the Classification Module have significantly increased, thereby improving the precision of the deterministic results of the model.

The CAN Model proposed in our study has some limitations that need to be addressed. Firstly, while positive echocardiographic results are necessary, they are not always sufficient for a final PAH diagnosis. In our study, we did not consider patients’ medical history and clinical manifestations [29], which could have enhanced the accuracy of the model. Secondly, there were significant differences in the sex distribution and mean age between PAH-positive and PAH-negative participants, which may have influenced the results of the deep learning and statistical analysis. This suggests that incorporating multiple demographic variables and echocardiographic images during model training could improve the test performance [30]. Finally, the outcome of PAH is clinically significant, but due to limitations in the dataset, our model could not predict the severity or progression of the disease. Future studies could focus on developing another prediction model that can complement the clinical consequences of PAH [31].

Conclusion

The proposed CAN model demonstrated superior performance compared to commonly used classification models on both the internal validation dataset and the external test dataset. The novel Chamber Attention Module introduced in the model enabled the network to focus on the chambers with higher attention and significantly improved the detection of PAH. Additionally, the attention vector obtained by the module was consistent with clinical experience, enhancing the interpretability of the classification results.

The clinical significance of CAN lies in its potential to enhance the diagnostic accuracy and efficiency for clinicians, particularly in challenging scenarios like remote and high-altitude regions where specialized expertise may be limited.

Our CAN model offers several practical implications for clinical practice:

(1) Efficient and Automated Diagnosis: The model’s automated nature allows for quicker and more efficient diagnosis of PAH. Clinicians can rely on the model’s preliminary results, which are further confirmed by clinical experts, streamlining the diagnostic process. This is particularly advantageous in regions with limited medical resources, where clinicians often face heavy caseloads.

(2) Reduced Workload: By assisting in initial diagnosis, the model can significantly reduce the workload on clinical experts. This is particularly advantageous in regions with limited medical resources, where clinicians often face heavy caseloads.

(3) Remote Healthcare Delivery: Our model’s integration into a software system could facilitate remote diagnosis. In areas with limited access to specialized doctors, nurses could conduct initial ultrasound examinations, and the model could provide preliminary results. For ambiguous cases, remote intervention by experts could provide accurate final diagnoses.

(4) Enhanced Diagnostic Confidence: The model’s attention mechanism highlights crucial features, allowing clinicians to focus on areas with higher diagnostic importance. This targeted approach increases diagnostic confidence and precision.

(5) Broad Applicability: The attention mechanism could be adapted for other cardiac diseases or conditions where specific structures play a critical role. This adaptability could lead to similar efficiency gains in diagnosing other cardiac conditions.

Incorporating our model into clinical practice would involve seamlessly integrating it into existing healthcare systems. Clinicians could upload echocardiograms to the software, receive preliminary results promptly, and consult with remote experts if needed. This integration aligns with modern telemedicine trends and has the potential to substantially improve patient outcomes by ensuring timely and accurate diagnosis and treatment for PAH.

Author contributions

Dezhi Sun, Wei Zhou and Yue Gao conceived of the research study. Dezhi Sun, Pan Shen, Zhijie Bai and Yangyi Hu developed and evaluated the deep learning models of the study. Yunming Li, Xi Chen, Chengcai Lai, Xianglin Tang, Yihao Wang and Bo Kang collected and preprocessed the echocardiographic data of the study. Xianbiao Yu, Rui Wang and Lina Guan labeled the echocardiographic images of the study. Dezhi Sun, Yangyi Hu, Zhexin Ni and Ningning Wang wrote the manuscript.

Compliance with ethics requirements

This study was approved by the Research Ethics Commission of Beijing Institute of Radiation Medicine (AF/SC-08/02.153). We certify that the study was performed in accordance with the 1964 Declaration of Helsinki and later amendments.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Supplementary material

The following are the Supplementary data to this article: