Hybrid vision transformer framework for congenital heart disease diagnosis

Abstract

Congenital heart disease is one of the most common birth defects and has been a source of significant morbidity and mortality among infants. Early diagnosis allows for early intervention and better patient outcomes. Traditional methods are effective yet have limitations such as high costs, dependency on expert knowledge, and noise. Recent developments in deep learning showcase the potential to address these limitations and automate diagnosis with high accuracies. This work proposes the use of a hybrid vision transformer (H_ViTs) model for the diagnosis of congenital heart disease based on heart sounds. The proposed vision transformer-based architecture leverages both local feature extraction with convolutional neural networks and global pattern modeling by vision transformer. Thus, the model is able to capture local dependencies with convolutional neural networks and global dependencies with vision transformers in heart sounds’ spectrogram representations. This study utilized a recently released open-source ZCHSound dataset of children’s pediatric heart sound recordings. To evaluate this model, we use accuracy, precision, recall, F-1 measure, and the confusion matrix as evaluation measures. Our proposed model obtained better results as compared to previously reported results on the same dataset. However, we notice comparatively poor results for multi-class classification, particularly for classes with limited data. In summary, the proposed hybrid model effectively detects congenital heart disease, though additional efforts are needed to improve robustness against class imbalance and noise.

Introduction

Reports show that congenital heart disease (CHD) is the most prevalent congenital defect worldwide. It remains a significant contributor to neonatal mortality. CHD is a broad spectrum of anatomical abnormalities affecting the heart or major vessels¹. According to the recent reports of World Health Organization (WHO), approximately 60,000 newborns are diagnosed with CHD annually. In 2017, the global burden of CHD was estimated to exceed 12 million individuals, corresponding to an age-standardized prevalence of 170.6 cases per 100,000 population¹. The prevalence is particularly elevated in low- and lower-middle-income countries in Africa and Asia, where incidence rates surpass 30 per 1,000 live births¹. Figure 1 shows survival rates to adolescence for infants born with CHD in high-income and low-income countries. The Figure 1 also shows that survival to adolescence is sufficiently higher in high-income countries, primarily due to improved access to pediatric cardiac surgery and healthcare facilities. These disparities are often attributed to limitations in initial prenatal screening, and the availability of some advanced diagnostic tools, underscoring the urgent need for scalable diagnostic problems and its solutions.

Fig. 1.

A comparison for babies born with critical CHD in high-income and low-income countries. Survival to adolescence is much higher in high-income countries¹.

CHD is a leading cause of mortality that is associated with congenital disorders. According to recent reports and research, approximately 4.5% of affected babies before birth die in utero, and 21% do not survive beyond the perinatal period². Similarly, among survivors, the risk of long-term complications is substantial; children with CHD have a nine-fold higher likelihood of neuro developmental or functional impairments¹. Moreover, it is also noticed that CHD remain asymptomatic for extended periods, which impedes timely detection and intervention². These complex and serious problems highlight the necessity for improved early screening protocols, as prompt diagnosis significantly enhances treatment planning. This also limits disease progression, and prevents life-threatening outcomes³.

Some modalities that are utilized in clinical practice for CHD screening, including echocardiography (Echo), electrocardiography (ECG), pulse oximetry (POX), chest X-ray imaging, and phonocardiography (PCG)^4–6. Research shows that Echo is the gold standard for diagnosis; its accuracy depends heavily on operator expertise and equipment quality, and its availability is limited in resource-constrained settings⁴. Another modality that is used frequently in diagonistics is ECG. It is affordable and widely accessible but it mostly lacks sufficient sensitivity to detect complicated structural abnormalities⁷. Similarly a cost effective approach is provided by POX, yet it is typically used in conjunction with other methods to reduce false negatives^8,9. Chest X-ray imaging remains a primary tool for general cardiac assessment, but it offers limited structural information and is subject to interpretive variability¹⁰. PCG is similarly economical and portable; however, its diagnostic accuracy depends heavily on clinician expertise, and recordings are particularly susceptible to noise¹¹.

Multiple diagnostic modalities are being employed in clinical practice for CHD screening, including echocardiography (Echo), electrocardiography (ECG), pulse oximetry (POX), chest X-ray imaging, and phonocardiography (PCG)^4–6. Echo, while regarded as the gold standard, exhibits accuracy that is significantly influenced by the operator’s proficiency and the quality of the equipment, with restricted accessibility in resource-constrained areas⁴. It is also noticed that ECG is most cost-effective and readily available modality, although it frequently lacks adequate sensitivity for intricate structural anomalies⁷. POX is an economical method for identifying significant CHD but this modality is employed alongside other modalities to mitigate false negatives^8,9. X-ray continues to serve as a primary instrument for general cardiac evaluation. However, it offers limited structural detail and is susceptible to interpretive variability¹⁰. Similalry, PCG is comparably economical and portable; nevertheless, its diagnostic precision is significantly reliant on clinician expertise, and recordings are particularly vulnerable to noise¹¹.

Despite the benefits of existing diagnostic approaches, no single modality provides a fully reliable, accessible, and operator-independent solution. This serious limitation is especially pronounced in resource-limited regions, where shortages of trained pediatric cardiologists and advanced imaging equipment frequently lead to delayed or missed diagnoses. We believe that Deep learning (DL) models can emerge as valuable tools for improving clinical decision-making by using automating feature extraction and reducing operator dependence. Among modern DL methods, vision transformers (ViTs) demonstrate outstanding performance in image, signal, and time-frequency analysis due to their ability to capture global context, and efficiently process structured data. Recent studies indicate that ViT-based models perform better than convolutional neural networks (CNNs) in medical imaging applications and offer greater resilience to noise, variability, and class imbalance as well.

A major gap exists in the use of AI for the diagnosis and treatment of patients with CHDs across their lifespan. The utilization of AI in CHD has been restricted by the lack of CHD-specific datasets with labels for model training, the complicated modelling demands caused by diverse clinical characteristics and age-related pathophysiological changes, and the fragmented nature of data within center-specific repositories. Furthermore, at baseline, information regarding particular rare forms of CHDs is limited, requiring multidisciplinary collaboration to compile sufficient datasets. Substantial deficiencies in clinical training, knowledge, experience, and familiarity with AI exist. CHD continues to be an inadequately explored area for both cardiovascular and ML experts. The region remains unexplored due to various intricate factors. Hospitals generate significant quantities of data, encompassing clinical information and data from electronic health records. The continuous progression of big data is crucial in healthcare management, as it enables the analysis of large datasets to improve illness treatment, determine appropriate therapeutic dosages, and generate predictions. Healthcare produces significant data; nevertheless, a considerable portion remains underused due to challenges in storing, organizing, and understanding complex datasets that often involve multidimensional and nonlinear relationships among variables.

This study introduces a ViT–based DL framework for diagnosis and classification of CHD. PCG data are well-suited for such applicationa because it contain clinically relevant physiological cues that are associated with structural heart abnormalities. These can be acquired through non-invasive, portable, and cost-effective methods. These attributes render PCGs particularly valuable in resource-limited settings where access to conventional diagnostic modalities is limited. Building on these strengths, an H-ViTs framework for CHD classification is developed and its performance is systematically evaluated against existing methods.

Major contributions of the paper are three fold:

• In contrast to many existing hybrid models that utilize computationally intensive CNN backbones such as ResNet or DenseNet, our proposed hybrid model employs a depthwise separable CNN that is more optimized for heart-sound spectrograms. Our proposed architecture substantially reduces the parameter count as well as computational cost while maintaining the ability to capture fine-grained local temporal and spectral patterns.

• Unlike traditional ViT-based models or hybrid methods that tokenize inputs using fixed-size patches, our proposed model generates transformer tokens from semantically meaningful CNN feature maps. This architectural ensures that the transformer processes representations already enriched with clinically relevant local structures.

• To the best of our knowledge, this work is among the first to introduce a CNN–ViT hybrid specifically designed and evaluated for CHD diagnosis using expert-annotated pediatric PCG dataset. Existing transformer-based PCG studies typically operate on some raw signals with attention mechanisms. In contrast, the proposed model integrates spectrogram-based CNN feature learning and transformer-based global modeling within a unified, end-to-end architecture.

Related works

CHD can be diagnosed using different methods, including POX, magnetic resonance imaging (MRI), ECG, Echo. Each modality provides unique advantages, from cost-effectiveness in POX to detailed anatomical structures in CT scans. On the other hand, some diagnosis types possess some challenges and disadvantages, such as the need for expert knowledge, cost, or ineffectiveness in CHD type classification. In this part of the paper, various modalities that address CHD recognition using machine learning (ML) are addressed.

CT scan

Ecabert et al.¹² introduced one of the earliest model-based methods for heart segmentation in CT scans. It comprised a 3D Generalized Hough Transform for initial localization, piecewise affine transformation, and deformable adaptation. This method achieved a mean surface-to-surface error of 0.82 mm, which showed high accuracy for segmenting CT scans of a heart into its major components, namely four chambers, myocardium, and major vessels. Later, Khan et al.² presented the Cardiac Deep Learning Model (CDLM) that employed a 3D U-Net architecture with graph matching for vessel classification. Overall, CDLM showed effectiveness in heart segmentation into chambers, blood pool, and vessels. CDLM demonstrated notable accuracy in aortic segmentation with high mean Dice scores across heart structures, surpassing earlier methods. While CT scans may seem promising in CHD diagnosis, collecting data for training and affordability of equipment may suggest that it may not be the most effective way for heart defect classification.

Echocardiograms

Jingjing et al.¹³ explored AI-assisted auscultation (AI-AA) for CHD diagnosis using heart sound recordings. The model used a CNN to classify time-frequency representations of heart sounds and obtained a sensitivity of 97%, a specificity of 89%, and an overall accuracy of 96%. This method aligns closely with traditional auscultation, offering accessible and effective remote CHD screening in certain resource-limited areas. Qiao et al.⁴ introduced a novel model for fetal screening. The model integrated a Multi-Scale Gated Axial-Transformer Network (MSGATNet) along with residual learning to segment and parse the four chambers of the fetal heart in ultrasound images. The proposed method addressed such challenges of fetal screening as movements, receiving precision of 96%, recall of 94%, and overall accuracy of 95%, surpassing 12 state-of-the-art models in fetal CHD diagnosis.

Electrocardiograms

Chen et al.⁷ developed CHDdECG, a DL model that integrates raw ECG waveforms with wavelet-transformed and human-concept features. The model achieved ROC-AUC scores of 0.915 (for internal tests) and 0.917/0.907 (external test sets). This model got a specificity of 0.881 on the internal test set, which might outperform traditional diagnostic approaches. Similarly, Du et al.¹⁴ proposed a Residual of Residual (RoR) Network for CHD diagnosis from ECG signals. The RoR model reached an accuracy of 92.45%, a sensitivity of 74.73%, and a specificity of 94.07% using 10-second ECG segments, with potential for increased sensitivity through longer recordings.

Phonocardiograms

Hassanuzzaman et al.¹⁵ proposed a DL model that integrates a one-dimensional CNN with an attention transformer that accepts raw PCG signals for the binary classification of CHD and non-CHD. They utilized a dataset of 484 patient recordings collected in clinical settings. Their model achieved promising results with an accuracy of 92.3%, a sensitivity of 98.3%, a specificity of 83.3%, an F1 score of 93.9 % and an AUC of 0.964.

Chen et al.¹⁶ introduced two lightweight CNN-based models for CHD diagnosis from pediatric PCG data. The first model was dense-block architecture, and the other one was clique-based architecture. Both models were utilizing raw PCG signals. This study used a pediatric heart sound dataset consisting of 528 recordings. The dense-block model showcased superior performance with an accuracy of 96.21%, a specificity of 98.08%, and a precision of 91.67% and 98.77% for CHD and non-CHD cases, respectively. The clique-based model showed worse but still promising performance with accuracy of 91.68%, specificity of 89.17%, sensitivity of 93.40%, and precision of 88.96% respectively. Authors also revealed that both models do not require high computational demands and thus can be used for real-time CHD diagnosis via IoT-enabled healthcare systems.

Yadav et al.¹⁷ proposed an approach for CHD classification from PCG signals by leveraging Fourier Transform and Cepctrum analysis. The method employed down-sampling of recorded signals from 2000 Hz to 1000 Hz for faster processing and bandpass filtering to reduce noise. Then, a Fourier transform was performed for frequency domain analysis. The ratio of the mean and maximum values of the Fourier transform signal was used as the discrimination between CHD and non-CHD cases. Additionally, they used the mean values of cepstrum coefficients extracted from heart sounds. Authors utilized the 2016 PhysioNet/CinC Challenge database with a total of 100 heart sound recordings. A support vector machine (SVM) was then trained and tested on these statistical features. The classifier achieved an accuracy of 95%, a sensitivity of 100 %, and a specificity of 90% respectively.

To summarize, we can conclude that CHD recognition using ML/DL is not a very explored area by ML and DL experts. A summary of the different methods reported so far is presented in Table 1. Table 1 summarizes prior ML/DL approaches for CHD diagnosis across multiple modalities, including CT/MRI, ultrasound, ECG, and PCG. While imaging-based methods achieve high accuracy, their dependence on expensive equipment and expert interpretation limits their use in resource-constrained settings. Signal-based approaches, particularly those using PCG recordings, offer a more accessible alternative and have benefited from recent advances in CNNs and transformer-based models. The availability of large, expert-annotated PCG datasets has further accelerated progress, highlighting PCG-based DL frameworks as a promising pathway toward scalable and automated early CHD screening.

Table 1.

Summary of ML/DL-based methods used for CHD recognition across different modalities.

Modality	Method Used	Key Idea/Description
CT/MRI	3D Model-Based Segmentation	Shape priors + deformable adaptation for anatomical partitioning of chambers and vessels.
CT/MRI	3D U-Net + Graph/Shape Constraints	Volumetric U-Net refined with anatomical graph constraints for improved boundary localization.
Ultrasound/Fetal Imaging	Multi-Scale Transformer Networks	Axial transformer with residual learning for fetal heart chamber parsing under motion and noise.
US	Deep Chamber Segmentation	Robust fetal heart chamber segmentation for early CHD diagnosis.
ECG	Waveform + Wavelet Features	Hybrid raw ECG + wavelet-domain representation for robust discriminative modeling.
ECG	Residual CNN/RoR Network	RoR architecture to capture long-range dependencies in ECG morphology.
PCG	CNN on Time–Frequency Inputs	CNN applied to spectrogram-like PCG representations for CHD vs. normal classification.
PCG	CNN + Transformer Hybrids	Local CNN feature extraction + transformer-based long-range dependency modeling.
PCG	Lightweight CNN Models	Dense/clique CNN architectures optimized for real-time screening on low-power devices.
PCG	FFT + Cepstrum + SVM	Traditional pipeline using Fourier and cepstral features for CHD discrimination.
PCG	Large-Scale PCG Collections	1259-subject pediatric PCG dataset (2020–2022) labeled by cardiologists; includes baseline ML benchmarks.

Methodology

Data

The performance of any ML-based methodology is significantly contingent upon the dataset used for experimental validation and then assessment. Various diagnostic methods are employed by medical practitioners to diagnose CHDs. The initial evaluation for CHD in infants includes cardiac auscultation, chest radiography, and Echo. Diagnostic procedures such as X-rays, cardiac MRI, dual-source tomography scans, and CT examinations are also used; nonetheless, they are time-consuming, intricate, costly, and necessitate skilled cardiologists.

A summary of the dataset used for CHD is shown in Table 3. From the table it is clear that different modalities are used by various researchers for CHD diagnosis. These modalities range from simple POX to very complex CT scans. Figure 2 shows a summary of how researchers explored these modalities to investigate CHD using ML/DL. Figure 2 shows that Echo is a signal that is mostly used by researchers for CHD recognition.

Table 3.

CHD Datasets Reported in Recent Literature (2020–2024).

Authors	Data Type	Dataset	Year	CHD Types
Pace et al.18	MRI	HVSMR2-0	2024	NOR, ASD, VSD, PDA
Xu et al.19	CT	ImageCHD	2024	16 variants of CHD
PhysioNet-ECG20	ECG	PhysioNet-ECG	2024	NOR, ABNOR
Tan et al.21	Echo	CHDEchoNet	2023	NOR, ASD, VSD
Chen et al.16	ECG	CHDdECG	2023	NOR, ASD, VSD, PDA, TOF
Gerke et al.22	X-Ray	DICOM	2022	NOR, VSD, ASD, PDA
Han et al.23	X-Ray	Chest X-rays	2022	NOR, ABNOR
Cheng et al.24	Echo	Doppler TTE	2022	NOR, ASD, VSD
Nurmaini et al.25	Echo	Seven-CHDs	2022	Seven CHDs
Roberts et al.26	POX	POX	2022	NOR, ABNOR
Arooj et al.27	Non-Clinical	UCI	2022	ASD, VSD, NOR
Liu28	PCG	ZCHSound	2022	NOR, ASD, VSD, PDA, PFA
Wang et al.29	Echo	Multiview Echo	2021	NOR, ASD, VSD
Karimi et al.30	MRI	Complex-CHD	2020	Complex CHDs
Luo et al.31	Non-Clinical	Shanxi	2020	NOR, ABNOR

Fig. 2.

Research papers reported so far for each modality using CHD-ML.

Table 2 provides a structured summary of the class-wise distribution within the ZCHSound dataset utilized in these experiments. The dataset consists of 1,259 pediatric subjects, manually screened by clinicians and categorized into 941 high-quality (clean) and 318 low-quality (noisy) recordings. It contains one normal class and four CHD subtypes: atrial septal defect (ASD), ventricular septal defect (VSD), patent foramen ovale (PFO), and patent ductus arteriosus (PDA). The normal class represents the largest proportion of the dataset (42.3%), while PDA is a pronounced minority class, comprising only 2.5% of samples and appearing infrequently in both recording-quality subsets. The other CHD subclasses have moderate representation and are present in both high- and low-quality recordings. This class imbalance contributes to reduced multi-class classification performance, particularly for minority classes, which experience higher misclassification rates even when weighted random sampling is applied. These results underscore the necessity of explicitly addressing dataset imbalance when interpreting multi-class outcomes and advise caution against over-extrapolation without consideration of the dataset’s inherent characteristics.

Table 2.

Distribution of diagnostic classes in the ZCHSound dataset.

Class	Total Samples (n)	Proportion (%)	High-Quality Subset	Low-Quality Subset
Normal	533	42.3	Majority	Minority
ASD	Moderate	–	Present	Present
VSD	Moderate	–	Present	Present
PFO	Moderate	–	Present	Present
PDA	32	2.5	Very few	Very few
Total	1259	100	941	318

Authors in¹¹ proposed a dataset of PCG data for CHD diagnosis. They present a comprehensive effort to create a large-scale, high-quality, open-source dataset of PCG recordings for research purposes. The dataset used in this study is publicly available at http://zchsound.ncrcch.org.cn/data. Since this is the latest dataset, we also use the same dataset to evaluate our model. The total number of participants who participated for data is 1259. The data was collected from 3 children’s hospitals. Participants’ ages range from newborns up to 14 years. To ensure correct diagnosis labeling, 2 cardiac experts reviewed the heart sounds based on their ECG results, which provide more reliable diagnostic information for decision-making. The study extracted 84 time-domain and frequency-domain features with the usual 60-20-20 dataset split.

All heart sound recordings were taken using a standard ChildCare G-100 smart stethoscope at an 8000 Hz sampling rate, and each recording was 11–30 seconds in duration and stored in standard.wav format. To provide reliability and consistency, all recordings were conducted on subjects in a supine position and underwent a stringent quality control procedure; trained clinicians manually screened the data to form two distinct subsets–a high-quality set of 941 clean recordings (both normal and various cases of CHD such as ASD, VSD, PDA, and PFO and a low-quality set of 318 noisy recordings from neonates. Furthermore, confirmatory cardiac US examinations and professional opinions were employed to strictly allocate diagnostic labels, thus providing an accurate foundation upon which to construct and test wise auscultation algorithms. The gender distribution is adequately balanced for both high-quality and low-quality sets.

Authors used conventional ML with four different classifiers during experiments: Random Forest (RF), SVM, K-Nearest Neighbors (KNN), and AdaBoost. The RF classifier achieved the best results for high-quality heart sound classification, achieving accuracy of 90.3% for Task 1-1 and 93.4% for Task 1–2. For Tasks 2-1 and 2-2, KNN reached accuracies of 62.5% and 55.7% respectively. Table 4 presents a summary of the setting the authors adapted in paper¹¹. For better comparison, we use the same setting and the same protocol.

Table 4.

Task Description Table¹¹.

Task Number	Description
Task 1-1	Binary classification of CHD vs. non-CHD in heart sounds.
Task 1–2	Multi-class classification of non-CHD vs. CHD subtypes in heart sounds.
Task 2-1	Binary classification of CHD vs. non-CHD in low-quality heart sounds.
Task 2-2	Multi-class classification of non-CHD vs. CHD subtypes in low-quality heart sounds.

The paper¹¹ that reported the dataset performed experiments in the following way:

• Task 1 is the binary classification of high-quality heart sound recordings (CHD vs. non-CHD).

• Task 2 is multiclass classification of high-quality audios by CHD types mentioned above. For classification of individual subtypes of CHD (non-CHD vs. CHD subtypes).

• Task 3 is binary classification of low-quality heart sound recordings.

• Task 4 is multiclass classification of low-quality heart sound recordings.

We also adapted the same strategy as adapted in the original paper.

Pre-processing

We preprocessed the heart sounds into Mel spectrograms, which represent spectral features. A Mel spectrogram is a visual representation of sound where the x-axis represents time and the y-axis represents frequency in terms of the Mel scale for frequencies and decibels for amplitude, similar to how humans hear. Mel spectrograms are different from traditional spectrograms and are superior to audio because they preferentially emphasize areas of frequency and intensity in more human-hearing-relevant terms and thus extract more features for DL models. Audio recordings were resampled at 16 kHz, normalized, and divided into fixed-length 5-second segments. Mel spectrograms were calculated with a window size of 1024 samples, a hop length of 512 samples, and 64 Mel frequency bins. The spectrograms were log-scaled and normalized to improve feature representation.

The Mel spectrogram is obtained by transforming a one-dimensional audio signal a(t) into a two-dimensional time–frequency representation that approximates human auditory perception. The signal is first divided into overlapping frames, which can be expressed as

where p[k] denotes a window function, H is the hop size, n is the frame index, and .

The short-time Fourier transform (STFT) of each frame is then computed as

where represents the complex-valued frequency spectrum of the n-th frame. The corresponding power spectrum is given by

To obtain the Mel spectrogram, the power spectrum is projected onto the Mel scale using a Mel filterbank matrix as

where each filter is a triangular weighting function centered on Mel-scaled frequencies defined by

This preprocessing pipeline yields a compact, noise-robust representation that emphasizes perceptually important low-frequency components, making it highly effective. Figure 3 shows the audio signal from the dataset, then converted it into a Mel spectrogram. The Mel spectrogram in Fig. 3 is obtained after applying the preprocessing method discussed.

Fig. 3.

Original PCG recording, after conversion to Mel spectrogram, enhanced version.

Model architecture

The architecture of the model combines CNN features and the ViTs. The model architecture can be seen in Fig. 4. The key modules are detailed below:

Fig. 4.

Architecture of the proposed hybrid vision transformer model.

CNN-based feature extraction

The core idea of the model proposed in this paper is a depthwise separable CNN that serves as the backbone for the extraction of spatial features from heart sound recordings. In this model, the depthwise convolutional stage processes each input channel independently, allowing the network to learn fine-grained local patterns that characterize the temporal and spectral structure of heart sound waveforms. The next step is followed by a pointwise convolution. The convolution integrates information across channels, enabling effective feature mixing while maintaining computational efficiency. Unlike a standard and conventional convolution methods, which applies a full filter across all C input channels and therefore requires parameters for M output channels, the depthwise separable formulation factorizes the operation into two smaller steps with a total parameter count of . This reduction decreases memory usage as well as significantly lowers the number of floating-point operations. Overall, this CNN backbone provides a compact yet expressive representation of the input spectrograms. Details regarding the CNN portion of the full architecture are summarized in Fig. 5.

• Conv2D Layer: The first convolutional layer employs a kernel to capture broad spectral-temporal structures present in the input spectrogram. This operation performs a full convolution across all input channels, producing an output feature map that is subsequently normalized using batch normalization to stabilize training dynamics. A ReLU nonlinearity is then applied to introduce elementwise activation, resulting in the transformation

  

  that forms the initial high-level representation for deeper processing.
• Depthwise separable convolution blocks: Each block consists of a depthwise convolution that filters each input channel independently using spatial kernels, followed by a pointwise convolution that mixes information across channels. This factorization reduces parameter count and computational burden compared to a standard convolution. Formally, for input , the depthwise stage computes

  

  while the subsequent pointwise projection yields

  

  Batch normalization and ReLU activation are applied after each convolutional operation to enhance stability and expressiveness.
• Max pooling layers: Following each depthwise separable block, max pooling layers are used to downsample the feature maps, thereby reducing spatial dimensions while retaining the most salient local activations. Given an intermediate feature map , a pooling operator with stride s produces

  

  effective compression of the resolution and enables deeper layers to focus on progressively higher-level representations.

Fig. 5.

CNN-based feature extraction architecture.

Vision transformer

The CNN architecture produces a structured feature map that captures spatially localized spectral and temporal variations within the spectrogram. This representations retain both fine-scale temporal fluctuations and broader frequency-based patterns characteristic. The resulting feature tensor is then reshaped into a sequence of tokens that actually ensure compatibility with transformer-based processing. The vision transformer module, designed to learn long-range contextual structures, operates on these tokens to model complicated dependencies spanning multiple cardiac cycles and spectral regions. In contrast to CNNs, which use fixed-size kernels and thus maintain a local perspective, the proposed architecture leverages a global receptive field through the self-attention mechanism.

The ViT module processes representations extracted from the CNN backbone using several interconnected components, each contributing to a different aspect of global feature integration and sequence modeling.

• Multi-Head Self-Attention Module (MHSA): This module lies at the core of the transformer encoder, enabling the model to attend to multiple locations within the token sequence simultaneously. Each attention head computes an independent attention distribution, allowing the model to capture distinct semantic, temporal, or spectral relationships. This provides a richer feature representation than a single-head formulation, as multiple subspaces of the embedding dimension can be processed concurrently. The computation across h heads is expressed as

  

  where each head examines the token sequence from a different representational perspective. This parallel multi-attention structure enhances the model’s ability to detect composite heart sound patterns, such as harmonically related murmurs or repeated pathological motifs that may stretch across long temporal windows.
• Residual Connections: Residual pathways are integrated into every transformer block to preserve low-level signal characteristics and ensure stable gradient propagation. They act as identity shortcuts, preventing degradation of learned features as the model deepens. Formally, a residual update can be represented as

  

  where represents a multi-head attention or MLP sublayer. These shortcuts allow the model to retain important CNN-derived information even after multiple layers of nonlinear transformation, resulting in smoother optimization and more robust gradient flow during training.
• Layer Normalization and MLP Blocks: Layer normalization stabilizes the intermediate token distributions by normalizing features along the embedding dimension. This operation effectively reduces internal covariate shift and improves convergence behavior, especially when combined with deep stacked attention layers. The MLP block expands the embedding dimension using a fully connected layer, applies a nonlinear activation (typically GELU), and then projects the dimension back to its original size. This results in a transformation of the form,

  

  which equips the model with a higher-order representational capability. The MLP is crucial for learning nonlinear relationships that self-attention alone cannot fully capture.
• Positional Encodings: Since transformers do not inherently encode positional information, positional encodings are added to the token embeddings to convey their ordering. These encodings capture the temporal and frequency-dependent structure implicit in the CNN-derived feature map. We employ learnable positional embeddings , which are added to the input sequence along with the special classification token:

  

  enabling the transformer to interpret each token in the context of its original spatial and temporal location in the spectrogram. The [CLS] token aggregates high-level information across the entire sequence and serves as the final classification representation.

The overarching CNN-ViT architecture follows the canonical transformer encoder design, where the tokenized CNN feature sequence is processed through several stacked transformer layers. Layer normalization is applied before each major operation (LNS strategy), and each sublayer output is wrapped with a residual connection. The mathematical flow through the encoder layers is given by:

1

2

3

Multi-head attention itself is composed of multiple independent self-attention branches, each learning relationships across different portions of the feature sequence. Each head projects the input tokens into query (Q), key (K), and value (V) matrices using linear transformations, enabling fine-grained relational modeling. The complete multi-head mechanism is expressed as

4

5

6

This formulation enables the transformer to identify intricate associations across the spectrogram, such as temporal dependencies between heartbeats and frequency-bound patterns indicative of pathological conditions.

Classification layer

Following the processing of tokens through the transformer layers, the model uses the [CLS] token output for classification. A fully connected layer processes the final output, projecting the learned features onto the target classes. The layer generates the class predictions based on the learned representation. We can represent this mathematically as

7

where denotes the embedding corresponding to the [CLS] token, and d is the hidden dimension.

8

A fully connected layer followed by a softmax function maps this representation to the target class probabilities as follows:

9

where and are the trainable weights and bias of the classification layer, C is the number of classes, and represents the predicted class probabilities.

10

Weighted random sampling

The imbalanced data is not only for representation of subtypes but also for abnormal-normal samples. The imbalance creates enormous difficulty in ML tasks, affecting performance and generalization ability. To resolve the problem, the weighted random sampling technique was employed. It provides a good approach to assigning sampling probability to minority classes. Particularly, the weighted random sampling assigns a weight to each sample according to the inverse proportion of class frequencies in each class in the training data set.

11

In the above formula, the weight of every sample is set to be inversely proportional to the number of samples in its own class . Because of this, less frequent class samples will have larger weights, thus an increased probability of selection when training. Through these normalized weights, the weighted random sampler perfectly balances the data set such that all the classes are proportionately represented when being trained.

Model training process

We performed two kinds of experiments. Details of experiemnts are as follows:

60-20-20 experimental protocol

In initial phase of experiments, we split the ZCHSound¹¹ dataset using the 60-20-20 random split method, where 60% data is used for training, 20% for validation, and 20% for testing. We use the same settings as in the original paper reporting ZCHSound dataset¹¹.

Let the dataset be represented as

where is an input data, its label, and N is the total number of input samples.

Now the strategy we adapted is: , validation set , and the testing set .

Using a random 60–20–20 split, we have

12

13

14

and

The last step actually ensures that the ML model is trained and tested on disjoint subsets of data.

10-fold cross validation experiments

To better test how robust and general our proposed hybrid framework is, we ran extra 10-fold cross-validation experiments. We split the dataset into ten equal, non-overlapping parts. For each round, nine parts were used for training and validation, and the last part was used for testing. This way, every sample was tested once. We averaged the results from all ten rounds to get the final results. This method gives a more reliable and fair estimate of how the model performs because it does not rely on just one train-test split and helps reduce random variation. The cross-validation results show that our model performs consistently across different splits, which supports the stability of what it has learned and the reliability of our findings, even when data is limited.

Results and discussion

This section of the paper discusses and summarizes the outcomes that have been reported. The dataset used in this study is the most recent (published in 2024), and aside from the original paper, no further work has been done on the same dataset. As a result, we are not offering a table that compares our study to earlier research that has been published. Results of the original work are already provided in Section “Methodology”.

• Class Imbalance: Although the ViT-based architecture shows strong performance in the binary CHD diagnosis task, its performance deteriorates in the multi-class scenario. This discrepancy is primarily driven by the significant class imbalance in the dataset, where subtypes such as PDA and PFO have far fewer recordings than ASD or VSD. ViTs rely heavily on large and balanced datasets because their self-attention mechanism lacks the inductive bias of convolution; therefore, attention maps for minority classes tend to be unstable and skewed toward majority-class patterns. This imbalance is reflected in the confusion matrix as well, where rare subtypes exhibit the largest misclassification rates.

• Pre-processing: The preprocessing step improves the quality of heart-sound representations, thereby enhancing model performance. The method converts raw PCG data into a time-frequency representation that is aligned with human auditory perception. In addition, frame segmentation and Mel-filter projection together produce a concise, information-rich representation that demonstrates increased robustness to noise.

• Why Vision transfermer model? ViTs are chosen over traditional DL models because they can better capture long-range and global relationships. This long-range global relationship is important for interpreting heart sound signals. While CNNs focus on local features, ViTs use self-attention to connect information from distant parts of the signal. Recent research shows that ViT-based models often perform better than CNNs and recurrent models in biomedical signal and imaging tasks, particularly when dealing with complex patterns. Similarly, transformer-based models are also robust to variability and generalize better than models. To take advantage of both local feature extraction and global context, our proposed model uses a hybrid CNN–ViT approach. Current paper¹¹ this database is uding conventional machine learning. Our reported results are better than conventional ML models and those results reported in¹¹.

• Confusion Matrices: For multi-class data and model evaluation confusion matrices are normally used. It is a two-dimensional table, where one dimension corresponds to the true labels and the other dimension corresponds to the predicted labels. In the confusion matrix, some abbreviations used are: TP - True Positive, FN - False Negative, TN - True Negative, and FP - False Positive. We report our results with confusion matrices along with other metrics. The confusion matrix (Fig. 6) shows that the model is able to discriminate almost perfectly between normal and abnormal states for binary classes 6 - (a). Similarly, Fig. 6(b) is for multi-class classification results. Figure 6(b) is analyzing the results with data distribution, where we see that a limited number of samples for CHD subtypes might contribute to the worse results. For example, the class PDA with the least number of samples in the dataset (32 samples) showed the worst results, while the normal class with 533 samples in the dataset showed the best results. Thus, this pattern indicates that the classification’s lower performance for Task 2 stems from insufficient data, which makes it difficult for the proposed model to effectively learn differences between CHD subtypes. Similarly, the confusion matrix also depicts near-zero misclassification, which suggests that the model is able to discriminate perfectly between normal and abnormal states.

• Precision, Recall, F-1 Measure: Figure 7(a) illustrates the binary classification performance of the proposed model for CHD, exhibiting consistently elevated precision, recall, F1-score, and accuracy for both normal and abnormal classes, culminating in an overall accuracy of 95%. These findings demonstrate consistent differentiation between healthy and abnormal cardiac sounds. Similarly, Fig. 7(b) illustrates the multi-class classification results, demonstrating performance discrepancies among CHD subtypes. The standard class is categorised with high precision and F1-score, however; diminished performance is noted for minority classes such as PDA and ASD, shown by poorer recall and F1-scores. VSD and PFO yield moderate outcomes, underscoring the influence of class imbalance and nuanced acoustic variations among CHD subtypes. The findings indicate that although the model is exceptionally proficient in binary CHD screening, multi-class diagnosis presents difficulties and necessitates enhanced imbalance management measures and more comprehensive subtype-specific data. Results on the same dataset and reported by the paper¹¹ (please see Section “Methodology”) show that we have better results as compared to previous results.

• The improvement in performance observed after transitioning from a fixed 60–20–20 data split to a 10-fold cross-validation framework can be attributed to several factors. Cross-validation facilitates more efficient utilization of the dataset by enabling approximately 90% of the data to be used for training in each fold, compared to only 60% in the original split. This greater exposure to training samples allows the model to learn more representative decision boundaries, which is especially advantageous in complex multi-class scenarios. Additionally, repeated training and evaluation across different folds reduces the variance associated with a single, potentially biased data partition, resulting in more robust and stable performance estimates. Furthermore, 10-fold cross-validation mitigates the adverse effects of class imbalance by distributing minority class samples more uniformly across training and validation sets, thereby enhancing the model’s ability to recognize underrepresented classes. These factors collectively contribute to improved generalization performance in both binary and multi-class classification tasks. Figure 8 demonstrate the performance improvements resulting from the proposed evaluation strategy in both binary and multi-class classification tasks. Figure 8a shows consistent gains in precision, recall, F1-score, and overall accuracy for binary classes. This indicates enhanced predictive consistency. Similarly, Fig. 8b displays that performance for multiclasses is also improved. Figure 8 overall exhibit a clear upward shift for both binary and multi-class models, reflecting improved discriminative capability and class separability. Collectively, these visual results indicate that the adopted strategy yields more robust and reliable performance across various classification scenarios.

• Figure 9(a) shows the ROC curve for binary classification, demonstrates that the suggested model attains an area under the curve (AUC) of 0.97. This is signify exceptional discriminative capacity between normal and abnormal heart sounds. Similarly it can also be noticed that the curve consistently exceeds the chance diagonal at all operating levels. This indicates significant sensitivity at minimal false positive rates. Conversely, Fig. 9(b) shows the multi-class ROC analysis. This Figure demonstrating that the model exhibits moderate but significant class-wise discriminating among CHD subtypes. We noticed highest AUC for VSD (0.85), succeeded by ASD (0.81), PFO (0.80), and the normal class (0.80). Class PDA demonstrates the lowest AUC (0.76). These results clearly indicate that the proposed framework is highly effective for binary classification, however its capacity to differentiate between specific CHD subtypes is limited by class imbalance and overlapping acoustic features.

Fig. 6.

Reported confusion matrices for binary and multi-class problem.

Fig. 7.

Precision, Recall, and F1-measure reported with 60-20-20 experiments.

Fig. 8.

Precision, Recall, and F-1 measure reported with 10-fold cross validations.

Fig. 9.

ROC-AUC results for binary and multi-class classification.

Conclusion

CHD is a global health challenge, especially among pediatric populations and in resource-limited settings. In resource limited areas access to expert clinicians and advanced diagnostic equipment is extremely limited. Research reveals that early diagnosis is essential for improving clinical outcomes. However, it is also a fact that conventional screening methods such as auscultation and imaging are often subjective, very expensive, and more dependent on human expertise. Our work proposes a hybrid ViT–based DL framework for CHD diagnosis using PCG recordings.

The proposed model combines a lightweight depthwise separable CNN for efficient local temporal–spectral feature extraction with a Transformer module that models long-range dependencies. Conversion of raw PCG signals into Mel spectrogram representations enables the model to capture clinically relevant acoustic patterns associated with both normal and pathological heart sounds. The CNN backbone extracts discriminative local features, while the transformer encoder utilizes self-attention to learn global contextual relationships across multiple cardiac cycles, resulting in a complementary and robust feature representation.

We perform extensive experiments on the large-scale, publicly available ZCHSound pediatric heart sound dataset using both a standard 60–20–20 split and 10-fold cross-validation. The model demonstrated strong performance in binary CHD classification, achieving high accuracy, precision, recall, F1-score, and ROC-AUC, and surpassing conventional ML baselines previously reported on the same dataset. These findings confirm the effectiveness of integrating convolutional inductive biases with transformer-based global modeling for heart sound. However, the model’s performance for multi-class classification was comparatively lower, primarily due to significant class imbalance and limited sample availability.

In summary, this study demonstrates that a hybrid CNN–ViTs architecture is a promising direction for automated CHD screening from PCG recordings. The proposed hybrid ViTs framework offers an effective balance between accuracy, computational efficiency, and clinical practicality, making it well suited for deployment in real-world pediatric screening. Future work will focus on improving multi-class classification performance through advanced data augmentation, imbalance-aware learning strategies, and noise-robust modeling, as well as validating the framework on larger and more diverse multi-center datasets.

Author contributions

Conceptualization, K.K (Khalil Khan); methodology, R.U.K (Rehan Ullah Khan), K.K; investigation I.U (Irfan Uddin); resources, I.U; data curation, K.K; writing–original draft preparation, K.K, I.U; writing–review and editing, R.U.K, I.U; visualization, I.U.; supervision, K.K.

Data availability

The datasets generated and/or analysed during the current study are available in the (http://zchsound.ncrcch.org.cn/data).

Declarations

Competing interests

The authors declare no competing interests.