SSL-DA: Semi-and Self-Supervised Learning with Dual Attention for Echocardiogram Segmentation

Abstract

Echocardiogram analysis plays a crucial role in assessing and diagnosing cardiac function, providing essential data to support medical diagnoses of heart disease. A key task, accurately identifying and segmenting the left ventricle (LV) in echocardiograms, remains challenging and labor-intensive. Current automated cardiac segmentation methods often lack the necessary accuracy and reproducibility, while semi-automated or manual annotations are excessively time-consuming. To address these limitations, we propose a novel segmentation framework, semi-and self-supervised learning with dual attention (SSL-DA) for echocardiogram segmentation. We start with a temporal masking network for pre-training. This network captures valuable information, such as echocardiogram periodicity. It also provides optimized initialization parameters for LV segmentation. We then employ a semi-supervised network to automatically segment the left ventricle, enhancing the model’s learning with channel and spatial attention mechanisms to capture global channel dependencies and spatial dependencies across annotations. We evaluated SSL-DA on the publicly available EchoNet-Dynamic dataset, achieving a Dice similarity coefficient of 93.34% (95% CI, 93.23–93.46%), outperforming most prior CNN-based models. To further assess the generalization ability of SSL-DA, we conducted ablation experiments on the CAMUS dataset. Experimental results confirm that SSL-DA can quickly and accurately segment the left ventricle in echocardiograms, showing its potential for robust clinical application.

Introduction

The heart is the central organ of the cardiovascular system, vital for pumping blood throughout the body and ensuring its return, thus maintaining effective circulation. According to the World Health Organization, cardiovascular diseases claim approximately 18 million lives each year [1]. Diagnostic imaging methods commonly used to assess cardiovascular diseases include magnetic resonance imaging (MRI), computed tomography (CT), and echocardiography. Among these modalities, echocardiography has become the most widely used due to its safety, ease of use, and high temporal resolution. Its ability to capture real-time images at a high frame rate makes it particularly suitable for assessing cardiac motion [2].

In clinical examinations, physicians segment the left ventricle (LV) to assess heart structure. This helps derive critical indicators, such as end-diastolic (ED) and end-systolic (ES) volumes, ejection fraction, and LV mass. These metrics are essential for evaluating cardiac function [3].

These measurements are usually obtained by manually tracing LV boundaries on the ED and ES frames of each cardiac cycle. Despite echocardiography’s widespread use, manual segmentation is labor-intensive, time-consuming, and prone to variability among observers. Additionally, echocardiography has inherent limitations, including low signal-to-noise ratios and motion artifacts [4]. Therefore, pre-processing plays a critical role in ultrasound image analysis. For example, Ansari et al. [5] addressed the issue of noise spots in ultrasound images by using Contrast Limited Adaptive Histogram Equalization (CLAHE) for image preprocessing, which significantly improved the model’s segmentation performance. Regaya et al. [6] performed denoising on the input data before training the model to enhance the image contrast, making the images more suitable for segmentation and achieving satisfactory results. To overcome the challenge of data scarcity, Ansari et al. [7] employed extensive data augmentation techniques during training, which improved the segmentation accuracy of the neural network. Afsa et al. [8], in addressing the prediction of cardiac arrest (CA) from multivariate time-series vital sign data, not only created a balanced dataset for the primary experiments but also designed four imbalanced subsets to strengthen the model’s ability to adapt to class imbalance. To ease the workload for clinicians and enhance the accuracy of echocardiographic analysis, automated diagnostic algorithms have been introduced. Early methods for LV segmentation mainly relied on models incorporating known cardiac anatomical principles [9]. Although these approaches showed promising results, most were validated only on small, private datasets. Recently, numerous deep learning-based models have been developed for cardiac function assessment. Initial deep learning models often required extensive manual annotation of static images from ED and ES frames, which sometimes led to discrepancies with expert evaluations [10, 11]. With advances in convolutional neural networks (CNNs), architectures such as U-Net [12, 13] and DeepLab have been successfully applied to LV segmentation, achieving impressive outcomes. Furthermore, the release of large datasets, such as EchoNet-Dynamic [14], has accelerated the progress in this field.

Despite the undeniable importance of echocardiography, datasets with carefully annotated, expert-validated labels are extremely limited. Clinicians typically annotate only the ED and ES frames in echocardiographic videos, and this annotation process is both tedious and repetitive, placing a considerable burden on echocardiographers. As a result, supervised labels for echocardiographic videos are scarce; for instance, in the EchoNet-Dynamic dataset, labeled frames constitute less than 1.2% of the total frames. Although CNNs and their variants perform well in LV segmentation tasks, they still have notable limitations. First, convolutional operations rely on local receptive fields, making it difficult to capture dependencies between distant features and extract global feature interactions [15]. Second, as shown by Wang et al. [16], the skip connection scheme used in U-Net struggles to effectively capture global multi-scale information, hindering its ability to bridge the semantic gap between encoder and decoder features.

To address these challenges, we propose SSL-DA, an end-to-end semi-supervised and self-supervised framework for left ventricle (LV) segmentation, integrating both channel and spatial attention mechanisms. The SSL-DA framework is designed to automatically segment the left ventricle in echocardiographic videos, even with limited annotations. It leverages a self-supervised pre-training network with temporal masking to initialize the segmentation model and introduces channel and spatial attention during semi-supervised training to enhance segmentation accuracy. Our contributions can be summarized as follows:

1. We propose an end-to-end automatic left ventricle (LV) segmentation framework, SSL-DA, which operates in two phases. In the first phase, a self-supervised pre-training network with temporal masking is employed to initialize the model, enabling it to learn the periodic features of echocardiographic videos from a large volume of unlabeled frames. In the second phase, we use a semi-supervised learning network focused on LV segmentation with sparse annotations, allowing the segmentation network to operate independently of the constraints of the cardiac cycle.

2. For network model design, we developed a novel U-Net architecture called CSAU-Net, specifically tailored for the LV segmentation task under sparse annotation conditions. By integrating global channel context attention mechanisms and enhancing spatial dependencies, CSAU-Net fuses spatial information across channels, extracts global channel context, and captures interdependencies among multi-scale encoder channels, effectively addressing semantic gaps.

3. We conducted experiments on the EchoNet-Dynamic and CAMUS datasets, and the results validate the effectiveness of the SSL-DA framework. Our approach demonstrates a significant improvement in segmentation accuracy under sparse annotation conditions, enabling rapid and accurate segmentation of the left ventricle in echocardiograms.

The remainder of this paper is organized as follows. The “Related Works” section provides an overview of related work. In the “Methods” section, we introduce the SSL-DA framework. The “Experiments” section presents experimental comparisons with state-of-the-art models. In the “Discussion” section, we analyzed the limitations and shortcomings of the SSL-DA framework. Finally, the “Conclusion” section concludes the paper with a summary of our findings.

Related Works

LV Segmentation

The earliest non-deep learning models for left ventricle segmentation primarily focused on identifying and segmenting the endocardial boundaries of the LV [13]. The active contour method proposed by Chen et al. [17] achieved relatively effective segmentation on early ultrasound images. However, it was highly dependent on specific data formats, resulting in poor scalability. Later, Barbosa et al. [9] introduced a fully automated method that performed well on the three-dimensional echocardiographic video dataset CETUS. Nevertheless, these models still showed significant discrepancies compared to human expert evaluations, and their robustness across other datasets remained limited.

With continuous advancements in deep learning, left ventricle segmentation technology has achieved significant breakthroughs. Convolutional neural networks (CNNs), especially fully convolutional networks (FCNs) like U-Net [18], have been extensively applied in medical segmentation. Variants of FCNs, such as SegNet [19], DenseNet [20], and DeepLab, have also shown impressive performance in medical image segmentation. Suyu et al. [21] were the first to combine deep learning with traditional snake models, achieving favorable segmentation outcomes. Smistad et al. [22] successfully applied U-Net for left ventricle segmentation in two-dimensional echocardiography. In addition, Oktay et al. [23] developed an anatomically constrained neural network model for three-dimensional echocardiographic segmentation, achieving state-of-the-art performance.

Despite these advancements, significant challenges remain in echocardiographic segmentation. Supervised labels are scarce and difficult to obtain, and CNN-based methods face limitations in capturing long-range dependencies and explicitly defining global multi-scale contexts.

Semi-and Self-Supervised Learning

The availability of supervised labels in echocardiography is extremely limited, as human experts must painstakingly annotate the ED and ES frames in each video segment. For example, Ouyang et al. [14] utilized fewer than 1.2% of the frames available in the EchoNet-Dynamic dataset, the largest of its kind, for training purposes. Early approaches to left ventricle segmentation, such as those by Smistad [22] and Ouyang, employed frame-by-frame solutions.

To better leverage the unlabeled frames, Saeed et al. [24] introduced contrastive learning methods, yielding promising results; however, temporal consistency remained relatively low. More recently, Tong et al. [25] applied masked autoencoders (MAE) for supervised pre-training of video networks, which accelerated model convergence and improved training performance. Despite these advancements, many current methods still use two-dimensional models for isolated frame analysis. Some methods combine two-dimensional and temporal sequence analyses. Both approaches significantly increase model training complexity.

Channel and Spatial Attention

Attention mechanisms have become essential components in many machine learning models. The Transformer model [26], with its ability to dynamically focus on different parts of the input data, has greatly improved the capacity to capture complex dependencies and has achieved remarkable success in natural language processing. Building on this success, transformers have also shown significant progress in computer vision tasks, such as image classification [27], object detection [28], and segmentation [29].

Inspired by the success of attention mechanisms, researchers have started combining channel self-attention and spatial self-attention to capture interactions across channels more effectively. For instance, Fu et al. [30] incorporated both channel and spatial attention modules in their DANet, enabling the modeling of semantic dependencies across spatial and channel dimensions, which notably improved segmentation performance. Similarly, Liu et al. [31] achieved performance gains by concatenating the outputs of these attention modules.

Recent research [32] introduced a dual attention method called DaVit, which applies spatial attention and channel attention modules sequentially, demonstrating that this dual attention approach can improve model performance. Similarly, Wang et al. [16] tackled the semantic gap in U-Net by using cross-channel attention, which leverages cross-attention across multi-scale encoder feature channels to capture inter-channel dependencies. This dual attention mechanism has enhanced segmentation performance and helped bridge the semantic gap between encoder and decoder features.

Methods

We propose a novel left ventricular segmentation framework, SSL-DA, as illustrated in Fig. 1. This framework mainly consists of two phases. The first phase involves a self-supervised pretraining network with temporal masking. This design allows the reconstruction framework $G_R$ to learn the periodic characteristics of echocardiographic videos, providing optimized initialization parameters for the segmentation framework $G_S$ in the subsequent stage. The second phase constructs a semi-supervised learning network focused on the left ventricular segmentation task with sparse annotations. The segmentation framework $G_S$ learns from sparsely annotated echocardiographic videos using a video-based segmentation network. This allows it to become independent of cardiac cycle constraints and learn left ventricular segmentation more effectively. It also avoids issues related to semantic gaps. Both phases of the base model introduce attention modules that fuse channel attention and spatial attention. The detailed descriptions of each component are as follows:

Fig. 1.

SSL-DA framework diagram: The red dashed box represents the self-supervised temporal masking pretraining network, while the black dashed box represents the semi-supervised LV segmentation network. After the reconstruction framework $G_R$ in the pretraining network completes its training, the network parameters are transferred to the segmentation framework $G_S$ for parameter initialization

Base Model

In this paper, we propose a CSAU-Net architecture, which is based on the 3D U-Net. This architecture employs a convolutional neural network (CNN) with residual units as the encoder and integrates a fused attention module, referred to as the CSA block. The specific structure is illustrated in Fig. 2, while the internal details of the CSA block are shown in Fig. 3. The encoder comprises five units with different structures.

Fig. 2.

The CSAU-Net architecture: the CSA block module extracts multi-scale features from different encoder stages, generates enhanced representations, and assigns them to the corresponding decoders

Fig. 3.

Internal structure of the encoder

In the self-supervised temporal masking pretraining network phase of the SSL-DA framework, we use CSAU-Net to perform echocardiographic reconstruction tasks, referred to as the reconstruction framework $G_R$. Subsequently, in the semi-supervised LV segmentation network phase, we employ CSAU-Net for LV segmentation, termed the segmentation framework $G_S$. After completing the first phase, the pretrained weights are transferred to the second phase. By leveraging the CSAU-Net architecture, the SSL-DA framework combines a multi-task learning paradigm, appropriate loss functions, and weight transfer, significantly enhancing both the performance and generalization ability of $G_R$.

Additionally, the model can be further lightweighted to address the increased computational cost introduced by the CSAU-Net architecture.

Self-Supervised Temporal Masking

Deep learning schemes can automate LV segmentation, but they are limited by the quantity and quality of labeled training data, which can be difficult or costly to obtain. For example, in the EchoNet-Dynamic dataset, labeled frames constitute less than 1.2% of the total frames. To better leverage the vast amount of unlabeled data, we introduce a self-supervised temporal masking pretraining network, as shown within the red dashed box in Fig. 1. The workflow of this pretraining network is as follows: first, a segment is extracted from the echocardiographic video, and certain frames are randomly masked. Then, the reconstruction framework $G_R$ is employed to reconstruct these masked frames. Through this method, the model learns valuable information from the periodic features of echocardiograms, maintaining strong reconstruction consistency within the given image clip. This process enhances the robustness of the subsequent segmentation model to noise and artifacts while providing optimized initialization parameters for the segmentation framework $G_S$. Additionally, this pretraining network helps alleviate the problem of over-segmentation caused by blurred cardiac structures. The specific reconstruction loss function is as follows:

$${\mathcal{L}}_{\mathit{rec}}=\frac{1}{N}\sum _{n=1}^{N}MSE\left(c^n,,,G_R^W,\left(c_m^n\right)\right)$$ 1

where $N$ is the batch size, $c^n$ represents the echocardiographic video clip of the n-th sample, $G_R^W$ denotes the reconstruction framework with the weight parameters $W$, $c_m^n$ represents the echocardiographic video clip of the n-th sample containing selected frames, and $m$ is a subset of frames randomly extracted from the echocardiographic video clip.

Semi-Supervised LV Segmentation with Channel and Spatial Attention

Semi-Supervised LV Segmentation

Annotating echocardiograms is a labor-intensive and time-consuming process. Therefore, human experts typically provide annotations only for keyframes. For instance, the EchoNet-Dynamic dataset has very sparse annotations, making left ventricular segmentation using this dataset quite challenging. Inspired by the methods of Çiçek et al. [33] and Maani et al. [34], we designed a network specifically for left ventricular segmentation in echocardiographic videos, as illustrated within the black dashed box in Fig. 1. In this network, the segmentation model inherits the initialization parameters from the reconstruction framework $G_R$ and uses echocardiographic video frames as input. For each input frame, the model computes the Dice loss only on the frames that have segmentation labels and performs backpropagation accordingly. The specific loss function of the segmentation model is as follows:

$${\mathcal{L}}_{\mathit{seg}}=\frac{1}{N}\sum _{n=1}^N{\mathcal{L}}_{\mathit{dice}}\left({\mathit{y}}^n,,,{\widehat{\mathit{y}}}^n\right)$$ 2

where $N$ is the batch size, ${\mathcal{L}}_{\mathit{dice}}$ epresents the Dice loss, $\widehat{\mathit{y}}$ is the LV segmentation result predicted by the segmentation network, and $\mathit{y}$ is the ground truth of the input segments.

Channel and Spatial Attention

Although U-Net and its variants have performed exceptionally well in numerous medical image segmentation tasks, they still present certain limitations. These include the inability to effectively capture long-range dependencies between features and the potential semantic gap caused by feature interactions between the encoder and decoder. We drew inspiration from the method proposed by Ates et al. [35] and designed an attention module that combines channel attention and spatial attention, as shown in Fig. 4. The CSA block consists of two components: channel attention and spatial attention, designed to address the semantic void problem. Channel attention extracts global channel context by fusing all spatial positions between any two channels, while spatial attention extracts global spatial context by capturing spatial interdependencies between any two positions in multi-scale encoder channels.

Fig. 4.

The CSA block specific structure: first, the combination of Avg Pool, Projection, and Reshape forms a multi-scale patch embedding module to obtain encoder tokens. Then, the CSA attention mechanism, incorporating channel attention and spatial attention modules, is applied to these encoder tokens to capture long-range dependencies. Finally, Layer Norm and GeLU are used to upsample these tokens, allowing them to be connected with the corresponding decoder tokens

By integrating this fused attention module, the model can extract long-range contextual information and capture rich global context in both channel and spatial dimensions. This enables the network to enhance local features while maintaining a comprehensive global feature distribution, thereby improving segmentation accuracy. Moreover, unlike the method in [36], which builds a parameter-heavy end-to-end network, and the method in [37], which introduces a MetaFormer module with significant computational cost, our fused attention method requires only a slight increase in parameters. This enhancement strengthens the connection between the encoder and decoder, improving the model’s segmentation performance.

Experiments

Dataset

EchoNet-Dynamic [15]

The EchoNet-Dynamic dataset was released by Ouyang et al. in 2020, consisting of 10,030 echocardiographic video sequences captured from apical four-chamber (A4 C) views. The dataset is divided into TRAIN, VALIDATION, and TEST sets, containing 7465, 1288, and 1277 videos, respectively. Each video has a fixed frame resolution of 112 × 112, with varying frame lengths that encompass multiple cardiac cycles; however, only the ED and ES frames are annotated.

CAMUS [38]

The CAMUS dataset comprises echocardiographic data from 500 patients, featuring both two-chamber and four-chamber views. For each patient, the dataset provides one two-chamber image and one four-chamber image, with each echocardiogram capturing a complete cardiac cycle and including segmentation annotations for all frames. The ED and ES frames correspond to the first and last frames of each echocardiogram, resulting in a total of 2000 images in the dataset. The training set includes data from 450 patients, with 900 images for both ED and ES phases, while the test set comprises data from 50 patients, with 100 images for each phase. Additionally, the dataset provides supplementary information, such as image quality (good, medium, and poor), ejection fraction grades, and patient demographics including gender and age.

SSL-DA Dataset

We collected a small echocardiogram dataset from the Department of Cardiology at Dongzhimen Hospital of Beijing University of Chinese Medicine, which consists of 36 samples. Each sample includes 10 echocardiogram video sequences captured from the A4 C view. The data were collected by different operators using Philips IE33 and Philips CX50 machines. The dataset also includes patient disease types and demographic information, such as gender and age for each patient.

Implementation Details

To ensure a fair comparison with state-of-the-art methods, this study strictly adheres to the original data distribution of the EchoNet-Dynamic dataset and evaluates the segmentation results using the Dice similarity coefficient (DSC). A higher Dice coefficient indicates a closer match between the segmentation results and the ground truth labels. We used the AdamW optimizer for training the CSAU-Net, setting the learning rate to 3 × 10⁻⁴ and the weight decay to 1 × 10⁻⁵. The self-supervised pre-training process was configured for 100 epochs, with data augmentation techniques including random rotation, padding, and color jitter. The semi-supervised segmentation training was set for 70 epochs, sampling each echocardiographic video twice per epoch to ensure the inclusion of annotated ED and ES frames. The experiments were conducted in an Ubuntu 20.04.6 environment using an NVIDIA GeForce RTX 3090, with the software framework being PyTorch 1.12.1 and CUDA 11.4.

Results

We compared our method with other state-of-the-art approaches on the EchoNet-Dynamic dataset, and the analysis results are presented in Table 1. Given that some samples in this dataset exhibit poor quality, which may affect the segmentation results, previous studies [39, 40] have excluded these test samples to enhance performance. However, to ensure a fair comparison, we did not exclude lower-quality samples in our work. Figure 5 illustrates the sample distribution of SSL-DA segmentation results.

Table 1.

The comparison of the Dice similarity coefficient in the EchoNet-Dynamic dataset with other published state-of-the-art methods for left ventricular segmentation, where SSL-DA achieves SOTA segmentation performance with relatively low parameter count

Methods	DSC (%)			Params (M)	Test set
	Overall	ES	ED
CoTr [41]	91.29	89.87	92.71	16.39	1277
TransBridge [41]	91.64	90.24	93.04	11.30	1277
ResU-Net [41]	92.34	91.17	93.51	7.90	1277
EchoNet-Dynamic [14]	92.00	90.68	92.78	39.64	1277
SepXception [42]	92.90	91.73	93.64	55.83	1277
MAEF-NET [39]	93.10	91.73	93.92	-	1226
MURAU-Net [40]	92.77/93.13	-	-	-	1277/1233
SimLVSeg [34]	93.23	92.22	93.86	18.83	1276
SSL-DA	93.34	92.27	94.01	20.00	1277

Fig. 5.

The figure shows the sample distribution of DSC calculation results for each ED/ES frame (n = 1277), indicating that the semantic segmentation performance at both ES and ED remains consistently aligned

As shown in Table 1, SSL-DA achieved the best overall, ES, and ED DSC scores with relatively low computational cost. Figure 6 presents the specific segmentation results of SSL-DA, demonstrating the accuracy of the method in delineating the left ventricle contours, which are finely detailed and smooth. This improvement comes from the self-supervised pre-training with temporal masking. It allows the segmentation network to learn the periodic features of echocardiographic videos. At the same time, the attention modules help reduce edge noise and artifacts.

Fig. 6.

Left ventricle segmentation results of echocardiography videos under the SSL-DA framework. The input consists of preprocessed echocardiograms, while the output represents the model-segmented echocardiograms, where the blue region indicates the left ventricle segmented by the model

To achieve a lightweight design, we modified the 3D U-Net architecture by reducing the number of repeated residual blocks in each encoder and decoder stage. For example, an encoder stage that originally employed six consecutive 128-channel residual blocks has now been simplified to just two blocks at the same channel width, significantly cutting down the total number of parameters while preserving the core functionality of the 3D U-Net. As shown in Table 2, the optimized model achieved a parameter reduction of 7.6 M and a 0.22 G reduction in FLOPs per frame, while maintaining segmentation accuracy.

Table 2.

The comparison of the Dice similarity coefficient, parameters, and FLOPs between lightweight SSL-DA (SSL-DA Lt) and SSL-DA on the EchoNet-Dynamic dataset

Methods	DSC (95% CI)%			Params (M)	FLOPs (G)
	Overall	ES	ED
SSL-DA (Lt)	93.33 (93.21–93.44)	92.26 (92.06–92.44)	93.99 (93.85–94.13)	12.40	1.02
SSL-DA	93.34 (93.23–93.46)	92.27 (92.07–92.45)	94.01 (93.88–94.15)	20.00	1.24

To further address the requirements of real-time clinical applications, we also evaluated both single-frame inference time and video frame rate. As shown in Table 3, the optimized model achieves an inference speed on a single RTX 3090 that is nearly equivalent to the baseline model for individual frame processing, and it can process video data at 46 FPS, thereby fulfilling real-time clinical demands.

Table 3.

Comparison of single-frame inference time and video frame rate for different models

Methods	Hardware setup	Input resolution	Inference time per frame (ms)	FPS
Base	NVIDIA RTX 3090	112 × 112	20.8	48
SSL-DA	NVIDIA RTX 3090	112 × 112	25.9	38
SSL-DA(Lt)	NVIDIA RTX 3090	112 × 112	21.6	46

Notably, on the EchoNet-Dynamic dataset, the lightweight model even retained SOTA segmentation performance. Through this optimization, we significantly reduced the computational cost, making the model more suitable for real-time clinical applications. Even in resource-constrained hardware environments, the optimized model can still deliver high-quality segmentation results, further enhancing its practical applicability.

Additionally, to validate the generalization capability of the SSL-DA method, we evaluated the model’s performance on the CAMUS dataset, with specific results shown in Table 5. Despite not undergoing any specific additional training, the SSL-DA method exhibited satisfactory segmentation performance on this target dataset.

Table 5.

Left ventricular segmentation DSC scores on the original standardized test set of the CAMUS dataset using different modules in an ablation study, where the model integrated with SSTM and CSA modules achieved the best segmentation performance

Methods	DSC (95% CI)%
	Overall	Small	Mid	Large
Base	88.36 (88.11–88.60)	85.45 (85.01–85.87)	88.09 (87.81–88.37)	90.26 (90.00–90.50)
Base + SSTM	89.20 (88.89–89.49)	86.77 (86.23–87.26)	88.55 (88.15–88.90)	90.77 (90.43–91.08)
Base + CSA	91.31 (90.97–91.62)	89.29 (88.69–89.81)	88.53 (88.04–88.98)	92.58 (92.18–92.93)
Base + SSTM + CSA	91.78 (91.62–91.94)	90.38 (90.15–90.60)	90.39 (90.17–90.59)	92.64 (92.44–92.83)

In addition to validating our approach on the EchoNet-Dynamic and CAMUS datasets, we also conducted experiments on the SSL-DA dataset collected in this study. This dataset covers a more diverse range of clinical cases, including challenging pathological conditions. These tests further demonstrate the robustness of our approach. As illustrated in Figs. 7 and 8, we invited multiple clinical experts to annotate the model’s segmentation results, allowing us to assess performance across different patient populations and imaging conditions.

Fig. 7.

Left ventricle segmentation results of echocardiography videos under the SSL-DA framework. The input consists of preprocessed clinical echocardiograms, while the output represents the model-segmented echocardiograms. The blue region indicates the left ventricle segmented by the model, while the red contour represents the left ventricular endocardial border annotated by clinical experts

Fig. 8.

Left ventricle segmentation results of echocardiography videos under the SSL-DA framework. The input consists of preprocessed low-quality and noisy clinical echocardiograms, while the output represents the model-segmented echocardiograms. The blue region indicates the left ventricle segmented by the model, while the red contour represents the left ventricular endocardial border annotated by clinical experts

In Fig. 7, the first and third rows display the original echocardiography inputs, whereas the second and fourth rows illustrate the model’s segmentation output alongside clinical expert annotations. Overall, the model’s segmentation results are largely consistent with expert delineations, suggesting that it captures essential anatomical boundaries in most cases. However, certain regions show discrepancies. On further examination, these variations may arise from differing interpretations of ambiguous anatomical borders, variations in experts’ familiarity with specific pathologies, or noise artifacts in lower-quality frames. Despite these localized mismatches, the model’s strong alignment with expert opinions indicates significant potential for reliable and reproducible performance in real-world clinical scenarios.

Similarly, Fig. 8 presents the model’s performance on low-quality images. Despite suboptimal imaging conditions, the predicted segmentations remain within clinically acceptable ranges, indicating notable robustness in real-world scenarios.

Ablation Study

Tables 4 and 5 present the ablation study results of SSL-DA on the EchoNet-Dynamic and CAMUS datasets, respectively, aiming to verify the effectiveness of each module. According to Tables 4 and 5, the overall DSC scores for the base method of SSL-DA on the two datasets were 93.19% (95% CI 93.07–93.31%) and 88.36% (95% CI 88.11–88.60%), respectively. After incorporating Self-Supervised Temporal Masking (SSTM), the DSC scores increased to 93.23% (95% CI 93.11–93.34%) and 89.20% (95% CI 88.89–89.49%).

Table 4.

Left ventricular segmentation DSC scores on the original standardized test set of the EchoNet-Dynamic dataset using different modules in an ablation study, where the model integrated with SSTM and CSA modules achieved the best segmentation performance

Methods	Test set	DSC (95% CI)%
		Overall	ES	ED
Base	1277	93.19 (93.07–93.31)	92.07 (91.88–92.27)	93.89 (93.73–94.03)
Base + SSTM	1277	93.23 (93.11–93.34)	92.22 (92.03–92.40)	93.86 (93.72–94.00)
Base + CSA	1277	93.32 (93.20–93.43)	92.25 (92.07–92.43)	93.97 (93.83–94.11)
Base + SSTM + CSA	1277	93.34 (93.23–93.46)	92.27 (92.07–92.45)	94.01 (93.88–94.15)

Furthermore, with the addition of channel and spatial attention modules, the DSC scores improved to 93.32% (95% CI 93.20–93.43%) and 91.31% (95% CI 90.97–91.62%), indicating that the CSA block can extract long-range contextual information, strengthen the connection between the encoder and decoder, and improve the model’s segmentation accuracy. When both modules were integrated simultaneously, performance was further enhanced, achieving DSC scores of 93.34% (95% CI 93.23–93.46%) and 91.78% (95% CI 91.62–91.94%). These results indicate that each module independently contributes to improved model performance, with the best results observed when both methods are combined. Additionally, the confidence interval analysis shows no overlap in the 95% confidence intervals for the different methods, further demonstrating the effectiveness of our proposed improvements.

We also performed a qualitative comparison of the model’s prediction results to further validate our CSA module. Figure 9 illustrates several segmentation outcomes, where the red rectangle in the middle images highlights regions where the baseline model struggled to yield accurate predictions. By contrast, the red rectangle in the rightmost images demonstrates the improvements introduced by the CSA module in those same areas. As shown in Fig. 9, integrating the CSA module leads to more consistent boundary delineations and preserves precise shape information. The dual attention design in the CSA module helps the model refine both global and local features simultaneously, thereby leading to more accurate boundary delineations and preserved shape information. Specifically, channel attention dynamically emphasizes the most salient feature maps relevant to cardiac structures, and spatial attention pinpoints the critical regions within each frame where these features are most pronounced. This dual attention mechanism inherently filters out irrelevant background noise and concentrates on meaningful morphological cues. Consequently, the resulting segmentation predictions exhibit better consistency and robustness.

Fig. 9.

Visualization comparison of left ventricle segmentation results on echocardiography videos between the baseline model and the model integrated with the CSA module

We tested the impact of different mask rates on model performance, as shown in Table 6. The experimental results indicate that when the mask rate is too low, the model has access to more complete information. However, it may fail to fully exploit hidden information, leading to suboptimal improvements in segmentation performance, particularly in cases where cardiac morphology is more complex. When the mask rate is too high, the model may become overly dependent on the limited available frames, which negatively affects segmentation performance. A moderate mask rate effectively prevents the model from over-relying on specific frames while maintaining good generalization ability. This is especially important in scenarios with varying heart rates and pathologies, where the segmentation results remain stable. Through ablation experiments, we selected 0.6 as the optimal mask rate, which was also used in our SSL-DA framework. We found that it minimizes information loss while benefiting from the regularization effect of masking.

Table 6.

Left ventricular segmentation DSC scores on the original standardized test set of the EchoNet-Dynamic dataset using different mask rates in an ablation study, where the best segmentation performance was achieved with a mask rate of 0.6

Methods	Mask rate	DSC (95% CI)%
		Overall	ES	ED
SSL-DA	0.4	93.26 (93.14–93.37)	92.20 (92.01–92.38)	93.91 (93.77–94.05)
	0.5	93.30 (93.19–93.42)	92.20 (92.02–92.39)	93.98 (93.83–94.12)
	0.55	93.32 (93.21–93.43)	92.27 (92.08–92.45)	93.97 (93.83–94.10)
	0.6	93.34 (93.23–93.46)	92.27 (92.07–92.45)	94.01 (93.88–94.15)
	0.65	93.32 (93.20–93.44)	92.27 (92.08–92.46)	93.97 (93.82–94.12)
	0.7	93.29 (93.17–93.41)	92.30 (92.10–92.48)	93.91 (93.76–94.05)
	0.8	93.25 (93.13–93.36)	92.21 (92.03–92.40)	93.89 (93.75–94.03)

Discussion

Data imbalance is a common issue in medical imaging, particularly in echocardiography datasets where pathological cases are often underrepresented compared to normal cases. This imbalance can lead to model performance bias, with models typically excelling at segmenting common cases but struggling with rarer pathological variations, potentially affecting the model’s generalizability and clinical reliability. In fact, the impact of data imbalance is not limited to medical imaging but spans various biomedical application domains. For instance, Chandrasekar et al. [43] discussed how data heterogeneity and imbalance in virus-like particles/nanoparticles (VLP) affect the reproducibility of reproductive biology experiments. This issue is similar to the challenges faced in echocardiography datasets, where heterogeneous patient demographics and varying image qualities may also impact model performance. Mone et al. [44] explored how imbalanced data in naphthoquinone drug structure–function analysis influences the prediction of synergistic effects against multidrug-resistant Staphylococcus aureus. Likewise, in echocardiography segmentation, the imbalance between normal and abnormal cases can hinder the model’s ability to accurately predict and segment, particularly under complex cardiac conditions. Dhage et al. [45] emphasized the challenges posed by imbalanced molecular datasets in targeting Alzheimer’s disease biomarkers. The authors highlighted how data augmentation and ensemble learning techniques can mitigate such imbalance effects, strategies which can directly be applied to echocardiography segmentation to improve robustness across different cardiac conditions.

In our work, to address data imbalance, we employed data augmentation techniques. Specifically, our data augmentation strategy includes random rotation, padding, and color jitter, which all enhance the diversity of underrepresented classes during training. In addition, to evaluate segmentation performance under imbalanced and diverse clinical conditions, we collected a clinical dataset from the Department of Cardiology at Dongzhimen Hospital of Beijing University of Chinese Medicine. This dataset contains ultrasound images from patients with different pathological types, acquisition devices, and operator settings. It encompasses common clinical cardiac abnormalities as well as low-quality imaging cases, providing a realistic scenario for assessing model robustness. Segmentation results, supported by clinical expert annotations, confirm that our method remains reliable even under these challenging circumstances.

Despite these measures, we acknowledge that data imbalance remains a persistent challenge. In future work, we plan to expand our dataset by incorporating more minority-class samples, particularly rare pathological cases. This will lead to a more balanced training distribution and further enhance the model’s overall generalization capacity.

In addition, although the CSAU-Net architecture employs dual attention mechanisms to enhance contextual feature representation, we acknowledge that these modules may introduce extra computational overhead. To address this concern and improve the model’s applicability in real-time clinical environments, we adopted a lightweight approach, including reducing the number of channels in both the encoder and decoder layers, as well as decreasing the number of convolution channels to lower the overall computational complexity. As shown in Tables 2 and 3, this lightweight version reduces the model parameters by 7.6 M, lowers FLOPs per frame by 0.22 G, decreases the inference time per image by 4.3 ms, and raises the FPS to 46—while still maintaining high segmentation accuracy that meets actual clinical demands.

Lastly, the current implementation of SSL-DA focuses on the A4 C view of echocardiography videos. At this stage, the model can only process data from the A4 C view and is not adaptable to multi-view scenarios. In the future, we plan to collect multi-view echocardiography datasets, explore multi-view fusion strategies, and train more generalized models based on these datasets. We also aim to further reduce the model complexity to enable faster and more accurate segmentation, better aligning with clinical requirements.

Conclusion

We introduce SSL-DA, a novel segmentation framework designed for automatic and efficient left ventricle segmentation in echocardiographic videos. Our method outperforms existing state-of-the-art techniques on the EchoNet-Dynamic dataset and shows strong generalization on the CAMUS dataset. SSL-DA uses a self-supervised pretraining network to extract latent information from echocardiographic videos. The semi-supervised segmentation model then integrates channel and spatial attention to address the semantic gap. This significantly enhances segmentation performance.

Both quantitative and qualitative evaluations confirm that SSL-DA exceeds the performance of current advanced methods for left ventricle segmentation.

Author Contribution

The study conception and design were proposed by Lin Lv and Feng-rong Sun. Material preparation, data collection, and analysis were performed by Lin Lv and Xing Han. The first draft of the manuscript was written by Lin Lv and all authors commented on previous versions of the manuscript. Data collection was conducted by Tong Jiang and Liangzhen You. All authors read and approved the final manuscript.

Funding

This work is supported by the Nature Science Foundation of Shandong Province (Grant No. ZR2019MF048), the Natural Science Foundation of China (Grant No. 12204273), and the National Natural Science Foundation Project (No. 82305056). It is also partly supported by the National Key R&D Program of China (Grant No. 2021YFB1407001).

Data Availability

EchoNet-Dynamic and CAMUS are publicly available datasets. The SSL-DA dataset is not publicly available due to privacy concerns.

Declarations

Ethics Approval

This study does not involve any human participants, animal subjects, or sensitive personal data, and therefore, ethics approval was not required.

Consent to Participate

The research was conducted using publicly available data that do not contain any personal or sensitive information.

Consent to Publication

The research was conducted using publicly available data that do not contain any personal or sensitive information.

Competing interests

The authors declare no competing interests.