A coordinated adaptive multiscale enhanced spatio-temporal fusion network for multi-lead electrocardiogram arrhythmia detection

Abstract

The multi-lead electrocardiogram (ECG) is widely utilized in clinical diagnosis and monitoring of cardiac conditions. The advancement of deep learning has led to the emergence of automated multi-lead ECG diagnostic networks, which have become essential in the fields of biomedical engineering and clinical cardiac disease diagnosis. Intelligent ECG diagnosis techniques encompass Recurrent Neural Networks (RNN), Transformers, and Convolutional Neural Networks (CNN). While CNN is capable of extracting local spatial information from images, it lacks the ability to learn global spatial features and temporal memory features. Conversely, RNN relies on time and can retain significant sequential features. However, they are not proficient in extracting lengthy dependencies of sequence data in practical scenarios. The self-attention mechanism in the Transformer model has the capability of global feature extraction, but it does not adequately prioritize local features and cannot extract spatial and channel features. This paper proposes STFAC-ECGNet, a model that incorporates CAMV-RNN block, CBMV-CNN block, and TSEF block to enhance the performance of the model by integrating the strengths of CNN, RNN, and Transformer. The CAMV-RNN block incorporates a coordinated adaptive simplified self-attention module that adaptively carries out global sequence feature retention and enhances spatial–temporal information. The CBMV-CNN block integrates spatial and channel attentional mechanism modules in a skip connection, enabling the fusion of spatial and channel information. The TSEF block implements enhanced multi-scale fusion of image spatial and sequence temporal features. In this study, comprehensive experiments were conducted using the PTB-XL large publicly available ECG dataset and the China Physiological Signal Challenge 2018 (CPSC2018) database. The results indicate that STFAC-ECGNet surpasses other cutting-edge techniques in multiple tasks, showcasing robustness and generalization.

Introduction

Cardiovascular diseases (CVDs) have become a major contributor to worldwide mortality, impacting more than 500 million individuals globally¹. By 2021, cardiovascular diseases (CVDs) caused 20.5 million deaths, which was almost 33% of the overall worldwide mortality rate². Cardiac arrhythmia is a prevalent heart problem³. Timely diagnosis through arrhythmia monitoring is essential for the prevention and early detection of cardiovascular diseases (CVDs).

The multi-lead electrocardiogram (ECG) is a cost-effective and non-invasive instrument that consists of two distinct types of features: intra-lead features and inter-lead features⁴. The former presents data regarding the P waves, QRS wave clusters, ST segments, T waves, and PR intervals in each lead⁵. It reflects the activity and function⁶, assisting in the screening and evaluation of potential cardiac abnormalities. In contrast, the latter denotes the correlation among distinct leads. For instance, leads V1-V6 show ST-segment elevation, whilst the remaining leads exhibit a normal rhythm⁷. The superiority of a multi-lead ECG over a single-lead ECG is apparent, since it collects a more extensive range of information and offers a greater degree of diagnostic precision. However, the interpretation of a multi-lead ECG necessitates the expertise of skilled cardiac specialists as well as a significant amount of time and effort⁸. Despite having extensive experience, human interpretation of multi-lead ECGs might still result in mistakes. In order to tackle these difficulties, the implementation of deep learning models for diagnostic aid seeks to improve the precision of multi-lead ECG interpretation and decrease the expenses associated with disease identification⁹. Hence, the use of deep learning models for automated detection of cardiovascular disorders is becoming more and more essential, leading to enhanced reliability in the clinical diagnosis of multi-lead ECG analysis.

The application of artificial intelligence in the medical field has become notably prevalent⁶⁵. Recent years have witnessed major advancements in the research of arrhythmia classification using ECG data. Machine learning researchers have employed many techniques, including Gaussian NB¹⁰, Random Forest¹¹, Logistic Regression¹², and Linear Discriminant Analysis¹³, to automatically classify ECG data. Scientists extract temporal variables from ECG data and then combine them using few-shot learning, support vector machines, decision trees, random forests, and nearest neighbor algorithms to create models that can diagnose arrhythmia¹⁴. Nevertheless, all of these machine learning algorithms require information discrimination, which refers to the process of extracting pertinent features. Furthermore, the accuracy of the model relies on the human interpretation of features and the requirement for workers to have a thorough understanding of medical expertise. These issues lead to the inconvenience of complex procedures in the machine learning process, as well as the inferior accuracy of the models compared to the expertise of cardiologists. This hinders the desired accuracy in automatically identifying arrhythmias using ECG data.

Deep learning is an approach that can be employed to identify arrhythmia in electrocardiogram (ECG) data without requiring manual extraction of features. This technology offers significant benefits as it can automatically extract crucial features from the ECG data without the need for professional knowledge in cardiology, hence saving time and effort⁶³. Presently, the application of deep learning techniques with ECG data for the diagnosis of arrhythmia is continuously progressing, demonstrating superior accuracy when compared to machine learning algorithms and signal processing algorithms, even surpassing the accuracy of cardiologists. Baloglu, Ulas Baran et al.¹⁵ proposed a 1D CNN model that can accurately diagnose myocardial infarction. When applied to all ECG lead data, the 1D CNN demonstrated excellent accuracy and sensitivity in MI detection. However, 1D CNN solely captures the sequence-space features of the sequence and does not capture the sequence-time features. In order to overcome the limitation of 1D CNN's ability to extract time-related features, researchers¹⁶ and ¹⁷ integrated recurrent cells into the framework of 1D CNN. The fusion model combining 1D CNN and recurrent cells is limited to capturing local ECG sequence features and lacks the ability to focus on extracting global features. Studies¹⁸ and ¹⁹ suggested a fusion of 1D CNN with attention, leveraging the benefits of the self-attention mechanism in Transformer to capture global features. Chun-Yen Chen et al.²⁰ proposed a multi-channel, multi-scale network that incorporates the features of the residual neural network. This network is capable of capturing information at many scales. Rongzhou Zhou et al.⁷ proposed a fusion model that combines 1D CNN and Transformer. They incorporated a spatial attention module and a channel attention module, which adaptively focus on significant features in both the global and local parts of the sequence space and sequence channel. Nevertheless, these methods are limited to process sequence-space features and sequence-time features, while disregarding picture information. In order to extract the intra-lead and inter-lead spatial features of the two-dimensional ECG, Jiahao Li et al.²¹ proposed a general model based on the two-dimensional ECG and ResNet's detached squeeze-and-excitation modules (DSE-ResNet). This model spliced the 12-lead ECG into a grayscale image and employs DSE-ResNet to extract both the intra-lead and inter-lead features of the two-dimensional ECG simultaneously. Rui Tao et al.²² proposed a flexible double-kernel residual block (DKR-block) to efficiently extract both intra-lead and inter-lead data. They then established a two-dimensional ECG classification model using the DKR block to precisely identify arrhythmia. Zicong Yang et al.⁶⁰ proposed a fusion model that combines ResNet with GRU, resulting in outstanding performance on the PTB-XL dataset.

Despite notable advancements in the accuracy of automatic arrhythmia diagnosis using deep learning algorithms based on multi-lead ECG data, there are still numerous unexplored opportunities and unresolved challenges in this field. A primary difficulty is that existing neural networks have the capability to extract either sequential spatio-temporal information or image spatial features from ECG data, but not both concurrently. Therefore, it is necessary to develop a deep-learning model that can extract both types of features simultaneously. ECG sequences have varying temporal features, and separate leads in a multi-lead ECG can capture diverse heartbeat sequences. Hence, it is imperative to incorporate a multi-scale temporal feature fusion mechanism into the neural network model in order to effectively capture and integrate these vital temporal features more precisely. Recurrent neural networks may have difficulties when dealing with lengthy sequence dependencies in ECG data and are unable to generate spatially selective attention maps for spatial features. It is valuable to investigate the utilization of an attention mechanism to overcome this constraint. 2D convolutional neural networks have constraints in capturing the entire perceptual field during downsampling, leading to insufficient acquisition of spatial features in images. Further research is required to enhance the functionality of 2D CNNs and optimize the extraction of spatial characteristics from images.

Based on the aforementioned discussions, we propose STFAC-ECGNet, comprising three modules: the CAMV-RNN block, the CBMV-CNN block, and the TSEF block. This model enables the simultaneous utilization of both electrocardiogram (ECG) signal sequences and two-dimensional ECG images to achieve a more robust representation. In addition, it considers the characteristics of multi-lead ECGs and utilizes a series of methodologies to enhance the features of ECG data. This results in more focused and comprehensive ECG features, ultimately enhancing the accuracy of arrhythmia detection. The experimental findings showcase the robustness and generalization capability of STFAC-ECGNet. The primary contributions of this study are as follows:

• 1. We propose the CAMV-RNN block, a module for sequence temporal and spatial feature fusion. It utilizes three branches for parallel computation, with one branch incorporating skip connections to enhance temporal features and achieve multi-scale temporal feature fusion. The CAMV-RNN block also introduces a coordinated adaptive simplified self-attention module, which adaptively performs global memory of the temporal and spatial features in the ECG sequence, enhancing the temporal information.

• 2. We constructed the CBMV-CNN block, which serves as a module for spatial feature fusion. It integrates spatial and channel attention mechanisms into a skip connection, enabling the combination of spatial and channel information to enhance the extraction of multi-lead ECG spatial features.

• 3. We assembled the TSEF (Temporal-Spatial Enhanced Fusion) block, which facilitates cross-dimensional connectivity and enables an enhanced multi-scale fusion of image spatial features and sequence spatio-temporal information.

• 4. The STFAC-ECGNet model architecture combines global feature maximization and global feature averaging fusion on Tensor feature values, enhancing the ability to capture ECG features in multi-lead electrocardiograms. This marks the first-ever endeavor in the field of electrocardiogram data to incorporate a strategy that combines multiple global feature maximization and global feature averaging fusion.

• 5. We conducted comprehensive experiments using the PTB-XL large-scale publicly accessible ECG dataset and the 2018 China Physiological Signal Challenge (CPSC 2018) database. The experimental results showcase the robustness and generalization capability of STFAC-ECGNet, outperforming other cutting-edge methods.

Related work

Machine learning for arrhythmia detection

Over the past few decades, machine learning based on electrocardiogram data have yielded significant research findings in the field of arrhythmia detection. Jianyong Yang et al.²³ proposed a hybrid feature selection method that combines a genetic algorithm (GA) with a support vector machine (SVM) to classify ECG arrhythmias. The study conducted by Mohebbanaaz, L. et al.²⁴ considered two classifiers: an optimized decision tree and an adaptively boosted optimized decision tree. These classifiers were specifically designed to classify six different types of heartbeats, encompassing normal beats and beats exhibiting moderate arrhythmia. A clustering-based feature reduction approach was developed in another study²⁵ for ECG-based biometric identification, aiming to achieve efficient and scalable classification.

Conventional machine learning methods encounter difficulties when it comes to arrhythmia detection. Preprocessing and characterizing the initial ECG signals is a procedure that requires a significant amount of time and effort. Constructing features appropriate for machine learning necessitates expertise in ECG analysis and extensive diagnostic experience, resulting in a significant reliance on expert subjective assessment throughout the entire process. The presence of subjectivity in this context might lead to biases and inconsistencies in preprocessing and feature engineering results, hence compromising the model's generalizability. Furthermore, variances among distinct ECG datasets encompass diverse data distributions, collection parameters, and quality criteria. These disparities necessitate multifaceted feature engineering approaches tailored to accommodate the unique characteristics inherent in each dataset. However, during various feature engineering procedures, ECG datasets may employ distinct signal processing methods, feature extraction algorithms, or feature selection strategies. This can lead to inconsistent features, making it challenging to generalize the model across diverse datasets. Furthermore, conventional machine learning methods encounter interference and artifacts when processing ECG data. The presence of these noises and artifacts can be attributed to various factors, including EMG interference, interference caused by respiratory movement, and loose electrodes. These factors can lead to a decline in classification performance and raise the likelihood of misclassifying data.

Deep learning for arrhythmia detection

Deep learning methods have demonstrated exceptional achievements in the field of diagnosing ECG arrhythmias in recent years. Automated arrhythmia detection becomes feasible by utilizing the powerful feature extraction and pattern recognition capabilities of deep learning technology. Debasish Jyotishi et al.²⁶ proposed a spatio-temporal learning network named ASTLnet for the automated diagnosis of cardiovascular diseases. This method utilizes the concurrent spatial variations in multi-lead ECG signals to acquire more accurate diagnostic representations. ASTLnet consists of two distinct modules: the Spatiotemporal Representation Learning (STRL) module and the Focus on Attentive Spatio-Temporal Aggregation (ASTA) module. Lang Qin et al.²⁷ proposed an ECG diagnostic neural network structure based on deformable CNN (deform-CNN). The structure makes full use of the characteristic learning capability of deformable convolution to learn the time domain and wire characteristics of multi-lead ECG signals. The MSGFormer network, proposed by Changqing Ji et al.²⁸, combines a multi-scale grid transformer architecture, self-attention based feature fusion, and a multi-scale grid attention mechanism. This integration aims to enhance feature extraction and capture intricate patterns in ECG signals, ultimately leading to improved classification accuracy. Quancheng Geng et al.²⁹ proposed an innovative multi-task deep neural network for electrocardiogram classification, which incorporates shared low-level feature extraction modules and task-specific classification modules. The authors also introduced context converter (CoT) blocks to dynamically model local and global information regarding ECG feature sequences. Yanfang Dong et al.¹⁸ proposed an end-to-end multi-label arrhythmia classification model named CNN-DVIT. This model integrates CNN with a deformable self-attention visual transformer structure, and introducted a spatial pyramid pooling layer to process ECG signals of varying lengths. Khiem H. Le et al.³¹ proposed a lightweight deep learning system called LightX3ECG, which uses three ECG leads (I, II, and V1) to accurately identify various cardiovascular abnormalities. A redesigned 1D-CNN backbone network was utilized for feature extraction, as well as a lead attention module was introduced for aggregation and classification. Pruning technique is employed to decrease the size of the system and is well-suited for deployments on platforms with hardware limitations.

Our proposed network introduces an innovative approach to enhanced multi-scale fusion, which effectively combines image spatial features and sequence spatio-temporal information for feature extraction in ECG data analysis. Additionally, we have incorporated a coordinated adaptive simplified self-attention module within the CAMV-RNN module. This module improves the network's ability to adaptively enhance the global memory of temporal features hence boosting the comprehension of time-related information. In comparison to various existing networks, our STFAC-ECGNet achieves superior performance on two publicly available datasets. These results highlight the effectiveness and potential of our proposed network in the field of ECG analysis.

Methods

This section presents an overview of the neural network architecture. We then provide a comprehensive description of the constituent components of our network. Specifically, the network comprises three primary components: the CAMV-RNN Block, the CBMV-CNN Block, and the TSEF Block.

Method overview

The structure of our proposed network is illustrated in Fig. 1. Our network employs a cross-dimensional connection strategy, utilizing a dual-branch structure that separates into an electrocardiogram (ECG) signal sequence processing branch and an ECG image processing branch. In the ECG signal sequence processing branch, in order to extract multi-scale features of different ECG signals and reduce the redundancy caused by similar features learned by the same recurrent neural network in different branches, thus reducing the importance of multi-scale learning, we introduce two bidirectional recurrent neural networks (BiLSTM³² and BiGRU³³) to compute the signal sequences in parallel. At the same time, BiGRU is easier to train and converges faster on small datasets because it is a lightweight network improved from BiLSTM, making it advantageous for handling small datasets⁶¹. On the other hand, BiLSTM has more parameters, making it more effective for handling large datasets⁶². In our paper, we used a small dataset, CPSC 2018, and a large dataset, PTB-XL, so this multi-scale design is better suited for datasets of different sizes. The upper branch undergoes the first BiGRU, followed by tensor global maximization and global averaging fusion. The skip connection branch then passes through the second BiGRU to enhance the temporal features of the sequence. Subsequently, the temporal features of the two branches are fused. In the lower branch of the ECG signal sequence processing, the BiLSTM is connected to the co-attentive adaptive attention module, enabling the fusion of spatio-temporal features with the upper branch. Meanwhile, a spatial channel attention module is incorporated into the skip connection of the ECG image processing branch to enhance the spatial features of the grayscale 2D ECG images. To achieve cross-dimensional connectivity, we propose the Temporal-Spatial Enhanced Fusion (TSEF) Block, which employs a strategy of dual global feature maximization and global feature averaging fusion to enhance the fusion of spatial–temporal features from images. For the task of arrhythmia classification, we incorporate a Multilayer Perceptron (MLP)³⁴ as the last layer of the network to predict the probability scores for each class.

Fig. 1.

STFAC-ECGNet network architecture. $\oplus$ represents feature fusion. STFAC-ECGNet consists of three modules: a CAMV-RNN block, b CBMV-CNN block, and c TSEF Block. $f_{\mathit{Max}}({\mathit{t}}_{\mathit{o}}^{\mathit{B}})$ represents the global maximization of the Batch dimension of the tensor ${\mathit{t}}_{\mathit{o}}^{\mathit{B}}\in {\mathbb{R}}^{W\times C}$, $f_{\mathit{Avg}}({\mathit{t}}_{\mathit{o}}^{\mathit{B}})$ represents the global averaging of the Batch dimension of the tensor ${\mathit{t}}_{\mathit{o}}^{\mathit{B}}\in {\mathbb{R}}^{W\times C}$. $f_{\mathit{Max}}({\mathit{t}}_{\mathit{o}}^{\mathit{W}})$ represents global maximized sequence feature extraction on the width dimension of ${\mathit{t}}_{\mathit{o}}^{\mathit{W}}\in {\mathbb{R}}^{B\times W\times C}$. $f_{\mathit{Avg}}({\mathit{t}}_{\mathit{o}}^{\mathit{W}})$ represents global average sequence feature extraction on the width dimension of ${\mathit{t}}_{\mathit{o}}^{\mathit{W}}\in {\mathbb{R}}^{B\times W\times C}$. $f_{\mathit{Max}}({\mathit{x}}_{\mathit{o}}^{\mathit{H}})$ represents global maximized spatial feature extraction on the height dimension of ${\mathit{x}}_{\mathit{o}}^{\mathit{H}}\in {\mathbb{R}}^{B\times C\times W\times H}$, while $f_{\mathit{Max}}({\mathit{x}}_{\mathit{M}}^{\mathit{W}})$ represents global maximized spatial feature extraction on the width dimension of ${\mathit{x}}_{\mathit{M}}^{\mathit{W}}\in {\mathbb{R}}^{B\times W\times C}$. $f_{\mathit{Avg}}({\mathit{x}}_{\mathit{o}}^{\mathit{H}})$ represents global average spatial feature extraction on the height dimension of ${\mathit{x}}_{\mathit{o}}^{\mathit{H}}\in {\mathbb{R}}^{B\times C\times W\times H}$, while $f_{\mathit{Avg}}({\mathit{x}}_{\mathit{A}}^{\mathit{W}})$ represents global average spatial feature extraction on the width dimension of ${\mathit{x}}_{\mathit{A}}^{\mathit{W}}\in {\mathbb{R}}^{B\times W\times C}$.

Spatio-temporal fusion of ECG signal sequences

The CAMV-RNN Block depicted in Fig. 1 enables the fusion of spatial and temporal information in electrocardiogram (ECG) signal sequences. It utilizes a multi-scale parallel computing approach to perform multi-scale spatio-temporal fusion on the ECG sequences. By incorporating skip connections, it enhances the temporal features of the ECG sequences. Moreover, the network architecture incorporates GRU cells, LSTM cells, and a coordinated adaptive attention module.

Bidirectional long short-term memory

LSTM has a limitation in that it can only process sequence data in one direction. To overcome this limitation, BiLSTM is an improved version of LSTM. BiLSTM is also highly effective in ECG disease detection³². By introducing bidirectionality, BiLSTM can consider both past and future temporal information of sequence data simultaneously. It consists of two LSTM layers, one processing the forward sequence and the other handling the reverse sequence. The bidirectional network can learn sequence patterns by considering inputs from both the beginning and end hidden states of the sequence, thus capturing more comprehensive contextual information. In the forward processing, BiLSTM traverses each timestep of the input sequence, progressively updating the forward hidden state and capturing the context information before the current timestep. The reverse process follows a similar methodology but traverses in the opposite direction, capturing context information from the current timestep and thereafter. Finally, the forward and reverse hidden states are combined to obtain a comprehensive representation that incorporates contextual information from the entire sequence.

Bidirectional gated recurrent unit

In the context of multi-scale time feature fusion, the upper branch of the CAMV-RNN network incorporates BiGRU. BiGRU is composed of Gated Recurrent Units (GRU)³⁷. GRU is a simplified recurrent neural network (RNN) structure. Compared to Long Short-Term Memory (LSTM), GRU combines the forget gate and input gate of LSTM into an update gate and a reset gate. This simplification reduces the number of parameters and computational complexity, making the GRU model easier to train and use. The GRU structure includes a hidden state and an update state. At each time step, GRU calculates the update state for the current time step based on the input data and the previous time step's hidden state. The update state determines the proportion of the old hidden state to retain and the introduction of new information. This update state is computed using a gate mechanism called the update gate, which has a value range of (0, 1). In other words, the update gate determines the ratio of retaining the old hidden state to introducing new information. In addition to the update gate, GRU introduces a reset gate. The reset gate determines which parts of the old hidden state should be forgotten. By utilizing the reset gate, GRU can selectively discard certain aspects of the previous hidden state, reducing reliance on past information. Using GRU networks for ECG disease detection has become an important trend in cardiology. It can save a significant amount of computational resources and achieve better sequence feature extraction, making it particularly well-suited for long time series like ECG signals. Similar to BiLSTM, BiGRU combines the forward and backward hidden states to better capture temporal information in sequential data.

Coordinated adaptive simplified self attention mechanism

The attention mechanism is a crucial technique in deep learning, utilized to dynamically allocate attention weights within a model. For instance, Channel Attention³⁸ is a type of attention mechanism employed in deep neural networks to dynamically adjust weights among channels. Its primary objective is to capture the correlation and significance among different channels, enhancing the network's modeling capability for various features. However, channel attention mechanisms solely focus on channel information and cannot extract features regarding spatial information.

In the lower branch of the CAMV-RNN, there exists a CASSAN (Coordinated Adaptive Simplified Self-Attention) block, as illustrated in Fig. 2. It is designed to capture global information from electrocardiogram (ECG) sequence data. The CASSAN block introduces a coordinated attention mechanism, reconstructing channels to visualize attention maps and extract spatial information in the feature space. Simultaneously, the model omits the scaling and masking operations of self-attention, significantly reducing computational complexity and parameter count. The CASSAN block comprises three parts (Query, Key, Value). Assuming the input tensor after BiLSTM is denoted as $\mathit{T}\in {\mathbb{R}}^{B\times W\times C}$, where B represents batch size, W denotes the sequence length, and C signifies the number of channels.

Fig. 2.

The coordinated attention mechanism aims to establish relationships between feature channels, reweight the importance of each channel, and obtain spatial information in the feature space. Here, "AdaptiveAvgPool2d" represents adaptive horizontal global pooling, ⊗ denotes tensor product, and "softmax" refers to the activation function.

The query attention map $\mathit{Q}\in {\mathbb{R}}^{B\times C_c\times W\times H}$, key attention map $\mathit{K}\in {\mathbb{R}}^{B\times C_c\times H\times W}$, and value attention map $\mathit{V}\in {\mathbb{R}}^{B\times C_c\times W\times H}$ undergo reshaping by and permutation, resulting in tensors with modified dimensions and reconstructed channels $C_c$, thereby enhancing channel information. Subsequently, $\mathit{Q}\in {\mathbb{R}}^{B\times C_c\times W\times H}$ and $\mathit{K}\in {\mathbb{R}}^{B\times C_c\times H\times W}$ pass through an adaptive average pooling layer $\mathcal{A}p$ to obtain spatially height-adaptive weights ${\mathit{Q}}_{\mathbf{w}}\in {\mathbb{R}}^{B\times C_c\times W\times 1}$ and ${\mathit{K}}_{\mathbf{w}}\in {\mathbb{R}}^{B\times C_c\times 1\times W}$, respectively, extracting global information adaptively. Then, the query attention map $\mathit{Q}\in {\mathbb{R}}^{B\times C_c\times W\times H}$ is concatenated with the spatially height-adaptive weight map ${\mathit{Q}}_{\mathbf{w}}\in {\mathbb{R}}^{B\times C_c\times W\times 1}$ along the third dimension to obtain the query attention map ${\mathit{Q}}_{\mathit{c}}\in {\mathbb{R}}^{B\times C_c\times W\times (H+1)}$, and the key attention map $\mathit{K}\in {\mathbb{R}}^{B\times C_c\times H\times W}$ is concatenated with the spatially height-adaptive weight map ${\mathit{K}}_{\mathbf{w}}\in {\mathbb{R}}^{B\times C_c\times 1\times W}$ along the second dimension to obtain the key attention map ${\mathit{K}}_{\mathit{c}}\in {\mathbb{R}}^{B\times C_c\times (H+1)\times W}$.

Meanwhile, to tackle the challenge of dimension mismatch, we employed the Broadcast method. This method may flexibly manage matrices with varying shapes by appropriately expanding their dimensions to achieve shape congruence. Consequently, element-wise operations become feasible between the elements. We compute the feature similarity by multiplying the concatenated query attention map ${\mathit{Q}}_{\mathit{c}}\in {\mathbb{R}}^{B\times C_c\times W\times (H+1)}$ with the concatenated key attention map ${\mathit{K}}_{\mathit{c}}\in {\mathbb{R}}^{B\times C_c\times (H+1)\times W}$, resulting in $\mathit{M}\in {\mathbb{R}}^{B\times C_c\times W\times W}$. After multiplying the ${\mathit{Q}}_{\mathit{c}}$ and ${\mathit{K}}_{\mathit{c}}$​ matrices, we divide by the scalar $\sqrt{C}$ to obtain the final lightweight feature similarity $\mathit{M}_{}^{_{}^{\mathbf{\prime }}}\in {\mathbb{R}}^{B\times C_c\times W\times W}$. Perform element-wise multiplication ∘ between $\mathit{M}_{}^{_{}^{\mathbf{\prime }}}\in {\mathbb{R}}^{B\times C_c\times W\times W}$ and the broadcasted spatial height-adaptive weight maps $Q_B^W\in {\mathbb{R}}^{B\times C_c\times W\times W}$ and $K_B^W\in {\mathbb{R}}^{B\times C_c\times W\times W}$, resulting in spatial feature similarity $\frac{\mathit{M}_{}^{_{}^{\mathbf{\prime }}}}{\sqrt{C}}\circ (Q_B^W+K_B^W)\in {\mathbb{R}}^{B\times C_c\times W\times W}$. Finally, sum the lightweight feature similarity and the spatial feature similarity to obtain the spatial similarity feature map ${\mathit{A}\mathit{t}\mathit{t}}_{\mathit{M}}\in {\mathbb{R}}^{B\times C_c\times W\times W}$. Subsequently, the feature map that exhibits spatial similarity is subjected to a Softmax function, to ensure that the weights are distributed between 0 and 1, therefore producing spatially similar weights. Afterward, the spatial similarity feature map ${\mathit{A}\mathit{t}\mathit{t}}_{\mathit{M}}\in {\mathbb{R}}^{B\times C_c\times W\times W}$ and the value attention map $\mathit{V}\in {\mathbb{R}}^{B\times C_c\times W\times H}$ undergo matrix multiplication, resulting in $\mathbf{A}\mathbf{t}\mathbf{t}_{\mathit{S}}^{_{}^{\mathbf{\prime }}}\in {\mathbb{R}}^{B\times C_c\times W\times H}$. Subsequently, $\mathbf{A}\mathbf{t}\mathbf{t}_{\mathit{S}}^{_{}^{\mathbf{\prime }}}\in {\mathbb{R}}^{B\times C_c\times W\times H}$ undergoes permutation, followed by a copying operation on the permuted tensor. Lastly, the dimensions of the copied tensor are adjusted to obtain ${\mathit{A}\mathit{t}\mathit{t}}_{\mathit{S}}\in {\mathbb{R}}^{B\times W\times C}$.

CAMV-RNN

The key to processing electrocardiogram (ECG) signals is the temporal-spatial fusion of the ECG signal sequences. Figure 1a shows the CAMV-RNN Block, which is a network structure that enables such fusion. This structure incorporates a multi-scale parallel computing approach to extract multi-scale temporal-spatial features from the ECG sequences. Additionally, the network architecture integrates BiGRU (Bidirectional Gated Recurrent Unit), BiLSTM (Bidirectional Long Short-Term Memory), and Coordinated Adaptive Attention modules to enhance the extraction capability of temporal features.

Assuming the input electrocardiogram (ECG) sequence data contains 12 leads, denoted as $\mathit{T}\in {\mathbb{R}}^{B\times W\times C}$, where B represents the batch size, C represents the number of leads, and W represents the sequence length. The data is processed through a dual-branch network to extract multi-scale temporal features from the sequence. In the upper branch, the tensor $\mathit{T}\in {\mathbb{R}}^{B\times W\times C}$ first passes through the first BiGRU layer to obtain ${\mathit{t}}_{\mathit{o}}^{\mathit{B}}$ $\in {\mathbb{R}}^{B\times W\times C}$. Subsequently, the tensor ${\mathit{t}}_{\mathit{o}}^{\mathit{B}}$ $\in {\mathbb{R}}^{B\times W\times C}$,undergoes global feature fusion across the batch dimension through global maximization $f_{\mathit{Max}}({\mathit{t}}_{\mathit{o}}^{\mathit{B}})$ and global averaging $f_{\mathit{Avg}}({\mathit{t}}_{\mathit{o}}^{\mathit{B}})$, resulting in the fused feature representation $\mathcal{U}\in {\mathbb{R}}^{W\times C}$ of the upper branch. In the skip connection branch, the ECG sequence features undergo batch normalization, dropout, and ReLU activation to enhance the temporal features of the sequence, producing the skip-connected feature values $\mathcal{J}\in {\mathbb{R}}^{B\times W\times C}$. This is followed by processing through the second BiGRU layer. The temporal features from the upper branch and the skip connection branch are subsequently fused. In the lower branch, the data initially passes through a BiLSTM layer and is then processed by a coordinated adaptive attention module, resulting in $\mathcal{L}\in {\mathbb{R}}^{B\times W\times C}$. The output is then fused with the spatio-temporal features from the upper branch, which are broadcasted along the $\mathit{B}$ dimension as ${\mathcal{U}}_B\prime \in {\mathbb{R}}^{B\times W\times C}$, and the features from the skip connection branch $\mathcal{J}\in {\mathbb{R}}^{B\times W\times C}$. Ultimately, the derived features from $({\mathcal{U}}_B\prime +\mathcal{J}+\mathcal{L})\in {\mathbb{R}}^{B\times W\times C}$ are processed through batch normalization, dropout, and ReLU activation to obtain the fused spatio-temporal features ${\mathcal{F}}_{\mathit{s}}\in {\text{R}}^{B\times W\times C}$ of the ECG sequence.

Spatial fusion of electrocardiogram images

CBMV-CNN

As shown in Fig. 1b CBMV-CNN achieves spatial fusion of electrocardiogram (ECG) image data. The 12-lead ECG signals are first converted into single-channel grayscale images. This conversion facilitates the extraction of features not only within the same lead but also across different leads from the two-dimensional grayscale ECG images. The variations in pixel intensities within the images play a vital role in the extraction of two-dimensional features. Next, CBMV-CNN adopts a single-scale pyramid structure combined with spatial and channel attention mechanisms in the skip connection. This approach effectively extracts spatial features within and between ECG leads while enhancing channel information. The introduction of the skip connection avoids excessive computation and enhances spatial features after convolutional feature extraction. Additionally, global average pooling and global maximum pooling are utilized to map the features and connect them to the next convolutional block, enhancing the spatial information obtained from convolutional feature extraction.

Convolutional block attention module

The attention mechanism is a widely used technique in the field of computer vision that effectively focuses on crucial features in input images and allocates limited computational resources to these important features during the training process, thereby improving the efficiency and accuracy of the model. CBAM (Convolutional Block Attention Module)⁴⁰ is a lightweight attention module composed of two sub-modules: the Channel Attention Module shown in Fig. 3 and the Spatial Attention Module shown in Fig. 4. These sub-modules are used to compute attention weights on the channel dimension and spatial dimension, respectively. In the channel attention module, the input features with height (H), width (W), and channel number (C) are separately passed through average pooling and max pooling layers. Then, these features undergo processing through two perceptron layers, resulting in two processed channel features. By combining these features using weights obtained through a sigmoid activation function, the channel attention features are obtained by multiplying the weights with the initial input features. In the spatial attention module, the input features are processed through connected average pooling and max pooling layers, considering the height, width, and channel dimensions. Subsequently, a 7 × 7 convolutional layer with a sigmoid activation function is applied to obtain spatial weights. Finally, the spatial attention features are obtained by multiplying these spatial weights with the input features.

Fig. 3.

Channel Attention Module.

Fig. 4.

Spatial Attention Module.

An enhanced multi-scale fusion of image spatial features and sequence spatial–temporal information

Based on Fig. 1, we utilize cross-dimensional connections to enhance the effectiveness of space–time feature fusion in a multiscale manner. In the depicted (c) TEST block, we first extract sequence length features along the width dimension from the output ${\mathit{t}}_{\mathit{o}}^{\mathit{W}}$ $\in {\mathbb{R}}^{B\times W\times C}$ of CAMV-RNN using functions $f_{\mathit{Max}}$ and $f_{\mathit{Avg}}$, resulting in the maximized sequence length feature $\mathit{m}\mathit{a}\mathit{x}{\mathit{t}}_{\mathit{o}}$ $\in {\mathbb{R}}^{B\times C}$ and the averaged sequence length feature $\mathit{a}\mathit{v}\mathit{g}{\mathit{t}}_{\mathit{o}}$ $\in {\mathbb{R}}^{B\times C}$. These two features are then concatenated to obtain the temporal fusion feature ${\mathit{t}}_{\mathit{R}\mathit{N}\mathit{N}}$ $\in {\mathbb{R}}^{B\times C}$. Subsequently, in the (b) CBMV-CNN block, we perform spatial maximization and spatial averaging feature extraction on the output ${\mathit{x}}_{\mathit{o}}^{\mathit{H}}$ $\in {\mathbb{R}}^{B\times C\times W\times H}$ of CBMV-CNN using functions $f_{\mathit{Max}}$ and $f_{\mathit{Avg}}$ along the height dimension, resulting in $x_M^W$ $\in {\mathbb{R}}^{B\times C\times W}$ and ${\mathit{x}}_{\mathit{A}}^{\mathit{W}}$ $\in {\mathbb{R}}^{B\times C\times W}$. We then apply the same functions $f_{\mathit{Max}}$ and $f_{\mathit{Avg}}$ along the width dimension on the resulting tensor, ultimately obtaining $\mathit{m}\mathit{a}\mathit{x}{\mathit{x}}_{\mathit{o}}$ $\in {\mathbb{R}}^{B\times C}$ and $avg{x}_o$ $\in {\mathbb{R}}^{B\times C}$. Finally, we combine the temporal fusion feature ${\mathit{t}}_{\mathit{R}\mathit{N}\mathit{N}}\in {\mathbb{R}}^{B\times C}$ with the spatial fusion features $\mathit{m}\mathit{a}\mathit{x}{\mathit{x}}_{\mathit{o}}$ $\in {\mathbb{R}}^{B\times C}$ and $\mathit{a}\mathit{v}\mathit{g}{\mathit{x}}_{\mathit{o}}\in {\mathbb{R}}^{B\times C}$ to achieve spatio-temporal fusion, yielding ${\mathit{x}}_{\mathit{o}}\in {\mathbb{R}}^{B\times C}$. Subsequently, a multilayer perceptron (MLP) is applied to ${\mathit{x}}_{\mathit{o}}$ to obtain the probabilities for different classes.

Experiment

Dataset

We utilized two datasets to assess the performance of our network, which are extensively employed for evaluating ECG algorithm detection. For the 12-lead ECG CPSC 2018 dataset⁴¹, we employed ten-fold cross-validation for data partitioning. This approach ensures that each data subset is utilized for testing, thereby validating the network's generalization capability. Regarding the 12-lead ECG PTB-XL dataset⁴², we employed stratified random sampling to divide it into ten parts. The publisher recommended the ninth part as the validation set, while the tenth part was designated as the test set⁴³. This partitioning scheme guarantees dataset diversity and impartiality.

CPSC 2018 dataset

The CPSC 2018 dataset serves as a pivotal resource for evaluating the performance of arrhythmia recognition algorithms. It aggregates 6,877 electrocardiogram (ECG) recordings from 11 hospitals, encompassing normal and 8 abnormal signal types, including atrial fibrillation (AF), first-degree atrioventricular block (I-AVB), left bundle branch block (LBBB), right bundle branch block (RBBB), premature atrial contraction (PAC), premature ventricular contraction (PVC), ST-segment depression (STD), and ST-segment elevation (STE). Each record comprises ECG signals from 12 leads along with annotations for arrhythmias, and additional patient information such as age, gender, height, and weight. Given its extensive scale, diversity, and high-quality annotations, this dataset has emerged as an industry standard in electrocardiogram signal processing research, holding paramount importance for early diagnosis and treatment of cardiac diseases. Table 1 displays detailed descriptions for each class.

Table 1.

The exact names of all types of arrhythmias in this list are as follows: (1) Atrial Fibrillation (AF) (2) First-degree Atrioventricular Block (I-AVB) (3) Left Bundle Branch Block (LBBB) (4) Right Bundle Branch Block (RBBB) (5) Premature Atrial Contraction (PAC) (6) Premature Ventricular Contraction (PVC) (7) ST-Segment Depression (STD) (8) ST-Segment Elevation (STE).

Superclass	Class	Description	Records	Duration(h)	Duration of a single segment(s)
					Mean	SD	Min	Max
NORM	Normal	Normal ECG	918	3.93	15.43	7.61	10.00	60.00
AF	AF	Atrial fibrillation	1098	5.11	15.01	8.39	9.00	60.00
QRS	I-AVB	First-degree atrioventricular block	704	2.89	14.32	7.21	10.00	60.00
	LBBB	Left bundle branch block	207	0.83	14.92	8.09	9.00	60.00
	RBBB	Right bundle branch block	1695	6.85	14.42	7.60	10.00	60.00
V	PAC	Premature atrial contraction	556	2.96	19.46	12.36	9.00	60.00
	PVC	Premature ventricular contraction	672	3.67	20.21	12.85	6.00	60.00
ST	STE	ST-segment elevation	825	0.87	17.15	10.72	10.00	60.00
	STD	ST-segment depression	202	3.35	15.13	6.82	8.00	60.00
	Total		6877	30.46	15.79	9.04	6.00	60.00

PTB-XL dataset

The PTB-XL ECG dataset is a large dataset that contains 21,799 clinical 12-lead ECG recordings from 18,869 patients. The distribution of these patients consisted of 52% male and 48% female. Each recording spans a duration of 10 s. The raw waveform data underwent annotation by a maximum of two cardiologists, potentially resulting in multiple ECG interpretations assigned to each recording. The five most important diagnoses among the 71 different ECG statements, which conform to the SCP-ECG standard, are categorized as ECG (NORM), conduction disturbance (CD), myocardial infarction (MI), hypertrophy (HYP), and ST/T changes (STTC). To ensure the comparability of deep learning networks trained on the dataset, folds 1–8 were used as the training set, fold 9 as the validation set, and fold 10 as the test set. The dataset includes comprehensive annotated information, making it a valuable resource for training and evaluating automated ECG interpretation algorithms. Additionally, the dataset provides extensive demographic information, infarct features, the likelihood of diagnostic ECG statements, and metadata on annotated signal properties.

Introduction of the comparison network

Baseline deep learning networks for arrhythmia classification in the PTB-XL dataset

Xresnet1d101⁴³ is an adaptive multi-label 12-lead electrocardiogram analysis method based on an improved ResNet architecture, demonstrating excellent performance on the PTB-XL dataset. FCN-Wang⁴⁴, comprising three convolutional layers and a global average pooling layer, is effective for disease classification in electrocardiogram sequence data. LSTM, designed to address long-term dependencies in recurrent neural networks, shows promising results in electrocardiogram sequence data. Inception1d⁴⁵ adapts the popular Inception architecture to time-series data. ECG-DNN⁴⁶ is an end-to-end deep learning network, outperforming cardiologists in identifying six types of abnormalities in 12-lead electrocardiogram recordings. DNN_zhu⁴⁷ is a resource-efficient and device-adaptive electrocardiogram diagnostic model based on a dynamic neural network with multiple classifiers. Resnet34_2d⁴⁸ is a 34-layer residual neural network structure used for image recognition, while Resnet34_1d is a one-dimensional residual neural network based on the improved Resnet34_2d architecture. Image_CNN⁴⁹ transforms electrocardiogram records into two-dimensional grayscale images and inputs them into a convolutional neural network.

State-of-the-art networks for arrhythmia classification in PTB-XL dataset

SincNet⁵⁰: Each channel of the 12-channel ECG signal is processed by a dedicated SincNet module. SE-ResNet1⁵¹ is utilized for single-lead electrocardiogram image classification, while SE-ResNet12 is employed for 12-lead electrocardiogram classification. LightX3ECG³¹: An effective and precise model for diagnosing diseases in I, II, and V1 lead electrocardiograms. 2D-ECGNet²² proposes a flexible Dual-Kernel Residual (DKR) block to effectively extract features within and across leads, establishing a two-dimensional ECG classification model to accurately detect arrhythmias. 1D-ECGNet²²: The 1D version of 2D-ECGNet. SE-ResNet-CoT-BiGRU²⁹: Shares a low-level feature extraction module and a task-specific classification module. Introduces a Context Transformer (CoT) block in the classification module to dynamically model local and global information of electrocardiogram feature sequences. Encoder-Decoder-KNN⁵²: A deep learning network integrating encoder-decoder and KNN, achieving excellent performance on two datasets.

Baseline deep learning networks for arrhythmia classification in CPSC2018 dataset

TI-CNN¹⁷: A neural network that combines CNN with LSTM. CNN-BiLSTM⁵³: Integrates the residual neural network ResNet with BiLSTM, exhibiting excellent performance in detecting diseases from electrocardiograms. ATI-CNN¹⁹: Achieves spatial and temporal fusion of electrocardiogram signal information through the integration of CNN, recurrent units, and attention modules. CNN-BiRNN-Attention¹⁸: Secured the first position in the CPSC 2018 ECG Challenge. CNN-Transformer⁵⁴: A network that embeds a transformer network within CNN to capture the temporal information of ECG signals. Deform-CNN²⁷: A neural network structure for electrocardiogram diagnosis based on deformable CNNs.

State-of-the-art networks for arrhythmia classification in CPSC2018 dataset

DAMS-Net⁷: This architecture combines CNN with a spatial attention module and a channel attention module using a CNN-transformer hybrid encoder. ASTLNet²⁶: Comprising the Spatio-Temporal Representation Learning (STRL) module and the Attention Spatio-Temporal Aggregation (ASTA) module. LightX3ECG³¹: An effective and precise model for diagnosing diseases in I, II, and V1 lead electrocardiograms. SE-ResNet-CoT-BiGRU²⁹: Shares a low-level feature extraction module and a task-specific classification module. Introduces a Context Transformer (CoT) block in the classification module to dynamically model local and global information of electrocardiogram feature sequences. CNN-DVIT³⁰: A combination architecture of Convolutional Neural Network (CNN) with depth-wise separable convolutions and a visual transformer structure with deformable attention. LFG-Net⁵⁵: A Guided Feature-Guided Network (LFG-Net) that enhances the feature representation capability of the network by focusing on guide features with high contributions.

Experiments platform

Our experiment was conducted using the FastAi framework⁵⁶. Our experiment was performed on a cloud computing platform, utilizing one RTX 3090 (24 GB) GPU and 16 vCPU Intel(R) Xeon(R) Platinum 8350C CPU @ 2.60 GHz.

Parameter setting

To enhance convergence speed and curtail training durations across two datasets, we configured the learning rate at 3e-3, employing the OneCycleLR⁵⁷ optimizer in tandem with a cosine annealing strategy for learning rate modulation. We further defined the batch size as 128. Our model incorporates a Cooperative Adaptive Attention Network, with a channel reconstruction parameter set to c = 8. In addition, our network utilizes a bidirectional recurrent neural network with the hidden state dimensionality established at $h=256$ Leveraging the super-convergence capability of the OneCycleLR⁵⁷ optimizer, we limited the training epochs to 10, thereby substantially reducing both the time and computational resource expenditure. Concurrently, the terminal layer of the Multi-Layer Perceptron (MLP) is designed with a single hidden state dimensionality of $h_{\mathit{MLP}}=256$, optimizing the model's performance and efficiency.

Evaluation metrics

In this investigation, we adopted classic evaluative indicators to validate the efficacy of STFAC-ECGNet, encompassing accuracy, precision, recall, F1-score, and Area Under the Curve (AUC). Accuracy quantifies the ratio of correctly predicted instances to the overall predictions, serving as a fundamental measure of the model's classification prowess. Recall, or the true positive rate, gauges the model's capacity to correctly identify positive instances within the corpus of actual positives. Precision assesses the accuracy of positive predictions, reflecting the model's specificity. The F1-score, amalgamating precision and recall, offers a balanced metric of the model's comprehensive effectiveness. Additionally, the AUC metric, delineating the area beneath the ROC curve, evaluates the model's predictive accuracy across varied decision thresholds.

$$Accuracy=\frac{TP+TN}{TP+FP+FN+TN}$$ 1

$$Recall=\frac{\mathit{TP}}{TP+FN}$$ 2

$$Precision=\frac{\mathit{TP}}{TP+FP}$$ 3

$$\text{F}1=2\times \frac{Precision\times Recall}{Precision+Recall}$$ 4

$${\mathit{AUC}}_{\mathit{macro}}=\frac{{\sum }_{i\in positiveClass}{\mathit{rank}}_i-\frac{m(1+m)}{2}}{m\times n}$$ 5

True Positives (TP) quantify the instances where the model accurately identifies positive samples. Conversely, False Positives (FP) reflect the instances wherein negative samples are mistakenly classified as positive. False Negatives (FN) occur when positive samples are erroneously predicted to be negative, and True Negatives (TN) account for the negative samples correctly recognized by the model. The notation ${\mathit{rank}}_i$ elucidates the sequential order of the model's predictive scores, where $\mathcal{m}$ delineates the total number of positive samples, and $\mathcal{n}$ specifies the total number of negative samples, and $\mathit{positiveClass}$ represents the index of positive samples.

Implementation details

In this investigation, data preprocessing was conducted on two electrocardiogram (ECG) datasets to facilitate data normalization. In an effort to optimize computational efficiency, the sampling frequency within the CPSC 2018 dataset was reduced from 500 to 150 Hz. Consistency in analysis length was ensured by upsampling the ECG data records to maintain each sequence at a maximum length of 60 s. We employed a ten-fold cross-validation approach to fully leverage each data segment in network evaluation, thus examining the model’s capacity for generalization and robustness through meticulous parameter selection and testing across datasets. For the PTB-XL dataset, characterized by 10-s recordings, a sampling rate of 100 Hz was selected, adhering to an expert-recommended train-test division. Recognizing the dataset's potential for symptoms to be classified into multiple categories, we accommodated samples corresponding to multiple labels through one-hot encoding. Furthermore, we introduced a 2.5-s sliding window with 50% overlap as a strategy for ECG data augmentation. Addressing the complexities of multi-label classification, the F1-score was established as the network metric, discarding argmax in favor of an adaptive testing methodology to ascertain the optimal threshold $\mathcal{s}\in [\text{0,1}]$, culminating in the identification of the network's superior F1-score and the optimal model configuration.

Result

To validate the proposed models, we employed two multi-lead electrocardiogram datasets mentioned in Sect. 4.1. Baseline models referenced in Sect. 4.2 and state-of-the-art models were utilized for comparison, juxtaposed with STFAC-ECGNet.

Results of PTB-XL dataset

Table 2 presents the measurement results of our models on the PTB-XL dataset's test set, where the top two rankings for each metric are indicated in bold italics. Our proposed CBMV-CNN, CAMV-RNN, and the combined STFAC-ECGNet have achieved excellent performance across all metrics. CBMV-CNN achieved the highest Accuracy (0.880), highest AUC (0.935), highest Recall (0.796), Precision (0.725), and F1-score (0.749). CAMV-RNN attained Accuracy (0.889), AUC (0.932), Recall (0.783), Precision (0.748), and F1-score (0.763). STFAC-ECGNet achieved the highest Accuracy (0.894), ranked second in AUC (0.933), Recall (0.756), Precision (0.778), and ranked second in F1-score (0.767). Therefore, our proposed networks outperform other baselines and state-of-the-art networks in terms of Accuracy, AUC, Recall, and F1-score, but there is room for improvement in Precision. Overall, using F1-score as the benchmark, STFAC-ECGNet only differs from the state-of-the-art network 2D-ECGNet by a margin of 0.003.

Table 2.

Diagnostic performance comparison with other baseline and advanced networks on the PTB-XL test dataset.

Network	Accuracy	AUC	Recall	Precision	F1-score
Xresnet1d10143	0.885	0.929	0.705	0.780	0.741
FCN-Wang44	0.880	0.926	0.786	0.729	0.756
LSTM35	0.876	0.927	0.800	0.706	0.750
ResNet-Wang43	0.877	0.749	0.795	0.712	0.751
Inception1d45	0.876	0.926	0.788	0.711	0.748
ECG-DNN46	0.884	0.924	0.684	0.793	0.734
DNN-zhu47	0.890	0.918	0.774	0.758	0.766
Resnet34_1d48	0.882	0.908	0.691	0.778	0.732
Resnet34_2d48	0.879	0.911	0.706	0.722	0.714
Image_CNN49	0.888	0.921	0.764	0.721	0.742
SincNet50	0.765	0.910	0.662	0.714	0.687
SE-ResNet151	0.862	0.889	0.599	0.753	0.667
SE-ResNet1251	0.880	0.923	0.694	0.772	0.731
LightX3ECG31	0.884	0.920	0.681	0.795	0.734
1D-ECGNet22	0.884	0.919	0.696	0.780	0.736
2D-ECGNet22	0.892	0.929	0.790	0.752	0.770
CBMV-CNN(ours)	0.880	0.935	0.796	0.725	0.759
CAMV-RNN(ours)	0.889	0.932	0.783	0.748	0.765
STFAC-ECGNet(ours)	0.894	0.933	0.756	0.778	0.767

Table 3 details the evaluation metrics of distinct networks across superclass divisions on the PTB-XL dataset. In the CD (Cardiac Dysrhythmia) diagnostic category, STFAC-ECGNet secured the top Accuracy and Precision, registering scores of 0.892 and 0.869, respectively. For the HYP (Hypertrophy) classification, CBMV-CNN excelled in Accuracy, achieving a remarkable 0.918. In the MI (Myocardial Infarction) category, CBMV-CNN led in AUC, Recall, and F1-score, with exemplary scores of 0.943, 0.863, and 0.853, respectively. STFAC-ECGNet matched CBMV-CNN's prime AUC achievement in this segment. Within the NORM (Normal) classification, STFAC-ECGNet attained a AUC of 0.949, a score equaled by CAMV-RNN, which also realized the highest Recall at 0.883. For the STTC (ST-T Change) group, STFAC-ECGNet clinched top scores in Accuracy, Precision, and F1-score, noted at 0.894, 0.846, and 0.845, correspondingly, with CAMV-RNN achieving an optimal F1-score. CBMV-CNN recorded peak performance in AUC and Recall, with notable scores of 0.938 and 0.856. In summary, using the F1-score as a reference, STFAC-ECGNet surpasses all competing advanced models in performance.

Table 3.

Diagnostic performance comparison with other baseline and advanced networks on the PTB-XL test dataset in five diagnostic categories. The bold font indicates the highest value of the metric for each diagnostic category across different metrics and comparison models.

SE-ResNet-CoT-BiGRU 29	Accuray	AUC	Recall	Precision	F1-score
CD	0.868	0.946	0.872	0.867	0.869
HYP	0.883	0.933	0.819	0.852	0.835
MI	0.912	0.898	0.734	0.814	0.772
NORM	0.905	0.924	0.849	0.877	0.863
STTC	0.869	0.926	0.813	0.834	0.823
AVG	0.887	0.925	0.817	0.849	0.833
Encoder-Decoder-KNN52	Accuray	AUC	Recall	Precision	F1-score
CD	–	–	0.844	0.771	0.806
HYP	–	–	0.849	0.792	0.820
MI	–	–	0.760	0.790	0.775
NORM	–	–	0.915	0.939	0.927
STTC	–	–	0.754	0.770	0.762
AVG	–	–	0.824	0.812	0.818
CAMV-RNN(ours)	Accuray	AUC	Recall	Precision	F1-score
CD	0.889	0.925	0.842	0.844	0.843
HYP	0.915	0.910	0.784	0.805	0.794
MI	0.886	0.939	0.845	0.853	0.849
NORM	0.877	0.949	0.883	0.879	0.881
STTC	0.882	0.934	0.855	0.835	0.845
AVG	0.890	0.931	0.842	0.843	0.842
CBMV-CNN(ours)	Accuray	AUC	Recall	Precision	F1-score
CD	0.885	0.928	0.857	0.832	0.844
HYP	0.918	0.910	0.717	0.856	0.780
MI	0.885	0.943	0.863	0.844	0.853
NORM	0.865	0.948	0.874	0.871	0.872
STTC	0.890	0.938	0.856	0.832	0.844
AVG	0.889	0.933	0.833	0.847	0.840
STFAC-ECGNet(ours)	Accuray	AUC	Recall	Precision	F1-score
CD	0.892	0.925	0.835	0.869	0.852
HYP	0.916	0.906	0.797	0.798	0.797
MI	0.893	0.943	0.841	0.856	0.848
NORM	0.878	0.949	0.882	0.875	0.878
STTC	0.894	0.936	0.844	0.846	0.845
AVG	0.895	0.932	0.840	0.849	0.844

Results of CPSC 2018 datasets

According to the data results from Table 4, our proposed network exhibits remarkable performance in the diagnostic categories of I-AVB, LBBB, RBBB, STD, and STE, as highlighted using bold italics font. STFAC-ECGNet achieves F1-scores of 0.940 and 0.762 in the RBBB and STE diagnostic categories, respectively. CAMV-RNN achieves F1-scores of 0.911, 0.940, 0.846, and 0.848 in the I-AVB, RBBB, STD, and average F1-score, respectively. STFAC-ECGNet achieves F1-scores of 0.905, 0.918, 0.940, 0.853, 0.765, and 0.852 in the I-AVB, LBBB, RBBB, STE, and average F1-score, respectively. Our proposed model surpasses baseline models and state-of-the-art models in these metrics. While our model may not exhibit a significant advantage in the Normal, AF, PAC, and PVC diagnostic categories, overall, the F1-score surpasses baseline models and advanced models. Additionally, we observed that the CAMV-RNN network is relatively more adaptable to the CPSC 2018 dataset compared to the STFAC-ECGNet network, possibly due to the smaller data size and larger model parameters. However, STFAC-ECGNet performs exceptionally well in the STE diagnostic category, outperforming CAMV-RNN by an additional 0.009 F1-score. Furthermore, STFAC-ECGNet also outperforms the baseline models and advanced models. When either the CASSAN module is omitted or the simplified self-attention ⁵⁸ (SSAN) module, lacking scale and mask operations, is removed, and only the CBAM module is retained in STFAC-ECGNet, the model exhibits improved adaptability to the dataset. This is because we performed upsampling on the original data, adjusting its length to 60 s. In the time series processing module, the CASSAN module or SSAN module tends to focus on the padding section, resulting in the original STFAC-ECGNet underperforming in various metrics compared to the version with only the CBAM module. The STFAC-ECGNet with only the CBAM module is superior to CAMV-RNN, as it introduces the CBMV-CNN module, enhancing the feature extraction capability for grayscale images. The CBAM module better concentrates attention on identifying the padding and non-padding areas of the image region.

Table 4.

Diagnostic performance comparison with other baseline and advanced networks on the CPSC test dataset. The top two rankings for each diagnosis are indicated in bold italics.

Network	Year	Normal	AF	I-AVB	LBBB	RBBB	PAC	PVC	STD	STE	AVG
TI-CNN17	2018	0.759	0.807	0.855	0.844	0.837	0.606	0.712	0.742	0.556	0.773
CNN + BiLSTM53	2019	0.755	0.846	0.870	0.869	0.780	0.748	0.829	0.790	0.655	0.800
ATI-CNN19	2020	0.789	0.920	0.850	0.872	0.933	0.736	0.861	0.789	0.556	0.812
CNN + BiRNN + Attention18	2020	0.795	0.897	0.865	0.821	0.911	0.734	0.852	0.788	0.509	0.797
CNN + Transformer54	2021	0.817	0.858	0.878	0.800	0.872	0.618	0.830	0.711	0.686	0.786
Deform-CNN27	2021	0.805	0.931	0.893	0.900	0.948	0.663	0.871	0.800	0.667	0.831
DAMS-Net7	2022	0.819	0.915	0.881	0.878	0.936	0.755	0.876	0.819	0.684	0.839
ASTLNet26	2022	0.790	0.923	0.867	0.892	0.937	0.757	0.837	0.798	0.558	0.818
LightX3ECG31	2023	0.755	0.940	0.892	0.887	0.944	0.631	0.792	0.785	0.578	0.800
SE-ResNet + CoT Attention + BiGRU29	2023	0.824	0.925	0.882	0.937	0.939	0.734	0.767	0.835	0.600	0.827
CNN-DVIT30	2023	0.831	0.924	0.877	0.905	0.935	0.704	0.842	0.823	0.610	0.829
LFG-Net55	2023	0.792	0.932	0.891	0.894	0.937	0.756	0.874	0.821	0.682	0.842
CBMV-CNN(ours)	2023	0.772	0.928	0.908	0.899	0.937	0.677	0.848	0.836	0.716	0.837
STFAC-ECGNet(ours)	2023	0.774	0.928	0.910	0.902	0.940	0.707	0.838	0.840	0.762	0.846
CAMV-RNN(ours)	2023	0.787	0.931	0.911	0.914	0.940	0.694	0.850	0.846	0.751	0.848
STFAC-ECGNet(only contains CBAM, without CASSAN and SSAN)(ours)	2023	0.797	0.929	0.905	0.918	0.940	0.688	0.868	0.853	0.765	0.852

Ablation experiments on CBMV-CNN

As shown in Fig. 5, this study investigates the impact of integrating the CBAM module on the performance of the CBMV-CNN network on electrocardiogram datasets (PTB-XL and CPSC 2018). In the PTB-XL dataset, the CBMV-CNN model without the CBAM module demonstrates an accuracy of 0.880 and an AUC of 0.933. Upon introducing the CBAM module, the accuracy of the CBMV-CNN network improves by 0.1%, and the AUC increases by 0.2%. Additionally, the CBAM module brings about a slight improvement in recall, increasing from 0.780 to 0.796. However, there is a slight decrease in precision, from 0.736 to 0.725, while the F1 score shows a slight increase, reaching 0.759. In the CPSC 2018 dataset, the introduction of CBAM leads to a significant improvement in model performance. The CBMV-CNN network with CBAM integration demonstrates a 0.2% increase in accuracy, a 1.0% increase in AUC, and a 2.5% increase in recall, with a slight decrease in precision and a 1.2% increase in F1 score. These findings indicate that integrating the CBAM module has a positive impact on the performance of the CBMV-CNN network on electrocardiogram datasets. This suggests that the CBAM module can better capture critical features in electrocardiogram data, thereby enhancing the model's classification performance and generalization ability in the context of deep learning and electrocardiography.

Fig. 5.

The CBMV-CNN conducted ablation experiments on two datasets.

Ablation experiments on CAMV-RNN

As show in Fig. 6, this study examines how the performance of the CAMV-RNN network on electrocardiogram datasets (PTB-XL and CPSC 2018) is affected by the introduction of our proposed CASSAN module and simplified self-attention mechanism module. The results obtained on the PTB-XL dataset indicate that integrating the CASSAN module into the CAMV-RNN architecture results in significant improvements in accuracy, precision, and F1 score, with enhancements of 0.4%, 0.11%, and 0.4%, respectively. Introducing the SSAN module increases accuracy and precision by 0.4% and 0.16%, respectively, while AUC remains unchanged. However, a decrease in recall leads to a decline in the overall evaluation metric, the F1-score. The comprehensive effect of introducing the SSAN module on F1-score is not as significant as without introducing either the CASSAN or SSAN block. On the CPSC 2018 dataset, the integration of the CASSAN module into the CAMV-RNN architecture similarly enhances accuracy, AUC, recall, precision, and F1 score by 0.1%, 0.1%, 0.6%, 0.4%, and 0.6%, respectively. Conversely, the improvements observed in various metrics with the introduction of the SSAN module are not as substantial as those achieved with the CASSAN module. In conclusion, the CASSAN module demonstrates more significant performance advantages over the SSAN module on these two datasets, possibly due to its more effective capture of features and patterns in the datasets.

Fig. 6.

The CAMV-RNN conducted ablation experiments on two datasets.

Ablation experiments on STFAC-ECGNet

In the ablation experiments illustrated in Fig. 7, we observed that the CBAM module consistently enhances the accuracy and F1 score of the model on both the PTB-XL and CPSC2018 datasets, highlighting the effectiveness of attention mechanisms in enhancing the model's ability to capture key features. Within the STFAC-ECGNet framework, the incorporation of the CASSAN module leads to superior outcomes when compared to integrating the SSAN module. Adding the CBAM module to any configuration demonstrates potential improvements in various performance metrics, but the combination of CASSAN and CBAM exhibits the best performance. Additionally, the different responses of the same model configuration to different datasets underscore the importance of dataset-specific feature selection and model tuning. On the PTB-XL dataset, the combination of CASSAN and CBAM performs well, while on the CPSC2018 dataset, a simpler configuration or an independent CBAM module alone is sufficient to achieve outstanding performance. These disparities can be attributed to differences in dataset scale and network parameters. In general, the proposed STFAC-ECGNet, incorporating the combined CASSAN and CBAM mechanisms, exhibits superior performance compared to baseline networks and state-of-the-art models.

Fig. 7.

The STFAC-ECGNet conducted ablation experiments on two datasets.

Networks parameters

According to the network parameter results in Table 5, the parameter configurations for the CAMV-RNN, CBMV-CNN, and STFAC-ECGNet models are as follows: The CAMV-RNN model has a total parameter count of 2.68 M, with an input size of 2.30 MB. The memory required for forward/backward propagation is 1376.53 MB, with parameter storage occupying 10.72 MB. The estimated total memory usage is 1389.55 MB. For the CBMV-CNN model, the total parameter count is 7.65 M, with the same input size of 2.30 MB. The memory required for forward/backward propagation is 14,167.20 MB, with parameter storage occupying 30.61 MB. The estimated total memory usage is 14,200.11 MB. Lastly, the STFAC-ECGNet model has the largest total parameter count, at 10.08 M, with an input size of 2.30 MB. The memory required for forward/backward propagation is 15,543.45 MB, with parameter storage occupying 40.27 MB. The estimated total memory usage is 15,586.02 MB.

Table 5.

Networks parameters.

Parameters	CAMV-RNN	CBMV-CNN	STFAC-ECGNet
Total params (M)	2.68	7.65	10.08
Input size (MB)	2.30	2.30	2.30
Forward/backward pass size (MB)	1376.53	14,167.20	15,543.45
Params size (MB)	10.72	30.61	40.27
Estimated Total Size (MB)	1389.55	14,200.11	15,586.02

Prototype visualization

Visualization of t-SNE on PTB-XL and CPSC 2018

As shown in Fig. 8a, t-SNE was employed to visualize the feature representations, thereby validating whether the diagnostic categories of electrocardiograms (ECGs) are well-clustered within the STFAC-ECGNet model. In the color-coded area chart on the Y-axis, 0 represents CD, 1 represents HYP, 2 represents MI, 3 represents NORM, and 4 represents STTC. From Fig. 8a, it is evident that in the high-dimensional space, NORM, CD, MI, and STTC all exhibit distinct boundaries. This is particularly noticeable in the STFAC-ECGNet model, which integrates both CASSAN and CBAM, showcasing superior overall arrhythmia diagnostic classification performance.

Fig. 8.

Visualization of the STFAC-ECGNet network on two different datasets using t-SNE: PTB-XL (a) and CPSC 2018 (b).

As shown in Fig. 8b, the color-coded area chart on the Y-axis indicates that 0 represents STE, 1 represents AF, 2 represents LBBB, 3 represents RBBB, 4 represents I-AVB, 5 represents PAC, 6 represents Normal, 7 represents STD, and 8 represents PVC. According to Fig. 8b, the color regions on the Y-axis correspond to different types of cardiac arrhythmias. In the high-dimensional space, except for PAC, all other cardiac arrhythmia types display distinct boundaries. This suggests that the PAC diagnostic category is mostly scattered, leading to relatively lower F1-score performance in model evaluation, whereas other diagnostic categories exhibit excellent performance. This finding confirms the reliability of the experimental results.

Confusion matrix plots for PTB-XL and CPSC 2018 diagnosis

The performance of STFAC-ECGNet is depicted in Table 6 through its confusion matrix, showcasing its effectiveness on the PTB-XL dataset. The true positive (TP) rates for CD (conduction disturbance), HYP (hypertrophy), MI (myocardial infarction), and STTC (ST/T change) all exceeded 0.90, specifically 0.94, 0.95, 0.92, and 0.90, respectively. However, this TP rate does not provide clear indications for diagnosing arrhythmias. It is imperative to substantially improve the true negative (TN) rates while maintaining high TP rates. The TN rate for NORM (Normal ECG) is the highest at 0.94, whereas the TN rates for the four diseases range from 0.63 to 0.82. This indicates that the classifier can better diagnose the five classes' true positives. Compared to the network proposed by Gokhan Kutluana et al.⁵⁹, our proposed network has increased the TN rates for CD category by 0.03, HYP category by 0.16, MI category by 0.10, STTC category by 0.06, and NORM category by 0.01.

Table 6.

Confusion matrix of diagnostic results on the PTB-XL dataset.

Condition	True positive	False positive	True negative	False negative
CD	1562 (0.94)	103 (0.06)	375 (0.75)	123 (0.25)
HYP	1813 (0.95)	87 (0.05)	166 (0.63)	97 (0.37)
MI	1478 (0.92)	132 (0.08)	441 (0.80)	112 (0.20)
NORM	981 (0.82)	218 (0.18)	905 (0.94)	59 (0.06)
STTC	1483 (0.90)	157 (0.10)	430 (0.82)	93 (0.18)

Table 7 illustrates the performance of STFAC-ECGNet on the CPSC 2018 dataset through the confusion matrix. We employed ten-fold cross-validation to comprehensively evaluate the model across tenfold experiments. The comprehensive performance indicates that our network achieves a true positive (TP) rate exceeding 0.90 for all diagnoses. Particularly noteworthy are its outstanding performances in atrial fibrillation (AF), first-degree atrioventricular block (I-AVB), left bundle branch block (LBBB), premature ventricular contractions (PVC), right bundle branch block (RBBB), and ST-segment depression (STD), reaching 0.93, 0.87, 0.89, 0.85, 0.94, and 0.77, respectively. Meanwhile, the true negative rates (TNR) of STFAC-ECGNet for the NORM, premature atrial contractions (PAC), and ST-segment elevation (STE) categories are relatively lower at 0.34, 0.52, and 0.54, respectively. This reaffirms the reliability and validity of the F1 score data reported in the experimental results.

Table 7.

Confusion matrix of diagnostic results on the CPSC 2018 dataset.

Condition	True positive	False positive	True negative	False negative
AF	4877 (0.96)	210 (0.04)	1023 (0.93)	76 (0.07)
I-AVB	5396 (0.97)	139 (0.03)	566 (0.87)	85 (0.13)
LBBB	5901 (0.99)	74 (0.01)	188 (0.89)	23 (0.11)
NORM	5116 (0.95)	244 (0.05)	447 (0.54)	379 (0.46)
PAC	5467 (0.97)	164 (0.03)	190 (0.34)	365 (0.66)
PVC	5209 (0.94)	347 (0.06)	534 (0.85)	96 (0.15)
RBBB	4304 (0.95)	208 (0.05)	1580 (0.94)	86 (0.06)
STD	5103 (0.94)	301 (0.06)	604 (0.77)	178 (0.23)
STE	5915 (0.99)	73 (0.01)	102 (0.52)	96 (0.48)

Limitations

Despite the significant improvements our proposed network has achieved in the task of disease detection using electrocardiogram (ECG) signals, we must also acknowledge its limitations. It is worth noting that we have not yet undertaken the implementation of denoising operations on the grayscale images of the electrocardiogram (ECG). However, integrating this functionality has the potential to significantly augment diagnostic performance⁶⁴. Furthermore, our approach utilizes multi-scale and cross-dimensional computations, which lead to substantial hardware resource consumption. The incorporation of recurrent neural networks also results in relatively slower inference speeds. Additionally, our study focuses only on the five most important diagnostic categories from the PTB-XL dataset, which includes 71 different ECG diagnostic categories. Therefore, we recognize that a major limitation of our current network design is its demanding computational resources and inadequate inference speed. A crucial direction for future research involves developing a more lightweight network structure specifically tailored to address the requirements of practical applications. Moreover, the complete interpretability of deep learning models has not been fully realized in this study, with only partial interpretability achieved. Enhancing this aspect is one of our primary objectives for our upcoming projects. Lastly, generalizing the model to different types of ECG datasets poses challenges, particularly related to data preprocessing. Each ECG dataset may vary in recording protocols and formats, necessitating preprocessing steps to standardize the data for input. While automation can assist in parts of this process, achieving complete automation across all preprocessing steps remains challenging. This variability in data preprocessing is indeed a significant challenge in extending the model to different types of ECG datasets.

Conclusion

Interest in innovative applications of deep learning methods in diagnosing cardiac arrhythmias is increasing This study proposes an enhanced spatio-temporal learning network (STFAC-ECGNet). Specifically, the STFAC-ECGNet comprises three main components: the Sequential Spatio-temporal Feature Fusion Module (CAMV-RNN block), the Image Spatial Feature Fusion Module (CBMV-CNN block), and the Enhanced Multiscale Fusion Module for Image Space-Sequential Spatio-temporal Features (TSEF block). First, the CBMV-CNN obtains more discriminative spatial feature representations by applying a spatial channel attention module on residual networks. Second, the CAMV-RNN, based on multiscale feature fusion, combines a cooperative adaptive simplified self-attention module with an effective bidirectional recurrent neural network to extract more effective temporal contextual feature representations. Various ablation experiments and visualization experiments were conducted to validate the effectiveness of our model design. The comparative experimental findings on the PTB-XL dataset and CPSC 2018 dataset provide further evidence of the robustness and generalization of our model in analyzing 12-lead ECG diagnostics, surpassing baseline networks and state-of-the-art networks. This study aims to assist doctors in expediting the review process of diagnoses made using 12-lead electrocardiograms. The integration of the STFAC-ECGNet network into human–computer interaction systems is proved effective to enhance the diagnosis rate of cardiac arrhythmia in 12-lead electrocardiograms. Future research should prioritize the development of lightweight models to achieve more efficient and robust arrhythmia diagnostic networks.

Author contributions

Z.Y.: Methodology, software development, validation, funding acquisition, writing—original draft, writing—review and editing. A.J.: Software development, writing—original draft, validation. Y.L.: Funding acquisition, project management. X.Y.: Validation. X.X.: Validation. J.W.: Funding acquisition, validation. Q.L.: Validation. X.G.: Validation. Y.L.: Proofreading, translation, and revision. All authors have read and agreed to the final version of the manuscript for publication.

Data availability

The PTB-XL dataset, which supports the findings of this research report, is publicly accessible on PhysioNet at https://physionet.org/content/ptb-xl/1.0.3/. The CPSC 2018 dataset can also be acquired from the ICBEB2018 website at http://2018.icbeb.org/Challenge.html.

Competing interests

The authors declare no competing interests.