Hybrid channel attention network for auditory attention detection

Abstract

Humans exhibit a remarkable ability to selectively focus on auditory stimuli in multi-speaker environments, such as cocktail parties. The Auditory Attention Detection (AAD) method aims to identify the conversation that a listener is attending to through the analysis of neural signals, particularly utilizing electroencephalography (EEG) data. However, current methodologies in this domain encounter several significant limitations. While many existing AAD methods use additional information–like spatial or frequency features–to improve decoding accuracy, they often miss the relationships between signals from different EEG channels. To address these shortcomings, this paper introduces a novel hybrid channel attention network for AAD. Our approach is the first to integrate spatial-temporal filtering, dynamic multi-scale feature fusion, and efficient cross-channel attention into a single unified architecture, enabling it to capture complex neural patterns of attention that previous methods overlooked. Our proposed network first extracts spatial-temporal features from raw EEG signals employing a dedicated spatial-temporal feature extraction module. The extracted features are then processed by a module that combines information across different time scales and uses an attention mechanism to identify important relationships between EEG channels. Experimental results demonstrate that our network achieves superior classification performance compared to baseline methods, particularly under conditions with short decision windows. Notably, while maintaining exceptional accuracy, the proposed architecture significantly reduces model parameters.

Introduction

Human auditory systems demonstrate remarkable selectivity in complex acoustic environments, a phenomenon prominently illustrated by the cocktail party effect, wherein listeners can concentrate on target speech streams despite the presence of competing voices^1–3. This ability is often impaired in people with hearing disorders, as conventional hearing aids have difficulty separating speakers in noisy environments⁴. Recent advancements in neural decoding have unveiled robust cortical activation patterns associated with attentional modulation, thereby providing a neurophysiological foundation for auditory attention detection (AAD) systems aimed at overcoming challenges posed by cocktail party scenarios⁵.

Electroencephalography (EEG)⁶ has established itself as the primary modality for monitoring neural dynamics, particularly in comparison to other methods such as electrocorticography⁷ and magnetoencephalography^8,9. EEG has undergone extensive validation regarding its efficacy in applications related to attention-related disorders¹⁰. Utilizing non-invasive electrode arrays, EEG captures fluctuations in scalp potentials that form non-linear temporal sequences. Using a mathematical tool called the Fast Fourier Transform (FFT), we break down these signals into five frequency bands¹¹: (1–3 Hz), (4–7 Hz), (8–13 Hz), (14–30 Hz), and (31–50 Hz)¹². Each band exhibits unique spatial-topographic distributions corresponding to specific cognitive states, with differential entropy (DE)¹³ and power spectral density (PSD)¹⁴ emerging as particularly effective methodologies for feature extraction.

Modern auditory attention detection (AAD) research focuses on two primary objectives: speaker identity discrimination and spatial attention tracking¹⁵. While stimulus-reconstruction paradigms that leverage clean speech references demonstrate potential, their real-world applicability is limited due to the prevalence of overlapping sound sources in natural environments. This limitation necessitates the development of EEG-exclusive frameworks for detecting spatial attention¹⁶.

Traditional linear analytical methods often struggle to model non-linear neural interactions, requiring extended temporal windows for reliable inference¹⁷. Contemporary approaches employ convolutional neural networks (CNNs) to exploit spectral characteristics¹⁸, achieving improved performance through 2D topographic mapping of discriminative features. To address the challenges of low signal-to-noise ratio scenarios,¹⁹ draws on the multi-scale convolutional encoder decoder structure, which utilizes spatiotemporal convolutional networks (S-TCN) as bottlenecks to model long-term dependencies. To fully utilize multi-scale contextual information, this study adopted the TFADCSU-Net model proposed by²⁰, which has a built-in multi-scale feature extraction layer (MSDEL) that can effectively capture global and local speech features.²¹ introduced a time-frequency attention (TFA) module after each multi-scale convolution block, which can dynamically assign weights to different time-frequency spectral components, enabling the model to accurately focus on key information.²² employed both local and global attention networks to jointly model speech signals, which can extract useful information more comprehensively and has better performance than a single self attention network.²³ proposed a novel approach utilizing microstate and recurrence quantification analysis features combined with a hybrid GRU-CNN architecture for AAD, demonstrating strong performance without requiring access to the auditory stimuli.²⁴ proposed AADNet, an end-to-end architecture that directly maps EEG to attention state, demonstrating significantly improved generalization to unseen subjects. However, these methods typically overlook the temporal evolution of EEG patterns. In contrast, attention-based temporal models effectively capture dynamic variations but frequently neglect essential spectral-spatial correlations. This methodological dichotomy highlights the need for hybrid architectures that synergistically integrate temporal-spectral features through multimodal fusion–a largely unexplored frontier in AAD research.

The primary objectives of this study are threefold: First, to design a novel neural architecture that seamlessly integrates spatial-temporal filtering, dynamic multi-scale feature fusion, and efficient cross-channel attention into a unified framework for AAD. Second, to validate that this hybrid approach effectively captures the complex neural patterns of auditory attention that are often overlooked by methods focusing on isolated feature domains. Third, to demonstrate that the proposed model achieves superior decoding performance, particularly under the challenging condition of short decision windows, while simultaneously maintaining a parameter-efficient structure suitable for potential real-time applications. Based on these objectives, we formulate the following hypotheses:

1. A network that explicitly models the interplay between spatial, temporal, and cross-channel features will yield significantly higher AAD decoding accuracy compared to state-of-the-art models that do not integrate these aspects jointly.

2. The incorporation of a dynamic multi-scale fusion mechanism will enable the model to robustly handle EEG patterns across varying temporal resolutions, leading to notably improved performance in short decision windows (e.g., 0.1 s).

3. The proposed efficient cross-channel attention mechanism will enhance feature discriminability without incurring substantial computational overhead, resulting in a model that is both more accurate and more parameter-efficient than existing benchmarks.

The experimental design and evaluations presented in this paper are structured to rigorously test these hypotheses. The major contributions of this paper are outlined as follows:

1. We introduce a novel network architecture for auditory attention detection that comprises the spatial-temporal extraction module, multi-scale adaptive fusion module, and cross-channel attention module. Our new network effectively leverages multi-scale features as well as inter-channel correlations to decode EEG data.

2. The results indicate that our network achieves remarkable decoding accuracy within very short decision windows–surpassing existing state-of-the-art (SOTA) models by 2.5 points on the DTU dataset and 1.1 points on the KUL dataset–all under a 0.1-second decision window. Additionally, in comparison to the recent model (DBPNet), our model has nearly 50 fewer parameters, which significantly enhances the efficiency of model inference.

The remainder of this paper is organized as follows. ‌Section 2‌ provides a concise introduction to the proposed methodology. Section 3 introduces the dataset processing methods and model training details. ‌In Section 4, we conduct comparative analyses between our network architecture and existing approaches, while empirically validating the efficacy of various constituent modules. ‌Ablation studies‌ are systematically ‌presented in Section 5 to quantify individual component contributions. ‌Finally, Section 6 concludes the paper with a comprehensive summary of findings‌

Proposed approach

Existing methodologies in EEG-based auditory attention detection (AAD) have predominantly focused on the isolated analysis of either temporal or spectral characteristics, often overlooking the critical interplay between multi-scale features and inter-channel correlations in neural recordings. To address this limitation, we propose a novel channel-attention neural architecture (Fig. 1) comprising three synergistic components: Spatial-Temporal Extraction Module, This module captures spatial and temporal patterns from multi-channel EEG inputs; dynamic multi-scale fusion module, by utilizing adaptive convolutional kernels and weighted feature integration, this module processes information across multiple temporal resolutions; cross-channel attention mechanism, this mechanism enhances local feature interactions while simultaneously establishing global dependencies through parallel attention branches.

Fig. 1.

Architecture of the proposed Hybrid Channel Attention Network for auditory attention detection. The network comprises three main modules: Spatial-Temporal Feature Extraction Module: Processes multi-channel EEG inputs to extract spatial and temporal features. Multi-Scale Adaptive Fusion Module (MSAFM): Integrates features across multiple temporal resolutions using adaptive convolutional kernels. Cross-Channel Attention Module (CCAM): Enhances local and global feature interactions through a partitioned self-attention mechanism. The model inputs are common spatial patterns (CSP) extracted from EEG signals, and the outputs are two predicted labels related to auditory attention.

Following standard preprocessing protocols, EEG signals are segmented into consecutive decision windows, represented as a matrix , where N denotes the number of electrode channels, and T represents the temporal samples per window. Each temporal slice corresponds to multi-channel neural measurements at the i-th window position, preserving both the spatial distribution and temporal evolution characteristics. The overall structure of the proposed network is illustrated in Fig. 1.

Following previous methods²⁵, we also employ the Common Spatial Pattern (CSP) technique for feature extraction to enhance the signal-to-noise ratio of the raw EEG signals^26,27. Similar to²⁸, and to avoid feature leakage, CSP feature extraction is performed only after dividing the data into training and testing sets. The corresponding formulas are as follows:

1

where means the csp method, the F means the extracted features from eeg data.

Spatial-temporal feature extraction module

EEG captures dynamic, time-varying electrical activity from neuronal cells, revealing functional patterns and interregional connectivity within the brain²⁹. The analysis of auditory-evoked neural responses can be enhanced by integrating both the temporal and spatial attributes derived from these signals. However, prior research has disproportionately emphasized localized temporal dynamics in EEG datasets, often neglecting the spatial distribution characteristics. To address this limitation, we implemented a spatial filter before traditional temporal filters, enabling the synthesis of spatiotemporally enriched EEG representations.

Our module separately processes spatial information (across different EEG channels) and temporal information (changes over time). Attention mechanisms are embedded in both the spatial and temporal modules, enabling dynamic calibration of these two-dimensional feature representations. The overall structure of Spatial-temporal feature extraction module is illustrated in Fig. 1.

Spatial convolution

The spatial convolution is employed to extract spatial features from the input EEG signals, explicitly modeling nonlinear relationships between variables. The spatial convolution consists of three main components: multi-level channel expansion, cross-variable aggregation, and the channel attention mechanism.

First, a convolution is applied to expand the number of channels by four times, enhancing the feature representation capability of the input EEG data. Subsequently, convolution is performed along the channel-wise dimension to achieve cross-variable feature fusion for multivariate time series. Finally, a channel attention module is introduced to adaptively calibrate channel importance, thereby enhancing the response of key features.

The process is described by the following formula:

2

Here, Conv2d represents the convolution operation, and GELU refers to the activation function.

The channel attention module combines information from all spatial locations to determine the importance of each EEG channel. By implementing a bottleneck structure with a reduction ratio of 16, the computational workload is reduced while maintaining performance and minimizing the number of parameters. The Sigmoid function outputs channel weights in the range of 0 to 1, enabling the soft selection of feature channels.

The process is described by the following formula:

3

where Conv2d means the convolution operation, GAP stands for adaptive averaging pooling.

Temporal convolution

The temporal convolution is employed to balance local and global features. We adopted depthwise separable convolution, which is decomposed into depthwise convolution and pointwise convolution. This approach significantly reduces the number of parameters while preserving the receptive field. The process is described by the following formula:

4

Here, means the spatial features, Conv2d means the convolution operation.

In addition, a attention mechanism has been incorporated. This mechanism generates attention weights in the time dimension through convolution, allowing the model to highlight key time-step features. Specifically, the attention mechanism produces attention maps along the timeline and adaptively learns the importance of different time points.

To reduce computational overhead, two convolutions are applied to compress the number of channels to 1. The operation x.mean(dim=3) is used to aggregate information across the time dimension, effectively capturing global temporal dependencies. The process is described by the following formula:

5

Here, means the spatial-temporal features.

After the spatial-temporal features are fused and projection, the final output is obtained by adding the postion information.

Multi-scale adaptive fusion module

Figure 1 illustrates the structure of the proposed module. To effectively fuse long-term and local features, this module adopts three parallel branches, each employing a different convolution kernel for feature extraction. The three branches employ kernels of different sizes (, , ), where the size of each kernel, , is determined by the following formula:

6

Where C represents the number of channels of the input signal, is the path specific offset (such as the contraction path = 1), and is the disturbance ratio (this module employd the = 0.2). The kernel size of each branch is as follows:

7

The extracted global contextual features and local features are fused using adaptive weights to enhance the dynamics of the receptive field. This module employs learnable relative weights, , to fuse the features from the three branches. This approach dynamically balances the feature contributions from different branches while automatically reinforcing the most important paths. The fusion method for the weights is provided in the following formula:

8

9

Where O represents the output of the multi scale adaptive module, the means the learnable adjustment parameters that automatically balance the specificity and robustness of features.

Cross-channels attention module

The module structure is shown in Fig. 1. Instead of relying on single-sequence global attention, the module adopts a combination of local attention mechanisms. The input sequence is divided into four segments, and the correlations between each segment are calculated separately. A dynamic QKV (Query, Key, Value) mechanism is employed to extract the correlations between these segments, enhancing the local interaction of features, achieving cross-segment information fusion, and improving the model’s representational capacity.

The specific process is as follows: Given an input sequence , it is evenly divided into four sub-segments: .

From these four sub-segments, combinations of three are selected to participate in the calculation. There are a total of possible combinations. The attention calculation format for each group is as follows:

10

Amog them, the defination of (a,b,c) is followed:

11

Simultaneously employing block combination attention effectively reduces computational complexity. The module integrates the outputs of four multi-head attention mechanisms. The specific formula for this integration is as follows:

12

Where represents the feature concatenation operation. The extracted features are subjected to convolution and pooling operations to achieve resolution reduction and feature enhancement of sequence data.

Experiments

Dataset

This section evaluates the performance of our proposed network employing two benchmark datasets widely utilized in auditory attention detection research: KUL^30,31 and DTU^32,33. Both the KUL and DTU datasets exclusively consist of EEG recordings collected from audio-only experimental paradigms. Key characteristics of these datasets are systematically compared in Table 1.

Table 1.

Summary of the KUL and DTU datasets employed for auditory attention detection.

Dataset	Subjects	Scene	Language	Duration per subject (minutes)	Total duration (hours)
DTU	18	audio-only	Danish	50	15.0
KUL	16	audio-only	Dutch	48	12.8

KUL dataset

Neural recordings were acquired using a BioSemi ActiveTwo system from 16 healthy participants in an acoustically controlled environment. The experimental protocol involved dichotic listening tasks, wherein subjects selectively attended to one of two competing narratives delivered via in-ear headphones at 60 dB. Four Flemish-language stories narrated by male speakers were presented under two spatialization conditions: Conventional diotic playback with separate ear assignments. HRTF-processed simulations, positioning sound sources at 90 lateral angles. Each participant completed eight 6-minute trials. EEG signals were recorded using a 64-channel setup at a sampling rate of 8,192 Hz, while auditory content was bandwidth-limited to 4 kHz.

DTU dataset

This dataset also employed the BioSemi ActiveTwo system but with a reduced sampling rate of 512 Hz. EEG recordings were collected from 18 normoacoustic participants performing auditory attention selection tasks. Participants attended to target speech signals spatially separated at 60 azimuth angles, presented concurrently with distractor narratives through ER-2 insert earphones at 60 dB SPL. The stimulus set consisted of Danish audiobook excerpts narrated by six speakers (3 male and 3 female). Each participant completed 60 experimental trials, each lasting 50 seconds

Data processing

To ensure equitable performance comparisons of our method, standardized preprocessing methods were implemented across the two benchmark datasets (KUL dataset and DTU dataset), tailored to their respective acquisition characteristics.

Preprocessing procedures differed between the two datasets. Processing of the KUL dataset commenced with re-referencing to the mastoids, followed by application of a 0.1–50 Hz bandpass filter, and concluded with down-sampling to 128 Hz. In contrast, the DTU data were initially filtered to suppress 50 Hz power line noise and harmonics. Ocular artifact suppression was subsequently performed through joint decorrelation, prior to re-referencing and final down-sampling to 64 Hz. Thus, for KUL and DTU dataset, each subject had 46080 points 8 trials = 368640 points, 3500 points 60 trials = 210000 points, respectively. Then, the data segments were obtained with a sliding decision window of length t1 with an overlap of 50.

To assess performance, our method was rigorously benchmarked against several established auditory attention detection architectures: SSF-CNN³⁴, MBSSFCC¹⁸, DBPNet²⁵, and DARNet²⁸. The evaluation employed standardized temporal resolutions using decision windows of 0.1 s, 1 s, and 2 s, enabling a multi-scale analysis framework. This framework provides a comprehensive characterization of temporal sensitivity across competing models.

Implementation details

In auditory attention detection (AAD) studies, classification accuracy has been widely recognized as the primary metric for evaluating model performance. Consistent with this established practice, we assessed our network framework employing two major standard datasets: the KUL dataset and the DTU dataset. The implementation methodology is exemplified through the KUL dataset, utilizing 1-second decision windows and detailing both training protocols and network architecture specifications.

The dataset was partitioned into training, validation, and test subsets at an 8:1:1 ratio, resulting in 4,600, 576, and 576 decision windows per subject, respectively. Optimization was carried out using the Adam algorithm, with a learning rate of and weight decay of , configured with a batch size of 32 and a maximum training duration of 80 epochs. An early stopping criterion was employed to halt training when the validation loss plateaued for 8 consecutive epochs. All implementations were executed in PyTorch.

EEG signals underwent preprocessing via the Common Spatial Patterns (CSP) algorithm to extract initial features. In order to avoid feature leakage, we perform CSP method feature extraction after dividing the training and testing data. The spatial-temporal feature extraction module is employed to extract features from the spatial domain and temporal domain. Following processing by the spatial module, the EEG data is represented as , After processing through the temporal module, the EEG data remains represented as , After projection and fusion, the final output data is represented as . After propessed through the multi-scale fusion module and the cross-channel attention mudule, the data dimension remains unchanged. Before being concatenated to obtain the fused feature, . Finally, X is passed through another fully connected layer (input: 4, output:2) to yield the final auditory attention prediction. The final predictions were derived from a 2-unit fully connected classification layer.

Results

Performance of our network

To assess the efficacy of our network, we conducted extensive evaluations across three temporal resolutions (0.1s, 1s, and 2s), benchmarking our results against existing state-of-the-art models as detailed in Table 2, utilizing literature-reported metrics. Our framework demonstrates superior performance across all tested datasets (KUL dataset and DTU dataset), establishing new benchmarks in auditory attention detection.

Table 2.

Comparative classification accuracy () of our method against state-of-the-art models on the DTU and KUL datasets under three decision window lengths.

Dataset	Scene	Method	Decision window
			0.1-second	1-second	2-second
DTU	Audio-only	SSF-CNN34	62.5±3.40	69.8±5.12	73.3±6.21
		CNN35	56.7	63.3	65.2
		STAnet36	65.7	71.9	73.7
		MBSSFCC18	66.28±6.55	77.78±7.16	80.67±7.75
		BSAnet37	—	83.1±6.75	85.6±6.47
		EEG-Graph38	72.5±7.41	78.7±6.47	79.4±7.16
		DBPNet25	74.0±5.20	78.9±6.91	80.2±6.79
		DARNet28	74.6±6.09	80.1±6.85	81.2±6.34
		Ours
KUL	Audio-only	SSF-CNN34	76.3±8.47	84.4±8.67	87.8±7.87
		CNN35	74.3	84.1	85.7
		STAnet36	80.8	90.1	91.4
		MBSSFCC18	81.19±8.47	88.81 ±8.67	91.56±7.87
		BSAnet37	—	93.7±4.02	95.2±3.08
		DenseNet-3D39	—	94.3±4.30	95.9±4.30
		DBPNet25	85.3±6.22	94.4±4.62	95.3±4.63
		DARNet28	89.2±5.50	94.8±4.53	95.5±4.89
		Ours

The “—” means no corresponding results were provided in the paper. The SD means the standard deviation. The data cited by the methods in the table are all from their papers.

For the KUL dataset, our network achieves mean classification accuracies of 90.3 (SD = 3.84), 95.8 (SD = 4.17), and 96.3 (SD = 3.56) for the 0.1s, 1s, and 2s windows, respectively. In evaluations of the DTU dataset, we observe comparable performance, with accuracies of 77.1 (SD = 5.35), 84.1 (SD = 4.76), and 85.6 (SD = 4.49) across the progressively longer windows.

These findings reveal two critical patterns: First, decoding accuracy shows a positive correlation with window length (from 0.1s to 2s), supporting previous studies³⁶ that attribute this trend to enhanced contextual information and outlier mitigation in extended temporal segments. Second, the incorporation of multi-scale features and inter-channel correlations significantly improves detection accuracy.

Building upon preliminary analyses, we note that the classification performance of the DTU dataset exhibits an 11 reduction compared to the KUL dataset–a pattern consistently reported in prior research. Three key factors contribute to this divergence: the spatial configuration of acoustic inputs, where KUL stimuli are strictly positioned at 90 left/right angles, while DTU recordings are captured within a narrower 60 lateral range; the variable room reverberation levels present in the DTU dataset, contrasting with the reverberation-free conditions of KUL dataset; and the fact that KUL dataset exclusively employs male vocal samples, whereas DTU dataset integrates both male and female speakers, introducing potential gender-related variability.

Notably, our network maintains exceptional baseline performance (77.1) even under the challenging 0.1s window constraint in the DTU dataset, demonstrating remarkable temporal sensitivity that surpasses that of comparative models.

Furthermore, we present the experimental results for different decision windows for each subject across the two datasets, as illustrated in Fig. 2. It is evident that the decoding accuracy on the KUL dataset surpasses that of the DTU dataset. Overall, as the decision window lengthens, the decoding accuracy tends to increase. However, in certain individual cases, such as Subject 15 in the DTU dataset and Subjects 9 and 10 in the KUL dataset, the decoding success rate remains unchanged despite the extended decision window. This phenomenon may be attributed to overfitting during the training process; thus, even with a longer decision window, the decoding accuracy did not improve.

Fig. 2.

Subject-wise classification accuracy across decision window lengths for the DTU (top) and KUL (bottom) datasets.

Statistical significance analysis

To quantitatively assess the superiority of our proposed model beyond average performance metrics, we performed statistical significance testing.

We conducted pairwise paired t-tests to evaluate the effect of decision window length on the performance of our proposed model on the DTU and KUL datasets. From Table 3, we can know that the classification accuracy significantly increased from the 0.1s window (77.06 ± 5.35) to the 1s window (84.11 ± 4.76) and from the 0.1s to the 2s window (85.61 ± 4.49) on the DTU dataset. A smaller but still significant improvement was observed between the 1s and 2s windows. These results demonstrate that while the most substantial performance gains occur when increasing the decision window from 0.1s to 1s, further extending the window to 2s continues to provide statistically significant improvements, albeit with a smaller effect size. Specifically, for the KUL dataset, classification accuracy showed significant improvements from the 0.1s window (90.31 ± 3.84) to the 1s window (95.81 ± 4.17) and from the 0.1s to the 2s window (96.31 ± 3.56). However, no significant difference was observed between the 1s and 2s windows. These results indicate that while extending the decision window from 0.1s to 1s leads to substantial performance gains, further extension to 2s does not provide additional significant improvement, suggesting a performance plateau effect at around 1 second on the KUL dataset.

Table 3.

Statistical significance (p-values) of performance differences across decision window lengths for our method on the DTU and KUL datasets.

	DTU	KUL

Furthermore, To provide rigorous statistical evidence for our performance claims, we conducted paired sample t-tests comparing the per-subject accuracy of our model against the MBSSFCC baseline on the DTU dataset. Only the MBSSFCC algorithm provided raw data in the paper, so we chose this method as the baseline. The results are summarized in Table 4.

Table 4.

Statistical comparison between our method and the MBSSFCC baseline on the DTU and KUL datasets.

Dataset	Method	Decision window
		0.1-second	1-second	2-second
DTU	MBSSFCC18	66.28±6.55	77.78±7.16	80.67±7.75
	Ours	77.1	84.11	85.62
		[74.44, 79.76]	[81.75, 86.47]	[83.38, 87.86]
		p =	p =	p =
KUL	MBSSFCC18	81.19±8.47	88.81 ±8.67	91.56±7.87
	Ours	90.33	95.76	96.26
		[89.00, 91.66]	[94.70, 96.83]	[95.08, 97.45]
		p =	p =	p =

The analysis reveals that the improvements offered by our model are not only substantial in magnitude but also highly statistically significant. For the critical 0.1-second decision window, our model’s improvement of more 10 percentage points (10.82) is significant with . This indicates an extremely low probability that this result occurred by chance. This trend of significant improvement holds for longer windows as well. The 6.33 improvement at the 1-second window is also significant with . Similarly, the 4.95 improvement at the 2-second window remains highly significant with .

Furthermore, we report the 95 confidence intervals for our model’s mean accuracy. The intervals, calculated as Mean±(t-critical Standard Error), provide a range within which we can be 95 confident the true population mean lies. The narrowness of these intervals, for example, [74.44, 79.76] for the 0.1s window, underscores the precision and reliability of our estimate.

For the KUL dataset, the statistical analysis revealed a consistent trend with the DTU dataset but with even higher levels of significance. For the 0.1-second window, our method demonstrated a 9.14 improvement over the MBSSFCC baseline, with this enhancement being highly statistically significant (). The improvements observed for both the 1-second and 2-second windows were also statistically significant (p < 0.001). It is noteworthy that on the KUL dataset, even the improvement at the 0.1-second window reached a high level of statistical significance, which aligns with but is more pronounced than the trend observed on the DTU dataset.

In conclusion, the statistical analysis provides overwhelming evidence that our proposed hybrid channel attention network delivers a statistically significant and practically meaningful performance enhancement over the strong MBSSFCC baseline across all tested temporal resolutions.

Ablation study

Component contribution analysis

This section evaluates the effects of different modules within the network, specifically the spatial-temporal feature extraction module, the multi-scale adaptive fusion module, and the cross-channel attention module. We conducted an ablation study on two datasets. The experimental results demonstrate that each proposed module enhances decoding accuracy.

As shown in Table 5, the removal of any module significantly reduces the model’s decoding accuracy. On the DTU dataset, the accuracy decreases by 1.5 (0.1s), 2.8 (1s), and 1.2 (2s) after removing different modules with the same decision window. Similarly, on the KUL dataset, the accuracy decreases by 5.2 (0.1s), 6.4 (1s), and 3.1 (2s). The numerical comparison indicates that the multi-scale module contributes the most among the three modules. Furthermore, the results suggest that as the decision window length increases, the accuracy rate also shows an upward trend.

Table 5.

Ablation study evaluating the contribution of each module to overall performance on the DTU and KUL datasets.

Dataset	Scene	Model	Decision window
			0.1-second	1-second	2-second
DTU	Audio-only	w/o spatial-temporal feature	75.6±6.21	81.2±6.35	82.1±6.13
		w/o multi-scale adaptive fusion	74.3±5.74	80.3±5.33	80.8±5.15
		w/o cross-channels attention	75.9±5.42	82.5±5.17	83.4±4.72
		Ours
KUL	Audio-only	w/o spatial-temporal feature	85.1±4.32	91.2±4.58	92.4±4.01
		w/o multi-scale adaptive fusion	83.9±4.56	89.4±5.01	90.2±4.62
		w/o cross-channels attention	87.2±3.98	92.5±4.72	93.7±4.71
		Ours

Module configuration

This section mainly introduces the impact of different model configurations on the final results, including different convolution kernel sizes, different branch numbers, and different of attention mechanisms.

From Table 6, it can be seen that we compare four different combinations of convolution kernels. The four methods can be divided into two categories, namely fixed convolution kernels and channe-bassed adaptive convolution kernels. In order to fully extract multi-scale features from EEG signals, different combinations of fixed convolution kernels were used, namely , and . The combination of achieved the best results. However, numerically speaking, there is not a significant difference between the several combinations. Compared with the channel-adaptive approach proposed by us, our method has improved the performance by 0.6, with significant numerical improvement, effectively proving the superiority of the channel adaptive convolution kernel approach.

Table 6.

Performance comparison of fixed vs. channel-adaptive convolutional kernel configurations on the DTU dataset (0.1s window).

Kernel sizes	Type	Decision window
		0.1-second
	Fixed	76.6±5.54
		76.3±5.61
		76.5±5.63
Ours	Channel-adaptive	77.1±5.35

We also tested the number of branches for feature fusion. In the Proposed Approach section, we employed three branches for fusion, and in this section, we compare the differences in performance between two branch fusion and three branch fusion. From Table 7, we can see that the parameters for the fusion of the two branches are the same as our proposed MSAFM. In the process of merging two branches, the combination of Branch2+Branch3 has the best accuracy, but compared with the combination of three branches, the accuracy is not high.

Table 7.

Performance of different branch combinations in the Multi-Scale Adaptive Fusion Module on the DTU dataset (0.1s window).

Branch	Parameter	Decision window
		0.1-second
Branch1+Branch2	= 1, = 0.2	75.7±5.51
Branch1+Branch3		76.1±5.44
Branch2+Branch3		76.4±5.53
Ours		77.1±5.35

Finally, we tested different channel attention mechanisms. We compared our proposed cross channel attention mechanism with commonly used multi-head attention mechanisms, and from the experimental results in Table 8, it can be seen that our proposed mechanism achieved higher accuracy.

Table 8.

Comparison of our Cross-Channel Attention Mechanism (CCAM) with standard multi-head attention on the DTU dataset (0.1s window).

Method	Dateset	Decision window
		0.1-second
Multi-head attention	DTU	75.3±5.46
CCAM		77.1±5.35

Discussion

Comparative analysis

To comprehensively evaluate the decoding effectiveness of our network, we compared it with several recent methods. Experimental results indicate that our model exhibits improved decoding accuracy relative to these algorithms. As shown in Table 2, on the DTU dataset, the proposed model achieves decoding accuracies of 77.1 (SD = 5.35) under the 0.1-second decision window, 84.1 (SD = 4.76) under the 1-second window, and 85.6 (SD = 4.49) under the 2-second window. For the KUL dataset, the model’s accuracies are 90.3 (SD = 3.84) for the 0.1-second window, 95.8 (SD = 4.17) for the 1-second window, and 96.3 (SD = 3.56) for the 2-second window.

It can also be seen from Table 2 that compared with other recent algorithm models, such as SSF-CNN, MBSSFCC, DBPNet and DARNet, the decoding accuracy of different windows in DTU data sets is improved by 14.6, 10.2, 3.1, 2.5 respectively. On the KUL dataset, the decoding accuracy of different windows is improved by 14.0, 11.3, 5.0, 1.1 respectively. The longer the decision window, the higher the detection accuracy. This is because a longer decision window provides more information for the model to make judgments, while also mitigating the impact of individual outliers on the prediction. From the results, the decoding accuracy of our model is uniquely improved under different decision windows, The longer the decision window, the higher the detection accuracy. This is because a longer decision window provides more information for the model to make judgments, while also mitigating the impact of individual outliers on the predictionwhich further proves the effectiveness of the proposed netwrok.

Error analysis

The auditory input itself has periods of low energy or high ambiguity (e.g., pauses, overlapping speech, plosive sounds). During these segments, the auditory-evoked neural responses are inherently weaker and noisier. Consequently, the EEG correlates of attention become less distinct, leading to a higher probability of misclassification. This effect is likely more pronounced in the DTU dataset, which contains room reverberation, explaining its overall lower performance compared to the anechoic KUL recordings. Subject-Specific Variability and Overfitting: The ablation results in Fig. 2 (Subject-wise Results) reveal that for certain individuals (e.g., Subject 15 in DTU, Subjects 9 and 10 in KUL), performance did not improve with longer decision windows. This suggests that the model might have overfitted to the specific neural patterns of the majority of subjects in the training set and failed to generalize to those with atypical neurophysiological responses. Factors such as individual anatomical differences, varying cognitive strategies for selective attention, or even suboptimal electrode contact for certain subjects could contribute to this variability. Limitations of the Pure EEG Paradigm: Our model, like all current AAD methods, relies solely on EEG correlates. In extremely challenging acoustic scenarios where the brain itself struggles to segregate speakers (e.g., same-gender, spatially close speakers with similar vocal characteristics), the attentional modulation in the EEG might be too subtle to decode reliably. In these “failure cases” for the human brain, our model is also expected to fail.

In summary, despite the overall high performance on these two datasets, an analysis of misclassified trials suggests common failure cases. Firstly, segments of the auditory stimulus with low signal-to-noise ratio (e.g., pauses, overlapping speech) provide weak neural cues, challenging for the model to decode. This is particularly relevant for the DTU dataset with its reverberant environment. Secondly, subject-specific variability is a factor, as seen in Fig. 2 where performance for a few subjects (e.g., DTU S15, KUL S9/S10) did not scale with window length, indicating potential overfitting to the majority or idiosyncratic neural responses. These insights highlight that future improvements could come from modeling transitional states, incorporating robust training on noisy data, and personalizing models to individual subjects

Ablation analysis

This section analyzes the role of the different modules within the network. All experiments were conducted under the same hardware and software conditions.

Table 3 presents the results of the proposed methods following the removal of different modules. On the DTU dataset, removing the spatial-temporal feature extraction module (1s window) results in a decrease in model detection accuracy of 2.9. Removing the multi-scale adaptive fusion module (1s window) leads to a 3.8 decrease, while the removal of the cross-channel attention module (1s window) results in a 1.6 decrease. On the KUL dataset, the removal of the spatial-temporal feature extraction module (1s window) causes a 4.6 decrease in detection accuracy; the removal of the multi-scale adaptive fusion module (1s window) results in a 6.4 decrease, and removing the cross-channel attention module (1s window) leads to a 3.3 decrease. In terms of numerical proportions, the multi-scale adaptive fusion module demonstrates the greatest improvement in model detection accuracy.

Computational cost

This section compares our proposed network with several other recent models. As illustrated in Table 9, we analyze the number of model parameters. Our model has the smallest number of parameters and the highest operational efficiency. Specifically, the number of parameters in our model is only 0.11 of that in the SSF-CNN model, 0.05 of the MBSSFCC model, and 51.6 of the DBPNet model. This ability to reduce the number of parameters while improving detection accuracy enhances the deployment and application of our model across various devices.

Table 9.

Number of trainable parameters for our model and competing architectures.

Model	Trainable parameters
SSF-CNN34	4.21M
MBSSFCC18	83.9M
DBPNet25	0.91M
Ours

Hardware deployment

Our proposed model exhibits several inherent characteristics that make it particularly suitable for real-time deployment in resource-constrained systems such as next-generation neuro-steered hearing aids and brain-computer interfaces (BCIs):

High accuracy under short decision windows: The core requirement for real-world applications is low latency. Our model’s superior performance, especially under the challenging 0.1-second decision window (77.1 on DTU, 90.3 on KUL), demonstrates its ability to make accurate inferences with minimal delay. This is essential for applications like hearing aids, which must rapidly adjust their directional focus to the attended speaker in a dynamic acoustic environment.

Significant reduction in computational complexity: As evidenced in Table 9, our model contains only 0.47M parameters, which is nearly 50 fewer than the recent efficient model DBPNet (0.91M) and several orders of magnitude smaller than other benchmarks. This drastic reduction in model size and complexity directly translates to lower computational overhead, reduced power consumption, and faster inference times. This is a critical advantage for deployment on embedded systems with strict power and processing constraints, such as hearing aid chips or wearable EEG headsets.

Architectural suitability for efficient inference: The use of depthwise separable convolutions and a localized cross-channel attention mechanism is specifically designed to be hardware-friendly. These operations are highly optimized for modern mobile and embedded AI accelerators, facilitating efficient on-device execution without the need for continuous cloud offloading, thereby ensuring user privacy and reliability.

In summary, the combination of high low-latency accuracy and a small computational footprint positions our Hybrid Channel Attention Network as a strong candidate for practical integration into real-time, wearable BCI and auditory assistance systems. In future work, we plan to collaborate with hardware engineers to implement and validate our model on dedicated edge computing platforms.

Generalization ability

Our results provide preliminary evidence of cross-linguistic potential. The model was trained and tested on two distinct datasets involving Dutch (KUL) and Danish (DTU), which possess different phonetic and prosodic features. The fact that our model achieved state-of-the-art performance on both suggests that it is learning fundamental neural correlates of auditory attention that may be transferable across Indo-European languages. However, more rigorous testing on typologically diverse languages (e.g., tonal languages like Mandarin) is necessary to fully claim language-agnostic robustness. This is a key objective for our future work. Our study inadvertently demonstrates a degree of robustness to variations in EEG acquisition parameters. Although both datasets used a BioSemi system, the KUL data was originally sampled at 8192 Hz and downsampled to 128 Hz, while the DTU data was sampled at 512 Hz and downsampled to 64 Hz. The consistent high performance across these different sampling rates and subsequent preprocessing pipelines indicates that our model is not overly sensitive to these specific technical variations. The next critical step is to validate the model on data collected with completely different hardware systems and with varying electrode montages. The DTU dataset inherently contains a more challenging acoustic environment compared to the anechoic setting of KUL, as it includes real-world room reverberation. The model’s ability to maintain a high baseline performance (77.1 at 0.1s) on DTU is a strong indicator of its inherent robustness to certain types of environmental noise. This suggests that the learned spatial-temporal features are discriminative enough to overcome some degradations in audio quality. However, we have not explicitly tested it in environments with non-stationary background noise. Integrating such noise during training or employing domain adaptation techniques would be a crucial advancement.

Conclusion

In this paper, we have proposed a novel hybrid attention network that advances the state-of-the-art in AAD by simultaneously addressing three limitations of previous approaches: (1) the lack of integrated spatial-temporal-channel modeling, (2) the use of fixed architectural hyperparameters like kernel sizes, and (3) the computational inefficiency of global attention mechanisms. Our innovations in dynamic multi-scale processing and efficient cross-channel attention have resulted in a model that not only achieves superior accuracy but also possesses the efficiency necessary for potential real-time deployment. The proposed model utilizes a spatial-temporal extraction module to extract features in both the spatial and temporal domains. The multi-scale module helps the model analyze both short-term and long-term patterns in the EEG signals. Moreover, our attention mechanism focuses on local groups of channels rather than all channels at once, which helps better capture local brain interactions.

The proposed network model is evaluated on two benchmark datasets (KUL dataset and DTU dataset), demonstrating superior decoding accuracy compared to other recent network models. Furthermore, we assess the model’s parameters, verifying both its efficiency and portability across different platforms. In future work, we plan to incorporate graph neural networks into the AAD problem to enhance the correlation among different nodes and further improve decoding accuracy. Specifically, map EEG channels to graph nodes and construct a graph based on electrode spatial adjacency. Using GAT/GCN to fuse spatial dependencies after MSAFM module, and then parallel or cascaded with CCAM module, introducing cross subject adaptive graph. Explicitly modeling spatial topology and functional networks through graph neural networks to enhance cross device generalization ability.

Furthermore, while this work focused on a unimodal EEG approach, human auditory attention is a multi-faceted cognitive process. Future research will greatly benefit from a multimodal fusion framework. As suggested, integrating eye-tracking could provide a strong spatial prior on attentional focus, while pupillometry could offer valuable insights into cognitive load. Combining these signals with our high-performance EEG decoder within an adaptive fusion network presents a compelling pathway towards building more robust and informative neuro-steered devices that operate in ecologically valid environments.

Author contributions

Yahao Wen conceived the experiments, conducted the experiments, performed the software development, carried out the formal analysis, did the investigation, provided the resources, and was responsible for visualization, supervision, and funding acquisition. Yahao Wen, Shuai Ma and Yongjie Wang were involved in validation. Yahao Wen and Yongjie Wang prepared the original draft of the manuscript. Shuai Ma and Chuang Liu reviewed and edited the manuscript. All authors reviewed the manuscript.

Data availability

The datasets generated and/or analysed during the current study are not publicly available due to privacy concerns and their intended use in a commercial software database but are available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.