Structure-aware acoustic scene classification: a feature decoupling framework using HPSS and asymmetric convolutions

Abstract

Addressing the computational efficiency and cross-device generalization challenges faced by acoustic scene classification in resource-constrained environments, this study proposes a structure-aware dual-stream feature disentanglement framework based on harmonic-percussive source separation and asymmetric convolutions. The framework achieves structured decomposition of acoustic signals through HPSS techniques, performs independent modeling targeting temporal-frequency dimension differentiated characteristics through cascaded asymmetric convolution kernels, and realizes adaptive feature fusion through a dual attention mechanism. Systematic validation on the DCASE 2020 and TAU Urban Acoustic Scenes 2022 datasets demonstrates that the proposed method significantly reduces parameter count and inference time while maintaining competitive classification accuracy, exhibiting superior robustness compared to Transformer-based methods in cross-device scenarios and consistent generalization capability under reduced input duration conditions. This study provides a solution balancing performance and efficiency for acoustic scene classification deployment on edge devices and mobile platforms, with the proposed structure-aware feature disentanglement concept offering a new perspective for enhancing the interpretability of deep learning models through leveraging physical prior knowledge in the audio signal processing domain.

Introduction

Acoustic scene classification is an essential component of environmental audio understanding that plays an important role in intelligent surveillance systems, robotic auditory perception, smart city development, etc. The rapid development of the DCASE challenge has forced the field to address the challenges of cross-device generalization and limited computing resources¹. The application of acoustic scene classification on mobile devices requires the model to have strong robustness under limited memory and computing capacity², while the difference in acoustic characteristics between devices makes it difficult to apply models trained on single devices to other devices^3,4. The application of lightweight architecture has been shown to have potential in simplifying the architecture of acoustic scene classification models^5,6, while the application of feature disentanglement methods in cross-city and cross-device scenarios has shown the significance of structured modeling methods⁷; however, how to improve the robustness of cross-device acoustic scene classification while maintaining classification accuracy is an urgent problem to be solved.

However, deep learning technologies offer diversified solutions for acoustic scene classification, which still suffer from the problem of balancing parameter efficiency with performance. Audio Spectrogram Transformer has shown superiority in global modeling, which can effectively extract long-range temporal-frequency relations⁸. Meanwhile, self-supervised pre-training strategies can improve the generalization capacity of the model with limited labeled data⁹. The multi-scale spectrogram modeling can effectively extract fine-grained textures and coarse-grained structures by using multi-scale patch embedding mechanisms¹⁰. In addition, dynamic token pruning and attention sparsification can significantly reduce the computational overhead with good performance¹¹. Convolutional neural networks can adaptively highlight discriminative temporal-frequency regions by using temporal-frequency attention mechanisms¹² . The combination of graph neural networks with pre-trained models can introduce structured representation methods¹³. The temporal-frequency domain joint scanning mechanisms can achieve the decoupling of spatial exploration and feature aggregation¹⁴. Meanwhile, adaptive temporal resolution learning can enable the model to adjust the windows according to the dynamic characteristics of each scene¹⁵. The Patchout mechanism has been introduced to reduce the computational cost of audio Transformers by stochastically removing input patches during training while maintaining classification performance¹⁶, and iterative audio pre-training with acoustic tokenizers has demonstrated effective transfer learning capability for downstream audio classification tasks¹⁷. However, although these methods show good performance in single-device scenarios, the characteristics of large parameter counts and high computational overhead limit their applications in real-world scenarios.

The intrinsic dual nature of harmonic and percussive components within audio signals creates a unique perspective for feature engineering, based on the inherent presence of both sustained tonal and transient impulsive components within acoustic environments. Harmonic percussive source separation (HPSS) enables an effective separation process based on the continuity of tonal components within the frequency axis and the locality of impulsive components within the time axis¹⁸, while phase-aware convex optimization techniques improve temporal frequency consistency constraints during the separation process¹⁹. However, existing HPSS approaches are largely geared towards music signal processing, while their suitability for environmental acoustic scene classification tasks remains largely untapped, particularly with regards to the absence of thorough analysis on the contribution of different acoustic components towards scene classification.

Lightweight network architecture design aims to achieve a balance between parameter efficiency and representation ability through architectural innovations, with asymmetric convolution being the main technical solution. Asymmetric convolution techniques rely on using different kernel configurations for different dimensions to exploit the anisotropic properties of audio signals in both the time domain and frequency domain to maintain the ability to capture patterns along each dimension while reducing the number of parameters²⁰. Depthwise separable convolution significantly reduces multiply accumulate operations by decomposing regular convolution into depthwise convolution and pointwise convolution²¹, while diverse branch blocks are employed to extract multi-receptive field features within a single convolutional layer²². The combination of asymmetric convolution with depthwise separable convolution has been shown to achieve better parameter efficiency for agricultural pest classification tasks²³, while lightweight network architectures designed for audio classification have been shown to achieve model compression using knowledge distillation and channel pruning techniques²⁴. Analytical studies on various model compression techniques have shown the trade-offs between classification accuracy and speed for quantization, pruning, and low rank decomposition-based techniques for audio classification tasks²⁵, while hardware-based neural architecture search techniques have been shown to incorporate latency and energy constraints into the search objectives for neural architecture search²⁶. The synergy of multi-scale dilated convolution with depthwise separable convolution has been shown to achieve outstanding performance for sound event detection tasks²⁷, while lightweight transformers have been applied to underwater acoustic target recognition tasks to highlight the computational benefits of depthwise separable convolution²⁸. Asymmetric multi-task networks based on Kolmogorov-Arnold networks have introduced a new paradigm for multi-task knowledge sharing²⁹.

Data augmentation and transfer learning methods have been significant contributors to the improvement of the models’ robustness. The data augmentation methods for the spectrogram simulate the partial occlusion effect, which is characteristic of real-world environments. The random masking approach has been useful in overcoming the issue of overfitting³⁰. Global embeddings, which have been pre-trained on large bioacoustic datasets, have provided universal features that have been useful for transfer learning across domains³¹. The application of end-to-end residual networks has provided the ability for fast adaptation to medical audio screening applications³². The use of mobile audio Transformers for the estimation of the acoustic parameters of a room environment, as part of the blind room acoustic parameter estimation, has provided the ability for the application of self-attention models for unsupervised scene understanding³³. The systematic review of the visual representation of the temporal-frequency domain and the features of textures has provided a comprehensive landscape of the techniques used for audio applications³⁴.

It is worth noting that the challenges of time–frequency feature extraction and robust classification under complex conditions extend beyond acoustic scene classification to related signal processing domains, where analogous architectural innovations have demonstrated significant effectiveness. Time–frequency aware hierarchical feature optimization has been applied to SAR jamming recognition through structured multi-scale feature extraction³⁵, and semantic graph reasoning over multi-domain fusion has demonstrated the benefits of integrating structural representations from heterogeneous feature domains for radar jamming recognition³⁶. In the domain of automatic modulation recognition, multi-level feature distribution learning has addressed the open-set recognition challenge through hierarchical feature space modeling³⁷, while local–global feature space constraint networks have achieved robust open-set performance by jointly capturing fine-grained and holistic feature representations³⁸. The scalable subspace approach with similarity completion has further advanced the class incremental recognition paradigm through efficient feature space expansion mechanisms³⁹. These developments in related signal classification tasks underscore the universal importance of structured feature extraction and adaptive classification strategies across time–frequency analysis domains, providing additional motivation for the structure-aware approach adopted in the current investigation.

However, despite the achievements of existing studies on acoustic scene classification from various aspects, there are still three major gaps in existing studies that need to be addressed: Deep learning-based methods typically regard audio spectrograms as general images for processing, ignoring the harmonic-percussive dual structure characteristics of audio signals, which leads to the lack of specificity and explainability of the extracted features. The advantage of transformer-based methods in global modeling capability cannot be fully utilized due to the quadratic complexity of self-attention mechanisms and the large number of model parameters, which makes it difficult to meet the requirements of edge deployment with strict real-time constraints. The enhancement of cross-device generalization capability is typically realized by increasing model capacity or using complicated domain adaptation mechanisms, which contradicts the goal of designing lightweight models and is not practical in real-world applications.

It is important to note that while existing dual-stream architectures for acoustic scene classification have explored parallel feature extraction from different input representations, these approaches typically employ symmetric convolutional kernels across both streams and rely on simple concatenation for feature fusion without leveraging the physical structural priors inherent in audio signals. The HPSS technique, although well-established in music information retrieval, has not been systematically integrated into a deep learning-based dual-stream framework for acoustic scene classification with component-aware asymmetric convolution and attention-based disentanglement fusion, leaving the potential of combining structured signal decomposition with differentiated deep feature learning largely unexplored.

In this study, a structure-aware dual-stream feature disentanglement framework is proposed, which can be divided into three parts: audio signals can be divided into two parts through harmonic-percussive source separation; asymmetric convolutional neural networks can be designed for differential modeling of the temporal-frequency characteristics of different audio components; and dual-stream features can be adaptively combined through disentanglement fusion mechanisms. Systematic experiments on the DCASE 2020 and TAU Urban Acoustic Scenes 2022 benchmark datasets validate the effectiveness of the proposed framework in both single-device and cross-device scenarios as well as under reduced input duration conditions.

Methodology

Overall framework design

The acoustic scene classification task aims to map input audio signals to a predefined set of scene categories . The proposed structure-aware dual-stream feature disentanglement framework achieves targeted feature extraction by fully exploiting the harmonic-percussive dual structure characteristics of audio signals, with the core innovation lying in combining structured signal decomposition with deep neural network representation learning to capture complementary discriminative information from different acoustic components through a dual-stream parallel processing mechanism.

The framework adopts Mel-spectrogram as the input representation, converting raw audio into a time–frequency representation through short-time Fourier transform, where 128 denotes the number of Mel filter banks and represents the number of time frames. The harmonic-percussive source separation module decomposes the time–frequency representation into harmonic component and percussive component , preserving sustained tonal features and transient impulsive features respectively. The dual-stream asymmetric convolutional neural network performs independent feature extraction on the two complementary components, where the harmonic stream and percussive stream adopt identical network structures but maintain independent parameters, enabling each stream to optimize convolution kernel configurations tailored to the temporal-frequency characteristics of its corresponding acoustic component. The feature disentanglement and fusion module adaptively integrates dual-stream features through a dual attention mechanism, modeling the contribution of different feature channels in the channel dimension and locating discriminative regions on the time–frequency plane in the spatial dimension. The fused features are mapped to category probability distribution through fully connected layers, optimized via cross-entropy loss function:

1

where denotes the one-hot encoding of true category labels. This end-to-end trainable architecture embodies a complete processing chain from structured signal decomposition to disentangled feature learning to adaptive fusion, with Fig. 1 illustrating the overall framework’s data flow and three zoomed-in detail regions elaborating key design components.

Fig. 1.

Dual-stream feature disentanglement framework with asymmetric CNN.

Figure 1 shows the entire data flow, with blue and orange representing harmonic and percussive processing, respectively. The zoomed areas highlight key architectural features. The asymmetric convolution block uses cascaded 3 × 5 and 5 × 3 filters to extend frequency and temporal receptive fields, respectively. The dual attention mechanism uses channel attention from global pooling and an MLP, and spatial attention from channel-wise pooling and 7 × 7 convolution. The dual-stream design visually represents channel expansion from 64 to 64, then to 128, to 256, and finally to 512, while highlighting shared structure and distinct parameters ().

HPSS-based dual-stream feature extraction

The harmonic percussive source separation techniques are based on prior knowledge of the audio signals within the time–frequency domain. For instance, the harmonic components are represented by horizontal bands along the frequency axis, while the percussive components are represented by vertical bands along the temporal axis. In this investigation, the median filtering-based harmonic percussive source separation technique has been employed to separate the structural components of the signals within the spectrogram domain using median filters of size 17 along the frequency axis and the temporal axis, respectively. The process of extracting the harmonic components preserves the energy components that are continuously distributed along the frequency axis, while the process of extracting the percussive components preserves the energy components that are locally concentrated along the temporal axis.

The separated harmonic and percussive components are converted into Mel-spectrogram representations as inputs to the dual-stream network. This harmonic component shows strong discriminative ability for acoustic scenes where there are musical background sounds and/or mechanical steady-state noise. Conversely, the percussive component of the signal focuses on the temporal localization and intensity characteristics of impulsive information, showing stronger discriminative ability for acoustic scenes with sudden acoustic events. This component-level distinction provides structured input for the subsequent differentiated modeling of the dual-stream network, where the varying proportions of harmonic and percussive components of different acoustic scenes allow the dual-stream features to show contrasting discriminative ability. To demonstrate the effectiveness of HPSS-based separation for different acoustic environments, Fig. 2 presents the HPSS-based separation for three acoustic scenes chosen from the DCASE 2020 dataset.

Fig. 2.

HPSS-based component separation for representative acoustic scenes.

Figure 2 depicts different patterns of harmonic percussive energy distribution for different scene types. In an airport, there are significant levels of harmonic energy at lower frequency regions, corresponding to sustained noise from engines, and percussive energy for transient sounds. In a park, there are balanced levels of dual components, where harmonic energy corresponds to bird chirps with frequency continuity, and percussive energy corresponds to footsteps with temporal localization. In a street traffic scene, there are significant levels of percussive energy across the entire frequency range, corresponding to braking, horns, and accelerating sounds. These patterns substantiate the dual role of discriminative features and provide a physical basis for architectural design.

Asymmetric convolutional neural network

The design of asymmetric convolutional neural networks incorporates the specific physical semantics of audio spectrograms along the temporal and frequency axes. Along the frequency axis, where the specific physical semantics of the structure of acoustic signals are represented, there are strong correlations between adjacent frequency components due to formant and harmonic effects. Along the temporal axis, where the specific physical semantics of the temporal evolution of acoustic events are represented, the correlations between adjacent time frames are affected by the duration of the events. The asymmetric convolution module developed in the current study employs a cascaded structure of 3 × 5 and 5 × 3 convolutional kernels. This structure extends the receptive fields along the frequency and temporal axes, respectively. The 3 × 5 convolution, with its extended frequency-axis receptive field spanning five adjacent Mel bins, is specifically designed to capture cross-frequency correlations arising from harmonic overtone series where fundamental frequencies and their integer multiples occupy predictable spectral positions, while the 5 × 3 convolution, with its extended temporal receptive field spanning five consecutive frames, captures the onset-sustain-decay dynamics of acoustic events that unfold over multiple time steps. This cascaded configuration ensures that each convolution operates along the dimension where the strongest structural correlations exist for its corresponding acoustic component, rather than applying a uniform square kernel that allocates equal modeling capacity to both dimensions regardless of the underlying signal characteristics, a design principle that is empirically validated by the ablation results in Table 1 where the asymmetric 3 × 5 and 5 × 3 configuration outperforms both the 3 × 3 and 5 × 5 symmetric alternatives.

Table 1.

Ablation study results.

Ablation component	Configuration	Accuracy (%)	F1-score (%)	Log loss
HPSS separation	Only original audio (no HPSS)	69.4	68.1	1.06
	Only harmonic component	68.2	66.8	1.12
	Only percussive component	67.5	66.0	1.17
	Dual-stream fusion	72.1	70.6	0.93
Asymmetric convolution	Standard 3 × 3 symmetric	70.4	68.9	1.01
	Large 5 × 5 symmetric	70.9	69.4	0.99
	Asymmetric 3 × 5 and 5 × 3	72.1	70.6	0.93
Feature fusion	Simple concatenation	70.3	68.7	1.02
	Channel attention only	71.2	69.6	0.97
	Spatial attention only	70.8	69.2	1.00
	Dual attention	72.1	70.6	0.93

The harmonic and percussive streams of the dual-stream network architecture make use of the same network structure while performing independent parameter optimization. Each stream is formed of a series of four blocks of asymmetric convolution. Each of the blocks consists of a cascaded structure of 3 × 5 convolution followed by batch normalization and ReLU activation, and then 5 × 3 convolution followed by batch normalization and ReLU activation. The feature map channel count progressively increases from an initial 64 channels to 512 channels across network depth, with global average pooling layers compressing spatial dimension feature maps into vector representations . The main advantage of the dual-stream network is that it can learn convolutional filters based on frequency continuity features as well as temporal transient features through independent parameter training of the harmonic and percussive streams, respectively, leading to improved discriminative feature learning compared to other network architectures.

Feature disentanglement and fusion

The feature disentanglement and fusion module achieves adaptive feature integration through a dual attention mechanism. The channel attention mechanism captures the statistical distribution of feature maps through global average pooling and global max pooling, generating channel dimension attention weights through a shared multilayer perceptron, with this mechanism highlighting feature channels with high contribution to scene discrimination while suppressing redundant channel influence, drawing inspiration from Gaussian-weighted self-attention in speech enhancement tasks⁴⁰. The spatial attention mechanism focuses on locally salient areas in the time–frequency domain of the feature map. Spatial-dimensional attention weights are learned using 7 × 7 convolutional layers after applying pooling in the channel dimension. This helps to focus more on discriminative time–frequency areas, which is analogous to dense convolutional neural networks that incorporate self-attention mechanisms for handling long-range dependencies in time-domain speech enhancement⁴¹. The application of channel and time–frequency attention has been shown to result in significant performance improvements in monaural speech enhancement⁴², with recent advances further demonstrating the effectiveness of triple attention mechanisms combined with temporal convolutional architectures for real-time speech enhancement⁴³, complex dual-path multi-axial transformers with frequency prompting for joint spectral-temporal modeling⁴⁴, and a comprehensive survey consolidating the evolution of deep learning-based speech enhancement techniques⁴⁵, collectively supporting the extension of attention-based feature integration to acoustic scene classification in the proposed framework.

The disentangled dual-stream features are integrated through a weighted fusion strategy:

2

where represents a learnable fusion weight, and and denote the channel attention weights for harmonic and percussive streams respectively. This fusion strategy allows the network to adaptively adjust fusion ratios based on the dependence degrees of different scene categories on harmonic and percussive components.

Feature disentanglement mechanisms have been proven to improve the performance of models in different domains. This is evident in the application of decoupling methods in deep neural networks for multi-agent collaborative inference, demonstrating the importance of decoupling in achieving efficient feature representation in models⁴⁶. The application of robustly optimized deep feature decoupling networks in fatty liver detection also achieves more reliable diagnostic results through the decoupling of features from influencing factors in the detection process⁴⁷. The decoupling of feature extraction and classification layers in neural network calibration also improves the quality of confidence estimation in the model⁴⁸. The success of deep learning decoupling in improving the interpretability of image recognition also underscores the importance of decoupling in machine learning models in general⁴⁹.

The classifier adopts a two-layer fully connected network configuration with an intermediate layer dimension of 256 and Dropout regularization (dropout rate = 0.5), with the output layer mapping to category probability distributions through Softmax. Training employs the Adam optimizer (learning rate = 0.001), applying SpecAugment data augmentation strategies with random masking in the time–frequency domain (time mask width 40 frames, frequency mask width 8 bins, 2 masks each) to enhance model robustness. The entire framework achieves joint optimization through end-to-end backpropagation, enabling the HPSS separation, dual-stream feature extraction, and disentanglement fusion modules to collaboratively learn optimal parameter configurations oriented toward scene classification tasks.

Results

Experimental setup

This study conducts systematic experimental validation using the TAU Urban Acoustic Scenes 2020 Mobile dataset (https://zenodo.org/record/3819968)⁵⁰. The dataset comprises 10 urban acoustic scene categories, encompassing Airport, Bus, Metro, Metro station, Park, Public square, Shopping mall, Street pedestrian, Street traffic, and Tram, with the training set containing 14,400 ten-second audio clips (1440 per class) and the test set containing 4185 audio clips. A distinctive feature of this dataset lies in its utilization of three different recording devices (Device A as the primary device, Devices B and C for cross-device generalization testing) with audio sampled at 44.1 kHz, with this multi-device configuration providing rigorous testing conditions for evaluating the practical deployment robustness of models.

Model performance is quantitatively assessed through three evaluation metrics, where classification accuracy reflects overall scene recognition capability, macro-average F1 score measures the balanced performance of the model across different categories, and Log Loss serves as the DCASE official evaluation metric to reflect the quality of predicted probability distributions. To comprehensively evaluate the effectiveness of the proposed method, nine representative baseline methods are selected for comparison, including three classical methods comprising the DCASE 2020 official Baseline (based on standard CNN architecture), ResNet-18 (deep residual network), and VGGish (Google pre-trained audio classification model), along with six recent advanced methods including Audio Spectrogram Transformer, EfficientAT, Self-Supervised AST, Temporal-Frequency Attention CNN, PaSST, and BEATs, with these baseline methods covering different technical approaches such as convolutional neural networks, standard and lightweight Transformer architectures, and attention mechanisms. The experimental implementation is based on the PyTorch 2.0 framework, employing the Adam optimizer (learning rate 0.001, weight decay 1e-4), with batch size set to 32 and training conducted for 100 epochs with early stopping strategy to prevent overfitting. Data augmentation strategies include SpecAugment (time mask width 40 frames, frequency mask width 8 bins, 2 masks each) and MixUp to enhance model robustness, with all experiments performed on an NVIDIA RTX 3090 GPU to ensure computational efficiency and result reproducibility.

Ablation study

In order to evaluate each of these components of the structure-aware two-stream feature decoupling framework, this study proposes three groups of ablation experiments to verify the effectiveness of HPSS separation, asymmetric convolution, and feature decoupling mechanisms, respectively. The rationale for conducting ablation experiments to verify the effectiveness of HPSS separation is based on the fundamental hypothesis of this study, that is, the discriminative ability of both harmony and percussion features for different acoustic scenes is complementary. This group of ablation experiments is used to evaluate the necessity of adopting two-stream architecture, and four different configurations are used to compare each other, that is, one-stream architecture based on raw audio, one-stream architecture based on harmony features, one-stream architecture based on percussion features, and two-stream architecture based on both harmony and percussion features. The rationale for conducting ablation experiments to evaluate asymmetric convolution is to explore the benefits of different modeling strategies for different components within the time–frequency domain. This group of ablation experiments compares the effectiveness of different configurations of convolutional kernels, that is, conventional 3 × 3 symmetric convolution, large 5 × 5 symmetric convolution, and asymmetric 3 × 5 and 5 × 3 cascaded convolution, and seeks to evaluate the trade-off between efficiency and effectiveness of asymmetric convolution while fixing other components of the architecture. The rationale for conducting ablation experiments to evaluate feature decoupling is to quantify the contribution of adopting an attention mechanism for feature integration. Four different strategies, that is, simple splicing, channel attention, space attention, and dual attention, are used to compare each other, and all ablation experiments are conducted on device A to keep all factors unchanged, and Table 1 summarizes all ablation study results.

The ablation results presented in Table 1 quantify the individual contributions of the three core components constituting the proposed framework. The HPSS dual-stream architecture demonstrates the most substantial performance gain of 2.7 percentage points over the single-stream baseline using original audio, with Log Loss reduced from 1.06 to 0.93, while the individual harmonic stream at 68.2% and percussive stream at 67.5% both fall below the original audio baseline of 69.4%, confirming that neither component alone captures the full discriminative information and that their complementary fusion is essential for recovering and exceeding the baseline performance. The asymmetric convolution configuration contributes an additional improvement of 1.7 percentage points over the standard 3 × 3 symmetric convolution and 1.2 percentage points over the 5 × 5 symmetric alternative, with the F1-score correspondingly increasing from 68.9% and 69.4% to 70.6%. The dual attention disentanglement mechanism yields a gain of 1.8 percentage points over simple concatenation, with channel attention alone at 71.2% and spatial attention alone at 70.8% each contributing partially, while their combination achieves 72.1% with a Log Loss of 0.93 that surpasses both individual variants.

The quantified contributions reveal a hierarchical importance ordering where HPSS-based structured decomposition provides the foundational performance gain, asymmetric convolution refines the feature extraction efficiency along differentiated time–frequency dimensions, and the dual attention mechanism further optimizes the feature space through synergistic channel-spatial interactions where channel attention selectively emphasizes discriminative feature channels and spatial attention focuses on salient time–frequency regions that distinguish acoustically similar scenes.

Beyond the architectural component ablation, the sensitivity of the HPSS decomposition to the median filter size, which governs the trade-off between harmonic and percussive separation granularity, is investigated by varying the filter size across a range of values while keeping all other architectural components fixed at their default configurations, as presented in Table 2.

Table 2.

Sensitivity analysis of HPSS median filter size.

Filter size	Accuracy (%)	F1-score (%)	Log loss
5	69.3	67.6	1.03
9	70.8	69.2	0.99
13	71.6	70.1	0.96
17 (default)	72.1	70.6	0.93
21	71.8	70.2	0.94
25	71.1	69.5	0.98

Table 2 reveals that the classification performance peaks at the default filter size of 17 with an accuracy of 72.1% and exhibits moderate sensitivity to this hyperparameter, with a total performance range of 2.8 percentage points across the tested values. The asymmetric degradation pattern is noteworthy, where smaller filter sizes cause more substantial performance decline than equidistant larger values, attributable to the fact that insufficient smoothing produces poorly separated components with significant cross-contamination between harmonic and percussive streams that undermines the fundamental premise of the dual-stream architecture, whereas excessive smoothing preserves the broad structural distinction between the two components while only sacrificing fine-grained spectral details. The mid-range filter sizes of 13, 17, and 21 all achieve accuracy above 71.5%, suggesting that the proposed framework is robust to moderate variations in this hyperparameter and does not require exhaustive task-specific tuning for practical deployment.

Performance comparison with state-of-the-art methods

In order to evaluate the competitiveness of the proposed approach for acoustic scene classification, this study undertakes comparative experiments with nine state-of-the-art approaches. The comparative experiments aim to evaluate the structure-aware dual-stream architecture from three different aspects: accuracy, computational complexity, and speed. All approaches are implemented and tested on Device A under the same conditions, with the same training strategy (Adam optimizer, learning rate = 0.001, batch size = 32), and data augmentation strategies (SpecAugment and MixUp). The state-of-the-art approaches used for comparative analysis cover various technical aspects, ranging from classical approaches such as DCASE 2020 Baseline, ResNet-18, and VGGish, to modern advanced approaches such as Audio Spectrogram Transformer, EfficientAT, Self-Supervised AST, Temporal-Frequency Attention CNN, PaSST, and BEATs. This wide range of state-of-the-art approaches used for comparative analysis ensures that the relative performance of the proposed approach is evaluated from different architectural aspects. The performance comparison results of all approaches are listed in Table 3.

Table 3.

Performance comparison with state-of-the-art methods.

Method	Accuracy (%)	F1-score (%)	Log loss
DCASE 2020 baseline	65.3	63.8	1.26
VGGish	67.9	66.3	1.15
ResNet-18	68.6	67.0	1.09
EfficientAT	69.9	68.3	1.03
TFCNN	70.5	68.9	0.99
PaSST	71.6	70.1	0.95
SSAST	72.7	71.0	0.88
BEATs	72.9	71.2	0.87
AST	73.1	71.4	0.86
Ours	72.1	70.6	0.93

From Table 3, the accuracy of the proposed method is 72.1%, which is higher than most of the baseline methods by 6.8% compared to the DCASE 2020 Baseline, 3.5% compared to ResNet-18, and 4.2% compared to VGGish, while surpassing the lightweight Transformer-based PaSST by 0.5 percentage points despite requiring substantially fewer parameters at 12.4 M compared to 86.2 M. This proves the effectiveness of the proposed structure-aware dual-stream architecture. Although the Audio Spectrogram Transformer has a higher accuracy of 73.1%, its number of parameters is much higher at 86 M compared to the proposed method’s 12.4 M parameters. The proposed method is much more efficient than the Audio Spectrogram Transformer. Moreover, the proposed method has a lower Log Loss of 0.93 compared to most of the baseline methods.

To assess the statistical significance of the observed performance differences, the proposed method and each baseline are independently trained ten times with different random seeds, and the paired t-test is applied to the accuracy measurements across the ten runs on the Device A test set, as summarized in Table 4.

Table 4.

Statistical significance analysis (paired t-test, 10 runs).

Comparison	Ours (Mean ± Std)	Baseline (Mean ± Std)	t-statistic	p-value
Ours vs. DCASE baseline	72.08 ± 0.47	65.34 ± 0.52	18.67	< 0.001
Ours vs. VGGish	72.08 ± 0.47	67.82 ± 0.55	12.93	< 0.001
Ours vs. ResNet-18	72.08 ± 0.47	68.53 ± 0.49	9.84	< 0.001
Ours vs. EfficientAT	72.08 ± 0.47	69.87 ± 0.43	5.72	< 0.001
Ours vs. TFCNN	72.08 ± 0.47	70.43 ± 0.51	3.41	0.008
Ours vs. PaSST	72.08 ± 0.47	71.53 ± 0.52	1.51	0.165
Ours vs. SSAST	72.08 ± 0.47	72.54 ± 0.61	− 1.27	0.236
Ours vs. BEATs	72.08 ± 0.47	72.84 ± 0.57	− 1.85	0.097
Ours vs. AST	72.08 ± 0.47	72.93 ± 0.58	− 2.08	0.067

Table 4 confirms that the performance improvements of the proposed method over the DCASE 2020 Baseline, VGGish, ResNet-18, EfficientAT, and TFCNN are statistically significant at the 0.05 significance level, while the differences with the PaSST, SSAST, BEATs, and AST do not reach statistical significance, which is consistent with the competitive rather than superior positioning of the proposed method relative to Transformer-based architectures and supports the interpretation that the observed performance advantages over CNN-based baselines are robust and reproducible rather than attributable to random training variability.

Quantification of the complexity of the model and the efficiency-performance trade-off is necessary for assessing the practical value of the deployed model. The experimental design intends to visualize the equilibrium of high accuracy and low complexity computationally through simultaneous depiction of comprehensive performance across three parameters: accuracy, parameters, and inference time. Inference time was conducted on an NVIDIA RTX 3090 GPU, with each test case subject to 100 consecutive inferences and then averaged to reduce measurement variability. A batch size of 1 was used to simulate real-world inference scenarios involving individual samples. A holistic analysis of these three parameters enables the elucidation of the trade-offs between efficiency and performance for each of the methods. Figure 3 shows the comparative results of the proposed method and the baseline methods for these parameters.

Fig. 3.

Efficiency-performance tradeoff of different methods.

Figure 3 shows the efficiency-performance trade-off of the proposed approach. As can be seen from Fig. 3a, the proposed approach has an accuracy of 72.1%, which is comparable to the Transformer-based methods AST, BEATs, SSAST, and PaSST, which range from 71.6 to 73.1%, while requiring only 12.4 M parameters compared to the 82.3–90.6 M parameter range of these Transformer-based methods. Figure 3b shows that the proposed approach achieves an inference time of 24.1 ms, which is substantially faster than PaSST at 44.6 ms and the larger Transformer methods AST, SSAST, and BEATs that range from 65.7 to 72.4 ms, while maintaining an accuracy advantage of 2.2 percentage points over the lightweight EfficientAT approach.

Beyond accuracy and parameter count, a comprehensive assessment of computational efficiency for edge deployment requires the quantification of floating-point operations, peak GPU memory consumption during inference, and CPU-based inference latency, as these metrics collectively determine the feasibility of deploying acoustic scene classification models on resource-constrained platforms where GPU acceleration may not be available. The computational complexity of each method is profiled under identical experimental conditions using a single 10-s audio clip as input, with FLOPs calculated via the ptflops profiling tool, GPU memory measured as peak allocation on the NVIDIA RTX 3090, and CPU inference time averaged over 100 iterations on an Intel Core i7-12700 K processor, as summarized in Table 5.

Table 5.

Computational complexity comparison of different methods.

Method	Params (M)	FLOPs (G)	GPU Mem (MB)	GPU time (ms)	CPU time (ms)
DCASE 2020 baseline	0.52	0.13	48	3.2	15.7
VGGish	62.4	1.58	278	12.6	108.2
ResNet-18	11.2	2.47	184	15.8	73.6
EfficientAT	4.47	0.89	116	8.4	39.2
TFCNN	8.15	2.17	158	11.3	56.4
PaSST	86.2	14.23	528	44.6	292.4
SSAST	82.3	19.82	652	65.7	415.8
BEATs	90.6	22.48	702	72.4	462.8
AST	86.0	21.35	684	68.3	441.6
Ours	12.4	2.86	198	24.1	98.3

Table 5 reveals that the proposed method requires only 2.86G FLOPs and 198 MB GPU memory, representing reductions of 86.6% and 71.1% respectively compared to the AST, and reductions of 79.9% and 62.5% compared to the lightweight PaSST that employs the Patchout mechanism to reduce Transformer computation, a disparity attributable to the quadratic complexity of self-attention mechanisms versus the linear scaling of convolutional operations. This efficiency advantage is further amplified under CPU inference conditions that better approximate edge deployment, where the proposed method completes inference in 98.3 ms compared to 441.6 ms for the AST, as the lack of hardware-optimized attention kernels on general-purpose CPUs further penalizes Transformer architectures. The moderate computational footprint of the dual-stream architecture relative to lightweight approaches such as EfficientAT can be attributed to the parameter-efficient design of asymmetric convolutions, where the cascaded 3 × 5 and 5 × 3 kernel configuration achieves an effective receptive field equivalent to a 7 × 7 symmetric kernel while requiring only 30 multiply-accumulate operations per position compared to 49 for the equivalent square alternative.

Cross-device generalization performance

The capability of cross-device generalization acts as a vital measure for evaluating the robustness of acoustic scene classification systems. The rationale for using this experimental methodology is rooted in the highly challenging requirements set by the DCASE challenge, which calls for high performance in model transfer from one recording device to another. In this work, the robustness of the acoustic scene classification model is evaluated through the implementation of four systems for training and testing configurations. The first configuration represents the standard scene, which trains on one device and tests on the same device, thereby providing a performance baseline. The second and third configurations represent cross-device scenarios, which involve training on one device and testing on another, thereby evaluating the robustness of the model with respect to domain shifts caused by recording devices. The fourth configuration represents a mixed training scenario, which trains on three devices and tests on each individual device, thereby evaluating the performance of the model with respect to joint training on multiple devices. The experimental methodology also considers baseline methods that encompass three technical aspects, thereby providing a comprehensive comparison of the relative merits of the proposed method with respect to the baseline methods, as presented in Table 6.

Table 6.

Cross-device generalization performance comparison.

Method	A → A	A → B	A → C	Mixed → A	Mixed → B	Mixed → C
AST	73.1	66.1	65.3	72.8	70.3	69.7
ResNet-18	68.6	62.1	61.4	68.2	65.7	64.9
EfficientAT	69.9	64.2	63.6	69.4	66.8	66.1
PaSST	71.6	64.7	64.0	71.2	68.7	68.0
BEATs	72.9	66.2	65.4	72.4	70.0	69.3
Ours	72.1	66.7	66.0	71.6	69.1	68.5

Training and testing configurations are denoted as “Train → Test”, where A, B, and C represent three different recording devices. “Mixed” indicates joint training on all three devices (A + B + C). All values represent classification accuracy in percentage.

Table 6 shows that significant robustness benefits can be achieved by using the proposed approach, especially for cross-device scenarios. The performance degradation from Device A to Devices B and C is found to be 5.4 and 6.1 percentage points, respectively, which is much smaller compared to the degradation of 7.0 and 7.8 percentage points for the Audio Spectrogram Transformer and that of 6.5 and 7.2 percentage points for ResNet-18, as well as 6.9 and 7.6 percentage points for PaSST and 6.7 and 7.5 percentage points for BEATs. This cross-device robustness is due to the effective extraction of recording device-agnostic acoustic structural characteristics using the HPSS separation technique and device-specific adaptation using independent parameter training within the dual-stream architecture.

In order to fully evaluate the adaptation capacity and performance balance of different methods, this experiment used the method of visual comparison on the results under mixed training configuration (joint training on Devices A + B + C). The rationale for choosing this mixed training configuration is that this configuration can exclude the bias from any single source device and may be closer to the real performance of models when confronted with different recording devices in real-world scenarios. The fundamental purpose of this visualization is to explore the performance distribution characteristics of different methods on these three devices, especially to identify which methods can achieve good performance on all devices and which methods show obvious device preference. This multi-dimensional performance comparison is significant for evaluating the application values of different models, because the type of recording device used by users in real scenarios is often uncertain and varied. Figure 4 shows the cross-device generalization performance comparison based on mixed training configuration.

Fig. 4.

Cross-device generalization performance comparison.

Figure 4 demonstrates the cross-device performance comparison result using the hybrid training approach. The accuracy of the proposed method is 71.6%, 69.1%, and 68.5% on devices A, B, and C, respectively, which is comparable to the Audio Spectrogram Transformer and BEATs while requiring only 14% of the AST model parameters and nearly 3 times faster inference speed. The performance degradation from device A to C is more pronounced for the Transformer-based methods PaSST, SSAST, BEATs, and AST as well as the CNN-based ResNet-18 and EfficientAT, whereas the proposed method exhibits the smallest cross-device variation among all compared approaches. The stability of the proposed method on cross-device performance can be ascribed to the effectiveness of HPSS-based extraction of device-independent acoustic features and the parameter efficiency advantage of the dual-stream structure.

Feature analysis and visualization

In order to deeply understand the influence of the two-stream feature decoupling mechanism on the discriminative ability of the model, this study comparatively analyzes the quality of representation changes before and after the fusion of features. The experiment adopts the test set of device A, where 100 random samples of each type of scene are randomly selected to extract the simple splicing fusion features of the two-stream network and the final features after the decoupling of the dual attention mechanism. The reason for selecting these two features to compare with each other is that the former simply concatenates the feature vector of harmonic flow with the feature vector of percussion flow without any selection mechanism, while the latter adopts the channel attention mechanism and spatial attention mechanism to realize the adaptive weighting of features and the focusing of features in the region. The most important aspect of the comparison between the two features is to verify whether the attention decoupling mechanism has improved the inter-class separability and intra-class compaction of the features, so as to verify the effectiveness of the proposed decoupling strategy in the optimization of the feature space structure. As shown in Fig. 5 below, the feature space distribution before and after decoupling.

Fig. 5.

t-SNE visualization of learned feature representations.

Figure 5 comparison shows the difference of spatial distribution before and after feature decoupling. Figure 5a shows that in the simple stitching feature space, semantically similar scene pairs such as Metro and Metro station, Street pedestrian and Street traffic have obvious overlap, and the classification boundary is fuzzy. Figure 5b shows that after the dual attention decoupling, the inter-class separation is significantly improved, and the same class samples form more compact clusters. This improvement is attributed to the selective enhancement of channel attention to discriminative feature channels and the ability of spatial attention to focus on key time–frequency regions.

For the purpose of gaining a comprehensive insight into the differences in discriminative capability and the types of misclassification for individual scene classes, the entire classification results on the test set of device A are analyzed in detail in this research work. The main focus of the analysis is to identify the types of scene pairs that are consistently misclassified by the classification model. This type of analysis essentially indicates the level of ambiguity in the acoustic feature space and hence offers clear guidelines for the subsequent optimization process. This level of analysis of the classification performance is critical for gaining an insight into the application characteristics of the classification model, as in practical application scenarios, it is not only important to achieve high accuracy in classification but also to achieve stable classification results in critical scenes. Figure 6 shows the classification performance analysis.

Fig. 6.

Confusion matrix for scene classification on Device A test set.

As depicted in Fig. 6, the confusion matrix shows the classification results for the test set of device A. High recognition accuracy is attained by the model for the Airport scene at 79.0%, the Park scene at 76.0%, and the Tram scene at 77.0%. These three scenes have distinct acoustic features. The significant confusion between the classes is 12.0% between Metro and Metro station, and 9.0% and 11.0% between Street pedestrian and Street traffic. The confusion between the classes is due to the acoustic features shared by the similar classes, which include running noise from the train or a mixture of features from vehicles and humans on the street.

Cross-dataset generalization evaluation

To further validate the generalization capability of the proposed framework under different evaluation conditions, additional experiments are conducted on the TAU Urban Acoustic Scenes 2022 Mobile dataset⁵¹, which shares the same ten urban scene categories and underlying recordings as the DCASE 2020 dataset but presents substantially different evaluation conditions. The segment duration is reduced from ten seconds to one second, significantly limiting the temporal context available for scene discrimination and increasing the total number of evaluation samples tenfold, while the device diversity in the development set is expanded from three real devices to three real devices supplemented by six simulated devices generated through impulse response convolution, providing a more rigorous testbed for assessing classification robustness under varied input duration and expanded device configurations. All methods are evaluated using the official development set train-test split with identical training configurations applied consistently across all approaches, as presented in Table 7.

Table 7.

Performance comparison on the TAU Urban Acoustic Scenes 2022 dataset.

Method	Accuracy (%)	F1-score (%)	Log loss
ResNet-18	55.4	53.7	1.51
EfficientAT	57.3	55.7	1.43
PaSST	58.2	56.6	1.36
AST	59.8	58.3	1.28
BEATs	60.6	59.1	1.25
Ours	59.1	57.6	1.32

Table 7 demonstrates that all methods exhibit notable performance degradation on the TAU Urban Acoustic Scenes 2022 dataset compared to the DCASE 2020 benchmark, which is attributable to the substantially shorter segment duration of one second that limits the amount of temporal context available for scene discrimination. The proposed method achieves an accuracy of 59.1%, narrowing the performance gap with the AST from 1.0 percentage point on DCASE 2020 to 0.7 percentage point on DCASE 2022, a finding consistent with the expectation that the HPSS-based structural decomposition retains its effectiveness on shorter audio segments where the global modeling advantage of Transformer architectures is diminished by the reduced sequence length. The consistent relative performance of the proposed method across both datasets confirms that the harmonic-percussive structural patterns captured by the dual-stream architecture generalize effectively across different recording conditions and device configurations rather than overfitting to the specific characteristics of a single benchmark. It is also noteworthy that the proposed method surpasses PaSST by 0.9 percentage points on the DCASE 2022 dataset despite requiring only 14.4% of its parameters, further confirming the efficiency advantage of the proposed architecture over lightweight Transformer alternatives under reduced input duration conditions.

Discussion

The proposed framework of structure-aware dual-stream feature disentanglement has been shown to attain an accuracy of 72.1%, which is an increase of 6.8 percentage points over the DCASE 2020 Baseline of 65.3%. The relative contributions of the different components of the framework have been made evident through ablation studies, where the HPSS dual-stream architecture has been seen to contribute to an increase of 2.7 percentage points over the baseline, with asymmetric convolution and dual attention disentanglement showing significant contributions to the performance of the framework. The effectiveness of different strategies of feature disentanglement has been evident in different domains as well. For example, in the context of domain adaptation research, iterative resampling decoupling has been used to attain domain transfer with class imbalance⁵², while in the context of domain generalization networks, robustness under unknown conditions has been attained⁵³, with the proposed framework being able to attain more interpretable component-level disentanglement with the help of acoustic physical priors. Decoupling techniques applied to long-tail classification have similarly demonstrated the effectiveness of separating representation learning from classifier training to mitigate class imbalance⁵⁴ , further corroborating the general principle that structured decomposition of the learning process yields performance benefits across diverse task configurations.

In the horizontal comparison using advanced techniques, the unique benefits of the proposed method are clearly illustrated through the efficiency-performance trade-off. For instance, the proposed method has an accuracy of 72.1%, slightly lower than that of the Audio Spectrogram Transformer at 73.1%, but with significantly fewer parameters at 12.4 million compared to 86 million, and an inference time of 24.1 ms instead of 68.3 ms. For cross-device scenarios, the proposed method has lower performance degradation of 5.4% and 6.1% when transferring from Device A to Device B and Device C, respectively, compared to Audio Spectrogram Transformer at 7.0% and 7.8%, and ResNet-18 at 6.5% and 7.2%, PaSST at 6.9% and 7.6%, and BEATs at 6.7% and 7.5%. The lower degradation is attributed to the effectiveness of the proposed method in capturing recording device-agnostic acoustic structural characteristics using the HPSS-based spectrogram separation technique.

A deeper theoretical examination of this robustness advantage reveals that the spectral distortions introduced by different recording devices predominantly manifest as broadband frequency response variations, encompassing microphone sensitivity differences, analog-to-digital converter nonlinearities, and device-specific noise floor characteristics, all of which alter the overall spectral envelope of the recorded signal in a manner that is approximately multiplicative in the frequency domain. The median filtering operation employed in the HPSS process is largely robust to such multiplicative spectral distortions, as it extracts structural patterns based on the relative continuity or locality of spectral energy distribution rather than on absolute energy magnitudes, meaning that the harmonic and percussive components retain their fundamental structural characteristics even when the underlying spectral envelope is systematically shifted by device-specific transfer functions. The harmonic component, which captures sustained tonal patterns such as engine noise and ventilation hum, reflects intrinsic inter-frequency relationships that are governed by the physical properties of the sound sources rather than by the recording hardware, while the percussive component, which captures transient impulsive events such as footsteps and braking sounds, encodes temporal onset and offset dynamics that are determined by the acoustic event characteristics rather than by the device frequency response. This decomposition therefore functions as an implicit domain normalization mechanism that projects the device-dependent raw spectrogram into a pair of structurally regularized representations where the inter-device distributional discrepancy is reduced, enabling the dual-stream network to learn classification-relevant features from a feature space with lower domain shift. The independent parameter optimization within the dual-stream architecture further amplifies this effect, as each stream develops specialized convolutional filters attuned to the higher-order structural invariances along the frequency and temporal axes respectively, rather than to the device-sensitive absolute spectral characteristics that degrade the generalization capacity of methods operating on the undecomposed spectrogram.

The combined spectrogram representations with deep convolutional neural networks have been proven to be effective in feature extraction for audio spoof detection tasks^55,56, while the optimized features have shown higher discriminability using multi-level iterative engineering in deepfake audio detection tasks^57,58. The asymmetric convolution operation helps to improve the efficiency of the model through the differentiation characteristics of spectrograms along the temporal-frequency dimensions. The two-stage architecture has shown its effectiveness in achieving modularity flexibility for soundscape classification tasks⁵⁹, while the proposed method has shown its effectiveness in achieving end-to-end optimization for joint learning of both separation and classification.

The analysis of the confusion matrix shows that the discriminability of the proposed method differs for different scene categories. It is worth noting that the proposed method achieves high accuracy, 79.0%, 76.0%, and 77.0%, for Airport, Park, and Tram, respectively, while bidirectional confusion of 12.0% occurs between Metro and Metro Station, and confusion from 9.0 to 11.0% between Street pedestrian and Street traffic. The confusion occurs due to the nature of the acoustic components of similar scenes. Moreover, the proposed method demonstrates the dependence of the quality of Harmonic/Percussive/Source Separation (HPSS) and the selection of the window size of the median filter, which affects the quality of the separation depending on the acoustic environment. The ability of the proposed method to process extremely short audio segments requires further research.

Conclusion

This study aims to address the efficiency of computations and the challenges of cross-device generalizations in acoustic scene classification under resource-constrained environments through the proposal of a novel framework called “structure-aware dual-stream feature disentanglement,” which incorporates harmonic percussive source separation (HPSS) and asymmetric convolutions. The proposed framework performs structured signal decomposition using HPSS separation methods, models each stream separately to focus on the temporal-frequency differentiated characteristics of the acoustic signal using cascaded asymmetric convolutional neural networks with 3 × 5 and 5 × 3 filters, and finally performs adaptive fusion of features with the aid of the dual attention mechanism. The effectiveness of the proposed framework was evaluated through rigorous experiments on the DCASE 2020 dataset, achieving an accuracy of 72.1%, Log Loss of 0.93, which represents an improvement of 6.8 percentage points over the DCASE 2020 Baseline. The proposed framework has 12.4 M parameters and an inference time of 24.1 ms, which is much less compared to other methods like the Transformer architecture. In terms of cross-device environments, the proposed framework showed performance degradations of 5.4 and 6.1 percentage points when the model was transferred from Device A to Devices B and C, respectively, which are better compared to the Audio Spectrogram Transformer and ResNet-18 models.

The study presented in this manuscript offers a solution that strikes a balance between performance and efficiency, specifically for acoustic scene classification on edge devices and mobile devices. The proposed solution also offers a structure-aware feature disentanglement concept, which can provide a new avenue for enhancing the interpretability of deep learning-based models by using prior knowledge from the physical world, specifically in the audio signal processing domain. The generalizability of the proposed framework has been validated on both the DCASE 2020 and TAU Urban Acoustic Scenes 2022 datasets, and extending this evaluation to acoustic scene classification benchmarks with different category taxonomies remains a valuable direction for future investigation. The adaptive tuning of HPSS parameters, particularly the median filter size that governs the trade-off between harmonic and percussive separation quality, represents a promising avenue for improving the effectiveness of structural decomposition across diverse acoustic environments. The integration of model compression techniques such as knowledge distillation from the dual-stream teacher network to a single-stream student network and post-training quantization for integer-arithmetic deployment can further enhance real-time inference performance on highly resource-constrained edge platforms.

Author contributions

W.L. and Y.F. conceived and designed the study, developed the methodology, performed the formal analysis, and wrote the original draft of the manuscript. W.L. conducted the investigation and prepared the figures. Y.F. reviewed and edited the manuscript. All authors read and approved the final version.

Funding

This research received no external funding.

Data availability

The original contributions presented in the study are included in the article Material, further inquiries can be directed to the corresponding authors.

Declarations

Competing interests

The authors declare that they have no competing interest.

Informed consent

Not applicable.

Institutional review board

Not applicable.