Abstract
Speech emotion recognition (SER) infers affective states from speech signals, but positional encoding for acoustic tokens remains underexplored in Transformer-based SER. Existing models often reuse encodings designed for text and do not explicitly account for the different sequential and two-dimensional structures of acoustic representations. We propose Fibonacci Position Embedding (FPE) and Fibonacci Target Shutter (FTS). FTS constructs overlapping candidate-index sets over time–frequency token grids, and FPE samples a Fibonacci index and applies its modulo-wrapped, dimension-dependent phase rotation to query and key vectors. The modules are integrated into STRE-Former, which fuses Wav2Vec, log-mel spectrogram, and MFCC representations through asymmetric cross-representation attention with representation-specific positional encodings. We also introduce an implementation-consistent conditional-entropy formulation that quantifies uncertainty in recovering a token location from its sampled positional representation; this quantity characterizes positional ambiguity rather than downstream modeling capacity. Experiments over 64 positional-encoding combinations on IEMOCAP and MELD identify dataset-dependent highest-observed configurations, reaching 74.21% weighted accuracy on IEMOCAP-4, 74.54% on IEMOCAP-6, and 49.44% on MELD. These empirical observations suggest that the relative behavior of positional-encoding strategies may depend on the acoustic representation and evaluation dataset, rather than supporting a single universally optimal scheme.
Keywords: speech emotion recognition, acoustic sensing, acoustic signal processing, multi-representation fusion, Fibonacci encoding, positional embedding, Transformer, affective computing
1. Introduction
Speech emotion recognition (SER) aims to automatically infer affective states from speech and is increasingly relevant to affective computing [1], human–computer interaction, and mental health monitoring. As acoustic sensing technology advances, robust SER systems can enable empathetic virtual assistants, assist clinical assessment of mental health conditions, support multimodal interaction systems in education and entertainment, and enhance customer service through emotion-aware dialogue systems.
Early SER methods relied on hand-crafted descriptors such as prosodic statistics and Mel-frequency cepstral coefficients (MFCCs) [2,3]. Deep learning shifted the field toward RNN- and CNN-based models that learn representations directly from spectrograms and frame-level features [4,5]. More recently, self-supervised pre-trained encoders such as Wav2Vec [6] and Transformer architectures [7] have achieved strong performance, while multimodal fusion has further enhanced robustness [8,9]. These advances have substantially improved performance on controlled benchmarks such as IEMOCAP [10] and MELD [11], but significant challenges remain in practical deployment, particularly in handling diverse acoustic conditions, speaker variability, and cross-cultural emotional expressions [12,13].
Despite these advances, the design of positional encodings in SER remains underexplored. Most works adopt absolute positional encoding (APE) or rotary encoding (RoPE) from natural language processing without examining whether one positional treatment is appropriate for all acoustic representations. Spectrograms and MFCCs form grids, whereas Wav2Vec embeddings form contextualized temporal sequences. Flattening these structures and applying a single text-derived encoding can discard representation-specific neighborhood information. The resulting gap is not the absence of positional encoding, but the limited systematic investigation of how representation-specific positional strategies behave across heterogeneous acoustic tokenizations.
To study this gap, we propose Fibonacci Position Embedding (FPE) and the Fibonacci Target Shutter (FTS) for spectro-temporal relation modeling in SER. FTS uses recursive golden-ratio boundaries to construct overlapping candidate-index sets on acoustic feature grids; FPE samples from these sets and converts the selected Fibonacci value into a bounded phase rotation. STRE-Former evaluates this design jointly with conventional encodings across three acoustic representation branches. We analyze the ambiguity introduced by candidate sampling and modulo-wrapped phase mapping without treating it as proof of downstream superiority.
1.1. Motivation for Fibonacci-Based Encoding
The Fibonacci design is motivated by two implementation properties rather than by an assumed intrinsic correspondence between the golden ratio and emotion. First, the convergence of adjacent Fibonacci ratios to provides a simple recursive rule for generating nested boundaries on a two-dimensional feature grid. These overlapping regions yield position-dependent candidate-index sets without requiring a learned ordering module.
Second, modular arithmetic maps the sampled Fibonacci values to bounded phases that can be applied as dimension-dependent rotations. The resulting position code is stochastic because FTS samples from each token’s candidate set. Section 3.3 models the resulting chain from token location to sampled index and modulo-wrapped phase. The associated conditional entropy measures positional recovery uncertainty; it does not measure relational richness or imply superiority over deterministic encodings.
For two-dimensional log-mel and MFCC branches, candidate sets are determined by the overlap between token patches and recursively generated regions. Nearby patches may therefore share candidate indices, while each patch can also receive more than one admissible index. This is the structural motivation for FTS; whether it improves recognition is evaluated empirically rather than assumed from the geometry alone.
1.2. Contributions
The main contributions of this work are:
We propose Fibonacci Position Embedding (FPE) and Fibonacci Target Shutter (FTS), which combine candidate-based Fibonacci index assignment with dimension-dependent phase rotations for spectro-temporal acoustic representations.
We introduce the STRE-Former architecture that integrates Wav2Vec, log-mel spectrogram, and MFCC representations via asymmetric co-attention with representation-specific positional encodings.
We provide an implementation-consistent conditional-entropy formulation that quantifies positional recovery uncertainty induced by FTS candidate sampling and the modulo-wrapped FPE phase mapping.
We conduct systematic ablation studies across 64 positional-encoding combinations on IEMOCAP and MELD, documenting representation- and dataset-dependent highest-observed configurations rather than claiming universal superiority.
2. Related Work
2.1. Temporal Modeling in Speech Emotion Recognition
Early SER approaches relied on hand-crafted temporal features such as pitch contours, energy trajectories, and MFCCs [2,14]. The advent of deep learning enabled automatic learning of hierarchical representations: RNNs and LSTMs [15] captured sequential dependencies [16], bidirectional LSTMs incorporated future context [17], and CNNs learned local temporal patterns via 1D and 2D convolutions over spectrograms [4,5,18].
The Transformer architecture [7] addressed the limitations of sequential models through self-attention that directly models relationships between all time steps. SpeechFormer [19] introduced hierarchical token semantic learning for SER, while DWFormer [20] proposed dynamic window-based attention with adaptive receptive fields. Self-supervised models such as Wav2Vec 2.0 [6] and HuBERT [21] further advanced the field by learning rich contextual representations from large-scale unlabeled speech, achieving strong fine-tuning performance on SER tasks [22]. The SUPERB benchmark [23] systematically evaluated self-supervised representations across diverse speech tasks including emotion recognition. More recently, large speech models such as Whisper [24] have demonstrated strong generalization across acoustic understanding tasks. Improved attention mechanisms with end-to-end multitask training have also contributed to SER performance gains [25,26].
Recent work has also pursued robust temporal aggregation through capsule, routing, and attention architectures. CENN uses a capsule-enhanced network for robust SER [27], and the sparse temporal-aware capsule network further constrains capsule-based temporal modeling to emphasize informative speech segments [28]. Content-dependent alternatives include pre-attentive processing with adaptive routing [29], while local-to-global feature aggregation [30] and residual attention [31] target complementary acoustic scales or emotion-salient features. These methods modify feature extraction or aggregation topology; they do not directly test how acoustic token coordinates should be encoded. FPE/FTS does not reorder the token tensor; instead, it changes the candidate positional index and phase rotation assigned to each token. It is therefore complementary to capsule routing and residual attention rather than a replacement for them.
These approaches improve temporal modeling through recurrence, hierarchical windows, pre-training, or learned attention, but they generally evaluate an entire architecture rather than isolate how positional encoding interacts with different acoustic representations. Consequently, their reported gains do not establish whether the same positional scheme is appropriate for contextualized waveform sequences and two-dimensional time–frequency features.
2.2. Multi-Representation and Conversational Emotion Recognition
Emotion recognition in conversational contexts has attracted significant attention, as emotional expression is inherently contextual and speaker-dependent [9]. Early work modeled inter-speaker and intra-speaker context through memory networks [32]. DialogueRNN [33] used recurrent networks to track speaker states and conversational dynamics, while DialogueGCN [34] modeled inter-speaker relationships through graph convolutions. Multimodal fusion of acoustic, visual, and lexical cues is broadly effective for emotion recognition [8], and cross-modal attention inspired by vision-language pretraining [35] has been adapted for SER.
Within audio-only SER, the Intelligent Fusion Network learns to combine multiple acoustic features with an emphasis on reproducibility and generalization [36]. Such adaptive fusion determines how feature streams contribute to a prediction, whereas the present work asks a different question: how positional information should be assigned within heterogeneous acoustic branches before their interaction. STRE-Former consequently uses representation-specific positional encodings followed by asymmetric co-attention; the fusion mechanism and the positional mechanism remain conceptually distinct.
Data augmentation such as SpecAugment [37] and contrastive learning [38] have further improved robustness against acoustic variability. Knowledge distillation [39] has been explored to transfer representations from large pretrained models to compact SER architectures [40]. Masked prediction pretraining for speech, inspired by BERT [41], has led to HuBERT [21] and related self-supervised frameworks.
These studies address conversational context, augmentation, or cross-modal fusion, whereas the present work uses only audio-derived representations and focuses on positional treatment within and across acoustic branches. We therefore use “multi-representation” for our model and reserve “multimodal” for systems that combine distinct sources such as audio, text, and video.
2.3. The Role of Positional Encoding in Speech Emotion Recognition
Positional encoding is crucial for Transformer-based SER because emotional expression unfolds dynamically over time, and the same acoustic pattern may carry different meanings depending on its temporal context. Without adequate positional information, self-attention cannot distinguish sequential order or distance relationships, impeding the separation of transient acoustic fluctuations from sustained emotional trends.
Speech signals present challenges beyond those of text. Spectrograms and MFCCs are inherently 2D time-frequency matrices representing acoustic sensor measurements: when flattened to 1D token sequences via raster-scan ordering, spatially adjacent patches can become distant in the token sequence, weakening explicit local acoustic structure. In a spectrogram of dimensions , a one-step temporal or frequency displacement maps to a different one-dimensional offset, so a single sequence index cannot explicitly distinguish the two axes. Patch-based vision Transformers [42,43] face an analogous 2D locality problem. In audio, the Audio Spectrogram Transformer (AST) represents a spectrogram as a sequence of image-style patches and adds positional information to those tokens [44]; subsequent work has directly compared positional-encoding approaches for audio spectrogram Transformers [45]. These studies establish the relevance of positional design for time–frequency patches, but they do not examine multi-representation SER or stochastic, position-dependent candidate-index assignment.
Existing strategies include Absolute Positional Encoding (APE) [7], Rotary Position Embedding (RoPE) [46], Transformer-XL’s relative encoding [47], relative position representations [48], and Attention with Linear Biases (ALiBi) [49]. APE supplies explicit absolute locations but does not directly encode relative displacement. RoPE rotates query and key pairs so that their interaction depends on relative position, while ALiBi introduces a head-dependent linear distance bias. Learned and adaptive encodings [50,51] provide additional flexibility but introduce data-dependent parameters. These methods were developed mainly for one-dimensional sequences; applying them after flattening a time–frequency grid does not by itself preserve two-dimensional neighborhoods. Our study therefore compares representation-specific combinations rather than assuming that any conventional or Fibonacci-based scheme is universally preferable.
3. Fibonacci Position Embedding
Fibonacci Position Embedding (FPE) converts FTS-assigned Fibonacci indices into dimension-dependent complex rotations for acoustic query and key vectors. Its stochastic candidate assignment differs from deterministic APE and RoPE mappings; the consequences of this ambiguity are characterized in Section 3.3 and evaluated separately from recognition performance.
3.1. Token Dispersal and Fibonacci Target Shutter
Given an input feature matrix, we divide it into patches with dimensions . Each patch is mapped to a d-dimensional token, yielding where .
3.1.1. FTS Overview and Intuition
FTS does not reorder the token sequence. It constructs a position-dependent set of admissible Fibonacci indices for each patch on a time–frequency grid and samples one index from that set. Self-attention still receives the tokens in their original flattened order; FTS changes only the phase index applied to each token’s query and key vectors. Overlap among the candidate sets allows nearby or intersecting recursively generated regions to share possible indices, while sampling prevents the assignment from being a deterministic one-to-one position code.
3.1.2. Mathematical Formulation
FTS recursively partitions the patch grid using the inverse golden ratio
| (1) |
Starting from the full grid bounds, the implementation cycles through left, upper, right, and lower rectangular strips. At each step, it updates the active horizontal or vertical boundary using and records the exposed region as . The recursion continues until the configured candidate-depth stopping condition is reached; all reported experiments use target_number.
For each patch with spatial support , its candidate set contains the indices of all exposed regions intersecting that patch:
| (2) |
During each positional-encoding construction, the implementation samples uniformly from using random.choice. The sampled index determines the Fibonacci phase assigned to patch i. Equations (9) and (10) give the corresponding probability model used in the uncertainty analysis.
3.2. Fibonacci Position Encoding
The position encoding uses a recursively generated, modulo-wrapped Fibonacci sequence:
| (3) |
where keeps the phase argument bounded. Let be the wrapped Fibonacci value selected for token position p. For an even attention-head dimension , the implementation forms complex dimension pairs with frequencies
| (4) |
and the rotation vector
| (5) |
After pairing adjacent real-valued query and key dimensions as complex values, FPE applies the same position-dependent rotation to both:
| (6) |
Attention logits are then computed from the real part of the complex query–key product before Softmax. This construction is the operational counterpart of Equation (17).
For the Fib+RoPE variant, the implementation combines the Fibonacci and two-dimensional RoPE rotation matrices before applying them to query and key pairs:
| (7) |
where the reported experiments use . When enabled, vertical and horizontal flips are independently applied to with probability 0.5 before the hybrid combination; both augmentation flags are enabled in the reported configuration.
Three-Layer Interpretation
Recurrence: nonlinear index generation. The Fibonacci recurrence supplies a nonlinearly varying sequence of index values. We use this sequence as a design choice for generating phase arguments; we do not assume that golden-ratio growth is intrinsically matched to emotional speech.
Modulo: bounded phase encoding. The operation maps Fibonacci values to bounded angles suitable for complex rotations. It can introduce exact or finite-precision phase collisions, but no length-extrapolation guarantee is inferred from this operation.
Stochastic ambiguity: candidate sampling and phase collisions. Ambiguity arises first because multiple Fibonacci indices may be admissible for one token and second because modulo wrapping can map different indices to similar phase vectors. Section 3.3 quantifies the resulting position-recovery uncertainty without assuming that greater ambiguity necessarily improves recognition.
3.3. Implementation-Consistent Positional Uncertainty Analysis
The implemented FTS/FPE pipeline does not construct idealized modules containing tokens. Instead, FTS produces a position-dependent candidate set, samples a Fibonacci index, wraps the corresponding Fibonacci value modulo , and applies dimension-dependent complex rotations to the query and key vectors. We therefore analyze this realized stochastic mapping directly. All logarithms in this subsection are natural logarithms.
3.3.1. FTS Candidate-Set Construction
Let an input contain N tokens and let the original token position be the random variable . We use a uniform prior over the evaluated positions:
| (8) |
For position p, FTS recursively partitions the two-dimensional token grid using golden-ratio boundaries. Let denote the spatial support of token p and the region associated with Fibonacci index k. The candidate index set produced by the implementation is
| (9) |
The implementation samples uniformly from this set. Thus, for the sampled index K,
| (10) |
This definition follows the candidate-list construction and uniform random.choice operation used by FTS; it does not impose a Fibonacci-valued module size.
3.3.2. Index-Level Positional Uncertainty
For a fixed token position, the uncertainty of the sampled index is
| (11) |
where the final equality follows from uniform sampling. Averaging over positions gives
| (12) |
Candidate-set entropy alone does not quantify whether a position can be recovered, because disjoint candidate sets may remain position-identifying. The marginal probability of index k is
| (13) |
Using and , the uncertainty remaining about the token position after observing the sampled index is
| (14) |
We normalize this quantity as
| (15) |
A value of zero means that the sampled index uniquely identifies the position over the evaluated grid, whereas a value of one means that the sampled index is independent of the original position.
3.3.3. Modulo-Wrapped Phase Uncertainty
FPE uses the modulo-wrapped Fibonacci value
| (16) |
For an even embedding dimension d, the r-th rotation frequency and the complex phase representation of index k are
| (17) |
The positional rotation observed for token p is therefore the random variable with . To account for finite precision and near-collisions after phase wrapping, we group indices into phase classes using a fixed deterministic clustering rule and the cosine-distance criterion
| (18) |
Let denote the resulting phase class. Its position-conditional and marginal probabilities are
| (19) |
The corresponding entropies and normalized phase-level positional uncertainty are
| (20) |
| (21) |
| (22) |
| (23) |
Proposition 1.
Because is a Markov chain and the phase representation and its numerical class are deterministic transformations of the sampled index, the data processing inequality gives
(24)
Equality between the index- and phase-level quantities holds when the index-to-phase mapping is injective over the evaluated index range. Thus, modulo wrapping and phase collisions cannot reveal more positional information than the complete sampled index; they can only preserve or increase positional recovery uncertainty.
3.3.4. Expected Positional-Relation Matrix
Conditional entropy measures recoverability, not relational geometry. To describe the geometry induced by the stochastic rotations, we separately define the expected positional-relation matrix
| (25) |
This expectation follows the independent index sampling performed for positions p and q. For deterministic RoPE, the sampling distribution reduces to a point mass and the corresponding expression uses its standard position-dependent rotation angles. Comparisons among APE, RoPE, and FPE must use the same sequence length, embedding dimension, collision tolerance, and numerical precision. Importantly, characterizes only the geometry induced by positional rotations. It is not the complete attention score, which also depends on the content-dependent query and key vectors.
3.3.5. Scope and Interpretation
The quantities and measure the uncertainty of recovering a token location from the realized FTS/FPE positional representation. APE and standard RoPE are deterministic position mappings; when they are injective over the evaluated range, their positional recovery uncertainty is zero. FTS/FPE can yield non-zero uncertainty because of candidate sampling and phase collisions. Higher recovery uncertainty must not be interpreted as richer relational information or better SER performance. FPE and FTS are examined empirically through the available component-oriented classification comparisons, while Equation (25) provides a descriptive positional-geometry diagnostic rather than a causal account of attention behavior.
4. Proposed STRE-Former Model
4.1. Feature Extraction
To comprehensively capture emotional information from acoustic speech signals, we extract three complementary representations that encode different aspects of the waveform and time-frequency sensor measurements.
4.1.1. Wav2Vec 2.0 Features
We employ the pre-trained Wav2Vec2-base-960h model [6] to extract contextualized waveform representations. The model consists of a CNN feature encoder with 7 convolutional layers that downsample the raw waveform (16 kHz sampling rate, 3 s = 48,000 samples) to a sequence of 149 latent representations, followed by a 12-layer Transformer encoder (768-dimensional hidden states, 12 attention heads) that captures long-range dependencies.
For each utterance, we extract the final hidden states from the last transformer layer, yielding . We use the pre-trained weights without fine-tuning to leverage the rich acoustic representations learned from 960 h of LibriSpeech data. The Wav2Vec2Processor is used to normalize the input waveform before feeding it to the model.
4.1.2. Spectrogram Features
We compute log-mel spectrograms from the Short-Time Fourier Transform (STFT). For frame index and frequency-bin index k, the magnitude STFT is
| (26) |
where n is the sample offset within a frame, H is the hop length, is the Hamming window, and N is the FFT size. Thus, , k, and n respectively denote the frame, frequency bin, and within-frame sample indices.
The extraction parameters are: sampling rate 16,000 Hz; Hamming window; window length 40 ms (640 samples); hop length 10 ms (160 samples); and FFT size 800 points, producing 401 non-redundant linear-frequency STFT bins over 0–8000 Hz. We then apply 200 mel filters and convert the mel power representation to a logarithmic scale. Each input is padded or truncated to 300 frames.
To create a 3-channel representation for the CNN branch, we compute delta and delta-delta features from the log-mel representation:
| (27) |
| (28) |
The resulting spectrogram contains the original log-mel spectrogram, delta, and delta-delta features as three channels.
4.1.3. MFCC Features
Mel-Frequency Cepstral Coefficients (MFCCs) capture the timbral texture of speech by mimicking human auditory perception. We compute MFCCs using the Discrete Cosine Transform (DCT) applied to log mel-filterbank energies:
| (29) |
where is the energy in the m-th mel filter bank, and M is the number of filters.
The extraction parameters are: number of MFCC coefficients 40; hop length 160 samples (10 ms); HTK formula enabled for compatibility; time frames 300 (matching spectrogram).
The extracted MFCCs form a matrix , which is normalized using L2 normalization along the feature dimension to ensure stable training.
4.1.4. Complementary Nature of Features
The three acoustic representations provide complementary views of the same waveform: Wav2Vec supplies contextualized acoustic representations learned through self-supervised pre-training; the log-mel spectrogram retains localized time–frequency energy, harmonic structure, and temporal changes; and MFCC compactly represents the spectral envelope and vocal-tract-related characteristics.
This multi-view representation enables the model to leverage both low-level acoustic details and high-level semantic information for robust emotion recognition.
4.2. Tokenization and STRE-Former Architecture
Each acoustic feature undergoes tokenization into patch-based representations , and . STRE-Former then processes each representation with a branch-specific positional encoding and self-attention module. For transparency, the evaluated configurations are not identical in attention-head count: APE, RoPE, and ALiBi branches use four heads, whereas Fib and Fib+RoPE branches use one. Consequently, comparisons involving these branches are comparisons between complete configurations rather than strictly isolated positional-encoding effects. For each branch, the attention output is , and feed-forward layers with ReLU produce , , and . The overall architecture is illustrated in Figure 1.
Figure 1.

Architecture of the proposed STRE-Former. Three representations derived from the same audio signal (Wav2Vec 2.0, log-mel spectrogram, and MFCC) are tokenized and processed by branch-specific transformer encoders. FTS assigns candidate-based Fibonacci phase indices to two-dimensional log-mel/MFCC patches without changing their flattened token order. An asymmetric cross-representation co-attention module fuses the three branches, and an MLP head predicts the emotion category.
4.3. Cross-Modal Co-Attention and Feature Fusion
To integrate complementary information across representation branches, we design an asymmetric co-attention mechanism. We adopt this structure as a practical fusion choice motivated by the different roles of frequency-domain and contextualized waveform representations. Log-mel and MFCC features provide spectral information, while Wav2Vec embeddings provide contextualized representations from pre-training. Frequency-domain features are used as queries over the Wav2Vec sequence; this is an architectural design choice rather than evidence that one representation is intrinsically more informative.
Log-mel spectrograms retain localized frequency-domain dynamics, while MFCCs compress the spectral envelope into perceptually motivated coefficients. Their concatenation forms a joint query for the Wav2Vec branch. We use frequency-to-waveform rather than bidirectional attention to limit computation; the contribution of this asymmetry is not isolated in the present experiments.
4.3.1. Co-Attention Mechanism
We concatenate the spectrogram and MFCC representations to form a joint query:
| (30) |
This query is projected and used to compute attention weights over the Wav2Vec sequence:
| (31) |
| (32) |
where depends on the positional encoding used in the Wav2Vec branch. To match dimensions, we apply a branch-specific adjustment layer:
| (33) |
The attention-weighted Wav2Vec representation is computed via matrix multiplication:
| (34) |
| (35) |
| (36) |
4.3.2. Feature Fusion
The three branch-specific representations are concatenated:
| (37) |
4.4. Emotion Classification and Training
4.4.1. Classification Head
The fused feature vector is fed through a multi-layer perceptron (MLP), which returns un-normalized class logits:
| (38) |
| (39) |
| (40) |
where C is the number of emotion classes (4 for IEMOCAP-4, 6 for IEMOCAP-6, and 7 for MELD). Dropout with probability 0.1 is applied in the MLP. Log-probabilities are computed inside the training criterion; at inference, the predicted class is . Thus, Softmax is not an additional trainable output layer.
4.4.2. Loss Function
The reported implementation uses a log-probability-based classification criterion on the logits. For one-hot label and log-probability , the implemented objective is
| (41) |
with and . This equation directly documents the criterion used in the reported experiments rather than assigning it a standard loss-function label. Its factor differs from the standard focal-loss factor , and the resulting terms are averaged over all classes. No class weights, learning-rate scheduler, or separate penalty are applied.
5. Experimental Setup
5.1. Datasets
We evaluate on IEMOCAP and MELD because they provide complementary acoustic and conversational conditions. IEMOCAP contains controlled dyadic interactions with comparatively clear utterance boundaries, whereas MELD contains natural multi-party conversations, more speakers and transitions, and a markedly imbalanced label distribution. Together they test the model in both controlled two-speaker and more complex conversational settings.
5.1.1. IEMOCAP Dataset
The Interactive Emotional Dyadic Motion Capture (IEMOCAP) database [10] contains approximately 12 h of audiovisual recordings from 10 professional actors (5 male and 5 female) in five dyadic sessions with scripted and spontaneous interactions. In the standard four-class setting, excitement is merged with happiness, giving 5531 utterances across anger, happiness/excitement, sadness, and neutrality. The six-class setting treats happiness and excitement separately and additionally includes frustration. Each utterance is annotated by multiple evaluators, and the majority label is used as ground truth. Audio is sampled at 16 kHz, and utterances are padded, truncated, or segmented into fixed-length inputs during preprocessing.
5.1.2. MELD Dataset
The Multimodal EmotionLines Dataset (MELD) [11] is derived from the television series “Friends” and provides natural multi-party conversations with seven emotion labels: anger, disgust, fear, joy, neutral, sadness, and surprise. Although MELD contains audio, text, and video, this study uses only its audio track. The official train/development/test splits contain 9989, 1109, and 2610 utterances, respectively. The test split is strongly imbalanced, ranging from 1256 neutral utterances to 50 fear utterances. This distribution, together with conversational speaker changes and natural recording variability, makes MELD complementary to the more controlled IEMOCAP setting.
5.2. Evaluation Metrics
We report Weighted Accuracy (WA), i.e., overall sample accuracy; Unweighted Accuracy (UA), the unweighted mean of class-wise recalls; Weighted F1 (WF1), the support-weighted mean of class-wise F1 scores; and Unweighted F1 (UWF1), the macro-averaged F1 score. Segment-level predictions are averaged within each utterance before utterance-level evaluation. All reported values are rounded to two decimal places for presentation consistency.
5.3. Implementation Details
All experiments are conducted on an NVIDIA GeForce RTX 4090 (NVIDIA Corporation, Santa Clara, CA, USA) using PyTorch 2.1.2 (Linux Foundation, San Francisco, CA, USA) and librosa 0.10.2. The model is trained for up to 500 epochs with AdamW, batch size 16, dropout 0.1, and early-stopping patience 100. The optimizer is instantiated without an explicit weight_decay argument and therefore uses the PyTorch 2.1.2 AdamW default of ; no second L2 penalty is applied in the loss. No learning-rate scheduler or class weighting is used. The configuration used for the reported experiments specifies one training repeat with RANDOM_SEED = 66. Because only one seed is reported, small score differences should not be interpreted as statistically stable.
The experiments are conducted in two stages. First, all 64 configurations are evaluated with a learning rate of to describe broad configuration-level trends and identify candidates for refinement. Second, the highest-observed candidate and the APE and ALiBi baselines are evaluated at and . RoPE is included in the common-budget first-stage screening but does not receive the same second-stage learning-rate refinement budget. Accordingly, first-stage results are used only for broad configuration-level observations, and comparisons between an unrefined RoPE score and a second-stage-refined maximum are not interpreted as strictly optimization-budget-matched rankings. The original 64-configuration analysis was designed as exploratory configuration screening rather than as a fully nested model-selection experiment. We therefore do not interpret the highest-observed configuration as an unbiased estimate of a universally optimal architecture. The revised analysis explicitly identifies selection variability and treats small differences among shortlisted configurations as descriptive observations.
Evaluation Protocol
We follow the widely adopted leave-one-session-out (LOSO) protocol on IEMOCAP. Specifically, each fold holds out one session for testing and uses the remaining sessions for training and validation, and the reported results are averaged across all folds. The reported configuration search is interpreted as exploratory screening rather than an unbiased nested model-selection evaluation.
6. Results and Analysis
6.1. Performance Comparison
Table 1 reports prior results alongside STRE-Former. The highest-observed refined STRE-Former triplets are (Fib, Fib+RoPE, Fib+RoPE) on IEMOCAP-4, (Fib, Fib+RoPE, RoPE) on IEMOCAP-6, and (APE, Fib+RoPE, Fib+RoPE) on MELD. Their respective WA values are 74.21%, 74.54%, and 49.44%. Under the reported protocols, the IEMOCAP scores are higher than the listed audio-only baselines, while the MELD score remains below SpeechFormer++ in WA. SpeechCueLLM [52] and AA-SLLM [53] are included only as contextual references because they use text or speech-language-model-scale pre-training beyond our audio-only setting. Cross-study rankings remain tentative because preprocessing, feature extraction, capacity, and evaluation protocols are not controlled.
Table 1.
Performance comparison of prior SER models and STRE-Former on IEMOCAP and MELD. Bold values indicate the best results for each metric. “Mod.” denotes source modality: A = audio only, T+A = text and audio, and LLM = large language model. For STRE-Former, the highest-observed refined configuration is reported for each dataset. Direct comparison is limited by differences in feature pipelines, data augmentation, model capacity, and evaluation protocols across studies.
| Model | Mod. | IEMOCAP(4) | IEMOCAP(6) | MELD | |||||
|---|---|---|---|---|---|---|---|---|---|
| WA | UA | WF1 | WA | UA | WF1 | WA | WF1 | ||
| CNN–LSTM [4] | A | 62.9 | 57.1 | 59.5 | 51.3 | 46.5 | 47.3 | – | – |
| LSTM-Attn [16] | A | 56.5 | 56.7 | 56.5 | 43.4 | 43.6 | 43.4 | – | – |
| MAEC [54] | A | 68.17 | 69.28 | – | – | – | – | 42.07 | – |
| SpeechFormer [19] | A | 62.9 | 64.5 | – | – | – | – | 41.9 | – |
| DWFormer [20] | A | 72.3 | 73.9 | – | – | – | – | – | 48.5 |
| SpeechFormer++ [55] | A | 70.5 | 71.5 | 70.7 | – | – | – | 51.0 | 47.0 |
| CubicKD [40] | A | – | – | – | 63.32 | 64.29 | – | – | – |
| TENT [56] | A | 72.43 | 73.88 | – | – | – | – | – | 45.1 |
| ENT [56] | A | 71.84 | 72.37 | – | – | – | – | – | 46.3 |
| AA-SLLM [53] | LLM | – | 85.50 | – | – | – | – | – | 55.23 |
| SpeechCueLLM [52] | T+A | – | – | – | – | – | 72.6 | – | 67.6 |
| STRE-Former (ours) | A | 74.21 | 68.31 | 73.96 | 74.54 | 70.48 | 74.29 | 49.44 | 45.87 |
From a computational perspective, we compare the complete highest-observed configuration (Fib, Fib+RoPE, Fib+RoPE) against complete APE and ALiBi baselines. All measurements are conducted on an NVIDIA GeForce RTX 4090 with batch size 16, using 50 warm-up iterations followed by 200 measurement runs. The configurations differ in attention-head count, so Table 2 should not be interpreted as an isolated measurement of positional-encoding overhead.
Table 2.
Configuration-level computational efficiency comparison. Bold values indicate the optimal result for each metric. The Fibonacci and conventional baselines differ in attention-head count. All measurements use an NVIDIA GeForce RTX 4090, batch size 16, 50 warm-up iterations, and 200 measurement runs.
| Configuration | Heads | Params (M) | FLOPs (G) | Latency (ms) | GPU Mem (MB) |
|---|---|---|---|---|---|
| Ours (F-FR-FR) | 1 | 161.79 | 22.94 | 13.87 | 1424.7 |
| APE | 4 | 162.90 | 26.78 | 9.89 | 1425.8 |
| ALiBi | 4 | 162.90 | 26.78 | 8.71 | 1424.8 |
As shown in Table 2, the evaluated Fibonacci configuration has fewer parameters (161.79 M vs. 162.90 M) and lower FLOPs (22.94 G vs. 26.78 G), but higher latency (13.87 ms vs. 9.89/8.71 ms); GPU memory is comparable (≈1425 MB). These differences reflect the complete implementations, including the one-head versus four-head design and candidate-based phase construction. The measurements do not isolate the computational effect of FTS or FPE alone. Figure 2 visualizes the observed configuration-level trade-offs.
Figure 2.

Configuration-level complexity comparison. The orange/red bars denote the proposed Fibonacci configuration (Ours, F-FR-FR), and the gray bars denote the conventional APE and ALiBi baselines. Red stars indicate the optimal or near-optimal value for each metric. The Fibonacci configuration has a lower measured parameter count and FLOPs but higher latency than the four-head APE and ALiBi configurations; GPU memory is comparable. Head counts and other implementation differences are not controlled in this figure.
6.2. Ablation Study: Positional Encoding Combinations
To examine positional-encoding configurations, we evaluate 64 combinations across the log-mel, MFCC, and Wav2Vec branches. Each triplet (Spectrogram, MFCC, Wav2Vec) uses APE, RoPE, Fib, or Fib+RoPE in each position. All 64 configurations use a learning rate of in this exploratory stage. This controls the learning rate but not all architectural factors, because conventional and Fibonacci branches differ in attention-head count. Figure 3 therefore reports complete-configuration results rather than isolated positional-encoding effects.
Figure 3.

Visualization of weighted accuracy (WA) across 64 positional encoding combinations on (a) IEMOCAP-4, (b) IEMOCAP-6, and (c) MELD. Each subplot represents different spectrogram encodings (APE, RoPE, Fib, Fib+RoPE), with rows indicating Wav2Vec encodings and columns showing MFCC encodings. Brighter colors indicate higher performance in each figure.
The observed scores vary across representation branches and datasets, and no configuration is highest in every setting. Because the configurations also differ in head count and are selected from a large search space, these results establish empirical associations rather than a causal effect of positional encoding alone. They motivate further capacity-matched evaluation of representation-specific positional strategies.
The first stage uses a common learning rate (), batch size 16, up to 500 epochs, and early stopping to identify broad trends. The second stage evaluates the shortlisted configurations and the APE and ALiBi baselines at and ; RoPE does not receive an identical second-stage refinement budget. The heatmaps show common-budget first-stage results before learning-rate refinement. We use these first-stage results only for broad configuration-level observations and do not rank an unrefined RoPE result directly against a second-stage-refined maximum as if their optimization budgets were matched. Since many configurations are screened and optimization exposure is not fully matched across encoding families, small differences among the highest values should not be interpreted as stable or strictly budget-matched rankings.
Several descriptive patterns appear in the first-stage results. Fib-Fib-Fib obtains 72.15% WA on IEMOCAP-4, 72.87% on IEMOCAP-6, and 47.89% on MELD. Fib+RoPE in the MFCC branch appears in each highest-observed refined triplet, whereas the selected log-mel and Wav2Vec encodings vary by dataset. These observations are hypotheses for follow-up capacity-matched and multi-seed experiments, not evidence that a particular encoding has an intrinsic advantage for one acoustic representation.
Figure 3 visualizes WA across the 64 complete configurations. APE-APE-APE obtains 69.73%, 65.53%, and 46.50% WA on IEMOCAP-4, IEMOCAP-6, and MELD, respectively; RoPE-RoPE-RoPE obtains 65.21%, 61.35%, and 47.42%. These values provide reference points within the search but do not isolate the encoding from the accompanying architecture.
Figure 3 and the refined results show that the highest-observed triplet differs among datasets. IEMOCAP-4 selects (Fib, Fib+RoPE, Fib+RoPE), IEMOCAP-6 selects (Fib, Fib+RoPE, RoPE), and MELD selects (APE, Fib+RoPE, Fib+RoPE). Fib+RoPE appears in the MFCC branch of all three selected triplets, while the log-mel and Wav2Vec choices vary. This is a descriptive pattern from the evaluated search and does not establish a universal representation–encoding rule. Table 3 reports the selected configurations and metrics.
Table 3.
Highest-observed refined positional-encoding configurations on each dataset. These values are selected from multiple configurations and should not be interpreted as stable rankings without repeated runs.
| Dataset | Spec | MFCC | Wav2Vec | WA (%) | WF1 (%) | UA (%) | UWF1 (%) |
|---|---|---|---|---|---|---|---|
| IEMOCAP-4 | Fib | Fib+RoPE | Fib+RoPE | 74.21 | 73.96 | 68.31 | 72.07 |
| IEMOCAP-6 | Fib | Fib+RoPE | RoPE | 74.54 | 74.29 | 70.48 | 73.49 |
| MELD | APE | Fib+RoPE | Fib+RoPE | 49.44 | 45.87 | 30.34 | 33.13 |
Table 4 reports the available three-state component comparison. “FPE-only” retains Fibonacci phase rotation but disables FTS candidate sampling by using sequential Fibonacci indices. “FPE+FTS” uses candidate-based sampling. An FTS-only state is not separately identifiable in the implementation because FTS outputs an index that is consumed by FPE; disabling FPE removes the downstream path through which the FTS index acts.
Table 4.
Available configuration-level comparison of Fibonacci components (WA, %). The No-Fibonacci reference uses APE in all three branches; FPE-only disables FTS candidate sampling; FPE+FTS denotes the highest-observed full configuration. The No-Fibonacci comparison is not capacity-matched.
| Dataset | No Fibonacci | FPE-Only | FPE+FTS | ΔFTS |
|---|---|---|---|---|
| IEMOCAP-4 | 69.73 | 70.11 | 74.21 | +4.10 |
| IEMOCAP-6 | 65.53 | 67.30 | 74.54 | +7.24 |
| MELD | 46.50 | 49.07 | 49.44 | +0.37 |
Within the available runs, candidate-based FTS assignment is associated with increases of 4.10 and 7.24 percentage points over FPE-only on the two IEMOCAP settings and 0.37 points on MELD. These are single-run configuration differences; without repeated capacity-matched runs, they document observed configuration-level differences rather than a stable causal effect.
6.3. MELD Class Distribution and Restored-Checkpoint Error Analysis
We complement the aggregate MELD result with the official emotion-label distribution and an error analysis from the retained checkpoint. These two sources use different counting bases and are therefore reported separately: Table 5 describes the 2610 raw utterances in the official test split, whereas Table 6 and Table 7 describe the 3808 evaluation samples processed by the restored inference pipeline.
Table 5.
Official MELD emotion-label distribution by split. Test proportion is calculated over the 2610 official test utterances.
| Emotion | Train | Development | Test | Test Proportion (%) |
|---|---|---|---|---|
| Neutral | 4710 | 470 | 1256 | 48.12 |
| Joy | 1743 | 163 | 402 | 15.40 |
| Surprise | 1205 | 150 | 281 | 10.77 |
| Anger | 1109 | 153 | 345 | 13.22 |
| Sadness | 683 | 111 | 208 | 7.97 |
| Disgust | 271 | 22 | 68 | 2.61 |
| Fear | 268 | 40 | 50 | 1.92 |
| Total | 9989 | 1109 | 2610 | 100.00 |
Table 6.
Overall MELD metrics from the restored-checkpoint evaluation (%).
| WA | Weighted F1 | UA | Macro F1 |
|---|---|---|---|
| 49.74 | 46.27 | 30.69 | 33.56 |
Table 7.
Class-wise MELD results from the 3808 samples processed by the restored-checkpoint evaluation.
| Emotion | Support | Precision (%) | Recall (%) | F1 (%) |
|---|---|---|---|---|
| Neutral | 1745 | 55.20 | 78.45 | 64.80 |
| Joy | 630 | 40.10 | 26.35 | 31.80 |
| Surprise | 364 | 38.64 | 23.35 | 29.11 |
| Anger | 507 | 38.60 | 33.73 | 36.00 |
| Sadness | 313 | 40.00 | 19.81 | 26.50 |
| Disgust | 166 | 54.00 | 16.27 | 25.00 |
| Fear | 83 | 30.43 | 16.87 | 21.71 |
The official test split has a 25.12:1 neutral-to-fear ratio (1256 vs. 50 utterances); across all three splits, the corresponding ratio is 17.98:1 (6436 vs. 358). Figure 4 visualizes this imbalance.
Figure 4.

Official MELD emotion-label distribution for the train, development, and test splits.
The retained checkpoint was evaluated twice under the same restored test-only procedure. Predicted labels, reference labels, and log-probabilities were identical element by element across the two evaluations. The restored pipeline produced WA 49.74%, weighted F1 46.27%, UA 30.69%, and macro F1 33.56% (Table 6). This result is close to, but not identical to, the historical 49.44% WA reported in the main comparison table; all class-wise analyses below use the restored 49.74% evaluation record.
Within the restored evaluation record, neutral constitutes 45.82% of the reference labels but 65.13% of model predictions. Class support has Spearman correlations of 0.93 with recall and 0.96 with F1, and the reported majority–minority macro-recall gap is 27.10 percentage points. These quantities indicate that errors are concentrated in lower-frequency classes and that predictions are strongly biased toward neutral. The row-normalized confusion matrix in Figure 5 is consistent with this pattern: 48.8–61.3% of disgust, anger, joy, surprise, sadness, and fear samples are assigned to neutral.
Figure 5.

Row-normalized confusion matrix from the restored MELD test-only inference. Each row represents a reference class and sums to one.
A 5000-replicate stratified bootstrap over the fixed restored-checkpoint predictions gives a 95% confidence interval of 48.42–51.05% for WA and 44.85–47.72% for weighted F1. These intervals quantify sample-level uncertainty for this fixed checkpoint; they do not represent variation across independently trained models. We use MELD’s original multi-party conversational recordings to analyze performance under its native acoustic variability and do not add artificial-noise experiments in this revision. Consequently, this analysis documents class imbalance and natural-condition errors but does not isolate the causal effect of individual noise factors.
6.4. Implementation-Consistent Interpretation of Positional Geometry
The positional analysis in Section 3.3 separates two questions that were conflated by a standalone positional-vector cosine comparison. First, and quantify how uncertain the original token location remains after the stochastic FTS/FPE mapping. Second, in Equation (25) describes the expected geometry of the implemented positional rotations by averaging over the actual candidate-index distributions. Neither quantity is a complete characterization of self-attention, because attention scores also depend on the learned, content-dependent query and key vectors.
Accordingly, we do not use pairwise cosine similarities between standalone APE, RoPE, and FPE vectors as evidence that one encoding “collapses” or preserves richer information. Any numerical comparison of positional geometry should be computed under matched sequence length, embedding dimension, numerical precision, and phase-collision tolerance. For stochastic FPE, it should additionally average over the FTS distribution rather than use an unspecified single index realization. Classification behavior is examined independently through the available component-oriented comparisons above.
7. Discussion
7.1. Dataset-Dependent Observed Configurations
The selected configurations differ across the three evaluations. IEMOCAP-4 uses Fibonacci encoding in the log-mel branch and Fib+RoPE in the MFCC and Wav2Vec branches; IEMOCAP-6 replaces the Wav2Vec encoding with RoPE; and MELD uses APE for log-mel with Fib+RoPE in the other two branches. These differences may reflect dataset characteristics, optimization variability, model-capacity differences, or selection from many candidates. The present results do not isolate which explanation is responsible.
The conditional-entropy formulation in Section 3.3 does not predict these rankings; it quantifies position-recovery uncertainty under the implemented stochastic mapping. Accordingly, the main empirical conclusion is limited to the evaluated search: no single triplet is highest across all datasets, and Fibonacci-based configurations are not uniformly better than conventional ones. Capacity-matched multi-seed experiments are required before attributing branch-specific advantages to a particular positional mechanism.
7.2. Performance Differential Across Datasets
WA is higher on IEMOCAP-4/6 (74.21%/74.54%) than on the historical MELD evaluation (49.44%), although values across different label sets and protocols are not directly comparable. The document-backed analysis in Section 6.3 quantifies two relevant MELD properties. First, the official test split has a 25.12:1 neutral-to-fear imbalance. Second, the restored-checkpoint record shows 78.45% recall for neutral but only 16.27–26.35% for disgust, fear, sadness, surprise, and joy, with 65.13% of predictions assigned to neutral. The confusion matrix therefore localizes much of the observed error to minority-class samples being assigned to the majority class. These results support class imbalance as an important associated factor, while the absence of controlled acoustic perturbations prevents causal attribution to noise, overlap, or speaker transitions individually.
7.3. Representation-Specific Encoding Observations
Across the three highest-observed triplets, the MFCC branch uses Fib+RoPE, while the selected log-mel and Wav2Vec encodings vary. This repetition is noteworthy but is based on only three dataset/task settings and a single reported run per setting. We therefore describe it as an observed search pattern rather than a stable preference.
One hypothesis is that two-dimensional log-mel and MFCC grids and contextualized Wav2Vec sequences place different demands on position modeling. However, the current experiments do not separately manipulate locality, conversational noise, or temporal context, and therefore cannot verify this mechanism. Testing this hypothesis requires matched-capacity encoders and controlled perturbations of token ordering, noise, and sequence length.
The smaller observed increment from FTS on MELD (+0.37 percentage points over FPE-only) further indicates that the configuration-level difference is dataset-dependent in the available runs. Multi-seed estimates are needed to determine whether this small difference is distinguishable from training variability.
7.4. Scope of the Positional-Uncertainty Analysis
The implementation-consistent analysis establishes a limited but precise result: because the modulo-wrapped phase is a deterministic transformation of the sampled Fibonacci index, observing the phase cannot reveal more about the original token position than observing the complete index. The resulting conditional entropy quantifies positional recovery uncertainty, not representational richness, attention quality, or expected classification accuracy. The expected relation matrix further describes only the geometry of the positional rotations. We therefore base practical claims on the dataset-specific ablation results and treat the information-theoretic quantities as implementation diagnostics rather than evidence of causal superiority over APE or RoPE.
7.5. Limitations and Scope
This work has several limitations. First, the reported training results are based on a single random seed; fixed-checkpoint resampling cannot replace independent multi-seed training, confidence intervals across runs, and significance testing. Second, the conventional and Fibonacci configurations differ in attention-head count, so the current comparisons do not isolate positional encoding from model capacity. Third, optimization exposure is not fully matched: RoPE is evaluated in the common first-stage search but does not receive the same second-stage learning-rate refinement budget as the shortlisted configurations and APE/ALiBi baselines. Fourth, the model uses fixed-length inputs, and generalization to variable-length utterances is not evaluated. Fifth, evaluation is limited to English datasets and does not establish multilingual or cross-cultural generalization. Sixth, the MELD analysis quantifies class imbalance and errors under the dataset’s native conversational conditions but does not include artificial-noise or controlled-imbalance experiments, so individual acoustic causes remain unresolved. Seventh, the official MELD distribution and restored-checkpoint class-wise record use different counting bases (2610 raw test utterances versus 3808 processed evaluation samples) and are intentionally reported separately. Eighth, the positional-uncertainty quantities measure location recoverability and rotation geometry rather than complete content-dependent attention behavior; their values depend on the token grid, candidate sets, embedding dimension, numerical precision, and collision tolerance. Ninth, FTS has not been compared with alternative structured index assignments, such as zig-zag scans, Hilbert curves, or other space-filling curves. Finally, cross-study comparisons remain incompletely controlled because preprocessing, features, model capacity, and evaluation protocols differ.
8. Conclusions
This paper presents FPE and FTS for candidate-based Fibonacci phase assignment in a multi-representation acoustic SER model. Across the evaluated exploratory search, the highest-observed configurations reach 74.21% WA on IEMOCAP-4, 74.54% on IEMOCAP-6, and 49.44% on MELD, with different triplets selected for the three settings. The implementation-consistent conditional-entropy formulation quantifies positional recovery uncertainty introduced by candidate sampling and modulo-wrapped phases; it does not establish richer representations or superior attention. Available component runs document larger configuration-level differences between FPE-only and FPE+FTS on IEMOCAP than on MELD. The restored MELD analysis further identifies strong prediction bias toward neutral and substantially lower recall for minority emotions, while fixed-checkpoint bootstrap intervals quantify sample-level uncertainty. Future work will address variable-length processing, multilingual evaluation, controlled noise and imbalance experiments, and comparisons with alternative two-dimensional index-assignment strategies.
Acknowledgments
The authors acknowledge Northeastern University at Qinhuangdao and Osaka University for providing research facilities and computational resources.
Author Contributions
Conceptualization, S.H. and C.F.; methodology, S.H. and J.Z.; software, S.H., J.Z., Q.Z. and Z.L.; validation, J.X., Z.X., S.W. and Z.Q.; formal analysis, S.H. and J.Z.; investigation, J.X., Z.X. and J.L.; resources, Q.Z., Z.L., S.W. and Z.Q.; data curation, J.X. and J.L.; visualization, Z.X. and J.L.; writing–original draft preparation, S.H.; writing–review and editing, J.Z. and C.F.; supervision, C.F.; project administration, C.F. All authors have read and agreed to the published version of the manuscript.
Institutional Review Board Statement
Ethical review and approval were waived because this study used previously collected benchmark datasets and did not involve new data collection from human participants.
Informed Consent Statement
Participant consent was waived because no new human-subject data were collected; this study used existing benchmark datasets under their respective access conditions.
Data Availability Statement
The IEMOCAP dataset is available from the USC SAIL Lab at https://sail.usc.edu/iemocap/, subject to the dataset access conditions (accessed on 1 July 2026). The MELD dataset is available at https://affective-meld.github.io/ (accessed on 1 July 2026). The code and processed experimental configurations will be released in a public repository upon acceptance, and the repository link will be provided in the final version.
Conflicts of Interest
The authors declare no conflicts of interest.
Funding Statement
This research was supported by the National College Students’ Innovation and Entrepreneurship Training Program (No. 202619145012) and the College Students’ Innovation and Entrepreneurship Training Program of Northeastern University at Qinhuangdao (No. CX260317).
Footnotes
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
References
- 1.Picard R.W. Affective Computing. MIT Press; Cambridge, MA, USA: 1997. [Google Scholar]
- 2.El Ayadi M., Kamel M.S., Karray F. Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognit. 2011;44:572–587. doi: 10.1016/j.patcog.2010.09.020. [DOI] [Google Scholar]
- 3.Eyben F., Wöllmer M., Schuller B. openSMILE—The Munich versatile and fast open-source audio feature extractor; Proceedings of the ACM Multimedia; Firenze, Italy. 25–29 October 2010; pp. 1459–1462. [Google Scholar]
- 4.Trigeorgis G., Ringeval F., Brueckner R., Marchi E., Nicolaou M.A., Schuller B.W., Zafeiriou S. Proceedings of the ICASSP, Shanghai, China, 20–25 March 2016. IEEE; Piscataway, NJ, USA: 2016. Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network; pp. 5200–5204. [Google Scholar]
- 5.Zhao J., Mao X., Chen L. Speech emotion recognition using deep 1D & 2D CNN LSTM networks. Biomed. Signal Process. Control. 2019;47:312–323. doi: 10.1016/j.bspc.2018.08.035. [DOI] [Google Scholar]
- 6.Baevski A., Zhou H., Mohamed A., Auli M. wav2vec 2.0: A framework for self-supervised learning of speech representations; Proceedings of the Advances in Neural Information Processing Systems (NeurIPS); Virtual. 6–12 December 2020; pp. 12449–12460. [Google Scholar]
- 7.Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser Ł., Polosukhin I. Attention is all you need; Proceedings of the Advances in Neural Information Processing Systems (NeurIPS); Long Beach, CA, USA. 4–9 December 2017; [Google Scholar]
- 8.Latif S., Rana R., Younis K., Qadir J., Epps J., Schuller B.W., Janjua M. Multimodal speech emotion recognition using deep learning. J. Intell. Fuzzy Syst. 2020;38:6933–6945. [Google Scholar]
- 9.Poria S., Cambria E., Bajpai R., Hussain A. A review of affective computing: From unimodal analysis to multimodal fusion. Inf. Fusion. 2017;37:98–125. doi: 10.1016/j.inffus.2017.02.003. [DOI] [Google Scholar]
- 10.Busso C., Bulut M., Lee C.C., Kazemzadeh A., Mower E., Kim S., Chang J.N., Lee S., Narayanan S.S. IEMOCAP: Interactive emotional dyadic motion capture database. Lang. Resour. Eval. 2008;42:335–359. doi: 10.1007/s10579-008-9076-6. [DOI] [Google Scholar]
- 11.Poria S., Hazarika D., Majumder N., Naik G., Cambria E., Mihalcea R. MELD: A multimodal multi-party dataset for emotion recognition in conversations; Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL); Florence, Italy. 28 July–2 August 2019; pp. 527–536. [Google Scholar]
- 12.Latif S., Rana R., Khalifa S., Jurdak R., Qadir J., Schuller B.W. Survey of Deep Representation Learning for Speech Emotion Recognition. IEEE Trans. Affect. Comput. 2023;14:1634–1654. doi: 10.1109/taffc.2021.3114365. [DOI] [Google Scholar]
- 13.Schuller B.W. Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends. Commun. ACM. 2018;61:90–99. doi: 10.1145/3129340. [DOI] [Google Scholar]
- 14.Schuller B., Rigoll G., Lang M. Proceedings of the 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Hong Kong, 6–10 April 2003. Volume 2. IEEE; Piscataway, NJ, USA: 2003. Hidden Markov model-based speech emotion recognition; pp. II–1. [Google Scholar]
- 15.Hochreiter S., Schmidhuber J. Long short-term memory. Neural Comput. 1997;9:1735–1780. doi: 10.1162/neco.1997.9.8.1735. [DOI] [PubMed] [Google Scholar]
- 16.Satt A., Rozenberg S., Hoory R. Efficient emotion recognition from speech using deep learning on spectrograms; Proceedings of the Interspeech; Stockholm, Sweden. 20–24 August 2017; pp. 1089–1093. [Google Scholar]
- 17.Mirsamadi S., Barsoum E., Zhang C. Proceedings of the 2017 IEEE International conference on acoustics, speech and signal processing (ICASSP), New Orleans, LA, USA, 5–9 March 2017. IEEE; Piscataway, NJ, USA: 2017. Automatic speech emotion recognition using recurrent neural networks with local attention; pp. 2227–2231. [Google Scholar]
- 18.Zhang S., Zhang S., Huang T., Gao W. Speech emotion recognition using deep convolutional neural network and discriminant temporal pyramid matching. IEEE Trans. Multimed. 2018;20:1576–1590. doi: 10.1109/tmm.2017.2766843. [DOI] [Google Scholar]
- 19.Chen W., Xing X., Xu X., Pang J., Du L. SpeechFormer: A Hierarchical Efficient Framework Incorporating the Characteristics of Speech; Proceedings of the Interspeech 2022; Incheon, Republic of Korea. 18–22 September 2022; pp. 4847–4851. [DOI] [Google Scholar]
- 20.Chen S., Xing X., Zhang W., Chen W., Xu X. Proceedings of the ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 4–10 June 2023. IEEE; Piscataway, NJ, USA: 2023. DWFormer: Dynamic Window Transformer for Speech Emotion Recognition; pp. 1–5. [DOI] [Google Scholar]
- 21.Hsu W.N., Bolte B., Tsai Y.H.H., Lakhotia K., Salakhutdinov R., Mohamed A. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio Speech Lang. Process. 2021;29:3451–3460. doi: 10.1109/taslp.2021.3122291. [DOI] [Google Scholar]
- 22.Pepino L., Riera P., Ferrer L. Emotion recognition from speech using wav2vec 2.0 embeddings; Proceedings of the Interspeech 2021; Brno, Czech Republic. 30 August–3 September 2021; pp. 3400–3404. [Google Scholar]
- 23.Yang S.w., Chi P.H., Chuang Y.S., Lai C.I., Lakhotia K., Lin Y.Y., Liu A.T., Shi J., Chang X., Lin G.T., et al. SUPERB: Speech processing universal performance benchmark; Proceedings of the Interspeech 2021; Brno, Czech Republic. 30 August–3 September 2021; pp. 1194–1198. [DOI] [Google Scholar]
- 24.Radford A., Kim J.W., Xu T., Brockman G., McLeavey C., Sutskever I. Robust speech recognition via large-scale weak supervision; Proceedings of the 40th International Conference on Machine Learning (ICML); Honolulu, HI, USA. 23–29 July 2023; pp. 28492–28518. [Google Scholar]
- 25.Li Y., Zhao T., Kawahara T. Improved end-to-end speech emotion recognition using self attention mechanism and multitask learning; Proceedings of the Interspeech 2019; Graz, Austria. 15–19 September 2019; pp. 2803–2807. [Google Scholar]
- 26.Fu C., Liu C., Ishi C., Ishiguro H. An end-to-end multitask learning model to improve speech emotion recognition; Proceedings of the EUSIPCO 2020, Amsterdam, The Netherlands; 24–28 August 2020. [Google Scholar]
- 27.Zhang H., Huang H., Zhao P., Zhu X., Yu Z. CENN: Capsule-enhanced neural network with innovative metrics for robust speech emotion recognition. Knowl.-Based Syst. 2024;304:112499. doi: 10.1016/j.knosys.2024.112499. [DOI] [Google Scholar]
- 28.Zhang H., Huang H., Zhao P., Yu Z. Sparse temporal aware capsule network for robust speech emotion recognition. Eng. Appl. Artif. Intell. 2025;144:110060. doi: 10.1016/j.engappai.2025.110060. [DOI] [Google Scholar]
- 29.Zhang H., Pang Z., Zhao P., Tang G., Shen L., Wang G. Pre-attentive speech signal processing with adaptive routing for emotion recognition. Biomed. Signal Process. Control. 2026;112:108782. doi: 10.1016/j.bspc.2025.108782. [DOI] [Google Scholar]
- 30.Lu C., Lian H., Zheng W., Zong Y., Zhao Y., Li S. Learning Local to Global Feature Aggregation for Speech Emotion Recognition; Proceedings of the Interspeech 2023; Dublin, Ireland. 20–24 August 2023; pp. 1908–1912. [DOI] [Google Scholar]
- 31.Zhao P., Zhang H., Yue Z., Li Z., Mi N., Liu X. Residual attention network for audio-based speech emotion recognition. Appl. Soft Comput. 2026;201:115637. doi: 10.1016/j.asoc.2026.115637. [DOI] [Google Scholar]
- 32.Hazarika D., Poria S., Zadeh A., Cambria E., Morency L.P., Zimmermann R. Conversational memory network for emotion recognition in dyadic dialogue videos; Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT); New Orleans, LA, USA. 1–6 June 2018; pp. 2122–2132. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Majumder N., Poria S., Hazarika D., Mihalcea R., Gelbukh A., Cambria E. DialogueRNN: An attentive RNN for emotion detection in conversations; Proceedings of the AAAI Conference on Artificial Intelligence; Honolulu, HI, USA. 27 January–1 February 2019; pp. 6818–6825. [Google Scholar]
- 34.Ghosal D., Majumder N., Poria S., Chhaya N., Gelbukh A. DialogueGCN: A graph convolutional neural network for emotion recognition in conversation; Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP); Hong Kong, China. 3–7 November 2019; pp. 154–164. [Google Scholar]
- 35.Lu J., Batra D., Parikh D., Lee S. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks; Proceedings of the Advances in Neural Information Processing Systems (NeurIPS); Vancouver, BC, Canada. 8–14 December 2019; [Google Scholar]
- 36.Zhang H., Zhao P., Tang G., Li Z., Yuan Z. Reproducible and generalizable speech emotion recognition via an Intelligent Fusion Network. Biomed. Signal Process. Control. 2025;109:107996. doi: 10.1016/j.bspc.2025.107996. [DOI] [Google Scholar]
- 37.Park D.S., Chan W., Zhang Y., Chiu C.C., Zoph B., Cubuk E.D., Le Q.V. SpecAugment: A simple data augmentation method for automatic speech recognition; Proceedings of the Interspeech 2019; Graz, Austria. 15–19 September 2019; pp. 2613–2617. [Google Scholar]
- 38.Chen T., Kornblith S., Norouzi M., Hinton G. A simple framework for contrastive learning of visual representations; Proceedings of the 37th International Conference on Machine Learning (ICML)—PMLR, Virtual; 13–18 July 2020; pp. 1597–1607. [Google Scholar]
- 39.Hinton G., Vinyals O., Dean J. Distilling the knowledge in a neural network; Proceedings of the NeurIPS Deep Learning Workshop; Montreal, QC, Canada. 11 December 2015. [Google Scholar]
- 40.Lou Z., Otake S., Li Z., Kawakami R., Inoue N. Proceedings of the ICASSP 2024—2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea, 14–19 April 2024. IEEE; Piscataway, NJ, USA: 2024. Cubic knowledge distillation for speech emotion recognition; pp. 5705–5709. [Google Scholar]
- 41.Devlin J., Chang M.W., Lee K., Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding; Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT); Minneapolis, MN, USA. 2–7 June 2019; pp. 4171–4186. [Google Scholar]
- 42.Dosovitskiy A., Beyer L., Kolesnikov A., Weissenborn D., Zhai X., Unterthiner T., Dehghani M., Minderer M., Heigold G., Gelly S., et al. An image is worth 16x16 words: Transformers for image recognition at scale; Proceedings of the International Conference on Learning Representations (ICLR); Virtual. 3–7 May 2021. [Google Scholar]
- 43.Liu Z., Lin Y., Cao Y., Hu H., Wei Y., Zhang Z., Lin S., Guo B. Swin transformer: Hierarchical vision transformer using shifted windows; Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV); Montreal, QC, Canada. 10–17 October 2021; pp. 10012–10022. [Google Scholar]
- 44.Gong Y., Chung Y.A., Glass J. AST: Audio Spectrogram Transformer; Proceedings of the Interspeech 2021; Brno, Czech Republic. 30 August–3 September 2021; pp. 571–575. [DOI] [Google Scholar]
- 45.Pepino L., Riera P., Ferrer L. Study of Positional Encoding Approaches for Audio Spectrogram Transformers; Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); Singapore. 23–27 May 2022; pp. 3713–3717. [DOI] [Google Scholar]
- 46.Su J., Lu Y., Pan S., Wen B., Liu Y. RoFormer: Enhanced Transformer with Rotary Position Embedding. Neurocomputing. 2024;568:127063. doi: 10.1016/j.neucom.2023.127063. [DOI] [Google Scholar]
- 47.Dai Z., Yang Z., Yang Y., Carbonell J., Le Q., Salakhutdinov R. Transformer-XL: Attentive language models beyond a fixed-length context; Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL); Florence, Italy. 28 July–2 August 2019; pp. 2978–2988. [Google Scholar]
- 48.Shaw P., Uszkoreit J., Vaswani A. Self-attention with relative position representations; Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT); New Orleans, LA, USA. 1–6 June 2018; pp. 464–468. [Google Scholar]
- 49.Press O., Smith N.A., Lewis M. Train short, test long: Attention with linear biases enables input length extrapolation; Proceedings of the International Conference on Learning Representations (ICLR); Virtual. 25 April 2022. [Google Scholar]
- 50.Ke G., He D., Liu T.Y. Rethinking positional encoding in language pre-training; Proceedings of the International Conference on Learning Representations (ICLR); Virtual. 3–7 May 2021. [Google Scholar]
- 51.Ito T., Cocchi L., Klinger T., Ram P., Campbell M., Hearne L. Learning interpretable positional encodings in transformers depends on initialization. arXiv. 20242406.08272 [Google Scholar]
- 52.Wu Z., Gong Z., Ai L., Shi P., Donbekci K., Hirschberg J. Proceedings of the Findings of the Association for Computational Linguistics: NAACL 2025, Vienna, Austria, 27 July–1 August 2025. Association for Computational Linguistics; Stroudsburg, PA, USA: 2025. Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances; pp. 2202–2218. [DOI] [Google Scholar]
- 53.Mai J., Xing X., Chen W., Fang Y., Xu X. AA-SLLM: An Acoustically Augmented Speech Large Language Model for Speech Emotion Recognition; Proceedings of the Interspeech 2025; Rotterdam, The Netherlands. 17–21 August 2025; pp. 4328–4332. [DOI] [Google Scholar]
- 54.Fu C., Liu C., Ishi C.T., Ishiguro H. Proceedings of the ICASSP 2021—2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Virtual, 6–11 June 2021. IEEE; Piscataway, NJ, USA: 2021. MAEC: Multi-Instance Learning with an Adversarial Auto-Encoder-Based Classifier for Speech Emotion Recognition; pp. 6299–6303. [DOI] [Google Scholar]
- 55.Chen W., Xing X., Xu X., Pang J., Du L. SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing. IEEE/ACM Trans. Audio Speech Lang. Process. 2023;31:775–787. doi: 10.1109/TASLP.2023.3235194. [DOI] [Google Scholar]
- 56.Shen S., Gao Y., Liu F., Wang H., Zhou A. Proceedings of the ICASSP 2024—2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea, 14–19 April 2024. IEEE; Piscataway, NJ, USA: 2024. Emotion Neural Transducer for Fine-Grained Speech Emotion Recognition; pp. 10111–10115. [DOI] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The IEMOCAP dataset is available from the USC SAIL Lab at https://sail.usc.edu/iemocap/, subject to the dataset access conditions (accessed on 1 July 2026). The MELD dataset is available at https://affective-meld.github.io/ (accessed on 1 July 2026). The code and processed experimental configurations will be released in a public repository upon acceptance, and the repository link will be provided in the final version.
