Explainability guided feature selection for speech emotion recognition using SHAP and permutation importance

Abstract

Speech Emotion Recognition (SER) systems commonly rely on high-dimensional handcrafted acoustic feature sets, which increase computational complexity, risk overfitting, and reduce interpretability. To address these challenges, we propose an explainability-driven, redundancy-aware feature selection framework that integrates SHAP-based attribution with permutation-based performance importance through a unified rank-sum criterion. A correlation-controlled redundancy penalty is further introduced to mitigate multicollinearity while preserving acoustically meaningful descriptors. The framework operates entirely in the original feature space, enabling transparent and interpretable model refinement. Experiments were conducted on four widely used emotional speech corpora (CREMA-D, RAVDESS, SAVEE, and TESS) under within-corpus evaluation settings. To assess the generalizability of the proposed framework, experiments were performed using multiple classifiers, including XGBoost, Support Vector Machine (SVM), and Random Forest (RF). Using the XGBoost classifier, the proposed framework achieved accuracies of 0.600 on CREMA-D, 0.752 on RAVDESS, 0.726 on SAVEE, and 0.998 on TESS, while reducing the feature space by up to 40–70%. Macro-F1 and UAR metrics exhibit similar trends, confirming stable performance gains. Additional experiments using SVM and RF further indicate that the proposed feature selection framework generalizes across different classifiers while maintaining competitive performance. The findings indicate that explainability can serve not only as a post-hoc interpretation tool but as an active optimization mechanism for compact and interpretable SER models.

Introduction

Speech Emotion Recognition (SER) has emerged as a fundamental component of affective computing, enabling machines to infer emotional states from spoken language [1]. By analyzing acoustic characteristics such as prosodic, spectral patterns, and voice quality, SER systems aim to bridge the gap between human affective expression and machine understanding. These systems play a crucial role in a wide range of applications, including healthcare monitoring [2], intelligent virtual assistants, customer service analytics, human–computer interaction [3], or education [4].

Despite its wide applications, SER remains challenging due to variability across speakers and contexts, overlapping acoustic patterns, recording inconsistencies, class imbalance, and the subjective nature of emotion labeling [5]. Traditional SER pipelines often extract hundreds of handcrafted features, encompassing prosodic, spectral, temporal, and voice-quality descriptors. While such rich feature sets can improve representational capacity, they also introduce the curse of dimensionality, increasing the risk of overfitting, inflating computational costs, and reducing model interpretability. As SER systems are increasingly deployed in sensitive domains such as healthcare, interpretability and transparency are becoming as important as predictive accuracy.

Over the past decade, numerous techniques have been developed for SER. These approaches can be broadly grouped into three categories. First, methods relied on hand-crafted acoustic features, including prosodic features (e.g., fundamental frequency, energy, and speech rate), spectral features (e.g., MFCCs and LPCs), and tonal descriptors. These features were typically combined with conventional machine learning classifiers such as GMMs, SVMs, and HMMs [6–8]. Second, with the rise of deep learning, end-to-end frameworks were introduced to automatically learn feature representations. Convolutional neural networks (CNNs), recurrent neural networks (RNNs), and BiLSTM architectures became widely adopted [9]. More recently, Transformer-based models have gained prominence [10]. For example, Chen et al. [11] proposed a hierarchical Transformer with a token-semantic module to reduce training time and model size, while Lu et al. [12] introduced a frame-to-segment Transformer aggregation strategy for utterance-level representation learning. Third, recent research has increasingly leveraged pre-trained models (PTMs) to extract emotion-relevant representations. Early PTMs were CNN-based, such as AlexNet [13] and ResNet [14], whereas more recent models adopt Transformer architectures, including wav2vec 2.0 [15] and HuBERT [16]. Trained in large-scale data using supervised or self-supervised learning (SSL), these models produce embeddings that achieve state-of-the-art performance across various speech tasks. In SER, approaches such as the upstream–downstream framework by Morais et al. [17] and emotion2vec proposed by Ma et al. [18] demonstrate the effectiveness of self-supervised representations for emotion-related tasks.

A major challenge in SER lies in training models on large, high-dimensional feature spaces where the relevance of individual features is not always clearly understood. This issue is further exacerbated by the limited size and diversity of available datasets, which restrict effective model training and generalization [19]. As a result, predefined feature sets often produce high-dimensional representations that hinder the model’s ability to capture meaningful emotion-related patterns.

In addition, transparency and explainability are critical in SER, particularly for safety-sensitive applications. Interpretable systems enable error analysis, bias detection, and more reliable decision-making [20]. Explainable AI (XAI) [21], aims to enhance the transparency and accountability of AI models, addressing concerns related to trust and fairness [22, 23]. In the context of SER, integrating XAI techniques supports the development of more interpretable and trustworthy emotion recognition systems. Among various XAI techniques, SHapley Additive exPlanations (SHAP) [24], offers a theoretically grounded framework based on cooperative game theory to quantify the contribution of each feature to a model’s prediction. SHAP has been successfully applied in domains such as finance, medicine, and computer vision, but its potential for systematic feature selection in SER remains underexplored.

To address the challenges of high-dimensional feature spaces and unclear feature relevance, we propose a supervised machine learning framework that integrates explainability-driven feature selection with performance-based validation. Specifically, we combine SHAP importance [24] with permutation-based feature importance [25] and redundancy control to iteratively refine the feature set in a model-dependent manner. This formulation ensures the retention of features that are both decision-relevant and performance-critical, while progressively eliminating redundant or less informative features.

The proposed architecture consists of three main components: (1) a feature refinement module that performs iterative importance scoring and redundancy-aware selection, (2) a supervised classification module for emotion recognition, and (3) an explainability module that quantifies feature contributions and provides feedback to guide subsequent optimization steps. Unlike conventional approaches where explainability is applied post hoc, our framework integrates explainability as a core iterative mechanism for feature optimization within the SER pipeline.

The main contributions of this paper are summarized as follows:

• I.We propose an explainability-guided feature selection framework that combines SHAP attribution and permutation importance within the original handcrafted acoustic feature space while incorporating redundancy-aware ranking.
• II.We perform stability-aware Top-K feature selection across multiple random seeds to analyze the trade-off between recognition performance and feature compactness.
• III.We evaluate the proposed framework across multiple classifiers (XGBoost, SVM, and RF) to assess classifier-independent generalization of the selected feature subsets.

Materials and methods

Dataset

Numerous public datasets are available for speech emotion recognition (SER), generally categorized into actor-based, induced, and natural emotional corpora. Model performance is strongly influenced by dataset authenticity, with natural datasets offering more realistic emotional expressions; however, they often present challenges related to accessibility and legal constraints [26]. Emotional speech datasets differ in language, number of emotions, speaker diversity, and recording conditions [27]. Among them, actor-based datasets are the most widely used, particularly in English, and typically include core emotions such as neutral, happy, sad, and angry.

In this study, we utilize four widely adopted English actor-based datasets: CREMA-D [28], RAVDESS [29], SAVEE [30], and TESS [31].

Experimental protocol

To ensure reproducibility, all experiments were conducted using identical train-test splits across all classifiers. For within-corpus evaluation, each dataset was divided into training and testing subsets using stratified sampling with a 70:30 ratio. Experiments were repeated using two random seeds to assess result stability.

Feature selection was performed exclusively on the training partition to prevent information leakage. SHAP importance and permutation importance were computed independently and combined using the proposed rank-sum criterion. Top-K feature subsets were then evaluated to analyze the trade-off between feature compactness and recognition performance.

Three classifiers were employed for evaluation: XGBoost (XGB) [32], Support Vector Machine (SVM) [33], and Random Forest (RF) [34]. All models were trained and evaluated under identical experimental settings for fair comparison.

Performance was evaluated using Accuracy (ACC), Macro-F1 score, and Unweighted Average Recall (UAR).

Proposed explainability-driven feature selection framework

Figure 1 illustrates the overall architecture of the proposed explainability-driven SER framework. The system consists of feature extraction, importance estimation using SHAP and permutation analysis, redundancy-controlled ranking, and multi-classifier evaluation using XGBoost, SVM, and RF.

Fig. 1.

Block diagram of the proposed explainability-driven SER framework, main idea is in light-blue blocks

Feature extraction

From each utterance, we extract a total of 296 handcrafted acoustic features using standard speech analysis techniques. The feature set is organized into four major categories. Prosodic features capture fundamental frequency (pitch), intensity (energy), and their statistical descriptors, reflecting variations in speech melody and loudness. Spectral features characterize the frequency-domain properties of speech and include Mel-frequency cepstral coefficients (MFCCs) with per-coefficient statistical functionals, as well as spectral centroid, bandwidth, contrast, and roll-off. Voice quality features describe micro-variations in phonation and include jitter, shimmer, harmonic-to-noise ratio (HNR), and glottal noise measures. Temporal dynamics features model changes over time in acoustic contours, including pitch slopes, energy dynamics, and related statistical measures. Prior to model training, all features are standardized to zero mean and unit variance.

Methods

Problem formulation

The overall methodology is mathematically formulated in (1)–(12), beginning with dataset definition, followed by importance estimation, redundancy-aware scoring, and concluding with optimal subset determination.

The handcrafted SER dataset is defined as shown in (1):

1

where represents a d-dimensional acoustic feature vector and denotes the emotion label.

As formulated in (2), the objective is to select an optimal subset of features such as:

2

where represents classification performance measured by Macro-F1, Accuracy, and UAR.

Combined SHAP–permutation importance

Feature relevance is estimated using two complementary criteria computed strictly on training data.

1. SHAP-Based Importance:

For a trained XGBoost classifier , the SHAP value for each feature is defined in (3):

3

representing the contribution of feature f to sample i for class c.

The global SHAP importance is computed according to (4):

4

• (b)Permutation-Based Importance:

Permutation importance quantifies performance degradation when feature f is randomly permuted and is calculated as shown in (5):

5

where denotes the baseline validation performance measured before feature permutation, and represents the validation performance after randomly permuting the i-th feature. The permutation importance therefore quantifies the performance degradation caused by disrupting the relationship between a feature and the target labels.

• (c)Rank-Sum Importance Score:

Features are ranked independently according to both measures of importance. The final ranking score is defined in (6):

6

Here, and denote the ordinal ranking positions of the i-th feature according to SHAP importance and permutation importance, respectively. Lower values of indicate higher importance.

Redundancy control formulation

To mitigate feature redundancy and multicollinearity, we introduce a redundancy penalty based on pairwise correlation defined in (7):

7

denote Pearson correlation between features and .

For a candidate feature f, redundancy relative to an already selected subset is defined as (8):

8

A redundancy-aware score is then defined as (9):

9

where controls the trade-off between importance and redundancy.

The correlation constraint is enforced as specified in (10). Feature selection proceeds greedily by iteratively selecting features minimizing , ensuring that:

10

for selected feature pairs whenever possible, where τ is a predefined correlation threshold.

This formulation ensures that selected features are:

• Highly influential (SHAP),

• Performance-critical (Permutation),

• Minimally redundant (Correlation control).

Optimal subset determination

Equation (11) defines the candidate subset sizes.

11

For each K, the top-K ranked features are selected and final classifiers (XGBoost, SVM, and RF) are trained and evaluated. The optimal subset is determined as (12):

12

with Accuracy used as a tie-breaking criterion.

Fig. 2.

Explainability-driven redundancy-controlled feature selection

The proposed framework differs from conventional SER pipelines in three aspects:

• I.It integrates SHAP as an active optimization mechanism rather than post-hoc interpretation.
• II.It combines explainability and performance validation through rank-sum scoring.
• III.It introduces redundancy-aware feature control to improve compactness while preserving discriminative acoustic information across multiple classifiers.

Together, these components establish a principled, explainability-driven feature refinement strategy that improves both interpretability and classification robustness in speech emotion recognition.

Results

Experimental overview

Experiments were conducted on four widely used emotional speech corpora (CREMA-D, RAVDESS, SAVEE, and TESS), under within-corpus evaluation settings. Performance was measured using Accuracy (ACC), Macro-averaged F1-score (Macro-F1), and Unweighted Average Recall (UAR) to account for class imbalance and per-class robustness (Table 1).

Table 1.

Summary of SER databases considered in this study

Dataset	Classes	Samples	Class distribution	Features
CREMA-D	6	7441	Angry 1271, disgust 1271, fear 1271, Happy 1271, sad 1270, neutral 1087	296
RAVDESS	7	1440	Neutral 288; all others 192 each	296
SAVEE	7	480	Neutral 120; all others 60 each	296
TESS	7	2800	All classes 400 each (balanced)	296

Baseline models were constructed using the complete handcrafted acoustic feature set (296 features), without applying any feature selection. This baseline serves as a full-feature reference model to evaluate whether the proposed explainability-guided feature selection strategy improves performance beyond the original feature space. For fair comparison, identical classifier configurations and train-validation splits were maintained between the baseline and proposed methods; the only difference was whether all features were used or a Top-K subset selected using SHAP and permutation-based ranking was applied. Such full-feature reference models are commonly used in feature selection studies to assess the effectiveness of dimensionality reduction methods [35].

Quantitative performance

Table 2 summarizes performance for the baseline and proposed methods. The proposed feature-selection framework generally improves performance across the evaluated datasets, with larger gains on CREMA-D and SAVEE and more modest improvements on RAVDESS, while reducing the feature space by up to 40–70%. To address potential model dependency, the proposed feature selection framework was additionally evaluated using multiple classifiers, including XGBoost (XGB), Support Vector Machine (SVM), and Random Forest (RF). Table 3 present class-wise precision, recall, and F1-score for the proposed model across the four datasets.

Table 2.

SER performance (baseline vs. proposed)

Corpus	Model	Baseline ACC	Baseline Macro-F1	Baseline UAR	Proposed ACC	Proposed Macro-F1	Proposed UAR
CREMA-D	XGB	0.593	0.591	0.595	0.600	0.598	0.602
	SVM	0.589	0.588	0.589	0.589	0.588	0.588
	RF	0.534	0.522	0.537	0.540	0.530	0.543
RAVDESS	XGB	0.744	0.728	0.730	0.752	0.737	0.739
	SVM	0.834	0.826	0.825	0.821	0.813	0.813
	RF	0.693	0.670	0.674	0.690	0.664	0.671
SAVEE	XGB	0.708	0.680	0.673	0.726	0.696	0.689
	SVM	0.761	0.744	0.740	0.764	0.747	0.744
	RF	0.708	0.683	0.666	0.715	0.692	0.675
TESS	XGB	0.995	0.995	0.995	0.998	0.998	0.998
	SVM	0.999	0.999	0.999	0.999	0.999	0.999
	RF	0.998	0.998	0.998	0.998	0.998	0.998

Table 3.

Class-wise precision, recall, and F1-score using the proposed XGBoost model

Class	Precision	Recall	F1-score
(a) CREMA-D
Angry	0.71	0.74	0.72
Disgust	0.56	0.49	0.53
Fear	0.62	0.49	0.54
Happy	0.59	0.59	0.59
Neutral	0.53	0.65	0.59
Sad	0.58	0.65	0.61
(b) RAVDESS
Angry	0.90	0.63	0.74
Disgust	0.70	0.81	0.75
Fear	0.79	0.79	0.79
Happy	0.71	0.64	0.67
Neutral	0.73	0.95	0.82
Sad	0.69	0.44	0.54
Surprise	0.74	0.84	0.79
(c) SAVEE
Angry	0.71	0.67	0.69
Disgust	0.58	0.39	0.47
Fear	0.62	0.56	0.59
Happy	0.61	0.78	0.68
Neutral	0.75	1.00	0.86
Sad	0.92	0.67	0.77
Surprise	0.73	0.61	0.67
(d) TESS
Angry	1.00	1.00	1.00
Disgust	1.00	0.99	1.00
Fear	1.00	1.00	1.00
Happy	1.00	0.99	1.00
Neutral	1.00	1.00	1.00
Sad	1.00	1.00	1.00
Surprise	0.98	1.00	0.99

Across four speech emotion recognition corpora (CREMA, RAVDESS, SAVEE, and TESS), we evaluated the proposed SHAP- and permutation-guided feature selection framework using all classifiers. On CREMA-D (6 classes, 7,441 samples), XGBoost showed consistent improvement after feature selection, with accuracy increasing from 0.593 to 0.600 and macro-F1 from 0.591 to 0.598. Random Forest also exhibited modest gains, improving accuracy from 0.534 to 0.540 and macro-F1 from 0.522 to 0.530. In contrast, SVM performance remained nearly unchanged, indicating that feature reduction had limited impact on this corpus for kernel-based classification.

On TESS (7 classes, 2,800 balanced samples), all classifiers achieved near-ceiling performance. XGBoost improved slightly from 0.995 to 0.998 accuracy, while SVM and RF maintained approximately 0.999 and 0.998 accuracy, respectively. These results suggest that the TESS corpus is highly separable even without aggressive feature reduction, leaving limited room for measurable improvement.

Larger improvements were observed on more challenging or smaller datasets. On SAVEE (7 classes, 480 samples), XGBoost improved from 0.708 to 0.726 accuracy, with macro-F1 increasing from 0.680 to 0.696. SVM also benefited slightly, improving from 0.761 to 0.764 accuracy and from 0.744 to 0.747 macro-F1. Random Forest demonstrated moderate gains, with accuracy increasing from 0.708 to 0.715 and macro-F1 from 0.683 to 0.692.

On RAVDESS (7 classes, 1,440 samples), XGBoost again showed consistent improvement, increasing accuracy from 0.744 to 0.752 and macro-F1 from 0.728 to 0.737. However, SVM experienced a slight performance decrease after feature selection, with accuracy dropping from 0.834 to 0.821 and macro-F1 from 0.826 to 0.813, suggesting that SVM may rely more heavily on the complete feature representation for this corpus. Random Forest showed marginal reductions as well, although overall trends remained relatively stable.

Overall, the results indicate that the proposed feature selection framework generalizes across multiple classifiers while providing the most consistent benefits when combined with XGBoost. The improvements are particularly noticeable in datasets with limited sample sizes or higher class confusability, whereas gains remain minimal in highly separable datasets where baseline performance is already saturated.

The class-wise evaluation results in Table 3 demonstrate the effectiveness of the proposed feature selection framework when combined with the XGBoost classifier across four benchmark speech emotion recognition datasets. On the CREMA-D dataset, the model shows relatively balanced performance, with the highest F1-score observed for angry (0.72), followed by sad (0.61) and neutral (0.59). However, the model exhibits comparatively lower performance for disgust (0.53) and fear (0.54), indicating that these emotions remain more challenging to distinguish due to their acoustic similarity with other emotional states. Among the evaluated classifiers, XGBoost provided the most consistent overall improvements after feature selection and therefore serves as the primary model for detailed class-wise analysis. For the RAVDESS dataset, the model achieves improved overall performance, with strong results for fear (F1 = 0.79) and surprise (F1 = 0.79), while neutral achieves the highest F1-score of 0.82 due to its very high recall (0.95). In contrast, sad shows relatively lower recall (0.44), resulting in a lower F1-score (0.54).

On the SAVEE dataset, the model demonstrates strong recognition capability, particularly for neutral (F1 = 0.86) and sad (F1 = 0.77), while disgust remains the most difficult class with the lowest F1-score (0.47). The higher recall for happy (0.78) also contributes to a competitive F1-score of 0.68. Finally, the TESS dataset shows near-perfect classification performance across all emotion categories, with F1-scores reaching 1.00 for most classes and 0.99 for surprise. This indicates that the proposed model generalizes extremely well on datasets with clearer emotional articulation and less acoustic variability. Overall, the results highlight the robustness of the proposed approach while also revealing that certain emotions, particularly disgust and fear, remain comparatively harder to recognize in more complex datasets.

The confusion matrices further illustrate the class-wise behavior of the proposed XGBoost-based model across the four datasets, as shown in Fig. 3. For CREMA-D, most correct predictions are concentrated along the diagonal, indicating reasonable classification performance. However, several misclassifications are observed, particularly between angry and disgust (0.246), suggesting that these emotions share similar acoustic characteristics. Additionally, sad is frequently confused with neutral (0.368), which explains the relatively lower recall for the sad class. Despite these confusions, neutral (0.953) and surprise (0.845) demonstrate strong recognition rates.

Fig. 3.

Confusion matrices of the proposed XGBoost-based model on four speech emotion recognition datasets: a CREMA-D, b RAVDESS, c SAVEE, and d TESS

In the RAVDESS dataset, the confusion matrix shows generally strong class separability. The fear (0.793) and disgust (0.810) classes achieve high correct classification rates, while neutral also demonstrates strong recognition performance. Some confusion still occurs between angry and disgust, as well as between happy and surprise, indicating overlapping vocal expressions in these emotional states. Overall, most values remain concentrated along the diagonal, reflecting stable model performance across the majority of emotion classes.

For SAVEE, the confusion matrix reveals more noticeable misclassification among certain emotions due to the smaller dataset size and speaker variability. For example, fear is often confused with happy (0.222), and disgust is split between disgust and neutral (0.389 each). Nevertheless, neutral is perfectly recognized (1.000), and happy (0.778) and sad (0.667) maintain relatively strong recognition rates compared to other classes.

Finally, the TESS confusion matrix demonstrates near-perfect classification performance. Almost all predictions lie on the diagonal, with most classes achieving 1.000 correct recognition, and only very minor confusion appearing in a few instances (e.g., disgust and happy at 0.992). This result indicates that the proposed model can effectively capture discriminative emotional features when the dataset contains clearer and more consistent speech patterns. Overall, the confusion matrix analysis confirms the robustness of the proposed model while highlighting the remaining challenges in distinguishing acoustically similar emotions in more complex datasets. Due to space limitations, class-wise results for SVM and Random Forest are not included, although overall performance comparisons are provided in Table 2.

Analysis of selected features

The selected feature sets show strong consistency across datasets and seeds. Table 4 summarizes the selected feature subset sizes (K), dominant feature groups, and representative top-ranked descriptors identified across the evaluated corpora.

Table 4.

Dominant feature groups and representative top-ranked descriptors across datasets

Dataset	Selected K	Dominant feature groups	Representative top-ranked features
CREMA-D	160	Spectral, Intensity, Voice Quality	mfcc01_q3, mfcc03_std, mean_hnr, median_intensity, q1_pitch
RAVDESS	120	Prosodic, Spectral, Temporal Dynamics	pitch_slope_without_octave_jumps, mfcc02_std, voiced_fraction, spectral_contrast, stddev_pitch
SAVEE	160	Prosodic, Intensity, Spectral	q1_pitch, mean_pitch, median_intensity, mfcc03_median, rms
TESS	80	Prosodic, Temporal Dynamics, Spectral	q1_pitch, q3_pitch, band_energy_difference, spectral_contrast, mfcc20_mean

The feature ranking analysis demonstrates that the proposed framework consistently prioritizes acoustically meaningful descriptors across datasets and selected feature subset sizes (K). Spectral and MFCC-based features were frequently selected in all corpora, while prosodic and temporal-dynamics descriptors became more prominent in RAVDESS and TESS. Voice-quality measures such as HNR and glottal-noise-related features were also repeatedly selected, indicating their importance for emotion characterization. The results further show that different datasets favor different compact feature subset sizes, reflecting variations in corpus complexity, speaker diversity, and class separability.

Notably, even in TESS, where baseline performance is near ceiling, the method still achieves marginal but consistent improvements, indicating that the selection process does not harm performance even in highly controlled settings.

Discussion

Effectiveness of explainability-guided feature selection

The experimental results demonstrate that explainability signals derived from SHAP values, when combined with permutation-based importance, can serve as an effective model-driven criterion for feature selection rather than merely a post-hoc interpretation mechanism. Across the four evaluated datasets, the proposed method consistently maintained or slightly improved classification performance while operating on substantially reduced feature subsets.

In the within-corpus setting, modest but consistent improvements were observed on CREMA-D and SAVEE, while improvements on RAVDESS remained relatively modest. On the TESS dataset, which already exhibits near-perfect separability, the proposed feature selection framework still achieved marginal improvements despite the baseline performance being close to the theoretical ceiling. These results indicate that the explainability-guided ranking successfully identifies acoustically informative features while removing redundant or weakly contributing descriptors.

Importantly, the selected features show strong alignment with well-known emotion-relevant acoustic characteristics. Frequently selected descriptors include pitch statistics, MFCC-based spectral features, intensity measures, and voice-quality indicators such as harmonic-to-noise ratio. This confirms that the proposed framework preserves interpretable acoustic cues rather than selecting features arbitrarily.

These observations indicate that the contribution extends beyond simple dimensionality reduction. Instead, the proposed framework reframes feature selection as an explainability-guided optimization process embedded directly within the SER pipeline.

Multi-model evaluation and feature generalization

While the within-corpus experiments demonstrate consistent performance improvements for XGBoost, the additional experiments using Support Vector Machine (SVM) and Random Forest (RF) provide further insight into the generalizability of the proposed feature selection framework across different classifiers.

The results indicate that the proposed feature selection consistently improves XGBoost performance across all evaluated datasets. In particular, noticeable gains are observed on smaller or more challenging datasets such as SAVEE and RAVDESS, suggesting that removing redundant and less informative acoustic features can improve discriminative capability and reduce overfitting.

For SVM, the impact of feature selection is more dataset-dependent. Slight improvements are observed in SAVEE, while performance on RAVDESS decreases marginally after feature reduction. This behavior suggests that SVM may rely more strongly on the complete high-dimensional feature representation in certain datasets. Random Forest demonstrates modest but generally stable improvements following feature selection, although its overall performance remains lower than XGBoost and SVM.

The consistently high performance observed on TESS across all classifiers indicates that the dataset is highly separable and contains relatively clear emotional distinctions, leaving limited room for measurable improvement through feature reduction. In contrast, larger improvements on CREMA-D and SAVEE suggest that the proposed framework is particularly beneficial in datasets with higher acoustic variability or limited sample sizes.

Overall, the findings demonstrate that the proposed explainability-driven feature selection framework generalizes across multiple learning paradigms while providing the most stable and consistent benefits when combined with XGBoost. These results further support the effectiveness of combining SHAP attribution, permutation importance, and redundancy-aware ranking for compact and interpretable SER systems.

Summary of findings

The experimental evaluation leads to several key observations:

• The proposed explainability-guided feature selection framework consistently preserves or slightly improves recognition performance across four widely used SER corpora.

• Selected features correspond closely to well-established acoustic indicators of emotion, including pitch statistics, MFCC descriptors, and voice-quality features.

• The proposed framework demonstrates consistent effectiveness across multiple classifiers, while the magnitude of improvement varies depending on dataset characteristics and classifier behavior.

• The framework offers a transparent and interpretable alternative to conventional dimensionality-reduction techniques such as PCA, which transform features into latent spaces that are difficult to interpret.

Overall, the results support the use of explainability-guided feature selection as a practical and interpretable strategy for building compact and reliable speech emotion recognition systems.

Conclusion

This study proposed an explainability-guided feature selection framework for speech emotion recognition that integrates SHAP-based feature attribution with permutation importance to identify informative acoustic descriptors. Unlike conventional dimensionality reduction methods such as principal component analysis, the proposed approach retains the original feature space and therefore preserves interpretability while enabling model-driven feature selection.

Experimental evaluation was conducted on four widely used emotional speech datasets (CREMA-D, RAVDESS, SAVEE, and TESS) under within-corpus evaluation settings. The results show that the proposed framework can maintain or improve classification performance while reducing the number of handcrafted acoustic features. Across multiple classifiers, including XGBoost, SVM, and RF, the selected-feature models achieved comparable or improved Accuracy, Macro-F1, and UAR relative to full-feature baselines, demonstrating classifier-independent effectiveness of the proposed framework. The most consistent improvements were observed with XGBoost, particularly on CREMA-D, SAVEE, and TESS. The selected feature subsets consistently retained acoustically meaningful descriptors, including MFCC-based spectral features, pitch statistics, intensity measures, and voice-quality indicators.

Overall, the findings demonstrate that explainability-guided feature selection offers a transparent and effective alternative to traditional dimensionality reduction approaches, enabling compact and interpretable SER models without sacrificing predictive performance.

Future work will investigate integrating the proposed framework with deep neural speech representations and adaptive model selection strategies to further improve robustness and applicability in modern SER systems.

Funding

This work was supported by Inha University and Basic Science Research Program (RS-2024-00348937) by National Research Foundation of Korea (NRF).

Data availability

Datasets are publicly available on the internet.

Declarations

Conflict of interest

The authors declare no conflicts of interest.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

AI usage

Generative AI tools were used for language refinement and clarity improvement. All scientific content, methodology, analysis, and conclusions were developed and validated by the authors.