A hybrid stacked ensemble learning framework for multilabel text emotion detection

Abstract

Understanding emotions in text is an important part of sentiment analysis, especially in areas like mental health monitoring, customer feedback analysis, hate speech and disaster management. Unlike basic sentiment analysis that classifies text as positive or negative, real human emotions are complex and often overlapping, requiring multi-label classification to accurately capture multiple emotional states within a single input. While transformer-based models have improved performance, challenges persist particularly in low-resource languages and culturally diverse contexts due to the scarcity of annotated data and the difficulty in generalizing in different languages. This study proposes Hyb-Stack, a hybrid stacked ensemble framework that integrates simple stacking and cross-validation stacking, combining predictions from four transformer-base models (BERT, DistilBERT, RoBERTa, and mBERT) using a Random Forest meta-classifier to enhance classification accuracy, adaptability, and cross-lingual generalisation. Hyb-Stack was evaluated on three datasets: a high-resource English corpus (SemEval-2018 Task 1 E-c) and two low-resource corpora, the Bahasa Indonesia hate speech and HaEmoC_V1, a newly constructed Hausa-language emotion corpus developed to address the lack of annotated data for this language. Experimental results demonstrate that mBERT outperforms individual base models, achieving F1-scores of 82.28 (HaEmoC_V1), 85.33 (Bahasa Indonesia), and 88.90 (SemEval-2018 English). Also, the EM-9 ensemble set (BERT + DistilBERT + mBERT) improves performance, yielding F1-scores of 89.48, 88.19, and 90.67 on the respective datasets, which surpasses both individual models and conventional ensemble strategies such as averaging and weighted averaging. These findings highlight the effectiveness of combining multiple transformers with an optimized decision layer to advance multi-label emotion classification on diverse linguistic contexts.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-026-38172-9.

Introduction

Emotion detection in text has become an essential aspect of sentiment analysis in various domains, including customer feedback analysis, mental health monitoring, and conversational agents¹. Unlike traditional sentiment analysis, which mainly categorizes texts as positive, negative, or neutral^2,3 emotion classification aims to identify a broader range of human emotions, such as joy, anger, sadness, fear, and surprise⁴. This deeper understanding of emotions is especially useful in areas like disaster management response or therapy support, where emotional context can guide timely and appropriate interventions⁵. By recognizing specific emotions, systems can prioritize urgent messages, offer personalized support, or trigger alerts for human intervention when necessary⁶. This is very important in serious situations where quick and accurate emotional understanding can help save time or even lives. This level of detail enables systems to more accurately interpret user intent, contextual meaning, and emotional states, thereby enhancing their responsiveness and intelligence⁷.

With the advancement of Natural Language Processing (NLP), various models have been developed to detect emotions in text, particularly in English and other high-resource languages⁸. Pre-trained transformer models like BERT and RoBERTa have achieved state-of-the-art results in various classification tasks, including sentiment and emotion detection⁹. However, despite these advancements, the benefits of such models are not equally distributed across all languages. Many low-resource languages lack sufficient annotated corpora and NLP tools, limiting the effectiveness and adaptability of emotion detection systems in multilingual and culturally diverse contexts¹⁰. This technology gap makes it difficult to create emotion detection tools in areas where they are most needed, like during emergencies or in communities with fewer resources. This has led to the marginalization of speakers of under-represented languages in the adoption of emerging artificial intelligence (AI) technologies.

Multi-label emotion detection increases the difficulty of the task, as a single input instance may simultaneously express multiple emotions. This requires models to accurately identify and classify overlapping emotion labels within the same textual context¹¹. Traditional single-label classification struggles with this task, as it cannot capture the difficulty and overlap of human emotions¹². Moreover, ambiguity in language, cultural variations in emotional expression, and the subjective nature of emotion perception further complicate the tasks¹³. These challenges require smarter methods, like deep learning, which can better understand the emotional meaning behind words¹⁴. Also, the lack of annotated datasets for multi-label emotion tasks, especially in low-resource languages, limits the performance and generalizability of existing systems¹⁵.

In the NLP domain, ensemble learning has been shown to improve both performance consistency and predictive accuracy¹⁶. Among the most prominent techniques are bagging, boosting, and stacking¹⁷. Their process and standard workflow are illustrated in Supplementary Fig. S1. Bagging methods, like Random Forests, aggregate predictions from multiple learners trained on bootstrapped datasets¹⁸. Boosting methods, such as AdaBoost and XGBoost, focus on sequentially correcting errors made by previous learners¹⁹. Stacking, in contrast, employs a meta-learner to combine the predictions of several base models, learning how to best integrate their outputs^20,21. By leveraging diverse models, ensemble approaches often achieve higher accuracy and stability compared to single models. They can also increase computational complexity and require careful tuning to maximize performance²².

Most ensemble strategies used in emotion detection are either limited to simple voting mechanisms or lack the architectural flexibility to adapt across multiple languages and resource levels²³. Hybrid ensemble methods, which integrate stacking with other ensemble paradigms, remain underexplored despite their potential to harness both diversity and coordination among base learners²⁴. Current research lacks ensemble models that are specifically optimized for multi-label emotion detection while also being adaptable to linguistically diverse contexts. Developing such models could significantly advance emotion detection capabilities, especially in complex and multilingual contexts. This would allow for better handling of overlapping emotions and variations in language use across different communities²⁵.

To address the limitations of existing models in multilingual and low-resource contexts, this paper proposes Hyb-Stack, a hybrid stacked ensemble framework for multi-label emotion detection. The architecture integrates simple stacking and cross-validation stacking, combining predictions from four transformer-base models (BERT, DistilBERT, RoBERTa, and mBERT) using a Random Forest meta-classifier to enhance classification accuracy, adaptability, and cross-lingual generalisation. This design enables effective emotion detection on both high-resource and low-resource languages, demonstrating strong adaptability to diverse linguistic and cultural contexts. Beyond its architectural novelty, Hyb-Stack provides a scalable and modular framework for building emotion recognition systems in both monolingual and multilingual environments. This flexibility allows researchers and practitioners to easily extend or customize the framework according to specific language needs or dataset characteristics. Such adaptability is crucial for addressing the wide variability found in emotional expressions on different languages and cultures.

This paper is structured as follows: “Related work” section provides a comprehensive review of related work in emotion detection and ensemble learning, establishing the foundational context for our research. “Methodology” section elaborates on the proposed Hyb-Stack methodology, including its architecture and key innovations. “Experimental results and analysis” section present the experimental setup, results, and a detailed analysis of the findings. Finally, “Conclusion” section concludes the paper by discussing the limitations of the current study and outlining potential directions for future research.

Related work

Emotion detection has evolved significantly, moving from traditional machine learning approaches to deep learning and transformer-based methods⁴. Early research primarily focused on sentiment polarity (positive, negative, neutral), but emotion classification requires finer distinctions across multiple categories such as joy, anger, sadness, fear, and surprise²⁶. Understanding these categories is essential because many real-world applications, including customer support, therapy assistance, and disaster management, demand systems that capture emotional representation rather than broad sentiment categories. Frameworks like Plutchik’s Emotion Wheel shown in Supplementary Fig. S2 provide theoretical grounding, illustrating how primary emotions combine to form complex states such as optimism or remorse²⁷. These psychological models highlight the importance of designing computational systems that can recognize co-occurring or overlapping emotions, especially in multilingual and noisy contexts.

Traditional machine learning methods such as Support Vector Machines (SVMs), Naïve Bayes, and Logistic Regression were widely applied in early emotion detection tasks^28,29. While effective for basic sentiment classification, they relied heavily on standard features and lexicons, which limited adaptability across domains and languages³⁰. To improve performance, researchers turned to ensemble methods such as bagging, boosting, and stacking¹⁷. Bagging techniques like Random Forests aggregated multiple learners to reduce overfitting³¹, while boosting approaches such as AdaBoost and XGBoost iteratively corrected classification errors³². Stacking combined base learners through a meta-classifier, offering improved generalization²⁰. Although these methods provided performance gains, they struggled with fine-grained emotion recognition and multi-label dependencies, where multiple emotions may co-occur in a single text³³.

The introduction of deep learning marked a turning point for emotion detection. Convolutional Neural Networks (CNNs) demonstrated strong capabilities in capturing local textual features, while Recurrent Neural Networks (RNNs) and their variants such as LSTMs and GRUs excelled at modelling sequential dependencies³⁴. Hybrid deep ensembles further improved performance by combining multiple architectures. For example, ensembles of CNNs and LSTMs captured different aspects of emotional semantics, improving robustness across diverse datasets³⁵. Despite these advances, traditional deep models faced limitations such as high computational costs, difficulties with long-range dependencies, and reduced generalization to unseen linguistic or cultural contexts³⁶. Moreover, most studies focused on single-label emotion recognition, which oversimplifies the complex and overlapping nature of human affect¹⁵.

Transformer-based architectures revolutionized natural language processing and pushed emotion detection to new levels of accuracy. Models such as BERT, RoBERTa, and XLNet, leveraging self-attention mechanisms, enabled richer contextual representation learning³⁷. These models steadily outperformed traditional deep learning systems on benchmark emotion datasets. Lightweight alternatives like DistilBERT³⁸ and fine-tuning techniques such as adapters offered efficiency gains, though often at the cost of reduced performance in complex tasks. Ensemble learning with transformer³⁸ emerged as a strategy to further enhance reliability and reduce variance. Studies showed that stacking multiple fine-tuned transformer variants improved generalization and robustness³⁹. However, most transformer-based ensembles to date have been restricted to high-resource languages like English and Chinese, single-label context, and computationally expensive pipelines²⁵.

Multi-label emotion detection introduces unique challenges that further expose the limitations of existing models. A single sentence can express multiple overlapping emotions, requiring systems to capture label correlations and dependencies¹¹. Traditional methods such as Binary Relevance treated labels independently, while classifier chains and label powerset methods modelled interdependencies but suffered from scalability and overfitting in high-dimensional label spaces^40,41. Ensemble strategies like RAkEL improved performance by decomposing label spaces into random subsets⁴², yet their reliance on shallow base learners restricted adaptability. Deep ensembles for multi-label emotion detection, including CNN–LSTM hybrids, demonstrated improved accuracy but were rarely tested in low-resource languages. Data scarcity, cultural variations in emotional expression, and class imbalance continue to limit the effectiveness of existing approaches^24,43,44.

Recent work has explored multilingual transformers such as mBERT and XLM-R to extend emotion detection to low-resource languages⁴⁵. These models exploit cross-lingual transfer learning, allowing systems trained on high-resource languages to generalize to underrepresented ones. While promising, existing multilingual approaches often overlook ensemble diversification, and stacking is rarely explored beyond shallow classifiers²⁰. Furthermore, meta-learners in stacking commonly logistic regression struggle to capture non-linear dependencies among labels, limiting performance in multi-label contexts²⁴.

Table 1 outlines key studies on ensemble and hybrid models, highlighting their base models, datasets, target languages, tasks, performance metrics, and key findings. It emphasizes the diversity of approaches and identifies gaps in multi-label and low-resource contexts addressed in this study.

Table 1.

Summary of representative ensemble and deep learning approaches for emotion and sentiment analysis.

Authors	Ensemble type	Base models	Dataset	Language	Task	Performance metrics
46	Deep Transfer Learning	RNN (customized), Sent2Affect	6 benchmark datasets	English	Text-based Emotion Recognition	F1 score = 70
47	Joint-Encoder (not ensemble)	Transformer + Co-attention	CMU-MOSEI	English	Emotion + Sentiment	F1 score = 88.19
48	Dual-Modality Fusion	LSTM + Sentiment Vectors	UCI Sentiment, Suicide Notes	English	Emotion Classification	Accuracy = 86.3%
49	Stacking	CNN, LSTM, BiLSTM, GRU + LR	Real-world ABSA datasets	English	Aspect-Based Sentiment Analysis	Accuracy = 69%
50	Transformer + Adapter Layers	Transformer	CMU-MOSEI	English	Emotion Detection	Accuracy = 95%
51	Not ensemble	BERT, RoBERTa, ELECTRA, XLM-R	Harnessing Twitter	English	Fine-Grained Emotion Detection	Accuracy = 93.41%
52	Neural Ensemble	BERT, ALBERT, RoBERTa, 2D CNN, Wav2Vec 2.0	MELD	English	Bimodal Sentiment Analysis	Accuracy = 77.0
53	Hybrid Deep Model	XLNet, BiGRU, Attention	IEMOCAP, CASIA	English, Chinese	Text Emotion Recognition	F-measure = 89.23 Accuracy = 91.71%
54	Ensemble (EmoDNN)	Deep NNs + Dropout CNNs	Short Texts	English	Multi-label Emotion Recognition	F-measure = 85.8 Accuracy = 85.8%
55	Transfer Learning	RoBERTa-large	11 emotion datasets	English	Emotion Detection	F1-score = 0.84
56	Hybrid Attention Model	BERT, CNN, BiGRU + Att	Hotel Reviews, Weibo	Chinese	Sentiment Analysis	F1-score = 88.1%
57	Not ensemble (semi-supervised)	CorMulT Transformer	CMU-MOSEI	English	Multimodal Sentiment Analysis	F1-score = 86.7%

The reviewed studies in Table 1 demonstrate considerable progress in ensemble and deep learning methods for emotion and sentiment analysis, but important gaps remain. A major limitation is the dominance of English, with only a few works extending to Turkish and Chinese, leaving low-resource languages such as Hausa largely unexplored. Research has also concentrated more on sentiment polarity classification than on fine-grained or multi-label emotion recognition, despite evidence that real-world texts often convey overlapping emotions. This imbalance restricts the adaptability of existing models in contexts where multiple emotions co-occur, such as social media communication.

To address these challenges, this study proposes Hyb-Stack, a hybrid stacked ensemble framework that combines simple stacking with cross-validation-based stacking to integrate diverse transformer models, including BERT, RoBERTa, DistilBERT, and mBERT. A Random Forest meta-learner is employed to capture non-linear label dependencies within the ensemble. The effectiveness of Hyb-Stack is evaluated on both high-resource (English) and low-resource (Hausa and Bahasa Indonesia) datasets, demonstrating its scalability and adaptability for multilingual emotion detection tasks. The proposed Hyb-Stack ensemble learning architecture is presented in “Methodology” section.

Methodology

In this section, we present a systematic methodology for developing multi-label emotion classification system using transformer-based models and ensemble learning techniques. The system is designed with both high-resource and low-resource text emotion classification datasets strategically partitioned into training, validation, and testing sets. Four transformer architectures were selected as base models based on their complementary characteristics: BERT provides robust bidirectional context understanding through its original transformer architecture⁵⁸, while RoBERTa builds upon this foundation with optimized pre-training procedures⁵⁹. The system also incorporates DistilBERT as a computationally efficient alternative that maintains competitive performance through knowledge distillation³⁸, alongside mBERT which, despite its multilingual design, is included to evaluate architectural consistency in English emotion classification³⁷. The methodology follows a structured pipeline consisting of data partitioning, feature embedding, base model training, and comprehensive performance evaluation as shown in Fig. 4.

Fig. 4.

Modelling of the Transformer-base models.

Dataset description

This study utilizes three datasets encompassing both high-resource and low-resource languages to evaluate the effectiveness of the proposed hybrid stacked ensemble learning framework (Hyb-Stack). The inclusion of diverse linguistic and cultural contexts enables a comprehensive and robust assessment of the model’s performance. Table 2 summarizes the datasets, specifying their language categories, sizes, and sources. Notably, the Hausa Emotion Corpus (HaEmoC_V1) was specifically developed in this study to address the paucity of annotated Hausa emotion datasets available for natural language processing (NLP) tasks.

Table 2.

Description of the datasets used in this work.

Authors	Datasets	High-resource language	Low-resource language	Train	Dev	Test	Total samples
60	SemEval-2018 Task 1 (Task E-c)	English	–	6838	886	3259	10,983
61	Indonesia Abusive and Hate Speech Twitter Text	–	Malaya–Indonesia	–	–	–	3293
Our work	Hausa Emotion Corpus (HaEmo_V1)	–	Hausa	–	–	–	12,811

Figure 1 presents the label distributions for the three datasets, HaEmoC-V1 (Hausa), Bahasa Indonesia Hate Speech Dataset, and the SemEval-2018 Task 1 English Emotion Dataset, with the visual comparison highlighting both the internal characteristics of each dataset and the cross-linguistic differences in emotional or toxicity-related expression across platforms.

Fig. 1.

Distribution of labels in datasets.

The three plots illustrated in Fig. 1 reveal that each dataset embodies unique statistical and linguistic characteristics as HaEmoC-V1 provides broad emotional diversity, the Indonesia dataset concentrates on varying intensities and targets of abusive content, and SemEval offers a moderately imbalanced yet rich emotional spectrum, distinctions which underscore the importance of tailoring preprocessing, modelling strategies, and evaluation approaches to the essential properties of each dataset.

Data acquisition and development of HaEmoC_V1

This section describes the dataset acquisition and organization process for the Hausa Emotion Corpus (HaEmoC_V1). The workflow, depicted in Fig. 2, outlines the key steps involved in data collection, pre-processing, verification, annotation and final organisation.

• A.Dataset Acquisition: The dataset was collected using the Tweepy API, which enabled the retrieval of tweets from native Hausa speakers on Twitter. This process resulted in a corpus of 19,200 raw tweets, containing unstructured data such as text, hashtags, URLs, emoji, and other social media-specific elements. Given the noisy nature of Twitter data, systematic filtering and pre-processing were necessary to ensure the quality and relevance of the dataset.
• B.Data Preparation and Pre-processing: To refine the dataset, multiple pre-processing steps were undertaken. First, text filtering was applied by removing URLs, hashtags, numbers, punctuation marks, and non-Hausa words to focus on the core textual content. Next, keyword-based filtering was used to prioritize tweets containing specific keywords related to disaster relief and emergency response (e.g., Tallafin Gaggawa, Tallafin kayan abinci, Tallafin Annoba), thereby enhancing relevance. Finally, tokenization and stopword removal were carried out, where the text was split into individual words or tokens, and common Hausa stopwords (e.g., da, ta, shi) were removed to eliminate non-informative elements.
• C.Dataset Verification: After completing the pre-processing stage, the initial labelling was conducted by 11 selected postgraduate students from different Universities in Northern part of Nigeria, all of whom are native Hausa speakers. Each annotator was assigned to label only one specific emotion category throughout the dataset. For example, an annotator X could only annotate anger or sadness but not multiple emotions simultaneously. To ensure consistency, the authors developed annotation guidelines to assist the annotators in maintaining accuracy during the labelling process. Following the initial annotation phase, the authors engaged three expert annotators specializing in text emotion analysis and fluent in Hausa to review and verify the initial labels. To determine the final emotion labels for each instance, a majority voting approach was employed, ensuring the reliability and objectivity of the dataset.
• D.Quality of Annotation: To assess the reliability of the multi-label annotations, two widely used inter-annotator agreement metrics were applied: Jaccard Coefficient and Fleiss’ Kappa. The overall agreement among the three expert annotators resulted in a Jaccard Coefficient score of 84% and a Fleiss’ Kappa score of 82%, indicating a high level of consistency and accuracy in the dataset annotations. These results confirm the high quality and reliability of the HaEmoC_V1 dataset for emotion classification tasks.
• E.Data Categorization and Final Dataset Construction: After pre-processing, the dataset was further refined and reduced to 12,811 cleaned tweets. These tweets were then manually annotated and categorized into predefined emotion classes, including; Anger, Sadness, Disgust, Fear, Surprise, Joy, Trust, Optimism, Pessimism, Anticipation, Neutral. These categorized datasets were essential for training transformer-based models in emotion classification, ensuring a high-quality benchmark for low-resource language sentiment analysis. The entire dataset preparation framework is illustrated in Fig. 2.

Fig. 2.

Dataset acquisition framework.

HaEmoC_V1 statistics

This section provides a detailed statistical overview of the Hausa Emotion Corpus (HaEmoC_V1), a multi-label dataset designed for emotion classification in the Hausa language. The dataset consists of 12,811 tweets, each labelled with one or more of 11 predefined emotion categories making the dataset suitable for multi-label emotion detection tasks. HaEmoC_V1 is structured as a textual dataset with no missing values, ensuring high data integrity for machine learning applications. Each instance includes a unique ID, a tweet text, and binary emotion labels, where 1 indicates the presence of an emotion and 0 indicates its absence. The dataset plays a crucial role in benchmarking transformer-based models for low-resource language processing, as shown in Table 3.

Table 3.

Dataset description of Hausa Emotion Corpus (HaEmoC_V1).

Corpus properties	HaEmoC_V1 description
Dataset characteristics	Textual dataset
Attribute characteristics	Text: Tweet content (string) Emotions: 11 binary attributes (0 or 1) for anger, sadness, disgust, fear, surprise, joy, trust, optimism, pessimism, anticipation, and neutral
Total number of instances	12,811
Missing values	No missing values
Total number of attributes	13 (1 text column + 11 emotion columns + 1 ID column)

Sample of HaEmoC-V1 dataset

Table 4 presents some samples of the instances from the newly constructed (HaEmoC-V1) dataset, showcasing its structure and labelling format. Each sample consists of a tweet in Hausa alongside its corresponding binary emotion labels, where 1 signifies the presence of an emotion and 0 indicates its absence. These examples help illustrate the dataset’s suitability for multi-label emotion classification in low-resource NLP tasks. Figure 3 illustrates the statistical analysis of the emotions in each category.

Table 4.

Sample of HaEmoC-V1.

ID	Tweet	English translation	ang	sad	dis	fea	sur	joy	tru	opt	pes	ant	neu
1	BBC idan bakuda abin posting ku saka mana hoton buhari mu zazzageshi mana	BBC if you have nothing to post, upload Buhari’s photo so we can bash him	1	1	1	1	0	0	0	0	1	0	0
2	@user @user Toh fah inji 'yan magana suka ce """"ana wata ga wata""""????	Well then, as the saying goes, “while one is happening, another is coming”????	0	0	0	0	1	0	0	0	0	0	1
3	tohh allah shi taimaka	Well, may God help	0	0	0	0	0	1	1	1	0	1	0
4	@user Lolz?? ta aure shi ko ya aureta!? VOA hausa una no get news again ??! Haba since last year!	Lol?? Did she marry him or he married her!? VOA Hausa, don’t you have news anymore?? Come on, since last year!	1	1	0	0	0	0	0	0	0	0	0
5	@user Ana ta diban palliatives????	They are still looting the palliatives????	0	0	1	1	0	0	0	0	0	0	0

ang, Anger; sad, Sadness; dis, Disgust; fea, Fear; sur, Surprise; joy, Joy; tru, Trust; opt, Optimism; pes, Pessimism; ant, Anticipation; neu, Neutral.

Fig. 3.

Dataset correlation.

Dataset correlation

This section reports the correlation analysis conducted on the HaEmoC_V1 dataset to examine the interrelationships among emotion labels. Figure 3 presents the correlation matrix, where darker shades denote stronger associations between emotions. In particular, trust and optimism (0.9) as well as pessimism and anger (0.8) exhibit strong positive correlations, whereas joy and sadness (− 0.32) show a moderate negative correlation. These findings provide valuable insights into patterns of emotion co-occurrence, which are particularly relevant for enhancing the performance of multi-label emotion classification models.

Transformer-based modelling framework

The proposed architecture illustrated in Fig. 4 presents a multi-label emotion classification framework that accommodates both high-resource and low-resource languages. The model utilizes datasets comprising English for high-resource language representation and HaEmoC_V1 and the Indonesia Hate Speech corpus for low-resource languages. The data is initially divided into training, validation, and testing sets. The actual training set undergoes feature embedding and is processed using a Binary Cross-Entropy (BCE) with Logit Loss Function, which is suitable for handling multi-label classification problems. The system then fine-tunes four transformer models specifically BERT, RoBERTa, DistilBERT, and mBERT as base models. Each model is trained independently, and their predictions are stored for ensemble learning approach. The ensemble combines the strengths of individual models to enhance overall performance and generalization. Finally, the performance of the models is evaluated using standard metrics including accuracy, precision, recall, and F1-score, providing a comprehensive assessment of the model’s effectiveness across different linguistic contexts.

Baseline transformer models

The baseline transformer models used in this study vary in size and architectural complexity, reflecting differences in their number of parameters and design. BERT-base consists of approximately 110 million parameters, featuring 12 transformer encoder layers, a hidden size of 768, and 12 self-attention heads. DistilBERT, a compressed variant of BERT, contains around 66 million parameters by reducing the number of encoder layers to six while retaining a hidden size of 768 and 12 attention heads. This reduction results in a lighter model, enabling faster training and inference. RoBERTa-base shares a similar architecture with BERT-base but benefits from optimized pre-training strategies, comprising approximately 125 million parameters, 12 layers, a hidden size of 768, and 12 attention heads. mBERT (Multilingual BERT-base) also has roughly 110 million parameters and the same architectural configuration as BERT-base, but it is pre-trained on multilingual corpora covering over 100 languages. These model sizes offer a balance between representational capacity and computational efficiency, facilitating fair comparison and effective ensemble integration across both high-resource and low-resource language scenarios.

Ensemble-based approach

After fine-tuning and evaluating the pre-trained transformer models individually as shown in Fig. 4, we applied multiple ensemble strategies including average and weighted average methods alongside our proposed hybrid stack ensemble learning (Hyb-Stack) approach. The Hyb-Stack method combines two stack ensemble techniques: simple stacking and cross-validation-based stacking, enabling a comprehensive comparison of ensemble performance. Hyb-Stack framework consists of three primary layers. First, the base learner layer consists of fine-tuned transformer models that generate probability distributions for each emotion label. Their predictions are then aggregated in the ensemble combination layer, where various strategies were systematically explored to maximize model diversity. Finally, the meta-classifier synthesizes these ensemble outputs to produce the final multi-label predictions. Equation 1 describes how the ensemble sets were generated using a systematic combination of the four models employed in this research.

1

where is the total number of base models; is the number of instances.

Hence, 11 ensemble method (EM) combinations were generated, designated EM-1 to EM-11. Each ensemble configuration generates a set of logits across the predefined emotion classes, and different aggregation techniques are applied for prediction.

Average ensemble approach

In the average ensemble, we average the logits output for each test instance across models. The final predicted class is the one with the maximum average logit value. This is computed as shown in Eq. 2 and Algorithm 1:

2

where bce_logit_ij represents the raw logit of the jth model for the ith instance. The final class is assigned by taking the index of the maximum averaged value.

Algorithm 1.

Average ensemble method

Weighted average ensemble approach

In the weighted average ensemble, each model’s logits are multiplied by its corresponding weighted F1-score before aggregation. The weighted average is presented in Eq. 3 and Algorithm 2:

3

where is the weighted F1-score of models , emphasizing high-performing models in the final prediction.

Algorithm 2.

Weighted average ensemble method

Hybrid stack ensemble (Hyb-Stack) approach

This is the proposed ensemble methods in this research. In this approach, the logits of all base models (BERT, DistilBERT, RoBERTa, and mBERT) are stacked into a feature vector, which is then passed to a meta-classifier specifically Random Forest. This model is trained on validation logits to learn the optimal combination of base model outputs. The Hyb-Stack output is calculated using Eq. 4:

4

where meta_clf is the trained meta-classifier and Concat represents vector concatenation of logits from all base models.

The full architecture in Fig. 5 and the workflow in Algorithm 3 illustrate the development from base-model predictions to meta-learner improvement, ensuring strong and adaptable emotion classification.

Algorithm 3.

Hyb-stack ensemble

Fig. 5.

Modelling using the proposed ensemble learning approach (Hyb-Stack).

Experimental results and analysis

In this section, we present an in-depth analysis of the models’ performance on the three datasets used in this study. The aim is to evaluate the effectiveness of the proposed hybrid stacked ensemble learning (Hyb-Stack) approach by comparing its performance with standard transformer models used as base learners, as well as with existing ensemble methods such as average and weighted ensemble learning. The results cover both low-resource and high-resource language datasets, enabling a comprehensive assessment of the model’s generalization capabilities. Through detailed error analysis, evaluation of ensemble contributions, and comparisons with prior studies, this section highlights the empirical strengths and practical advantages of the proposed system.

Experimental setup and evaluation metrics

The experiments were conducted on Google Colab Pro using a T4 or V100 GPU based on their availability (16–40 GB VRAM), supported by 16 GB RAM and an Intel Core i7-10110U CPU (3.10 GHz base, 3.59 GHz boost) to accelerate training. The implementation used Python 3.8 with key libraries: PyTorch for deep learning, HuggingFace Transformers for pre-trained language models, and NumPy/scikit-learn for data processing and evaluation. All models were trained with the AdamW optimizer (learning rate = 2e−5) and a batch size of 16, balancing computational efficiency and performance. Training proceeded for 11 epochs with early stopping (monitoring validation loss) to avoid overfitting while ensuring convergence. For text pre-processing, we used each transformer model’s native tokenizer, truncating/padding sequences to 128 tokens for uniformity. These hyper-parameters followed established fine-tuning practices while optimizing for available hardware.

For baseline comparisons, we evaluated four widely adopted transformer models: BERT, RoBERTa, DistilBERT, and mBERT, selected for their proven contextual understanding and multilingual performance³³. Our proposed Hyb-Stack ensemble combined these architectures to leverage their complementary strengths, aiming to enhance generalization and predictive accuracy. Performance was assessed using micro/macro-averaged F1-scores, precision, recall, and accuracy, providing a comprehensive evaluation of both class-wise and overall model behaviour.

Comparative performance of the base models

Table 5 summarizes the performance of individual transformer models on the test datasets, evaluated using weighted F1-score serving as the primary metric for model effectiveness. Table 6 compares ensemble configurations, demonstrating performance improvements achieved through different combination strategies including average, weighted average, and our proposed Hyb-Stack Ensemble.

Table 5.

Transformer models result.

Models	Datasets	Precision	Recall	F1-score	Accuracy
BERT	HaEmoC_V1	80.42	78.39	79.65	84.27
	Bahasa Indonesia Hate Speech	83.56	82.74	84.88	87.19
	Sem-Eval-2018 Task 1 E-c English	85.67	82.48	85.21	89.34
DistilBERT	HaEmoC_V1	82.91	77.36	81.84	87.68
	Bahasa Indonesia Hate Speech	84.77	84.22	84.93	88.34
	Sem-Eval-2018 Task 1 E-c English	87.59	85.15	86.44	94.02
RoBERTa	HaEmoC_V1	75.62	71.93	73.88	81.17
	Bahasa Indonesia Hate Speech	80.84	77.38	79.16	85.92
	Sem-Eval-2018 Task 1 E-c English	86.53	84.79	85.66	91.45
mBERT	HaEmoC_V1	84.37	83.65	82.28	86.73
	Bahasa Indonesia Hate Speech	86.21	83.86	85.33	91.91
	Sem-Eval-2018 Task 1 E-c English	90.13	85.26	88.90	94.68

Table 6.

Performance of the proposed Hyb-Stack ensemble learning approach.

EM sets	Models	Datasets	Avg. EM			Wt. Avg. EM			Hyb-Stack ensemble
			Prec	Rec	F1	Prec	Rec	F1	Prec	Rec	F1
EM—1	BERT + RoBERTa	HaEmoC	80.14	78.91	79.62	82.30	81.84	84.51	85.12	83.78	85.27
		BHIS	83.47	82.03	84.69	84.51	84.29	86.90	87.65	85.09	84.96
		SE-18-T1 E-c	85.76	82.65	85.12	86.41	85.98	87.32	88.44	86.73	86.11
EM—2	BERT + DistilBERT	HaEmoC	82.11	80.90	81.55	85.77	83.65	82.12	86.39	85.78	84.41
		BHIS	84.58	83.10	82.94	83.37	82.28	84.51	87.06	84.32	85.78
		SE-18 T1 E-c	87.81	85.12	84.67	86.73	84.65	85.44	88.24	87.95	86.73
EM—3	BERT + mBERT	HaEmoC	83.29	81.41	82.58	85.13	82.66	86.79	87.01	86.88	87.36
		BHIS	84.35	80.13	83.22	84.80	81.11	83.55	86.49	87.63	86.07
		SE-18 T1 E-c	85.78	83.98	84.69	83.03	81.90	84.15	89.74	88.21	87.92
EM—4	RoBERTa + mBERT	HaEmoC	86.52	84.44	83.05	85.90	84.22	83.76	86.88	87.31	85.64
		BHIS	84.26	82.94	81.71	84.00	80.59	84.91	85.82	84.37	83.74
		SE-18 T1 E-c	83.92	81.61	80.35	89.21	87.53	86.17	89.88	88.14	87.33
EM—5	DistilBERT + mBERT	HaEmoC	82.45	79.97	81.02	85.78	84.60	83.12	87.06	82.97	84.59
		BHIS	84.99	82.33	82.88	86.01	83.85	85.42	88.90	84.93	86.15
		SE-18 T1 E-c	87.17	85.29	86.33	88.84	85.90	86.01	89.53	86.90	88.47
EM—6	DistilBERT + RoBERTa	HaEmoC	84.60	80.75	81.64	84.31	83.21	82.59	85.17	84.67	83.92
		BHIS	80.43	78.23	79.99	83.15	80.31	81.58	84.78	85.82	83.17
		SE-18 T1 E-c	87.76	86.20	85.96	87.12	83.38	84.27	86.58	84.71	85.37
EM—7	BERT + RoBERTa + mBERT	HaEmoC	86.98	80.85	83.63	86.74	82.58	84.91	87.24	83.39	85.40
		BHIS	85.05	82.18	83.01	82.72	80.79	81.64	89.13	87.93	86.06
		SE-18 T1 E-c	84.61	81.45	80.22	86.92	84.04	85.13	87.89	85.94	87.67
EM—8	BERT + DistilBERT + RoBERTa	HaEmoC	85.89	82.99	84.70	83.37	82.43	83.11	88.21	88.67	86.23
		BHIS	86.20	84.11	85.94	87.09	86.25	84.44	86.80	85.42	85.58
		SE-18 T1 E-c	83.67	86.71	88.84	87.32	84.07	85.73	89.22	87.33	88.68
EM—9	BERT + DistilBERT + mBERT	HaEmoC	87.79	86.58	85.14	85.61	87.73	86.25	90.91	89.26	89.48
		BHIS	86.23	85.99	84.82	88.67	86.70	86.30	89.43	87.81	88.19
		SE-18 T1 E-c	89.12	87.34	86.41	89.54	87.12	87.73	91.88	89.55	90.67
EM—10	DistilBERT + RoBERTa + mBERT	HaEmoC	83.83	83.40	82.94	84.01	82.36	83.67	87.40	89.23	87.81
		BHIS	83.99	80.86	81.39	85.60	82.54	84.91	86.08	87.02	85.44
		SE-18 T1 E-c	85.06	85.81	86.59	86.77	84.28	86.10	87.02	85.27	86.36
EM—11	BERT + DistilBERT + RoBERTa + mBERT	HaEmoC	82.74	85.80	84.31	83.51	80.33	84.84	86.33	85.65	84.96
		BHIS	84.16	82.40	83.91	88.73	84.83	86.50	88.05	86.27	87.69
		SE-18 T1 E-c	89.53	85.49	86.10	87.93	86.08	85.34	88.86	87.58	89.62

EM, Ensemble Method; Avg. EM, Average Ensemble Method; Wt. Avg. EM, Weighted Average Ensemble Method; Hyb-Stack Ensemble, Hybrid Stacked Ensemble; Prec. , Precision; Rec. , Recall; F1, F1-score, HaEmoC, Hausa Emotion Corpus, BHIS, Bahasa Indonesia Hate Speech Dataset, SE-18-T1 E-c, Sem-Eval-2018 Task 1 E-c English.

After completing the experiment of the four transformer-based models on all three datasets as presented in Table 5, the results revealed that mBERT consistently achieves the highest F1-scores, indicating robust cross-lingual performance. On the low-resource HaEmoC_V1 dataset, mBERT attains an F1-score of 82.28%, outperforming BERT (79.65%), DistilBERT (81.84%), and RoBERTa (73.88%). Performance improves for all models on the Bahasa Indonesia Hate Speech dataset, where mBERT again leads with an F1-score of 85.33%, followed closely by DistilBERT (84.93%) and BERT (84.88%), while RoBERTa records a lower score of 79.16%. The highest F1-scores are observed on the SemEval-2018 English dataset, reflecting greater language resource availability, with mBERT achieving 88.90%, followed by DistilBERT (86.44%), RoBERTa (85.66%), and BERT (85.21%). Overall, the progression of F1-scores from Hausa to Bahasa Indonesia and then to English highlights the influence of language coverage in pretrained models and underscores the effectiveness of multilingual representations for emotion and hate speech classification.

Figure 6 illustrates a line chart showing the error rates of each model. The chart highlights clear contrasts in model behaviour. RoBERTa begins with the highest error rate of 26% on HaEmoC_V1 but improves substantially on subsequent datasets, showing adaptability with larger, data-rich resources. mBERT demonstrates the lowest overall error rates, improving steadily to 11% on SemEval-2018, confirming its strong cross-lingual generalization. DistilBERT maintains competitive error rates throughout, consistently outperforming BERT while remaining computationally efficient. BERT shows stable but less dynamic error rates of 15–20% on all the datasets. These trends emphasize that mBERT generalizes best, RoBERTa adapts most dramatically, and DistilBERT balances performance with efficiency.

Fig. 6.

Error rate of transformer-based models.

Training and validation performance

Figure 7 illustrates the training and validation curves for mBERT, the best-performing base model. The curves show a consistent reduction in training and validation loss on all the 11 epochs, accompanied by steadily improving validation accuracy. The absence of divergence between training and validation confirms effective convergence without overfitting, further validating mBERT strong generalization.

Fig. 7.

mBERT model’s training and validation performance.

Hybrid stack ensemble learning results (Hyb-Stack)

Table 6 summarizes the results of eleven ensemble configurations evaluated with the three datasets using average ensemble, weighted ensemble, and the proposed Hyb-Stack method.

The experimental results in Table 6 demonstrate that the Hyb-Stack ensemble consistently outperforms simple and weighted averaging baselines across all evaluated datasets. By integrating the four transformer models (BERT, RoBERTa, DistilBERT, and mBERT), Hyb-Stack achieves substantial improvements in precision, recall, and F1-score for multilingual and multi-label classification. F1-scores range from 83.05 to 89.48% on the HaEmoC dataset, 81.39% to 88.19% on the BHIS dataset, and 85.13% to 90.67% on the SemEval-2018 English dataset, with the EM-9 configuration (BERT + DistilBERT + mBERT) achieving the highest scores of 89.48%, 88.19%, and 90.67%, respectively. Overall performance improves from Hausa to Bahasa Indonesia and then to English, reflecting differences in language resource availability, while weighted averaging offers moderate gains over simple averaging but remains inferior to the Hyb-Stack approach.

Figure 8 presents the error rates of 11 ensemble models (EM-1 to EM-11). The chart shows that the EM-9 ensemble set achieved the highest performance, with the lowest error rate of 10.55%, whereas EM-6 recorded the weakest performance with an error rate of 15.85%. Most models produced moderate error rates, ranging between 12.58% and 14.55%, with EM-11, EM-3, and EM-8 also performing competitively. The substantial performance gap between EM-9 and the other models suggests that its ensemble configuration, including the choice of base models, stacking methodology, and hyperparameter tuning was more effective in minimizing errors.

Fig. 8.

Error rate of the H-stack ensemble approach.

Confusion matrix

The confusion matrix of the best-performing ensemble model (EM-9) illustrated in Fig. 9 demonstrates strong classification performance, with most predictions concentrated along the diagonal, reflecting high accuracy. Misclassifications were relatively limited and mainly occurred between semantically close emotion classes, such as joy vs. surprise and sadness vs. fear. These confusions highlight the inherent overlap in emotional expressions and illustrate the challenges of multi-label emotion classification.

Fig. 9.

Best performing ensemble model based on the proposed Hyb-Stack ensemble method.

Comparison with other studies

Recent advances in emotion classification have increasingly relied on ensemble learning to combine the strengths of multiple models. Table 7 compares seven state-of-the-art ensemble approaches, highlighting their architectures, key innovations, and performance metrics. The analysis reveals how strategic model combinations and meta-learning techniques achieve varying success in different languages (e.g., Arabic, Bengali) and domains (e.g., social media content such as tweets). Our proposed Hyb-Stack ensemble emerges as the new performance leader, demonstrating the effectiveness of hybrid transformer architectures with advanced ensemble strategies.

Table 7.

Comparison of ensemble learning approaches for emotion classification.

Authors	Ensemble technique used	Ensemble key features	Performance of ensemble models (F1-score) (%)
4	Weighted Average Ensemble	Combines Bangla-BERT-2, XLM-R, Indic-DistilBERT, and Bangla-BERT-1; uses weighted Softmax averaging for prediction; optimized for Bengali emotion texts	80.24
23	Weighted Stacked Generalization	Combines BiLSTM and BERT (IndoBERT-large-p2); stacking uses logistic regression as meta-learner, optimized on PRDECT-ID (Indonesia)	75
33	Average Voting Ensemble (AVEDL)	Uses BERT, DistilBERT, and RoBERTa; trained on tweets and ERSS call transcripts; supports 5 emotion classes	85.20
62	Voting Ensemble Classifier (VEC) with AraBERT embeddings	Combines SGD Classifier, Linear SVC, Multinomial NB, and Ridge Classifier; leverages AraBERT for Arabic sentiment analysis	73.77
63	Ensemble of Random Forest (RF), Decision Tree (DT), and Support Vector Machine (SVM) with AdaBoost	Uses Adaptive Boosting (AdaBoost) to combine predictions from RF, DT, and SVM; optimized for tweet sentiment classification	87.32
35	Multi-Level Ensemble (Combines Bagging-based, Boosting-based, and Multinomial Naive Bayes)	Combines Random Forest, Balanced Random Forest, XGBoost, and Naive Bayes models	62
Our work	A new approach—hybrid stack ensemble (Hyb-Stack)	Combines BERT, RoBERTa, DistilBERT, and mBERT using Random Forest as meta-classifier; optimized for multi-label and multilingual emotion classification	90.67

Conclusion

This study presented Hyb-Stack, a hybrid stacked ensemble learning framework for multi-label text emotion detection that integrates simple stacking with cross-validation-based stacking. The framework was evaluated on three datasets, HaEmoC_V1 (a newly developed Hausa emotion corpus), Bahasa Indonesia Hate Speech, and SemEval-2018 Task 1 E-c (English) to examine whether ensemble learning can consistently improve performance over individual transformer-based models across languages with differing resource availability. Experimental findings show that the proposed Hyb-Stack framework, particularly the EM-9 ensemble configuration (BERT, DistilBERT, and mBERT), achieved higher macro-F1 scores than the strongest individual baseline across all evaluated datasets. These results indicate that hybrid stacking can enhance predictive performance, with more noticeable benefits in low-resource language settings. The strong performance of mBERT highlights the value of multilingual pre-training for cross-lingual emotion modelling, while the ensemble results suggest that combining complementary transformer representations can yield more robust emotion predictions.

Limitations and future work

While the proposed Hyb-Stack framework demonstrates consistent performance improvements across the evaluated datasets, some scope-related considerations should be noted. The experimental analysis was conducted on three publicly available datasets, which, although diverse in language and domain, do not cover all possible forms of emotional expression. In addition, the study focused on the emotion categories defined within these datasets, and did not explore more fine-grained or context-dependent emotional phenomena. Finally, as with most ensemble-based approaches, the framework involves additional model components, which may require further optimization for large-scale or real-time applications.

Future research will focus on extending the Hyb-Stack framework to a wider range of emotion categories and further enriching the HaEmoC_V1 corpus with larger and more diverse data sources. Additional studies may explore the framework’s ability to process code-mixed and code-switched texts, emoticon- and emoji-based emotional expressions, and mixed-emotion content commonly found in social media. Evaluating the approach on other underrepresented African and Indigenous languages would provide a stronger assessment of its generalizability. Methodological enhancements, such as dynamic ensemble selection, alternative meta-learning strategies, and the integration of larger multilingual transformers (e.g., XLM-R), also represent promising directions. Finally, addressing bias mitigation, interpretability, and deployment considerations will be essential for improving the robustness and real-world applicability of ensemble-based emotion detection systems.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

Hassan Adamu: Conceptualization, investigation, data curation, methodology, formal analysis, writing—original draft. Masrah Azrifah Azmi Murad: conceptualization, validation, writing—review and editing, supervision. Nurul Amelina Nasharuddin: Conceptualization, writing—review and editing, supervision.

Funding

This research was funded by Universiti Putra Malaysia.

Data availability

The data that support the findings of this study are available from the corresponding author Hassan Adamu (gs66245@student.upm.edu.my) upon reasonable request.

Declarations

Competing interests

The authors declare no competing interests.