Multi-modal collaborative learning for facial expression recognition in e-learning platforms

Abstract

In recent years, the task of recognizing facial expressions (FER) has become an important research topic in the development of intelligent educational systems. However, facial expressions are highly dynamic and exhibit substantial variability, which makes accurate recognition difficult. To address this challenge, we introduce a novel multi-modal Collaborative Learning (MaCLe) approach based on the CLIP model for FER applications in online learning scenarios. The implementation of collaborative prompt learning enhances generalization and dynamics. The effectiveness of the proposed model is validated on three widely used benchmarks, including RAF-DB, FER2013 and AffectNet. MaCLe achieves recognition accuracies of 95.45, 80.37 and 77.66% on these datasets, respectively. Ablation studies have demonstrated that an optimal depth of J = 7 and prompt length of 2–4 tokens enable this model to effectively balance accuracy and generalization. Furthermore, even with the integration of multi-modal prompt generation, MaCLe increases FLOPs by only approximately 0.1% in comparison to CoOp and Co-CoOp. Meanwhile, it demonstrates faster convergence, achieving results in 5 epochs compared to 10, and exhibits a lower parameter overhead with only a 2.85% increase compared to CLIP. The results indicate that MaCLe significantly enhances recognition accuracy and improves training efficiency and scalability within e-learning platforms.

Introduction

Facial expressions are one of the most instinctual modes of human communication. Facial muscles convey emotions, intentions, and cognitive states across various contexts. A prominent theory regarding emotional expression was proposed by Ekman^1–3. Human emotions can be categorized into six fundamental types: sadness, joy, fear, anger, surprise, and disgust. Research on understanding emotional and psychological conditions from facial expressions has become a major focus across both academia and industry in recent years. As a result, facial expression recognition (FER) has gradually evolved into a key technology for enabling effective human-computer interaction (HCI), particularly in the context of online education applications^4–6. In remote learning environments, students’ psychological states and learning engagement are only evaluated by facial expression recognition owing to the lack of face-to-face communication and feedback. Therefore, it is crucial to develop intelligent teaching assistance systems using computer vision technologies. It can help assess learning status and emotional fluctuation by recognizing students’ facial expressions. Furthermore, it can also be used to analyze their behaviors and provide personalized learning.

Recently, FER technologies have achieved significant developments in the fields of computer vision and deep learning⁷. However, there are several limitations to online education applications. First, it remains challenging to obtain high-quality data for the training and evaluation of FER systems⁸. Most public FER datasets cannot accurately reflect students’ natural emotional responses. More importantly, it is difficult to establish a standard for emotion classification when using a single image recognition model. This is because students from different cultural backgrounds may have different emotional responses to the same expression owing to a high degree of subjectivity. Therefore, the lack of a widely accepted standard for the measurement and evaluation of facial expressions significantly limits the scalability and generalizability of FER systems in online education. Second, facial expressions not only have static facial features but also exhibit distinct dynamics⁹. In the context of online learning, students’ emotional status may change rapidly within a few seconds. However, traditional static image classification models cannot capture dynamic variations, resulting in inaccurate recognition of emotional states. Therefore, an FER system for online education needs to ensure model capability to accurately extract critical features. Meanwhile, it needs to identify the dynamics of expressions to enhance sensitivity in recognizing complex states. Third, conventional facial expression recognition methods have poor adaptability and limited generalization capability¹⁰. Most approaches neglect the significance of visual prompt learning. This uni-modal prompting strategy lacks the capability to adapt to the dynamics of visual features. Consequently, achieving a coordinated alignment between textual and visual modalities is difficult. Thus, the performance of the model in complex educational scenarios is limited. Currently, the limitation of effective mechanisms for simultaneously learning prompts in both visual and textual modalities remains a significant challenge, especially in online education applications.

To address the limitations of FER applications, deep neural networks have gradually emerged as a promising approach¹¹. Among the various neural network-based frameworks, Convolutional Neural Networks (CNNs) demonstrate superior performance in image processing tasks. For instance, ResNet-50 can effectively reduce the influence of the vanishing gradient problem in deep networks owing to its residual connection architecture, which enhances stability and robustness in tasks related to image classification and feature extraction¹². However, these models cannot effectively focus on the most informative facial regions relevant to emotional expressions. Therefore, researchers have introduced attention mechanisms. The Convolutional Block Attention Module (CBAM) incorporates both channel and spatial attention mechanisms to enhance feature refinement¹³. This module can concentrate on the most informative regions, such as the eyes, eyebrows, and mouth. Thus, the ability to detect subtle expressions will improve. In addition, facial expressions are dynamic processes that evolve over time in online educational settings. Consequently, single-image analysis is insufficient to capture emotional fluctuations. To address this problem, Temporal Convolutional Networks (TCNs) are widely used because of their strong parallel computation capabilities and proficiency in modelling long-range dependencies¹⁴. TCNs can construct the temporal evolution of facial expressions while maintaining the efficiency of the convolutional structures. Furthermore, students’ emotional transitions can be effectively recognized. The integration of spatial and temporal modelling can enhance the practicality and stability of FER systems in online learning. Contrastive Language-Image Pretraining (CLIP) has shown excellent multi-modal capabilities. Its text and image encoders coexist to achieve an accurate alignment between the vision and language modalities. However, prompt learning applied to the text encoder is insufficient to meet the adaptation requirements of the image encoder. Thus, we proposed a Multi-modal Collaborative Learning (MaCLe) strategy to adjust the representations of both the text and image encoders for optimal alignment in downstream tasks.

The contributions of this study are summarized as follows:

1. We introduce a novel multi-modal collaborative prompt learning approach based on the CLIP framework to optimize vision-language representation alignment, which is the first to apply multi-modal prompt learning to CLIP fine-tuning for facial expression recognition.

2. We designed a coupling function to explicitly coordinate prompt learning between the text and image encoders, enabling effective joint adaptation across the two modalities in CLIP.

3. We performed extensive evaluations using the three widely adopted FER benchmarks: RAF-DB, FER2013 and AffectNet. The experimental results demonstrate that our method consistently outperforms conventional deep learning-based approaches for facial expression recognition.

Related work

Rapid advances in artificial intelligence have sparked renewed interest in online education^15–17. Intelligent analysis of student emotions and behavior helps teachers understand students’ concentration in real time and adjust their teaching strategies accordingly. This will enhance the effectiveness and personalization of online learning. Florestiyanto¹⁸ et al. pointed out that computer vision technology had a significant role in enhancing online learning environments. It can provide classroom monitoring and support for teachers. Emotion recognition can analyze students’ emotional feedback and help teachers change their teaching methods, which keeps students engaged and increases their academic performance. Liu¹⁹ et al. reported that visual classroom behavior recognition primarily relied on deep-learning-based identification techniques. In related fields, object detection frameworks represented by the YOLO family have been successfully applied to recognize diverse animal behaviors. To accurately capture students’ emotional states in online education. Salloum²⁰ et al. developed a refined convolutional neural network model for students’ emotions recognition. The experimental results show that the proposed model achieved a test accuracy of 95%. Effective facial expression recognition makes it suitable for real-time student emotion monitoring in educational environments and helps teachers adjust their teaching based on emotional responses. In addition, Mortezapour²¹ et al. introduced novel hybrid end-to-end deep learning models to automatically detect student engagement levels in online classrooms. Their findings demonstrated that the EfficientNetV2-L+LSTM model was the most proficient, with an accuracy of 62.11%. Thus, the validity of a hybrid architecture of a convolutional neural network and time series model for detecting students’ online learning engagement status was verified. However, obtaining valid data for training and testing in complex environments is difficult. Thus, this study introduced pretrained vision-language (V-L) models to improve the robustness and adaptability of facial expression analysis systems.

FER techniques are widely used to automatically detect and analyze emotional states from facial images or video^22–24. However, it is difficult for traditional CNN to capture the local details of an expression. To address this limitation, Ye²⁵ et al. introduced an attention mechanism based on a convolutional neural network and long short-term memory (CNN-LSTM). Compared with traditional methods, this hybrid model achieved higher accuracy in expression recognition. These results verify that the integration of attentional mechanisms and local features enhances FER performance. Nan²⁶ et al. introduced truncated ConvNeXt approach (Conv-cut) as a feature extractor for FER. Meanwhile, they designed detailed extraction modules and self-attention mechanisms to enhance the learning of complex expression features. Their experimental results demonstrated that the proposed model was superior to those of other related studies. In addition, there are uncertainties in real-time applications owing to ambiguous facial expressions or low-quality images. Wang²⁷ et al. proposed a simple and efficient Self-Cure Network (SCN) to reduce the influences of noisy data by automated recognition of uncertain samples during training process. In detail, the SCN model can identify samples with high losses and reduce their weights to avoid overfitting noise samples. The proposed model enhances the classification accuracy and generalization performance. In addition, to improve the adaptability of FER to complex environments, researchers have integrated traditional features and other modal information. RCL-Net²⁸ is an FER strategy based on an attention mechanism and local binary feature fusion (LBP). The RCL-Net combines texture features with depth features and shows superior performance in complex environments, particularly in variable lighting applications.

Compared with traditional facial expression recognition (FER) models trained solely with image supervision, CLIP has attracted increasing attention due to its ability to learn rich vision–language representations from large-scale image–text pairs^29–31. Such representations can partially alleviate the reliance on manually annotated FER datasets and have demonstrated strong performance in few-shot and zero-shot recognition scenarios. However, as a general-purpose pre-trained model, CLIP does not necessarily achieve optimal performance when directly applied to facial expression recognition tasks^32,33. FER typically involves subtle emotional cues, ambiguous inter-class boundaries, and significant variations in data distribution caused by factors such as low resolution, occlusion, and cultural subjectivity, which may differ considerably from the data used during CLIP pretraining. Consequently, effective task-specific adaptation is required to better align both the textual semantics of emotion descriptions and the visual representations of fine-grained facial features with the FER domain. Existing prompt-learning approaches, such as CoOp and Co-CoOp, primarily improve adaptation by optimizing context prompts on the text side while keeping the image encoder largely fixed. This design limits their effectiveness for FER, where accurate recognition depends heavily on adapting the visual feature space to capture fine-grained local patterns and to distinguish visually similar expression categories. Meanwhile, previous multimodal FER methods often enhance performance by incorporating additional modalities or handcrafted features. Although effective to some extent, these approaches generally lack an explicit mechanism to jointly coordinate prompt learning across vision and language within a unified CLIP-style alignment framework, which may restrict their generalization ability when label definitions or expression dynamics vary across different scenarios³⁴. Hence, we propose MaCLe, a multi-modal collaborative prompt-learning framework that jointly adapts both the text and image encoders through coupled prompts. By enabling collaborative optimization across modalities, MaCLe aims to achieve stronger vision–language alignment and more robust FER performance, particularly in online learning settings.

Method

As shown in Fig. 1, the proposed framework consists of a visual encoder and a text encoder. Following the standard CLIP architecture proposed by Radford et al.³⁵, we adopt the Vision Transformer (ViT) as the backbone for image encoding. Both the input image and its corresponding textual descriptions are processed through the CLIP encoders to obtain aligned visual and textual embeddings.

Fig. 1.

Overview of the proposed MaCLe framework.

Revisiting clip

Image encoder V with Z transformer layers spits the input image I into M fixed-size patches. Then, these patches are mapped into patch embeddings using a projection layer. Patch embeddings E_i with a learnable class token c_i are input into the (i + 1)^th transformer block V_i+1. Subsequently, they are processed through all the Z transformer layers. Finally, the class token output c_z from the last transformer block V_z is projected by image projection into a shared visual-linguistic latent embedding space.

1

The final image feature x is derived by mapping the class token c_z from the last transformer layer into the shared visual–language latent space using the ImageProj module.

2

To generate features for textual descriptions, words are split by the text encoder L. Meanwhile, textual descriptions project it into the word embedding. These embeddings W_i are then input into the corresponding (i + 1)^th transformer layer in the textual encoding branch. Similar to the image encoder, the final word embedding output from the last transformer block is projected using text projection into the shared visual-linguistic laten space.

3

The final textual feature z is obtained by mapping the class token output from the last transformer layer into the shared visual–language latent space through the TextProj module.

4

For zero-shot classification, traditional methods utilize manually crafted text prompts, where each class label has C classes. The cosine similarity scores between the image and textual embeddings were calculated. Then, it is scaled by a temperature parameter. The predicated value y for image I can be found and selected as the highest score.

5

Macle

To effectively fine-tune CLIP on facial expressions images, we proposed a multi-modal collaborative prompt tuning approach. Prior research has revealed that traditional approaches to facial expression recognition, especially those relying on a single modality, suffer from notable performance constraints. These approaches find it difficult to incorporate prompt learning within the visual branch. Furthermore, they lack flexibility in dynamically adapting to the representation spaces of vision and language modalities. To address these problems, we emphasized the importance of multi-modal prompt learning to achieve a comprehensive and adaptive prompt mechanism.

Thus, we introduced two types of prompting layers: static prompts (SP) and dynamic prompts (DP). Compared with other standard CLIP-based methods that focus on single textual prompts, our approach jointly learns the prompts in both the language and vision branches. Better cross-modal coordination and adaptation can be achieved through this method.

Recent research suggests that injecting prompts into deeper layers of Transformer architectures is critical for progressively modelling complex feature representations³⁶. Therefore, we introduced learnable vectors into the first J layers (J < K) of both visual and language encoders. These multi-modal hierarchical prompts are constructed to preserve the prior knowledge contained in the CLIP model while simultaneously enabling the learning of contextual representations that are specific to the target task.

Hybrid learnable prompts

To improve the effectiveness of facial expression recognition in online educational scenarios, we incorporate both static and dynamic prompts. The static prompts correspond to a group of learnable base tokens shared by all image inputs, and their parameters are optimized through auxiliary training data. However, previous research³⁷ suggests that static prompts have limited adaptability in classification tasks. These prompt embeddings are fixed throughout the training process and are designed for general-purpose information.

However, to improve the model’s ability to handle data distributions, we introduced dynamic prompts. Unlike static prompts, these are generated by a Dynamic Prompt Generator (DPG) for each test image. In our framework, the DPG is a frozen, pretrained backbone that extracts class-specific tokens. These tokens are then projected into the dynamic prompt space through a learnable linear transformation. These prompt embeddings are dynamically generated based on the test image to provide information related to it. The detail process can be described as follows. Firstly, static prompts can be defined as a set of learnable vectors as follows:

6

Here, these prompts are shared for all images, providing general-purpose information. In addition, MaCLe introduced DPG to capture expressive cues of each image. It can be described as follows:

7

where the DPG is a frozen, pretrained backbone that extracts class-specific tokens and W_d is a learnable projection mapping backbone outputs to prompt space. Compared with the previous MaPLe method that used the same prompt for all images³⁶, each image can obtain a dynamic prompt based on their characteristics. Therefore, the final input prompt for layer i can be calculated as follows:

8

The integration of static and dynamic prompts allows the MaCLe method to achieve more accurate guidance for both encoders when facing facial expressions recognition in complex educational settings.

Collaborative language prompting

To facilitate the learning of language-context prompts, we incorporated b learnable tokens (vectors) into the language branch of the CLIP. The input embedding is structured as , where denotes the fixed tokenized input embeddings. These learnable prompt tokens are integrated into the transformer blocks of the language encoder up to a designated depth J. Beyond the J-th layer, the subsequent transformer layer processes the output of the preceding layers to compute the final textual representation z.

9

When J = 1, the learnable prompt tokens P are applied to the input of the first transformer layer in the language encoder. Therefore, after J^th transformer layer, the subsequent layers and final text representations can be calculated as follows:

10

Collaborative vision prompting

Analogous to the prompt mechanism used in the language branch, we incorporate b learnable tokens into the visual pathway of CLIP and merge them with the image embedding inputs. These visual prompt vectors are then propagated into successive transformer blocks of the image encoder V, with the injection continuing until a predetermined depth J.

11

Our deep prompt provides the ability to learn flexibly within different levels of visual features. We found that the shared prompts across stages performed better than the independent prompts because of the continuous transformer block processing. Inter-feature correlations were improved. Thus, the later stages contribute less to providing independent complementary information than the earlier stages. Shared prompts are more effective than independent prompts.

Vision language prompt coupling

In online education, multi-modal strategies for prompt tuning are crucial. To achieve complete contextual optimization, both the visual and textual branches of CLIP must be jointly adjusted. A simple solution is to combine deep visual prompts and language prompts directly, where language prompts P and visual prompts P are learned under the same training plan. This design is independent of vision-language prompting. Although this approach meets the requirement of prompt completeness, it lacks cross-modal interactions. Because the two branches do not influence each other in learning task- relevant contextual prompts. This leads to insufficient alignment between facial expression images and their textual descriptions. In MaCLe, the cross-modal collaboration is explicitly implemented by a trainable coupling function F that connects the prompt representations of the two modalities. Instead of optimizing the language and vision prompts separately, F enables the visual prompts to be conditioned on the learned language prompts, thereby introducing an explicit interaction path between modalities during optimization.

To address this limitation, the proposed MaCLe strategy simultaneously optimizes CLIP’s visual and textual components of CLIP by utilizing shared prompts across modalities. Language prompt tokens were incorporated into the visual component. To ensure cross-modal consistency and cooperation, visual prompts P are generated by projecting the language prompts P using a proposed vision-language coupling function. Thus, the visual prompts P can be obtained by the following function.

12

Where the coupling function is designed as a linear layer. It can map the input from the language embedding dimension d_l to the visual embedding dimension d_v. This mechanism plays a role in bridging the two modalities, facilitating bidirectional gradient propagation. This design allows the supervision from the visual branch to be back-propagated to the language prompt tokens through F, enabling coordinated updates of prompts across modalities under the same FER objective.

13

Therefore, the proposed MaCLe method addressed the limitations of previous prompt learning methods. Firstly, MaCLe introduces a learnable coupling function that projects language prompts into the visual prompt space to ensure the consistency between both modalities. In contrast, CoOp and Co-CoOp only improve textual prompts and the visual encode can not be adapted. Although MaPLe introduced deep prompt learning at the visual and language encoders, both model prompts are independently improved. Secondly, CoOp, Co-CoOp and MaPLe employ only static prompts, which limit their ability to recognize complex or unclear expressions in FER. However, MaCLe simultaneously employs static prompts (global semantics) and dynamic prompts (sample-adaptive), enabling the model to learn both universal emotional semantics and fine-grained sample-specific variations. Finally, prompts are injected only at the input layer of the text encoder for CoOp and Co-CoOp. For MaPLe, the visual and textual prompts are optimized independently, and visual prompts are not influenced by textual semantics, resulting in a lack of cross-modal coordination. However, MaCLe jointly injects prompts into the first J layers of both the visual and textual encoders, enabling the model to strengthen vision–language alignment across multiple hierarchical level.

Experiments

Datasets

RAF-DB (Real-world Affective Faces Database) is a widely used benchmark for facial expression recognition, consisting of 29,672 natural facial images. To guarantee reliable labels, each sample is annotated through a crowdsourced protocol in which roughly 40 independent evaluators contribute to the final expression annotation for every image. Because the images originate from unconstrained online sources, RAF-DB exhibits extensive variations in age, gender, ethnicity, head pose, illumination, occlusions (such as glasses, facial hair, and hand-induced self-occlusion), as well as a wide range of post-processing operations including filters and special visual effects. These characteristics make RAF-DB a highly representative benchmark for in-the-wild facial expression recognition. RAF-DB provides rich and comprehensive annotations, including a 7-dimensional expression distribution vector for each image, enabling both discrete and distribution-based emotion modeling.

FER2013 (Facial Expression Recognition 2013) contains around 35,000 grayscale images of size 48 × 48 pixels. These images were labelled into seven different emotional categories. FER2013 provides a consistent and controlled environment for evaluating FER models.

AffectNet is one of the largest and most comprehensive facial expression datasets for affective computing. It is designed to support both categorical and dimensional emotion modeling. The dataset was constructed to address the lack of large-scale, annotated, in-the-wild facial expression databases, particularly for the continuous emotion model involving valence and arousal, which is markedly underrepresented in existing corpora. AffectNet was constructed by gathering over one million facial images from the web. The collection process queried three mainstream search engines with 1,250 emotion-related keywords spanning six languages to capture broad linguistic, cultural, and contextual variability. From this pool, roughly 440,000 images were subsequently hand-labeled to produce high-quality annotations. Each annotated image contains seven discrete emotion categories and continuous valence and arousal scores representing emotional intensity.

Datasets and preprocessing

All images are resized to 224 × 224 and normalized using the mean and standard deviation provided by the pre-trained CLIP model. No additional data augmentation is applied during training to ensure fair comparison under the few-shot setting. For testing, we adopt a standard center-crop strategy.

Few-shot sampling protocol

Following common few-shot learning practices, we randomly select 16 samples per class for training. The sampling process is repeated with a fixed random seed to ensure reproducibility across experiments.

Backbone network

Our method is built upon the pre-trained ViT-B/16 CLIP model. The vision and text encoders are kept frozen during training, and only the prompt parameters are optimized. The input resolution of the vision encoder is fixed to 224 × 224.

Prompt design and initialization

We adopt a multi-layer prompt tuning strategy with prompt depth JJJ. For the few-shot setting, JJJ is set to 7. The initial language prompt P0P_0P0 is initialized using the pre-trained CLIP word embeddings with the template “This is an image of < emotion>.” Prompts in subsequent layers are randomly initialized from a normal distribution. The lengths of both visual and language prompts are set to 2. When training on ImageNet with all 1,000 categories as source model, the prompt depth is reduced to J = 3 to balance model capacity and training efficiency.

Training configuration

The proposed MaCLe method is trained using the Adam optimizer for 5 epochs with a learning rate of 0.0035 in the few-shot setting. For ImageNet pre-training, the model is trained for 2 epochs with a learning rate of 0.0026. No learning rate scheduler or gradient clipping is applied. All experiments are conducted on a single NVIDIA RTX 4090 GPU with 24 GB memory.

Testing protocol

During inference, the learned prompts are fixed, and no test-time augmentation is used. Model performance is evaluated on the standard test split of each dataset.

Evaluation metrics

Multiple metrics are employed to comprehensively evaluate the performance of FER systems. The four core evaluation criteria commonly used are accuracy (Acc), precision (Prec), recall (Rec), and F1-score. Acc reflects the proportion of correctly classified expressions to the total number of samples. It can be calculated as follows:

14

Where TP (True Positives) is correctly recognized facial expressions, TN (True Negatives) represents correctly excluded non-expressions, FP (False Positives) corresponds to incorrectly recognized expressions, and FN (False Negatives) refers to expressions that failed to be recognized.

Prec measures the proportion of true positive predictions among all predicted positive samples and can be described as follows:

15

Rec represents sensitivity and can be used to evaluate the model’s ability to identify all relevant instances of a class. It can be calculated as follows:

16

The F1-score is the harmonic mean of precision and recall and comprehensively evaluates the system performance. It provides a balance between false positives and false positives and can be described as follows:

17

The system’s accuracy, specific expression classification ability, and robustness can be obtained using these metrics.

Implementation details

In our experiments, we adopted a few-shot training strategy, where 16 samples per class were randomly selected for training. Prompt tuning was performed based on the pre-trained ViT-B/16 CLIP model. The proposed MaCLe method was trained for 5 epochs using the Adam optimizer with a learning rate of 0.0035. This process was performed on a machine equipped with an NVIDIA RTX 4090 GPU (24 GB memory). This configuration was used to accelerate convergence during gradient descent and reduce issues such as vanishing or exploding gradients. All experiments were conducted in the PyCharm development environment, ensuring a consistent and smooth training process.

For MaCLe, we set the prompt depth J = 7, with both the language and visual prompt lengths set to two. The initial language prompt P₀ was initialized with the pre-trained CLIP word embedding using the template “This is a picture of a/an < angry mood>”. The prompts in the subsequent layers were randomly initialized from a normal distribution. When training the model as a source model on the full 1000 classes of ImageNet, prompt depth J was adjusted to 3. The model was trained for two epochs with a learning rate of 0.0026.

Results and discussion

The proposed MaCLe framework was designed to enhance facial expression recognition capabilities in online classroom environments. Thus, to verify the effectiveness and robustness of the proposed strategy, we conducted experiments on three datasets: RAF-DB, FER2013, and AffectNet. This study ensured comprehensive learning and evaluation of the model’s performance.

Results on RAF-DB

Figure 2 presents a comparative analysis of confusion matrices for facial expression recognition on the RAF-DB dataset using the ResNet-50 model and the proposed MaCLe model. In this experiment, the MaCLe model demonstrated higher classification accuracy than the ResNet-50 model. This improvement may be attributed to the multimodal collaborative learning framework, where visual and textual modalities share prompt representations, potentially enhancing the model’s ability to capture discriminative features.

Fig. 2.

Confusion matrices of different models tested on the RAF-DB dataset: (a) ResNet-50; (b) MaCLe.

The MaCLe model achieved improved classification accuracy across all seven expression categories in the RAF-DB dataset. For relatively easier-to-classify expressions such as “Happy” and “Neutral,” the accuracy improved from 89% to 88% with ResNet-50 to 99% and 96% with MaCLe, respectively. Notably, improvements were also observed for complex expressions that are more difficult to distinguish, such as “Fear” and “Disgust,” where the accuracy increased from 73% to 90% and from 70% to 93%, respectively. Additional gains were observed for “Angry” (from 85% to 95%), “Sad” (from 81% to 95%), and “Surprise” (from 83% to 97%). These findings suggest that MaCLe is capable of enhancing recognition accuracy for both simple and complex expressions under the given experimental conditions.

To further evaluate the performance of the proposed approach, we compared MaCLe with several traditional deep learning models on the RAF-DB dataset. As summarized in Table 1, as a baseline convolutional model, ResNet achieves an accuracy of 81.28% on this dataset. However, it is lower than that of models that incorporate attention mechanisms or multimodal feature extractions strategies, including GReFEL (92.47%), DDAMFN++ (92.34%), S2D (92.57%) and DCJT (92.24%). The proposed MaCLe shows better performance and achieved higher accuracy of 95.45%. This improvement can be attributed to the MaCLe model’s CLIP-based architecture, which integrates not only pre-trained visual features but also introduces multi-layered language and visual prompt mechanisms.

Table 1.

Performance comparison of different models on the RAF-DB dataset.

Dataset	Model	Accuracy (%)	F1-score (%)	Recall (%)	Precision (%)
RAF-DB	ResNet	81.28	79.02	79.54	79.14
	GReFEL38	92.47	91.32	91.07	89.63
	DDAMFN + + 39	92.34	90.15	91.04	89.64
	S2D40	92.57	90.02	92.45	92.08
	DCJT41	92.24	90.13	92.26	91.45
	MaCLe (proposed model)	95.45	93.92	94.86	93.25

Results on FER2013

As shown in Fig. 3, the MaCLe model demonstrates improved classification accuracy across all seven expression categories. For relatively straightforward expressions such as “Happy” and “Neutral”, the accuracy increases from 82% to 76% with ResNet-50 to 88% and 82% with MaCLe, respectively. For more challenging categories such as “Fear” and “Disgust”, larger performance gains are observed, with the accuracy improving from 61% to 81% and from 58% to 80%, respectively. Improvements are also observed for “Angry” (from 57% to 78%), “Sad” (from 66% to 74%), and “Surprise” (from 60% to 82%). These findings suggest that MaCLe offers enhanced recognition ability across both simple and complex facial expressions under the specific conditions of the FER2013 dataset.

Fig. 3.

Confusion matrices of different models tested on the FER2013 dataset: (a) ResNet-50; (b) MaCLe.

As reported in Table 2, the baseline ResNet-50 model achieves an overall accuracy of 57.92% on FER2013. This result highlights the challenges in facial expression recognition under conditions of label noise and low-resolution input. Several advanced models achieve higher accuracy, including LHC-Net (74.42%) and Local Learning Deep + BOW (75.42%), with Segmentation VGG-19 and EmoNeXt reaching 75.97% and 76.12%, respectively. These improvements are generally attributed to their use of local feature learning, attention mechanisms, or data augmentation strategies. The proposed MaCLe model attains a higher accuracy of 80.37%, representing an increase of 4.25% and 4.40% over EmoNeXt and Segmentation VGG-19, respectively. Compared to ResNet-50, the improvement is approximately 22.45%. While these results demonstrate the potential of MaCLe in improving recognition performance on FER2013, further studies in real-world online learning scenarios are needed to validate its generalization ability.

Table 2.

Performance comparison of different models on the FER 2013 dataset.

Dataset	Model	Accuracy (%)	F1-score (%)	Recall (%)	Precision (%)
FER2013	ResNet	57.92	57.55	58.11	56.35
	LHC-Net42	74.42	74.34	74.83	72.33
	Local Learning Deep+BOW43	75.42	75.13	75.58	73.65
	Segmentation VGG-1944	75.97	75.81	76.05	73.83
	EmoNeXt45	76.12	75.07	77.45	74.82
	MaCLe (proposed model)	80.37	74.41	80.49	73.99

Results on AffectNet

Figure 4 presents a comparative analysis of confusion matrices for facial expression recognition on the AffectNet dataset using the ResNet-50 and MaCLe models. As shown in Fig. 4(a), the ResNet-50 model achieves relatively low classification accuracy on several challenging categories, with “Disgust” and “Neutral” recognized at 30% and 26%, respectively. This may be attributed to the presence of overlapping visual features and low-quality or ambiguous facial images in the dataset. In contrast, for comparatively easier categories such as “Happy” and “Sad”, the model attains higher accuracies of 79% and 77%, suggesting that some emotions are more visually distinguishable. However, other categories like “Angry” (56%), “Fear” (58%), and “Surprise” (51%) still pose challenges in the context of FER for online learning environments.

Fig. 4.

Confusion matrices of different models tested on the AffectNet dataset: (a) ResNet-50; (b) MaCLe.

Figure 4(b) shows that the MaCLe model yields consistently improved performance across all emotion categories. Notably, the recognition accuracy for “Disgust” increases from 30% to 72%, and for “Neutral” from 26% to 83%, indicating enhanced robustness in distinguishing complex or subtle expressions. For “Happy” and “Sad”, the MaCLe model achieves 88% and 74%, comparable to or slightly better than the ResNet-50 baseline. These results suggest that MaCLe may offer more reliable emotion recognition under the specific conditions of the AffectNet dataset.

To further assess the performance of the proposed approach, we conducted a comparative evaluation with several advanced FER models, as summarized in Table 3. The AffectNet dataset presents greater challenges due to its scale, annotation noise, and emotional diversity, thus requiring models with strong generalization and semantic understanding capabilities. The ResNet model achieves a baseline accuracy of 59.50%, while improved models such as S2D (67.62%), DDAMFN++ (67.36%), POSTER++ (67.49%), and LFNSB (66.57%) demonstrate enhanced performance through the incorporation of attention mechanisms, cross-modal fusion, and temporal modeling strategies.

Table 3.

Performance comparison of different models on the AffectNet dataset.

Dataset	Model	Accuracy (%)	F1-score (%)	Recall (%)	Precision (%)
AffectNet	ResNet	59.50	59.5	59.5	59.5
	S2D40	67.62	67.60	67.64	67.62
	DDAMFN + + 39	67.36	67.34	67.38	67.36
	POSTER + + 46	67.49	67.47	67.51	67.49
	LFNSB47	66.57	66.56	66.58	66.57
	MaCLe (proposed model)	77.66	77.64	77.77	77.66

The MaCLe model achieves an accuracy of 77.66%, representing an improvement of 10.04% over the best-performing baseline model (S2D) and 18.16% over ResNet-50. While these results highlight the potential of MaCLe in addressing complex FER tasks, it is important to note that these evaluations are based on offline experiments. Further validation under real-world conditions and in cross-cultural online learning settings is necessary to confirm its generalizability and robustness.

Ablation studies

Prompt depth

To investigate the impact of prompt depth J on the performance of the proposed MaCLe model, we conducted an ablation study on the prompt depth of both the language and vision branches. As shown in Fig. 5, it can be observed that the increasing prompt depth improves the performance, especially for vision prompting, which indicates that there is significant correlation between deep contextual modeling and visual features. In contrast, language prompting shows slight performance improvement. There is a slight decline beyond layer 5. This suggests that deep prompt in the language branch may hinder performance. For Naive V-L prompting, there is continuous improvement from layer 1 to layer 7. However, there is a minor decline at layer 12 due to the absence of semantic alignment mechanisms.

Fig. 5.

Ablation on prompt depth in MaCLe model.

In conclusion, as randomly initialized prompts are introduced into deeper layers of the feature space in the pretrained model, the model performance becomes sensitive. Compared with earlier approaches that employed only shallow language prompts (e.g., J = 1), the proposed method demonstrates superior performance when using deeper language prompting. Overall, the proposed MaCLe model achieved its optimal performance at a prompt depth of 7 and consistently outperformed deep language prompting across the entire comparison.

Prompt length

To explore the effect of prompt length on the performance of the MaCLe model, we conducted a series of comparative experiments. As shown in Fig. 6, as the number of prompt tokens increase, the performance on base categories remains relatively stable. However, the accuracy of the new categories exhibits a significant decline, which suggests that excessively long prompts may lead to overfitting. Thus, the model’s generalization ability for new categories will be affected.

Fig. 6.

Ablation on prompt length in MaCLe model.

In addition, the performance reaches a peak around 2–4 prompt tokens, indicating this range of prompt length provides a better performance between base and new category performance. However, when prompt length is over 6 tokens, there is a significant decline in performance. These results reveal that the proposed MaCLe model has strong stability in maintaining performance on base categories. However, to maintain performance and its generalization ability, the prompt length need be selected within the range of 2–4 tokens.

Prompt complexity

We conducted a comprehensive analysis of the computational complexity of MaCLe and compared it with several widely used approaches. As shown in Table 4, it can be observed that although MaCLe introduces additional computation steps during multimodal prompt generation, its floating-point operations (FLOPs) are only slightly higher by approximately 0.1% than those of CoOp⁴⁸ and Co-CoOp⁴⁹. In addition, Co-CoOp exhibits noticeably slower performance, and its frames per second (FPS) fail to improve with increasing batch sizes. In contrast, MaCLe effectively avoids this performance bottleneck and demonstrates superior efficiency in both inference and training. Furthermore, MaCLe shows excellent convergence behavior, requiring only half the number of training epochs compared to Co-CoOp (5 epochs vs. 10 epochs). These findings indicate that MaCLe achieves prompt efficiency without introducing significant computation. Based on CLIP, MaCLe increases the number of trainable parameters by only approximately 2.85%, thereby offering a favorable performance-to-cost ratio in practical applications.

Table 4.

Comparison of computational complexities among different methods.

Method	Params	Params % CLIP	FPS (with BS) 1	FPS (with BS) 4	FPS (with BS) 100	HM
Independent V-L	31,488	0.02	62.5	239.4	1383.8	77.90
CoOp	2048	0.002	13.8	55.3	1353.0	71.66
Co-CoOp	35,360	0.03	64.6	114.7	15.1	75.83
MaCLe	3.56 M	2.79	59.8	239.5	1359.7	78.10

Multimodal configuration

To further substantiate the effectiveness of the proposed multimodal collaborative design, we conduct an ablation study on different multimodal prompting configurations. Specifically, we compare (i) vision-only inference, (ii) text-prompt-only adaptation (CoOp-style), (iii) naive dual-branch prompting without cross-modal interaction, and (iv) MaCLe with vision–language prompt coupling and dynamic prompts. This study isolates the contribution of explicit cross-modal collaboration and verifies that performance gains are not solely driven by prompt depth/length choices. As shown in Table 5, it can be observed that unimodal settings achieve relatively limited performance, indicating that relying on a single modality is insufficient for robust facial expression recognition. Introducing prompts into both the visual and language branches leads to noticeable performance improvements. However, the naive dual-branch prompting strategy still underperforms due to the lack of explicit cross-modal interactions. In contrast, the proposed MaCLe framework consistently achieves the best performance across all evaluated metrics. This demonstrates that explicitly coupling visual and language prompts, together with the dynamic prompt design, enables more effective multimodal collaboration and leads to more robust and accurate recognition results.

Table 5.

Ablation study on different multimodal prompt configurations.

Setting	Vision input	Text input	Prompting strategy	Coupling (F)	Dynamic prompt
(1) Vision-only	✓	✗	none / frozen CLIP	✗	✗
(2) Text-prompt-only (CoOp-style)	✓	✓	text prompts only	✗	✗
(3) Naive V–L prompting (dual prompts)	✓	✓	prompts in both branches	✗	✗
(4) MaCLe (ours)	✓	✓	dual prompts	✓	✓

Limitations and future work

Recent FER research has explored advanced architectures such as Transformer-based models, temporal attention mechanisms, and multimodal fusion strategies that integrate additional cues such as audio or physiological signals. Transformer-based FER methods aim to capture long-range dependencies and richer spatial representations, while temporal attention mechanisms are designed to leverage sequential dynamics in video data. Multimodal fusion approaches combine complementary modalities (e.g., vision–audio, vision–text) to enhance expression understanding, especially in dynamic scenarios. While these methods achieve strong performance in fully supervised settings with abundant labeled data, they typically rely on task-specific architectures, extensive temporal modeling, or additional modality inputs, which limits their applicability in few-shot or low-label scenarios. In contrast, the proposed MaCLe framework focuses on prompt-based adaptation on top of a frozen CLIP backbone, enabling efficient learning with significantly fewer trainable parameters using only limited labeled examples. This fundamental difference means that MaCLe is complementary rather than directly comparable to these recent FER approaches.

Although MaCLe demonstrates promising results across benchmark datasets, several limitations remain. First, the current evaluation is limited to curated datasets and does not include real-time deployment in actual online classrooms. Second, facial expression labels are inherently subjective and may not fully reflect emotional diversity across different cultural contexts. Lastly, the model’s adaptability to noisy or occluded facial inputs warrants further exploration. Future work will focus on cross-cultural generalization, real-time implementation, and integration with multimodal student feedback for enhanced educational impact.

Conclusion

This study presents a novel multi-modal collaborative learning framework for facial expression recognition (FER) in online education. Based on the CLIP framework, MaCLe integrates visual and textual prompts to effectively capture both the spatial and dynamics of student facial expressions. The experimental results across the RAF-DB, FER2013, and AffectNet datasets demonstrate that the proposed MaCLe model achieves superior recognition performance, particularly for complex expression categories. In addition, ablation studies show that prompt depth and length are critical factors for performance. More importantly, MaCLe obtains optimal performance with a prompt depth of 7 and prompt length of 2–4 tokens owing to the balance between representation richness and generalization. Although MaCLe introduces multi-modal prompt learning, its FLOPs increase only marginally by 0.1% compared to CoOp and Co-CoOp. These results indicate that MaCLe has a superior performance in terms of inference speed owing to its faster convergence. Therefore, the proposed MaCLe model can maintain a stable recognition performance of facial expressions and significantly enhance the recognition of new and complex facial expressions. However, it does not introduce significant computation. Future research will further expand MaCLe to cross-cultural facial expression understanding and enhance its adaptability to global online education systems.

Author contributions

J.M. and Y.Y. conceived the study. J.M. designed the methodology and developed the software. H.L. and Z.H. validated the results. J.M. performed the formal analysis and conducted the investigation. J.M. provided resources. H.L. and R.C. curated the data. J.M. prepared the original draft. H.L. and Y.Y. reviewed and edited the manuscript. J.M. generated the visualizations. Y.Y. and H.L. supervised the project. Y.Y. and H.L. administered the project. J.M. acquired the funding. All authors reviewed and approved the final manuscript.

Data availability

The datasets used in this study are publicly available. RAF-DB is available: http://www.whdeng.cn/RAF/model1.html, FER2013 is available at https://www.kaggle.com/datasets/msambare/fer2013, and AffectNet is available at http://mohammadmahoor.com/pages/databases/affectnet/.

Declarations

Competing interests

The authors declare no competing interests.