Evaluating the emotional accuracy of AI-generated facial expressions in neurotypical individuals

Abstract

This study examines the ability of generative artificial intelligence to produce facial expressions representing basic emotions in a neutral context using black-and-white cartoon imagery. Mentalization, the capacity to recognize and interpret one’s own and others’ mental states, is critical for social interaction and emotional regulation. We explored the emotional validation of artificial intelligence (AI)-generated images by assessing the agreement between human interpretations of emotions and those generated by an AI model. Thirty-four participants evaluated images depicting six basic emotions: sadness, anger, happiness, surprise, fear, and disgust. Our findings revealed significant variability in human agreement, with higher concordance for sadness, anger, and happiness (87%, 73%, and 69%, respectively) and lower agreement for fear, surprise, and disgust (3%, 9%, and 14%, respectively). No significant gender differences were found in emotion recognition accuracy, although a positive correlation between age and accuracy indicated that older participants may possess greater emotional insight. These results suggest that while AI can effectively replicate certain emotional expressions, its capacity to convey more nuanced emotions remains limited. The study underscores the need for advanced training data that encompass a broader range of emotional expressions and cultural nuances to enhance the applicability of AI in mental health and interpersonal communication. Furthermore, this research is particularly relevant for improving AI's ability to generate clear and accurate emotional cues could advance its utility as a tool for social and emotional learning.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10791-025-09630-1.

Introduction

Mentalization has been defined as the capacity to recognize and interpret one’s own and others' mental states—including thoughts, emotions, beliefs, and intentions [1]. One method of identify how someone is feeling is through the identification of facial expressions[2]. Researchers have identified 6 basic emotions that are constant across cultures [2]. Human emotions and intentions are often inferred through facial expressions, a skill deeply rooted in our capacity for mentalization [3]. This capacity underpins core aspects of social cognition, emotional awareness, and empathy, and is essential for effective social interaction and emotional regulation. Traditional mentalization research has used tools like the Reading the Mind in the Eyes Test (RMET) to assess individuals' abilities to interpret emotions based on certain facial features [4]. Impairments in the ability to interpret emotions from facial expressions can affect numerous psychiatric and neurological conditions, highlighting mentalization as a critical focus in therapeutic interventions aimed at enhancing emotional insight and relational understanding [5]. One of the primary challenges in studying emotions lies in the inherently subjective nature of emotional responses, which can vary markedly across individuals [6]. Thus, achieving substantial agreement among individuals is challenging [7] and is even more complex when comparing generative artificial intelligence to human emotional responses [8].

Recent advances in artificial intelligence (AI) are challenging the historically human-centric domain of emotion generation. With the development of large language models (LLMs) capable of interpreting and generating language, there is growing interest in how AI might recognize and develop faces of emotions. However, the question remains: can AI models develop facial expressions that express emotions that humans agree with? These factors present significant challenges for categorizing emotions. To address this, psychology has developed categorical emotion states (CESs) or discrete models, which identify fundamental emotions, as proposed by Ekman and Mikels [9]. Ekman’s model includes that 6 basic emotions: sad/angry/happy/surprised/afraid/disgusted [2].

In recent years, alongside advancements in automatic emotion recognition, there has been substantial progress in generative artificial intelligence (AI) capable of creating images based on prompts regarding emotions [10]. This progress underscores the necessity of additional validation processes for generated content [11]. Validation can consider various visual elements, including format, color, and texture, as well as connotative aspects such as meaning or intrinsic emotional qualities. However, the emotional validation of AI-generated images has yet to be examined.

Understanding this capability could be beneficial in several ways. It may enhance the development of AI systems for applications in mental health, therapy, or user interaction, where accurate emotional representation is crucial. Furthermore, insights gained from this research could inform the design of more effective AI-driven communication tools, improving their ability to interpret and respond to human emotions in various contexts.

The present study aims to explore whether generative AI can create facial expressions in a neutral context, specifically using black-and-white cartoon imagery, to eliminate potential biases introduced by color. This investigation is grounded in the premise that AI should be able to generate emotional expressions based on universally agreed-upon emotional descriptors. Furthermore, if AI-generated images do not align with human emotional interpretations, the study seeks to identify which specific emotions are more prone to discrepancies in agreement with human perceptions. This work holds particular relevance for research involving neurodivergent populations like autism populations, as individuals with autism may experience challenges in recognizing and interpreting emotional expressions [12]. However, the double empathy hypothesis suggests that difficulties in social interaction between neurotypical individuals and those with autism may stem from bidirectional misunderstandings rather than deficits within autistic individuals [13]. This perspective highlights the importance of creating tools that are accessible and interpretable across diverse populations to bridge these mutual gaps in understanding. Enhancing AI's capacity to generate clear, culturally neutral emotional expressions could support the development of tools to evaluate and teach theory of mind and emotional understanding in autism interventions. For the present study, the authors aimed to create images that might be useful in future research to assess theory of mind abilities. However, before applying these images in such a population, the authors sought to validate their clarity and interpretability within a neurotypical sample. This step ensures that the AI-generated images provide a robust foundation for future applications in diverse populations, and aligns with the principles of the double empathy hypothesis by prioritizing mutual understanding and shared meaning. This research focuses on the decoding of emotional facial expressions generated by AI. We aim to evaluate the extent to which neurotypical adults can accurately interpret the emotions encoded in cartoon images produced by ChatGPT-4o. We are not examining the AI's ability to recognize human emotions.

Research questions/hypotheses

• Research Question 1: To what extent do neurotypical adults agree on the emotion conveyed by AI-generated cartoon facial expressions for six basic emotions (sadness, anger, happiness, surprise, fear, disgust)?

• Research Question 2: Are there specific AI-generated images that elicit consistently high or low agreement in emotion identification among neurotypical adults?

• Research Question 3: Does participant gender influence the accuracy of emotion identification in response to AI-generated facial expressions?

• Research Question 4: Is there a relationship between participant age and the accuracy of emotion identification in response to AI-generated facial expressions?

Method

The present study was approved by the institutional review board for a study evaluating social cognition in adults.

Participants

The sample consisted of 34 neurotypical adult participants (18 male, 16 female) with a mean age of 34 years (SD = 13.42). Participants were recruited through social media platforms and provided informed consent prior to their participation. Participants reported no history of neurological or psychiatric conditions that might affect emotion recognition.

Procedure and design

The study employed a within-subjects design, where each participant viewed and rated all 40 AI-generated images. Participants completed the rating task individually and at their own pace via an online survey platform, Qualtrics. The order of image presentation was randomized for each participant to minimize potential order effects. Participants were provided with clear instructions on how to view each image and select the emotion they believed the AI-generated twin was displaying.

Material development

The development of the image stimuli involved an iterative process of prompt refinement with ChatGPT-4o. Initially, broader prompts were tested (e.g., "Create a sad cartoon face"), but these often resulted in variations in style, number of characters, and the inclusion of contextual elements. To achieve a standardized set of stimuli focusing specifically on facial expressions, the "specific prompt" was developed. This prompt explicitly controlled for:

Number of characters: Ensuring a consistent dyadic interaction (two twins).

Perspective: Specifying that both full faces should be visible to the viewer.

Emotional focus: Clearly designating one twin to display a specific basic emotion and the other a neutral expression.

Demographic variation: Including options for gender and age ("men/women/girls/boys") to explore potential nuances in AI-generated expressions across perceived demographics (though only one demographic set was ultimately used for this study to maintain a manageable number of stimuli).

The decision to use black and white cartoon imagery was made to minimize potential confounding factors associated with color and photorealistic details, allowing raters to focus primarily on core facial features indicative of emotion. The cartoon style was chosen for its simplicity and potential for clearer representation of basic emotional configurations, reducing ambiguity that might arise from more complex or stylized artistic interpretations. No specific constraints were placed on the drawing style generated by ChatGPT-4o beyond the elements specified in the prompt, allowing for the exploration of the AI's inherent stylistic tendencies within the defined parameters. Several initial image generations were reviewed informally by the research team to ensure the AI was consistently adhering to the prompt and producing usable stimuli before finalizing the set of 40 images.

The decision to utilize black and white cartoon imagery was also made to prioritize the clarity of core emotional expressions by minimizing potential confounding variables present in realistic faces, such as variations in age, attractiveness, identity, and ethnicity. Ethnicity, in particular, can influence the expression and perception of emotions, and by using standardized cartoons, we aimed to focus on universally recognized facial configurations. Cartoons offer a more controlled representation of basic facial movements, potentially making them more accessible for initial evaluation and for future application in diverse populations. Furthermore, the absence of color and complex shading aimed to focus participants' attention on the fundamental facial muscle movements intended to convey each emotion. While acknowledging that realistic faces may provide different insights into emotional recognition, potentially revealing important aspects related to ethnic cues in emotional expression and perception, this initial choice allowed for a controlled examination of the AI's ability to generate basic emotional signals in a less complex visual format. Future research will explore the use of more realistic stimuli to investigate these nuances further.

Measures

The primary measure in this study was the level of agreement between human raters and the intended emotion as defined by the AI prompt. For each of the 40 AI-generated images, participants were asked to identify the emotion being displayed by selecting one option from the following list: sad, angry, happy, surprised, afraid, disgusted, or other (with an open-ended text box for specification). Participants were instructed to choose the single best label that corresponded to the emotion they perceived in the AI-generated facial expression. The primary dependent variable in this study was the categorical emotion label selected by each participant for each image. The analysis focused on the frequency of each selected emotion category and the level of agreement between the participants' choices and the emotion that the AI was prompted to generate. No Likert scales or slider ratings were used for the primary emotion identification task.

AI ratings were not directly presented to the participants. The "AI rating" refers to the emotion that the prompt was designed to elicit from ChatGPT-4o.

On the human side, in addition to the emotion label selected, we collected demographic information including age and gender via a brief pre-task questionnaire. This allowed us to explore potential relationships between participant characteristics and their accuracy in rating the AI-generated emotions, as reported in the Results section.

Data analysis

The primary dependent variable in this study was the emotion label selected by each participant for each of the 40 AI-generated images. The independent variables explored were the intended emotion of the AI (sad, angry, happy, surprised, afraid, disgusted), participant gender, and participant age.

Agreement on the AI-generated emotional expressions was the central focus of our analysis and was assessed in two main ways:

For each of the 40 images, agreement was calculated as the percentage of the 34 participants who selected the emotion label that matched the emotion the AI was prompted to generate for that specific image. For example, if the AI was prompted to create a "happy" image, the agreement for that image was the percentage of participants who labeled it as "happy." This allowed us to identify specific instances where the AI's intended emotion was either clearly or poorly recognized by the human raters.

For each of the six basic emotions, overall agreement was calculated as the average percentage of correct identifications across all the images generated for that emotion. For instance, if there were 7 images intended to depict "anger," the overall agreement for "anger" was the average percentage of participants who correctly labeled those 7 images as "anger." To explore the influence of participant gender on the accuracy of emotion identification, we calculated an overall accuracy score for each participant. This score represented the proportion of the 40 AI-generated images for which the participant's emotion rating matched the emotion intended by the AI prompt. We then conducted an independent samples t-test to compare the mean accuracy scores of male and female participants, with participant gender as the independent variable and overall accuracy score as the dependent variable. Finally, to examine the relationship between participant age and the accuracy of emotion identification, we used Pearson correlation analysis. In this analysis, participant age was the independent variable, and the overall accuracy score (as defined above) was the dependent variable.

The analysis proceeded in two distinct steps to evaluate the emotional accuracy of the AI-generated facial expressions:

1. Identification of High and Low Agreement Images: First, we identified specific images that demonstrated the highest and lowest levels of agreement among the human raters regarding the expressed emotion. This was achieved by calculating the percentage of raters who selected the emotion intended by the AI prompt for each of the 40 images. We then selected the top 5 images with the highest agreement (defined as 90% or higher) and the top 5 images with the lowest agreement (defined as less than 15%). This step allowed us to examine the characteristics of AI-generated expressions that were either consistently or inconsistently recognized by human observers.

• (b) Calculation of Overall Agreement by Emotion: Second, we calculated the overall (average) agreement for each of the six basic emotions. For each emotion category (e.g., all "sad" images), we determined the percentage of human raters who correctly identified the intended emotion. This provided a measure of the overall accuracy of the AI in generating recognizable facial expressions for each specific emotion.

Following these primary analyses, further statistical tests were conducted using IBM SPSS software:

• Descriptive statistics were calculated to summarize the demographic characteristics of the sample (gender and age distribution).

• Independent samples t-tests were used to explore the effect of participant gender on overall accuracy scores (agreement with AI ratings).

• Pearson correlation analysis was employed to examine the relationship between participant age and overall accuracy scores.

Key Changes Made:

• Added a "Data Analysis" subsection with clear subheadings (a) and (b).

• Explicitly stated the two-step analysis process.

• Detailed how the high and low agreement images were identified (percentage thresholds).

• Clarified how overall agreement was calculated for each emotion category.

• Retained the information about the subsequent statistical tests (t-tests and correlations) under the "Following these primary analyses…" section to maintain the logical flow.

Results

Research participants were predominantly male (53%, 18 out of 34) with an average age of 34 years (SD = 13.42). We analyzed the five images that showed the highest level of agreement with AI (i.e., 90% or higher) and the five images with the lowest agreement (i.e., less than 15%). Among the images with the greatest agreement, two depicted disgust, one depicted surprise, one depicted sadness, and one depicted anger. In contrast, the images with the lowest agreement included four fear images and one surprise image. Overall, AI-generated images showed the highest agreement with human raters for sadness (87%), anger (73%), and happiness (69%), while disgust (14%), surprise (9%), and fear (3%) showed the lowest levels of agreement.

The independent samples t-test revealed no significant difference in accuracy scores between males (M = 0.35, SD = 0.19) and females (M = 0.35, SD = 0.21), t(29) = 0.003, p = 0.958, indicating that gender did not impact the accuracy of ratings. A Pearson correlation analysis revealed a moderate, positive correlation between accuracy score and age, r = 0.43, p = 0.015, indicating that higher accuracy scores were associated with older participants (see supplemental for our images).

Discussion

The current study sought to investigate the ability of generative AI to create facial expressions representing basic emotions, specifically within a neutral context using black-and-white cartoon imagery. Based on the findings from our neurotypical sample, we observed significant variability in human agreement with the AI-generated emotional expressions evaluated, suggesting potential early insights into both the promise and current limitations of AI in mimicking human emotional recognition.

Our results demonstrate that the highest levels of agreement between human raters and AI-generated images were found for the emotions of sadness, anger, and happiness, with agreement percentages of 87%, 73%, and 69%, respectively. This aligns with existing literature indicating that these emotions are more universally recognized and consistently expressed across cultures [2]. Conversely, the lowest agreement was observed for fear and surprise, emotions known to be less reliably interpreted, potentially due to subtle visual cues or contextual dependencies that make them harder to convey effectively [6].

The low agreement for disgust (14%) further underscores the challenges AI faces in generating nuanced or culturally specific expressions. This limitation could significantly impact the use of AI-generated imagery in therapeutic or educational contexts where emotional accuracy is critical [5]. Holland et al. note that human emotional interpretation is inherently subjective, suggesting that such variability could reflect not only AI's current shortcomings but also differences in individual human perception [7].

The study’s lack of significant gender differences in recognition accuracy suggests that emotional perception of AI-generated expressions is not strongly influenced by gender. This is noteworthy given previous studies that suggest subtle gender-related differences in emotional recognition [1]. Additionally, the observed positive correlation between participant age and recognition accuracy highlights how life experience and emotional insight may play a role in interpreting emotional cues, aligning with findings that indicate age-related improvements in emotional understanding [3].

This work has important implications for research with autistic populations, as historically researchers have noted that individuals with autism often exhibit variability in recognizing and interpreting emotional expressions [12]. Autism research has traditionally focused on deficits in theory of mind or emotion recognition. However, the double empathy hypothesis [13] reframes this as a bidirectional challenge in mutual understanding between autistic and neurotypical individuals. This hypothesis suggests that social interaction difficulties are not solely rooted in autistic individuals but in the dynamic interaction between different cognitive and perceptual styles.

In this context, generative AI tools offer a unique opportunity to bridge these mutual gaps in understanding by creating standardized emotional expressions that can be used in autism interventions. The ability to generate culturally neutral and easily interpretable emotional cues is particularly relevant for adults with autism, who may benefit from tools designed to enhance their social communication and theory of mind skills. For example, AI-generated imagery could be used to develop interactive exercises that help individuals practice identifying and responding to emotional expressions, thereby fostering social-emotional learning.

For the present study, the authors sought to create a set of AI-generated images that could eventually be applied in research with adults with autism to evaluate theory of mind abilities. Before introducing these tools to autistic populations, it was essential to validate the images within a neurotypical sample to ensure their clarity and interpretability. This step establishes a foundational benchmark, ensuring that the images are robust and accessible across diverse populations. Aligning with the double empathy hypothesis, this approach prioritizes mutual understanding and shared meaning, offering the potential to design tools that not only support autistic individuals but also promote greater inclusivity and understanding within broader social contexts.

The study’s findings emphasize that while AI shows promise in replicating certain emotional expressions, further refinement is needed to enhance its accuracy and utility. Specifically, the consistent low recognition of fear suggests a critical area for improvement in AI models. Future research should investigate the specific visual features that AI struggles to accurately reproduce for this emotion. Furthermore, exploring why some surprise expressions were highly recognizable while others were not could provide valuable insights into the nuances of generating more consistently accurate emotional cues. Incorporating diverse training data that capture cultural and contextual nuances could improve AI's ability to convey more complex emotions, ultimately enhancing its relevance in fields such as autism research, mental health, and social communication interventions.

The current study, while focusing on a neurotypical sample, represents a critical initial step towards evaluating the potential of AI-generated facial expressions for social and emotional learning tools, particularly for autistic individuals. Our decision to first assess the interpretability of these images within a neurotypical population was driven by the need to establish a clear baseline of recognition for these stimuli. By understanding how neurotypical individuals perceive these AI-generated emotions, we can better identify any inherent ambiguities or limitations in the AI's output, independent of neurodevelopmental differences. This foundational data will be crucial for comparison in our subsequent research with autistic participants. By contrasting the responses of neurotypical and autistic individuals, we aim to gain valuable insights into potential differences in emotional processing and recognition of AI-generated cues. This comparative approach will ultimately inform the development of more targeted and effective AI-supported interventions designed to enhance social-emotional understanding for autistic individuals, potentially contributing to a deeper understanding of the double empathy challenge in the context of novel AI tools.

Limitations

While this study provides valuable insights into the capability of generative AI to produce facial expressions representing basic emotions, several limitations must be acknowledged. First, the study's sample size was relatively small (n = 34), which may limit the generalizability of the findings. A larger and more diverse sample would enhance the robustness of the conclusions drawn and allow for more nuanced analyses across different demographics, including variations in cultural background, socioeconomic status, and levels of emotional intelligence. However, this study was intended as a pilot investigation to establish the feasibility of using AI-generated images for emotion recognition research and to identify initial trends in human agreement. Future research with larger and more diverse samples is necessary to confirm and extend these preliminary findings. Second, the use of black-and-white cartoon imagery, while designed to eliminate color biases, may not accurately reflect the complexities of real-world emotional expressions. Human facial expressions are influenced by numerous factors, including context, cultural norms, and individual differences in emotional expression. Future research should explore the effectiveness of AI-generated images using more realistic facial representations to assess whether these findings hold true in more lifelike scenarios. Additionally, the method of data collection relied on self-reported measures of emotion recognition, which can be subjective and influenced by individual biases. While the study attempted to standardize the prompts given to participants, factors such as personal experiences with the depicted emotions could impact their interpretations. Including a more objective measure, such as physiological responses or neuroimaging, could provide complementary data to assess emotional recognition more accurately. Moreover, the study focused on a limited set of six basic emotions, as proposed by Ekman and Friesen (1971). This narrow scope may overlook the complexity of human emotions, which encompass a broader spectrum beyond these basic categories. Future research should consider integrating additional emotions, including those that are more complex or culturally specific, to better evaluate AI's capabilities in generating a wide range of emotional expressions. Finally, the current study does not explore the implications of the findings in applied settings, such as mental health interventions or AI-driven communication tools. Understanding how these AI-generated emotional expressions affect user interactions, therapeutic outcomes, or emotional engagement in real-world contexts remains an important area for future research.

Conclusion

In conclusion, our research highlights both the potential and limitations of AI-generated emotional expressions in replicating human emotional recognition. While certain basic emotions are well represented, others require further refinement. This underscores the need for ongoing exploration to enhance the validity of AI in emotional contexts and its applications in fields such as mental health, education, and social communication. The findings also have significant implications for autism research and neurodiversity. Understanding and supporting diverse ways of perceiving and interpreting emotions is critical, especially for individuals on the autism spectrum who may approach social interactions differently from neurotypical individuals. The double empathy hypothesis reframes challenges in social communication as mutual misunderstandings between differing cognitive and perceptual styles rather than deficits within autistic individuals alone. This perspective encourages the development of AI tools that foster inclusivity and shared understanding, addressing the needs of neurodiverse populations while also educating neurotypical individuals about alternative ways of perceiving emotions and social cues. Future studies should prioritize training AI models with datasets that encompass diverse cultural, contextual, and neurodiverse perspectives. Such advancements will not only improve the accuracy and utility of AI-generated emotional expressions but also contribute to bridging gaps in mutual understanding, ultimately enhancing the role of AI as a tool to complement human emotional intelligence and foster inclusivity in research, therapy, and everyday interactions.

Future research should explore the role of individual differences in emotion processing by incorporating measures such as alexithymia (e.g., using the PAQ-S) and autism spectrum traits (e.g., using the AQ-10) in participant samples. Investigating whether these factors predict or moderate the accuracy of emotion recognition for AI-generated facial expressions could provide valuable insights into the nuances of human-AI interaction in the realm of social-emotional cues. Understanding these relationships could also have implications for the design of AI-supported tools for individuals with varying neurodevelopmental profiles.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

AFP wrote the manuscript text. All authors, including KAL and RA reviewed the manuscript.

Funding

This research was funded by the University of Texas Health Science Center Houston, Department of Psychiatry and Behavioral Sciences Seed grant, an Autism Speaks Postdoctoral Fellowship Grant (#13904), and the National Center for Advancing Translational Sciences (NCATS) of the National Institutes of Health, through UTHealth-CCTS grant number [5TL1TR003169-05 and T32TR004904, and the Eunice Kennedy Shriver National Institute Of Child Health & Human Development of the National Institutes of Health under Award Number K99HD118079]. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the Center for Clinical and Translational Sciences or the National Institutes of Health.

Data availability

Data is available upon request.

Declarations

Ethics approval and consent to participate

Informed consent was obtained from all individual participants included in the study by the Internal Review Board at UTHealth Houston. For participants under the age of 18, informed consent was obtained from parents or legal guardians, along with assent from the participants themselves. The protocol was approved by the Internal Review Board at UTHealth Houston in accordance with the Declaration of Helsinki. Informed consent was obtained from all participants prior to their participation in the study.

Competing interests

The authors declare no competing interests.