Collective and augmented intelligence outperform artificial intelligence on emotion recognition tests

Abstract

As artificial intelligence (AI) systems increasingly match human performance on standardized mental state recognition tasks, the question is no longer whether AI is human-level, but which humans define that level. This question remains underexplored. This study evaluates GPT-5 mini against the full spectrum of human ability, not just average performance, on standardized forced-choice emotion and mental state recognition tasks including the Reading the Mind in the Eyes Test (RMET) and the Multiracial Reading the Mind in the Eyes Test (MRMET). At the individual level, GPT-5 mini outperforms human participants across nearly all performance levels on both the RMET and MRMET. Yet this advantage is reversed when independent responses are aggregated. Using bootstrap-resampled plurality voting to aggregate independent responses, we find that human responses significantly outperform those of GPT-5 mini. This wisdom-of-crowds effect cannot be replicated through repeated sampling from AI models. Furthermore, an augmented approach that aggregates bootstrapped human and AI responses together outperforms either source alone. These findings suggest that evaluating AI against average human performance risks mistaking AI mediocrity for human excellence. We discuss the implications of these findings for combining human and machine intelligence to surpass what either achieves in isolation.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-026-45331-5.

Introduction

Emotion and mental state recognition, broadly defined as the ability to discern and interpret others’ emotional and mental states^1,2, is widely considered an important aspect of social cognition, as it shapes how individuals perceive, navigate, and respond to their environments. To measure this skill, researchers commonly use standardized benchmark tasks such as the Reading the Mind in the Eyes Test (RMET)³, although the ecological and construct validity of these tasks remains a subject of ongoing debate^4,5. Despite these debates, performance on these standardized tasks is consistently associated with social functioning and clinical outcomes^6,7. And these results point to a reality in which emotion and mental state recognition is not a baseline faculty possessed equally by all, but a skill that varies substantially, ranging from individuals who struggle significantly to those who excel with high acuity⁸.

This variability, long acknowledged in the literature, becomes methodologically consequential as artificial intelligence (AI) enters the field. As researchers increasingly report that AI models match human participants on standardized emotion and mental state recognition tasks^9–11, the question is no longer whether AI is “human-level,” but which human participants define that level.

This question has gained more relevance with the rapid and widespread deployment of automated systems. AI systems now regularly operate in high-stakes contexts, such as screening individuals for depression in clinical settings or identifying potential deception at border security checkpoints^12,13. Although these applications are ethically contested, scientifically debated, and subject to growing regulatory oversight, including the European Union’s AI Act¹⁴, their proliferation underscores the need for rigorous, empirically sound comparisons between AI models and human capabilities that account for the full range of human ability, not just central tendencies such as average human performance.

To assess whether AI models can match the full spectrum of human ability, our study employs GPT-5 mini, which achieved the highest accuracy among available models at the time of testing (see Supplementary Materials for comparison analyses). Testing a high-performing model provides a demanding benchmark and examines the upper limits of current systems. Indeed, recent years have seen rapid progress in this domain, with large language models (LLMs) reportedly matching or exceeding average human participant performance on text-based emotion and mental state recognition benchmarks^9,15, and multimodal large language models (MLLMs) achieving comparable results on image-based tasks, including GPT-5 mini’s predecessor GPT-4o^11,16. If even a leading model cannot match human participant performance across the full distribution, such findings would suggest that claims about AI surpassing human capabilities must be refined to specify exactly which stratum of human ability is being outperformed. Our study directly tests this proposition.

Current AI assessment practices reveal a significant gap. Typically, evaluations of AI performance rely on central tendency metrics, such as group averages, to determine whether models surpass human abilities. Strachan et al.¹¹, for example, reported that GPT-4o exceeded average human participant performance on standardized forced-choice emotion and mental state recognition tasks on the RMET, and similar patterns have been observed across other emotion and mental state recognition benchmarks^17,18. However, by concentrating solely on mean differences between AI models and human participant groups, such approaches neglect whether models can match or exceed the performance levels of human participants at the highest end of the skill distribution. This oversight matters because exceptional human participants often serve as benchmarks in sensitive, high-stakes fields like clinical psychology or medical diagnostics⁸. Relying solely on comparisons to average human ability risks misrepresenting AI mediocrity as excellence, potentially leading to the deployment of suboptimal systems in contexts requiring specialized emotion and mental state recognition skills. In other words, “human-level” is not a fixed benchmark but varies depending on the comparison level. To directly address this gap, our first research question (RQ1) evaluates how GPT-5 mini’s performance compares across the entire spectrum of human ability, rather than against only the average, using shift-function and stochastic dominance analyses.

Building upon insights gained from considering individual-level performance, we broaden our investigation to collective human intelligence, which is the ability of a crowd to perform more effectively than its average member¹⁹ and is often referred to as the “wisdom of crowds”²⁰. This collective intelligence perspective is particularly valuable because consequential societal decisions are frequently made by groups rather than by individuals alone²¹. Indeed, substantial empirical evidence demonstrates that aggregating multiple independent responses generally results in greater accuracy than decisions by single individuals, as aggregation effectively mitigates individual biases and errors^22,23. Yet this collective advantage emerges from experiential diversity, including varied experiences, heuristics, and knowledge structures that humans bring to their tasks. AI models generate variation through repeated sampling, not lived experience, and whether such sampling yields comparable gains when independent judgments are statistically aggregated remains untested. Our second research question (RQ2) asks whether bootstrap-resampled aggregation of GPT-5 mini responses via plurality voting can match the collective accuracy of similarly aggregated human responses.

Beyond adversarial comparisons of humans versus AI, we also ask whether combining their responses yields additional benefits. Augmented intelligence refers to a collaborative framework in which humans and AI work together to enhance the intelligence of the resulting system²⁴. This augmented intelligence approach is timely given the increasing interest in human-AI collaboration. However, direct human-AI interactions frequently encounter biases and judgment errors such as algorithm aversion or excessive reliance on AI recommendations, potentially reducing the effectiveness of collaborative outcomes^25,26. Our study circumvents these challenges by separately resampling and aggregating human and GPT-5 mini responses through plurality voting, combining their independent judgments without direct interaction. This methodological approach leverages the complementary strengths of human intuition and AI consistency, providing insights into optimal human-AI collaborative frameworks. Our third research question (RQ3) thus assesses whether this augmented approach leads to greater accuracy compared to either source alone.

To systematically investigate our research questions, we used the RMET³ and the Multiracial Reading the Mind in the Eyes Test (MRMET)²⁷, as standardized forced-choice emotion and mental state recognition tasks. These tasks require participants to infer mental states from photographs of the eye region by selecting from four predefined mental state labels, thus offering a controlled measure of emotion and mental state recognition accuracy. GPT-5 mini responses were generated through OpenAI’s application programming interface (API). Throughout this paper, unless otherwise indicated, discussions of AI mainly reflect findings from GPT-5 mini. For human performance benchmarks, we used publicly available datasets comprising responses from 17,680 human participants for RMET and 9,295 for MRMET²⁷. Consistent with established analytical approaches in this literature¹¹, we used mixed-effects logistic regression models to account for variability within and between participants, alongside item response theory (IRT) to assess item-level characteristics. To compare individual-level performance across the entire spectrum of human ability, we used shift-function^28,29 and stochastic dominance analyses³⁰. For collective and augmented intelligence comparisons, we used mixed-effects logistic regression with plurality voting aggregation across varying sample sizes.

Methods

Participants and procedures

In this study, we used publicly available data from Kim et al.²⁷, who collected 17,680 and 9,295 responses for the Reading the Mind in the Eyes Test (RMET) and Multiracial Reading the Mind in the Eyes Test (MRMET), respectively. They described original demographic information, recruitment methods, and consent information. As this study utilized publicly available, anonymized secondary data, it was exempt from Institutional Review Board (IRB) review. For our study, we generated 100 instances without any previous examples (i.e., zero-shot runs) from GPT-5 mini (model version: gpt-5-mini-2025-08-07; knowledge cutoff: May 31, 2024) in November 2025 using specified settings. Zero-shot learning is defined as a procedure in which the model is given a natural language instruction describing the task but is not provided with any explicit examples or demonstrations³¹. By precluding reliance on example demonstrations, we aimed to ensure that performance reflects the model’s baseline capabilities rather than adaptation to exemplars in the prompt context. Human participant data was complete without any missing observations. GPT-5 mini produced complete data for all instances.

We acknowledge that benchmark contamination poses interpretive challenges for LLM evaluations on publicly available instruments³². Both the RMET (published 2001) and MRMET (published April 2024) were technically available before GPT-5 mini’s training data cutoff (May 31, 2024). However, contamination risk depends not merely on availability but on corpus prevalence, which reflects the frequency with which test materials appear across training documents. The RMET has accumulated over two decades of widespread citation and dissemination, including inclusion in educational curricula and extensive online discussion, creating substantial opportunity for training exposure. In contrast, the MRMET was published in April 2024, leaving a short window for dissemination into training corpora prior to the May 31, 2024 cutoff; by that cutoff, it likely had a minimal academic footprint and comparatively negligible online presence. Accordingly, contamination risk is plausibly lower than for RMET. While we cannot definitively exclude contamination for either instrument, these prevalence differences suggest that MRMET results are less susceptible to memorization effects. We return to contamination considerations and their implications for interpreting comparative GPT-5 mini and human participant performance in the Discussion.

Under this zero-shot design, the model analyzed each image from the RMET and MRMET datasets. Complete procedural specifications are provided in Supplementary Materials (Tables S2–S3); we summarize the key elements here. Each image was accompanied by a brief instruction asking the model to select the best answer from four options (see Table S2 for verbatim prompt specification). The four answer options were presented in lowercase letter format (a, b, c, d) in their original dataset order. Images were transmitted as Base64-encoded data (see Table S3 for encoding details), and responses were parsed using structured JSON output with automated answer extraction.

We ran the model with the inference parameters of temperature = 1.0 (controlling randomness in token sampling), top-p (nucleus sampling) = 1.0 (no probability mass truncation), and no frequency or presence penalties. The OpenAI API does not support user-specified random seeds for this model; stochasticity arises solely from the temperature-based sampling process during inference. All API calls used model version gpt-5-mini-2025-08-07 via the OpenAI Python SDK (version 2.6.1). Complete parameter specifications are provided in Supplementary Materials (Sect. 1).

This design yielded 3,600 item-level responses for RMET (100 instances × 36 items) and 3,700 for MRMET (100 instances × 37 items), totaling 7,300 GPT-5 mini responses. When the model produced an invalid response or declined to answer, we retried up to three times. Under this procedure, each item in each instance ultimately yielded exactly one valid response; no items failed after the maximum retry attempts. Combined with human participant data (N = 17,680 for RMET; N = 9,295 for MRMET), the total observations entered into our models were 640,080 for RMET (17,780 × 36) and 347,615 for MRMET (9,395 × 37).

Regarding the GPT-5 mini instances, we note a methodological consideration. Because all instances share identical model weights, they are not independent draws in the classical sense³³. Nevertheless, sampling multiple instances and analyzing them as replicated responses is a common evaluation practice in recent human-AI comparison studies; thus, we follow this convention to enable direct comparability with prior work^11,34. Regardless, to empirically characterize the effective diversity of our GPT-5 mini sample, we calculated between-instance agreement rates at the item level. All 100 GPT-5 mini instances produced identical responses for 36.1% of RMET items and 32.4% of MRMET items (i.e., the percentage of items on which every instance selected the same answer). The mean pairwise agreement rate (the average proportion of items on which any two instances produced matching responses) was 0.86 for RMET and 0.83 for MRMET, corresponding to mean pairwise correlations of and 0.77 and effective sample sizes of and 1.3 independent observations, respectively (; Supplementary Table S4). We conducted the same agreement analysis for other major models (e.g., Gemini and other GPT-5 variants) and observed comparably high within-model agreement. Accordingly, we interpret GPT-5 mini effects and associated uncertainty cautiously given the limited effective sample size implied by these agreement rates.

Materials

To assess performance on standardized forced-choice emotion and mental state recognition tasks, we employed two instruments. RMET is a widely used test for inferring mental states and emotions from eye-region photographs³. It uses 36 black-and-white photographs of the eye region. Human participants select from four options the one that best describes the person’s mental state. RMET demonstrates convergent validity, discriminant validity from vocabulary and processing speed, and split-half reliability²⁷ of .

MRMET²⁷ addresses RMET limitations, including lack of racial diversity and vocabulary accessibility. It uses 37 high-resolution, color photographs with diverse individuals. MRMET demonstrates psychometric properties comparable to RMET, including equivalent convergent and discriminant validity, similar split-half reliability (), and reduced vocabulary dependence²⁷.

While RMET and MRMET are commonly described as measures of emotion and mental state recognition, both tests use a forced-choice format requiring participants to match eye-region photographs to verbal labels. Performance on these tasks may be influenced by vocabulary knowledge, verbal reasoning, and visual processing skills in addition to mental state attribution abilities⁴. Notably, MRMET was designed with accessible vocabulary (at or below fifth-grade reading level) and demonstrates reduced vocabulary dependence compared to RMET²⁷, partially mitigating this concern. To further address these concerns, we also ran Item Response Theory (IRT) analyses, providing item-level psychometric characterization that allows examination of whether human participants and GPT-5 mini exhibit similar patterns of item difficulty (see Supplementary Tables S12–S13). While construct and ecological validity debates remain ongoing, our focus on relative performance patterns across standardized benchmarks allows comparison even as these broader questions are explored. We address these validity considerations further in the Discussion. We conducted all statistical analyses using R version 4.4.1 (lme4 version 1.1.35.5; emmeans version 1.10.6; rogme version 0.2.1). Code and data are available upon request.

Analytical strategies

Analyses were organized by research question. For RQ1, we compared individual-level accuracy between human participants and GPT-5 mini using mixed-effects logistic regression with crossed random intercepts for respondents and items, and we compared the distributions of respondent accuracies using bootstrap shift-function and stochastic dominance analyses. For RQ2 and RQ3, we simulated crowds (bootstrapped plurality-vote aggregations) via bootstrap resampling and plurality voting and modeled plurality-voting accuracy as a function of group type and log-transformed sample size using mixed-effects logistic regression.

Generalized linear mixed-effects model (GLMM)

To compare accuracy between GPT-5 mini and human participants (RQ1), we used generalized linear mixed-effects models (GLMMs) with a logit link, as outcomes were binary for the test items (correct = 1 and incorrect = 0) (Snijders & Bosker, 2012). Each respondent (human participant or GPT-5 mini instance) answered the same set of test items, and each item was answered by multiple respondents, creating a crossed data structure. GLMMs suit this structure, accounting for random effects at both respondent and item levels. These models accommodate unbalanced and crossed designs through partial pooling³⁵.

Formally, letting denote item-level accuracy (1 = correct, 0 = incorrect) for respondent on item , we modeled the probability of a correct response, , as:

where indicates respondent group (human condition [reference] vs. GPT-5 mini condition), is a respondent-level random intercept, and is an item-level random intercept. We implemented this model in R using the lme4 package as glmer(Accuracy ~ Group + (1 | Participant) + (1 | Item), family = binomial). Here, participant refers to the unique identifier associated with the human participants or GPT-5 mini instances. We compared multiple model specifications, including fixed-effects-only, random-item-only, and random-participant-only models, using standard model selection criteria (see Supplementary Materials, Sect. 2).

Given the large number of observations, estimation was performed using the penalized quasi-likelihood approximation to ensure model convergence. Because this estimation provides a faster but less accurate approximation than the Laplace approximation and can be biased for binary outcomes, we interpret small effects and associated uncertainty with caution. To control the family-wise error rate across multiple pairwise comparisons, we applied the Bonferroni correction to the resulting p-values. We derived odds ratios using the emmeans package; odds ratios greater than 1 indicated higher accuracy than the reference human condition.

Item response theory (IRT)

To examine item-level performance differences between human participants and GPT-5 mini, we conducted IRT analyses using two-parameter logistic (2PL) models. IRT models estimate two parameters for each item: discrimination , which indicates how well an item differentiates between respondents of varying ability levels, and difficulty , which represents the latent trait level at which the probability of a correct response equals 0.50³⁶. We fitted multigroup 2PL models using the mirt package in R³⁷, with separate parameter estimates for human participants and GPT-5 mini instances.

To enable meaningful cross-group comparisons, we identified anchor items showing minimal performance differences between groups (absolute accuracy difference < 0.13 for RMET; < 0.09 for MRMET) and no evidence of differential item functioning (DIF). DIF analyses tested whether items showed consistent difficulty differences across ability levels (uniform DIF) or whether difficulty differences varied by ability level (non-uniform DIF), using Firth’s penalized logistic regression to handle data separation³⁸. We applied the Benjamini-Hochberg correction to control the false discovery rate at 0.05. Complete IRT results, including item-level parameters and DIF statistics, are presented in Supplementary Tables S12–S13.

Bootstrap shift-function test for quantile performance

To compare GPT-5 mini and human participant performance across the full distribution of ability (RQ1), rather than only at the mean, we used a bootstrap shift-function analysis with the rogme R package²⁸. Shift functions reveal how two distributions differ by comparing them at multiple quantiles, identifying where and by how much one distribution departs from the other. We compared human participant performance with GPT-5 mini performance across quantiles ranging from 0.03 to 0.97, following standard practice for shift-function analyses²⁹. Extreme quantiles were excluded because the Harrell-Davis estimator exhibits increased estimation variance at distribution tails, and accuracy scores near floor or ceiling limits yield less reliable comparisons³⁹. This approach directly tests whether GPT-5 mini performance varies depending on which segment of the human ability distribution serves as the comparison benchmark.

We calculated performance scores by averaging item accuracy per human participant and per GPT-5 mini instance. For example, if a human participant or GPT-5 mini instance responds correctly to 35 of 36 RMET items, accuracy equals 35/36 = 0.972. We used the Harrell-Davis quantile estimator. To assess between-group differences across distributions, we computed uncertainty intervals for each quantile comparison using percentile bootstrap techniques. This approach provides uncertainty estimates without parametric assumptions. We used the Benjamini-Hochberg method for multiple comparisons adjustment across quantiles and carried out analyses independently for RMET and MRMET tasks. Throughout the manuscript, when interpreting shift-function results, descriptive terms such as “lower-performing” and “higher-performing” refer to positions along the quantile distribution (e.g., quantiles 0.03–0.25 and 0.75–0.97, respectively) rather than discrete performance categories.

First and second-order stochastic dominance tests

To further evaluate distributional differences between GPT-5 mini and human participant performance (RQ1), we conducted first- and second-order stochastic dominance analyses^30,40. While shift-function analysis identifies where distributions differ at specific quantiles, stochastic dominance tests whether one distribution systematically outperforms the other across the entire performance range, requiring consistency across all thresholds rather than at selected points.

First-order stochastic dominance (FSD) exists when one distribution consistently offers higher probabilities of achieving values greater than or equal to any given level. Formally, letting and denote the cumulative distribution functions (CDFs) of distributions and , exhibits FSD over if for all , with strict inequality for at least one . This implies that, at any performance threshold, the dominant distribution has a higher probability of surpassing that threshold. Second-order stochastic dominance (SSD) compares the cumulative areas under the distributions, accounting for both mean and variance. Unlike FSD, which requires one CDF to lie entirely below the other, SSD allows the CDFs to cross while still establishing dominance. exhibits SSD over if for all , with strict inequality for at least one . SSD suggests more consistent and stable performance across various thresholds.

We used bootstrap resampling with 2,000 replicates to construct confidence bands around the empirical CDFs for both groups across all accuracy levels. Unlike conventional hypothesis testing where the null hypothesis posits no difference, stochastic dominance tests reverse this logic by assuming that dominance holds under the null. Consequently, p >.05 indicates that dominance is supported (insufficient evidence to reject it), whereas p <.05 indicates that dominance is violated. We tested dominance in both directions (GPT-5 mini over human participants, and human participants over GPT-5 mini) and performed analyses independently for RMET and MRMET tasks.

Plurality voting: human, artificial, and augmented collective intelligence

To evaluate whether GPT-5 mini crowds can match human collective intelligence (RQ2) and whether human-AI collaboration yields greater accuracy (RQ3), we simulated group decision-making via bootstrapped plurality voting and analyzed the resulting accuracy. Throughout this paper, “crowds”, in this sense, refers to bootstrapped plurality-vote aggregations of responses that were generated in an ad hoc manner without communication among respondents, rather than to empirically observed interacting groups. Our results therefore speak to the benefits of statistically aggregating individually produced responses, not emergent properties of group deliberation. We operationally defined three conditions as follows:

Human collective intelligence (human crowds)

Human crowds were constructed by bootstrapping separately from each dataset’s pool of human participants (N = 17,680 for RMET; N = 9,295 for MRMET). This condition operationalizes collective intelligence as aggregation-based decisions, where plurality-voting benefits emerge from diversity among sampled human participants.

Artificial collective intelligence (GPT-5 mini crowds)

GPT-5 mini crowds were constructed by bootstrapping from the pool of 100 zero-shot GPT-5 mini instances. Because resampled units are stochastic samples from the same model, diversity is limited; thus, any plurality-voting gains are expected to show faster diminishing returns with increasing sample size than in human crowds.

Augmented collective intelligence (human crowds + GPT-5 mini crowds)

Human + GPT-5 mini crowds were defined as hybrid decision-making units comprising both human participants and GPT-5 mini instances. To simulate an augmentation scenario where GPT-5 mini supplements rather than replaces human judgment, we fixed the crowd composition at 20% GPT-5 mini instances and 80% human participants. This proportion was selected so that AI serves as a minority contributor to predominantly human collective judgment, consistent with the study’s augmentation-over-replacement framing. However, we also conducted a sensitivity analysis varying the AI proportion from 10% to 90% (in 10-percentage-point increments) which confirmed that 20% represents a robust augmentation point across both tests (see Supplementary Materials for complete results). For a given sample size n, we pooled votes from 0.2 × n GPT-5 mini instances and 0.8 × n human participants (e.g., for n = 10, the crowd consisted of 2 GPT-5 mini instances and 8 human participants). When 0.2 × n was non-integer, we set the number of GPT-5 mini instances to round(0.2 × n) and assigned the remaining crowd members to human participants (i.e., human participants = n − round(0.2 × n)).

Simulation procedure

For all three conditions (Human, GPT-5 mini, and Human + GPT-5 mini), we simulated crowds across sample sizes (n) ranging from 1 to 100. For each sample size and condition, we performed 500 independent resamples with replacement from the respective source pools. In each resampled group, we applied a plurality voting rule, wherein the item choice receiving the most votes was designated as the collective answer. For example, if seven of ten crowd members selected “skeptical,” then “skeptical” became the crowd’s decision for that item. Ties were resolved via random selection among tied options.

Formally, letting denote plurality-voting accuracy (1 = correct, 0 = incorrect) for resample on item at sample size , we modeled the probability of a correct plurality decision, , as:

where indexes group type (Human, GPT-5 mini, Human + GPT-5 mini), is an item-level random intercept, and is a resample-level random intercept. We implemented this model in R as glmer(Accuracy ~ Group * log(CrowdSize) + (1 | Item) + (1 | ResampleID), family = binomial). The interaction term tested whether the rate of performance improvement differed between group types. Log transformation accounted for diminishing returns in collective decisions⁴¹. We performed separate analyses for RMET and MRMET tasks and calculated marginal slopes using the emtrends function in emmeans.

Findings

RQ1: GPT-5 mini’s performance across the human ability spectrum

Our first research question examines how GPT-5 mini’s performance compares to human participants across the full spectrum of ability levels, rather than only against average performance. We address this question through a series of complementary analyses. Generalized linear mixed-effects models (GLMMs) estimate overall group differences in the probability of a correct response (log-odds), accounting for item and respondent variability. Shift-function analyses reveal how these differences manifest across specific quantiles of the performance distribution; stochastic dominance tests assess whether one group systematically outperforms the other across the entire distribution; and item response theory (IRT) analyses identify item-specific patterns underlying the aggregate differences.

GLMM analysis

We begin our analysis by examining overall performance differences between the human and GPT-5 mini conditions. Descriptive statistics and average scores are presented in Table 1. On the Reading the Mind in the Eyes Test (RMET), GPT-5 mini achieved 0.83 (SD = 0.03) across 100 zero-shot instances, outperforming the human average of 0.71 (SD = 0.14). On the Multiracial Reading the Mind in the Eyes Test (MRMET), GPT-5 mini attained 0.83 (SD = 0.04), outperforming the human average of 0.63 (SD = 0.13).

Table 1.

Summary statistics by conditions for RMET and MRMET.

Dataset	Group	M	SD	Min	Max	n
RMET	Human	0.71	0.14	0.08	1.00	17,680
	GPT-5 mini	0.83	0.03	0.78	0.92	100
MRMET	Human	0.63	0.13	0.03	1.00	9,295
	GPT-5 mini	0.83	0.04	0.73	0.95	100

Accuracy scores represent proportion correct on assessments of RMET or MRMET. AI models evaluated using zero-shot prompting without prior examples or training. Sample sizes for AI models represent instances of API calls. Standard deviations for human participants reflect individual differences in performance; standard deviations for AI models reflect variation across instances. RMET = Reading the Mind in the Eyes Test (36 items); MRMET = Multiracial Reading the Mind in the Eyes Test (37 items). M = mean accuracy; Min = minimum accuracy; Max = maximum accuracy; n = sample size.

GLMMs confirmed these differences while accounting for item and respondent variability (Table 2). Overall, GPT-5 mini achieved higher accuracy than human participants on both RMET and MRMET, consistent with prior findings in the literature. On the RMET, GPT-5 mini showed significantly higher log-odds than human participants (β = 0.668, SE = 0.077, 95% confidence interval (CI) [0.517, 0.819], p <.001, adjusted). On the MRMET, GPT-5 mini showed significantly higher log-odds than human participants (β = 1.177, SE = 0.075, 95% CI [1.030, 1.324], p <.001, adjusted).

Table 2.

Mixed-effects logistic regression results for RMET and MRMET.

	RMET	SE*	MRMET	SE*
	Estimate		Estimate
Fixed-effects
Intercept (Human)	0.961***	0.077	0.684***	0.154
GPT-5 mini	0.668***	0.077	1.177***	0.075
Random effects()
Participant	0.390	0.625†	0.343	0.586†
Item	0.215	0.463†	0.871	0.933†
Model statistics
AIC	730,078.4		396,005.2
BIC	730,123.9		396,048.2
Log-likelihood	−365,035.2		−197,998.6
N Individual	17,780		9,395
N Item	36		37
Total Observations	17,780 × 36 = 640,080		9,395 × 37 = 347,615

N Individual = total number of human participants and GPT-5 mini instances. N Item = total number of test items. Models fitted using generalized linear mixed-effects regression with binomial error distribution and logit link function. Model included fixed effects for group (Human condition vs. GPT-5 mini condition) with crossed random intercepts for participants and items using the following syntax: Accuracy ~ Group + (1 | Participant) + (1 | Item). The intercept represents the baseline performance for human participants on the logit scale. GPT-5 mini coefficient represents the difference in log-odds from human baseline. Random effects include crossed random intercepts for participants (both human participants and independent GPT-5 mini instances) and items (individual test items from each assessment), accounting for individual differences and item-specific difficulty. GPT-5 mini performance reflects zero-shot evaluation without prior training examples. Models were estimated using the penalized quasi-likelihood approximation. RMET = Reading the Mind in the Eyes Test; MRMET = Multiracial Reading the Mind in the Eyes Test; AIC = Akaike Information Criterion; BIC = Bayesian Information Criterion. SE = standard error. †Standard deviation. ***p < .001

While these results establish GPT-5 mini’s average advantage over human participants, such comparisons may obscure important differences across the ability spectrum. We next examine performance at specific quantiles of the distribution.

Shift-function analysis

We conducted a shift-function analysis with bootstrap resampling, comparing GPT-5 mini and human performance distributions across quantiles. Figure 1 shows these differences across quantiles.

Fig. 1.

Performance and distributional differences between GPT-5 mini and human groups on RMET and MRMET.

Table 3 presents quantile-specific comparisons between GPT-5 mini and human participants, employing the Harrell-Davis quantile estimator and percentile bootstrap, with p-values adjusted using the Benjamini-Hochberg method. Reported differences represent quantile estimates (Δ GPT-5 mini – Human, meaning GPT-5 mini minus Human scores), where positive values indicate higher accuracy for GPT-5 mini within each quantile comparison. The analysis revealed that GPT-5 mini generally showed higher accuracy than human participants, though this advantage varied in magnitude and direction across the performance spectrum. Notably, the pattern of this advantage differed systematically across ability levels and between tests.

Table 3.

Comparison of RMET/MRMET performance between GPT-5 mini and human groups across quantiles.

Dataset	Quantile	GPT-5 mini	Human	Difference	95% CI	p	Adjusted p†
RMET
	0.03	0.778	0.389	0.389***	[0.389, 0.408]	< 0.001	< 0.001
	0.05	0.779	0.444	0.335***	[0.333, 0.361]	< 0.001	< 0.001
	0.10	0.794	0.517	0.278***	[0.254, 0.303]	< 0.001	< 0.001
	0.25	0.806	0.624	0.182***	[0.167, 0.194]	< 0.001	< 0.001
	0.50	0.830	0.722	0.108***	[0.085, 0.111]	< 0.001	< 0.001
	0.75	0.840	0.806	0.035***	[0.028, 0.055]	< 0.001	< 0.001
	0.90	0.861	0.861	0.000	[−0.003, 0.007]	0.717	0.717
	0.95	0.871	0.889	−0.018	[−0.028, 0.004]	0.075	0.151
	0.97	0.884	0.917	−0.032***	[−0.056, −0.004]	< 0.001	< 0.001
	Mean			0.142
MRMET
	0.03	0.750	0.375	0.375***	[0.357, 0.406]	< 0.001	< 0.001
	0.05	0.763	0.405	0.358***	[0.335, 0.379]	< 0.001	< 0.001
	0.10	0.783	0.459	0.323***	[0.302, 0.344]	< 0.001	< 0.001
	0.25	0.810	0.541	0.270***	[0.253, 0.271]	< 0.001	< 0.001
	0.50	0.832	0.649	0.183***	[0.163, 0.189]	< 0.001	< 0.001
	0.75	0.854	0.730	0.124***	[0.108, 0.143]	< 0.001	< 0.001
	0.90	0.888	0.795	0.093***	[0.061, 0.122]	< 0.001	< 0.001
	0.95	0.906	0.838	0.069***	[0.042, 0.105]	< 0.001	< 0.001
	0.97	0.928	0.851	0.077***	[0.030, 0.105]	< 0.001	< 0.001
	Mean			0.208

Shift-function analysis using the Harrell-Davis quantile estimator with percentile bootstrap confidence intervals (10,000 bootstrap samples). Quantiles represent the performance distribution from lowest (0.03) to highest (0.97) performers. Difference calculated as GPT-5 mini minus Human. Positive values indicate GPT-5 mini advantage; negative values indicate Human advantage.RMET = Reading the Mind in the Eyes Test; MRMET = Multiracial Reading the Mind in the Eyes Test; CI = confidence interval. †Benjamini-Hochberg method applied for multiple comparisons adjustment for p-values across 9 quantiles.***p <.001. Mean shows the mean of the quantile differences. Differences were computed from unrounded estimates; minor discrepancies (0.001) between displayed column values reflect independent rounding.

On the RMET, accuracy differences were most pronounced at lower quantiles (Δ GPT-5 mini – Human = 0.389 at the 0.03 quantile, p <.001), indicating that GPT-5 mini performs substantially better than lower-performing human participants (Table 3). This advantage progressively diminished at higher quantiles, converging to no difference at the 0.90 quantile (Δ = 0.000, p =.717). Notably, at the highest quantiles, the pattern reversed. Human participants showed significantly higher accuracy at the 0.97 quantile (Δ GPT-5 mini – Human = −0.032, p <.001). This pattern suggests GPT-5 mini’s advantage is concentrated at lower and middle quantiles (0.03–0.75), while human participants at the highest quantiles (0.90–0.97) match or exceed GPT-5 mini on this RMET task. The mean difference of the quantile differences (Δ GPT-5 mini – Human) across all quantiles was 0.142, favoring GPT-5 mini.

In contrast to the RMET, GPT-5 mini’s advantage on the MRMET persisted across all quantiles. GPT-5 mini showed the largest differences at lower quantiles (Δ GPT-5 mini – Human = 0.375 at the 0.03 quantile, p <.001) (Table 3). Unlike the RMET, these differences remained significant throughout the entire distribution (p <.001 for all quantiles), including at the 0.97 quantile (Δ GPT-5 mini – Human = 0.077, p <.001). This pattern indicates that GPT-5 mini’s advantage on the MRMET persists even among the highest-performing human participants, though the gap narrows at higher quantiles. The mean difference of the quantile differences (Δ GPT-5 mini – Human) across all quantiles was 0.208.

Figure 1 presents three complementary visualizations for both RMET and MRMET. Panels A and D show box plots of accuracy distributions for the human and GPT-5 mini conditions, with quantile-based sampling applied to accommodate large sample sizes. Panels B and E display violin plots illustrating density estimates of accuracy, with vertical lines marking quantiles from 0.03 to 0.97 (thicker lines indicate medians). Panels C and F present shift-function plots comparing GPT-5 mini and human accuracy differences (Δ GPT-5 mini – Human) across quantiles. These visualizations confirm that GPT-5 mini’s advantage is most pronounced at lower quantiles and gradually diminishes toward higher quantiles. While insightful, shift-function analysis compares distributions at discrete quantiles rather than across the entire performance range. To test whether either group consistently achieves higher performance across all accuracy thresholds, we next employed stochastic dominance analyses.

Stochastic dominance analysis

We conducted first-order stochastic dominance (FSD) and second-order stochastic dominance (SSD) tests using bootstrap resampling (2,000 replicates), comparing the distributions of GPT-5 mini and human participants on both RMET and MRMET. Importantly, in both FSD and SSD tests, the null hypothesis states that one distribution dominates the other; rejecting the null provides evidence against that dominance claim, whereas failing to reject indicates that we do not have sufficient evidence to refute the dominance hypothesis. Accordingly, large p-values (e.g., p >.05) should be interpreted as “no evidence against dominance.” This framework requires careful interpretation of p-values for directional hypotheses. Figure 2 visualizes these dominance results for both datasets.

Fig. 2.

First and second-order stochastic dominance analysis.

For the RMET, the FSD test compared cumulative distribution functions (CDFs) of GPT-5 mini and human participants. Testing whether human participants dominated GPT-5 mini yielded (p <.001), rejecting human stochastic dominance. Testing whether GPT-5 mini dominated human participants yielded (p =.506), providing no evidence against GPT-5 mini’s first-order stochastic dominance (Supplementary Table S8). Although the shift-function analysis revealed that the highest-performing humans matched or slightly exceeded GPT-5 mini at extreme quantiles (Δ = −0.032 at the 0.97 quantile), this localized reversal was small enough to fall within the bootstrap confidence bounds of the dominance test. We acknowledge that this CDF crossing constitutes a strict theoretical violation of first-order stochastic dominance; however, the bootstrap-based statistical test, which evaluates whether observed violations exceed what would be expected from sampling variability in finite samples, found this departure non-significant. In other words, the dominance test asks whether one group’s CDF consistently lies below the other’s across all accuracy levels; minor, isolated crossings that could arise from sampling variability do not constitute sufficient evidence to reject dominance. In practical terms, across the full distribution of performance, GPT-5 mini maintained a higher probability than human participants of reaching any given accuracy level or above on the RMET.

The SSD test further confirmed this pattern, rejecting human dominance over GPT-5 mini (, p <.001) while providing no evidence against GPT-5 mini’s second-order stochastic dominance (, p =.936; see Supplementary Table S8). This finding indicates GPT-5 mini achieved higher overall accuracy while maintaining greater consistency across most of the performance distribution.

The MRMET results paralleled those of the RMET, consistent with first- and second-order stochastic dominance of GPT-5 mini. The FSD test rejected human dominance (, p <.001) while providing no evidence against GPT-5 mini dominance (, p >.998; see Supplementary Table S8). The SSD test similarly rejected human dominance (, p <.001) while providing no evidence against GPT-5 mini dominance (, p =.911; see Supplementary Table S8). Unlike the RMET, where the highest-performing human participants matched or exceeded GPT-5 mini at extreme quantiles, GPT-5 mini’s advantage on the MRMET persisted across the entire distribution.

These results are consistent with GPT-5 mini achieving both first- and second-order stochastic dominance on both RMET and MRMET. At the distribution level, GPT-5 mini generally has a higher probability than human participants of achieving any given accuracy threshold. Second-order dominance further reveals both higher overall accuracy and greater stability across most of the performance distribution, with lower variance and more stable predictions. These global dominance patterns are consistent with GPT-5 mini’s advantage being concentrated among lower- and middle-performing individuals, with the advantage diminishing (and in the case of RMET, reversing) at the highest performance quantiles.

IRT analysis

To complement the distributional comparisons above with item-level characterization, we conducted supplementary IRT analyses using two-parameter logistic models. Latent trait comparisons confirmed GPT-5 mini’s overall advantage. The effect size was large on the RMET (d = 1.58) and also large, though smaller in magnitude compared to the RMET, on the MRMET (d = 1.01). Differential item functioning (DIF) analyses further identified specific items where human participants significantly outperformed GPT-5 mini. On the RMET, significant DIF favoring human participants emerged for “playful” (human: 67.6% vs. GPT-5 mini: 2.0%, p =.032), “serious” (67.4% vs. 3.0%, p <.001), “thoughtful” (83.2% vs. 39.0%, p <.001), and “hostile” (57.5% vs. 16.0%, p <.001); GPT-5 mini also showed zero accuracy for “despondent” (human: 71.4% vs. GPT-5 mini: 0.0%). On the MRMET, significant DIF favoring human participants emerged for “indifferent” items (human: ranging from 42.5% to 63.3% vs. GPT-5 mini: 4.0% to 46.0%, p <.01) and “confident” (human: 89.2% vs. GPT-5 mini: 59.0%, p <.001). These item-specific blind spots reveal that certain emotions and mental states remain systematically difficult for the model to recognize, a pattern with implications for human-AI collaboration that we revisit in RQ3 (see Supplementary Tables S12–S13 for complete results). Having established GPT-5 mini’s performance relative to human participants across the ability spectrum, we now turn to our second research question examining collective intelligence.

RQ2: collective intelligence (human crowds vs. GPT-5 mini crowds)

To address our second research question, we examined whether GPT-5 mini crowds can match or surpass human crowds when aggregating judgments through plurality voting. For each group size (n = 1 to 100), we performed 500 bootstrap resamples with replacement from the respective participant pools and applied plurality voting to determine collective responses. Accuracy was assessed via GLMMs with fixed effects for condition and log-transformed group size, including their interaction, and random intercepts for items and resampled groups. Separate models were fitted for RMET and MRMET.

Table 4 presents the GLMM fixed effects for both tasks. On the RMET, the intercept (human baseline) was significant (β = 7.108, SE = 0.294, p <.001), with GPT-5 mini showing lower baseline log-odds (β = −4.338, SE = 0.036, p <.001) and a negative interaction with log-group size (β = −4.167, SE = 0.036, p <.001). This negative interaction indicates minimal scaling benefits for GPT-5 mini crowds. The main effect of log-group size was positive and substantial for human participants (β = 4.396, SE = 0.032, p <.001).

Table 4.

Mixed-effects logistic regression results for collective and augmented intelligence.

	RMET	SE*	MRMET	SE*
	Estimate		Estimate
Fixed-effects
Intercept (human)	7.108	0.294	2.983	0.302
GPT-5 mini	−4.338***	0.036	−0.015	0.012
Human + GPT-5 mini	0.144***	0.048	1.062***	0.015
log(crowd size)	4.396***	0.032	1.542***	0.013
GPT-5 mini × log(crowd size)	−4.167***	0.036	−1.235***	0.019
Human + GPT-5 mini × log(crowd size)	−0.300**	0.044	0.383***	0.020
Random effects ()
Resampled Participant	0.086	0.294†	0.028	0.166†
Item	3.068	1.752†	3.362	1.834†
Model statistics
AIC	141,803.8		280,876.0
BIC	141,894.8		280,967.3
Log-likelihood	−70,893.9		−140,430.0
N unique groups	18,000		18,000
N Item	36		37
Total observations	18,000 × 36 = 648,000		18,000 × 37 = 666,000

Models examine the effect of group size on accuracy using plurality voting aggregation. Model specification is Accuracy ~ Group * log(CrowdSize) + (1 | Item) + (1 | ResampleID). The intercept represents baseline Human crowd performance. GPT-5 mini and Human + GPT-5 mini coefficients represent differences in log-odds from Human baseline. Group size is log-transformed to account for diminishing returns. Interaction terms test whether the benefit of increasing group size differs by group type. Random intercepts account for item-level variability and between-group variability across 500 bootstrap resamples per condition. RMET = Reading the Mind in the Eyes Test; MRMET = Multiracial Reading the Mind in the Eyes Test; AIC = Akaike Information Criterion; BIC = Bayesian Information Criterion. SE = standard error. †Standard deviation. **p <.01, ***p <.001.

Table 5 shows the marginal probabilities and odds ratio analysis. On the RMET, it revealed near-ceiling accuracy for human crowds (99.918%, 95% CI [99.855%, 99.954%]) but lower accuracy for GPT-5 mini crowds (94.105%, 95% CI [90.003%, 96.587%]). Odds ratios (OR) confirmed that bootstrapped human crowds significantly outperformed GPT-5 mini crowds (OR = 76.553, p <.001). Slope analysis (Table S11) underscored this pattern, showing that human crowds improved steeply with size (slope = 4.396, 95% CI [4.333, 4.459]), while GPT-5 mini showed minimal gains (slope = 0.230, 95% CI [0.196, 0.263]). In other words, bootstrapped human crowds demonstrate a strong aggregation benefit under plurality voting, while GPT-5 mini crowds show minimal scaling gains.

Table 5.

Performance differences between human, GPT-5 mini, and augmented intelligence groups: predicted probabilities and odds ratios.

Dataset	Model	Accuracy (%)	SE*	95% CI	Contrast	Odds ratio†	p‡
RMET	Human	99.918	< 0.001	[99.855, 99.954]	-	-	-
	GPT-5 mini	94.105	0.016	[90.003, 96.587]	Human/GPT-5 mini	76.553	< 0.001
	Human + GPT-5 mini	99.929	< 0.001	[99.874, 99.960]	Human/(Human + GPT-5 mini)	0.866	0.007
					GPT-5 mini/(Human + GPT-5 mini)	0.011	< 0.001
MRMET	Human	95.200	0.014	[91.600, 97.300]
	GPT-5 mini	95.100	0.014	[91.500, 97.200]	Human/GPT-5 mini	1.015	0.619
	Human + GPT-5 mini	98.300	0.005	[96.900, 99.000]	Human/(Human + GPT-5 mini)	0.346	< 0.001
					GPT-5 mini/(Human + GPT-5 mini)	0.341	< 0.001

Marginal probabilities calculated from the mixed-effects logistic regression models at maximum group size (n = 100). Values represent estimated accuracy back-transformed from the logit scale. The Human + GPT-5 mini condition represents combined predictions from both groups. RMET = Reading the Mind in the Eyes Test; MRMET = Multiracial Reading the Mind in the Eyes Test; CI = confidence interval. SE = standard error. †Odds ratio for group contrasts. ‡Bonferroni-adjusted p-values for three comparisons.

On the MRMET, a different pattern emerged. The human baseline was significant (β = 2.983, SE = 0.302, p <.001), with GPT-5 mini showing a non-significant baseline difference (β = −0.015, SE = 0.012, ns) and a negative interaction with log-group size (β = −1.235, SE = 0.019, p <.001). The main effect of log-group size was positive for human participants (β = 1.542, SE = 0.013, p <.001; Table 4). Table 5 shows that marginal probabilities for human crowds (95.200%, 95% CI [91.600%, 97.300%]) and GPT-5 mini crowds (95.100%, 95% CI [91.500%, 97.200%]) were nearly identical, with no significant difference between them (OR = 1.015, p =.619). Most importantly, slope analysis (Table S11) revealed that human crowds still improved with size (slope = 1.542, 95% CI [1.520, 1.568]), as did GPT-5 mini crowds, albeit more modestly (slope = 0.307, 95% CI [0.280, 0.334]). In other words, while bootstrapped human crowds demonstrate substantial aggregation benefits with increasing group size, GPT-5 mini crowds show minimal scaling gains.

RQ3: augmented intelligence (combined human and GPT-5 mini crowds)

Our third and final research question investigated whether augmented intelligence, formed by combining votes from human participants and GPT-5 mini, outperforms human-only or GPT-5 mini-only crowds. Using the same plurality voting and GLMM framework, we included the human + GPT-5 mini condition in the models.

As shown in Table 4, on the RMET, the augmented intelligence (human + GPT-5 mini) condition showed a significant positive baseline (β = 0.144, SE = 0.048, p =.002) and a negative interaction with log-group size (β = −0.300, SE = 0.044, p <.001), indicating that augmented crowds scaled slightly less steeply than human-only crowds but overall achieved higher accuracy. In Table 5, marginal probabilities analysis confirmed the highest accuracy for augmented crowds (99.929%, 95% CI [99.874%, 99.960%]), surpassing both human crowds (OR = 0.866, p =.007) and GPT-5 mini (OR = 0.011, p <.001). Table S11 shows that the trend slope for augmented groups (4.097, 95% CI [4.030, 4.163]) was slightly lower than human crowds (slope = 4.396, 95% CI [4.333, 4.459]) and substantially exceeded GPT-5 mini (slope = 0.230, 95% CI [0.196, 0.263]). Despite scaling slightly less steeply than human-only crowds, augmented intelligence crowds achieved the highest overall accuracy, indicating that the augmented baseline advantage more than compensated for the modestly lower scaling rate.

For the MRMET, augmented intelligence showed a significant positive baseline (β = 1.062, SE = 0.015, p <.001) and interaction (β = 0.383, SE = 0.020, p <.001) in Table 4. In Table 5, marginal probabilities yielded an accuracy of 98.300% (95% CI [96.900%, 99.000%]), significantly outperforming both human crowds (OR = 0.346, p <.001) and GPT-5 mini (OR = 0.341, p <.001). While human and artificial crowds achieved comparable accuracy on the MRMET, their combination yielded significantly better performance. This result highlights the value of algorithmic diversity. Table S11 shows that trend slopes were highest for augmented groups (1.925, 95% CI [1.890, 1.956]), compared to human crowds (slope = 1.542, 95% CI [1.520, 1.568]) and GPT-5 mini (slope = 0.307, 95% CI [0.280, 0.334]).

Figure 3 visualizes marginal trends by condition and group size for both tasks. Bootstrapped human crowds show effective scaling with increasing group size, while GPT-5 mini crowds show minimal improvement with aggregation. Augmented intelligence crowds demonstrate strong improvements with scale, consistently achieving the highest overall accuracy across both RMET and MRMET. These results address RQ3, showing that augmented intelligence, operationalized through independent aggregation of human and GPT-5 mini predictions, yields superior accuracy on these tasks compared to either human or GPT-5 mini crowds alone.

Fig. 3.

Artificial, collective, augmented intelligence.

Discussion

Our findings addressing the three research questions reveal important patterns in the capabilities and limitations of GPT-5 mini on forced-choice emotion and mental state recognition tasks. First, GPT-5 mini exceeded human performance at most levels on the Reading the Mind in the Eyes Test (RMET) and the Multiracial Reading the Mind in the Eyes Test (MRMET), with shift-function analysis and stochastic dominance tests confirming its overall distributional advantages. While the highest-performing human participants matched or exceeded GPT-5 mini accuracy on the RMET, GPT-5 mini’s advantage persisted across all ability levels on the MRMET. Second, aggregating responses via plurality voting revealed a notable reversal. Bootstrap-sampled human crowds achieved large accuracy improvements while bootstrapped GPT-5 mini crowds showed minimal aggregation gains. Finally, augmented intelligence, which combines human participant and GPT-5 mini responses, produced the highest accuracy, supporting the ensemble-learning principle that combining agents with complementary error patterns surpasses either alone⁴². We discuss each finding’s implications and limitations below.

First, our findings have shown that GPT-5 mini at the individual level, and across almost all levels of performance from the 3rd percentile to the 97th percentile, demonstrated stronger performance. Importantly, we have shown that GPT-5 mini, in both the RMET and MRMET, can outperform individuals at the human group mean level, with a large effect size. However, GPT-5 mini’s individual performance warrants careful interpretation, particularly regarding benchmark contamination, the possibility that test materials appeared in pre-training data⁴³ and the nature of the tests used in this study. The RMET has been publicly available since 2001 and extensively discussed online, creating substantial opportunity for inclusion in web-scraped training corpora. We cannot preclude that GPT-5 mini’s strong RMET performance partly reflects recall rather than true mental state inference. However, the MRMET provides convergent evidence against pure memorization, serving as a natural sensitivity analysis for contamination effects under our zero-shot design. Released in April 2024, just weeks before GPT-5 mini’s May 2024 training cutoff, with limited web distribution, MRMET items likely had minimal presence in training corpora. If GPT-5 mini’s performance were primarily driven by memorization, we would expect substantially degraded accuracy on MRMET compared to RMET. Instead, GPT-5 mini demonstrated strong performance on both instruments (RMET: 83%; MRMET: 83%), with similar patterns of stochastic dominance and large IRT effect sizes (d = 1.58 for RMET; d = 1.01 for MRMET). This consistency across high-contamination-risk (RMET) and low-contamination-risk (MRMET) instruments suggests that GPT-5 mini’s capabilities reflect learned generalization rather than mere retrieval.

Beyond contamination concerns, the distribution of GPT-5 mini’s advantage reveals that accuracy differences with human participants vary systematically across ability levels. On the RMET, GPT-5 mini’s advantage is approximately 40% at the 3rd percentile, converges to zero by the 90th percentile, and reverses at the 97th percentile when human participants exceed by 3% (p <.001). On the MRMET, its advantage persists from approximately 38% at the 3rd percentile to around 8% at the 97th percentile (p <.001). This pattern shows that GPT-5 mini’s diminishing, and on RMET even inverted, performance at higher quantiles reflects limited differentiation potential, not true parity in underlying ability. It remains crucial to acknowledge that this does not imply equal skill levels, as expert human participants may possess nuanced competencies that forced-choice tests cannot capture. These findings are specific to standardized, static image assessments and may not extend to less structured contexts such as open-ended inference or real-time social interaction. Thus, our findings may speak only to forced-choice emotion and mental state recognition tasks with static images, yet to be generalized across domains and contexts.

Our second finding reveals a marked but task-dependent reversal in collective intelligence. When aggregating judgments via plurality voting, bootstrapped human crowds on the RMET improved substantially as simulated group size grew, reaching near-ceiling accuracy (99.92%) while GPT-5 mini crowds stagnated at 94.11% (odds ratio (OR) = 76.55, p <.001). The trend slopes (see Supplementary Table S11) were 4.396 for human crowds versus 0.230 for GPT-5 mini. However, on the MRMET, a different pattern emerged. Bootstrapped human and GPT-5 mini crowds achieved comparable accuracy (95.2% vs. 95.1%, OR = 1.02, ns). This divergence between GPT-5 mini’s individual-level dominance and its diminished collective performance warrants explanation.

We posit that two complementary mechanisms account for GPT-5 mini’s attenuated aggregation benefit. First, a ceiling constraint. With individual accuracy already at 83% (SD = 0.03), there are approximately 17% points of improvement available, compared with approximately 29 points for human participants starting at 71%, which bounds the maximum possible gain. Hence, whatever headroom remains can be realized only if errors are sufficiently independent across instances. Under the Condorcet jury theorem, which broadly holds that majority-vote accuracy among independent voters with above-chance individual accuracy tends to approach certainty as group size increases, 100 truly independent voters at p =.83 would approach near-perfect accuracy²⁰; the observed plateau at 94.11% on the RMET falls well short, illustrating the substantial cost of correlated errors for GPT-5 mini. In other words, with the relatively strong base-level starting points and with a high error correlation, it would be reasonable to see that GPT-5 mini is not able to capitalize on the benefit of wisdom-of-crowd effects.

Second, this divergence in collective intelligence outcomes might stem from how diversity emerges differently in human versus GPT-5 mini crowds. Bootstrapped human crowds embody experiential diversity arising from varied backgrounds, experiences, and perspectives among independently sampled participants. Aggregating such independent perspectives via plurality voting cancels idiosyncratic errors, enabling collective judgments that exceed individual capability, consistent with the wisdom-of-crowds phenomenon²⁰. GPT-5 mini crowds, by contrast, showed substantially attenuated aggregation benefits due to pseudo-independence. Our between-instance agreement analysis revealed that all 100 instances produced identical responses for 36.1% of RMET items and 32.4% of MRMET items, yielding an effective sample size of only 1.2–1.3 independent observations rather than the nominal 100. This limited diversity is reflected in the variance estimates. GPT-5 mini exhibited SD = 0.03 (RMET) and SD = 0.04 (MRMET), compared to SD = 0.14 (RMET) and SD = 0.13 (MRMET) for human participants, a 3- to 5-fold reduction. Because each API call processes information through identical model weights, temperature-based sampling introduces only token-level stochasticity while the underlying decision boundaries remain fixed, yielding highly correlated error patterns.

In a sense, repeated GPT-5 mini sampling resembles asking one highly consistent expert repeatedly rather than polling diverse independent judges. This mechanism accounts for the RMET divergence from individual-level performance to collective-level intelligence, where human crowds reached 99.92% while GPT-5 mini crowds plateaued at 94.11%; the comparable MRMET performance (~ 95% for both) likely reflects a task-specific ceiling that neither group could exceed. Importantly, aggregating GPT-5 mini responses reinforces rather than eliminates its systematic errors, errors that, as our IRT analyses revealed, concentrate on specific emotional states like “playful,” “despondent,” and “indifferent” that human participants recognize more reliably. This explains why augmented intelligence succeeds. Human participants contribute complementary error patterns and item-specific recognition abilities that GPT-5 mini cannot generate through repeated sampling. Resampling one model many times, no matter how accurate, does not replicate the error diversity that independent human judges bring to a collective vote.

Can combining human diversity with GPT-5 mini’s consistent baseline break through the ceilings that constrain each approach independently? Augmented intelligence, created by combining votes from human participants and GPT-5 mini independently without allowing direct interaction, achieved near-perfect accuracy on both tasks, 99.93% for RMET (marginally exceeding human crowds’ 99.92%) and 98.3% for MRMET (breaking through the ~ 95% ceiling that neither group could breach alone). This demonstrates that even a limited injection of AI judgments, complementary to human errors, is sufficient to drive performance beyond what either system achieves alone. A sensitivity analysis across AI proportions from 10% to 90% confirmed the robustness of this finding. On the RMET, augmented crowds significantly outperformed human-only crowds at 10% and 20% AI (odds ratios < 1, p <.01) but not at higher proportions; on the MRMET, augmented crowds outperformed both human-only and AI-only crowds at all proportions from 10% to 80% (p <.0001), with the strongest effects at 20–30% AI. Performance peaked at low AI proportions (20–30%) and declined steadily beyond that point, reflecting the growing dominance of correlated AI errors, converging with AI-only performance at 90% (see Supplementary Materials). Augmented intelligence not only achieved higher accuracy but reached these levels with smaller group sizes, scaling nearly as steeply as human crowds and far more steeply than GPT-5 mini crowds. By aggregating human participant and GPT-5 mini judgments independently, our approach also avoided the overreliance and algorithmic aversion that typically compromise human-AI collaboration, aligning with distributed cognition theories⁴⁴.

Overall, augmented intelligence related findings suggest a direction for intelligent systems development that prioritizes augmentation over replacement. Rather than developing autonomous multimodal large language models (MLLMs) to substitute for human expertise, the goal might shift toward hybrid configurations where AI contributes alongside human judgment. In a similar vein, recent theoretical work increasingly cautions that AI systems can threaten cognitive autonomy and self-perception when they displace human judgment across domains⁴⁵, and framing MLLMs as collaborators rather than surrogates aligns with human-centric frameworks that seek to preserve agency and identity amid fast-evolving AI technologies. Although we do not test these outcomes directly, our results lend empirical weight to this stance. Our approach also addresses established challenges in human-AI collaboration. Cognitive offloading and algorithmic aversion are well-documented across medical diagnosis⁴⁶, education⁴⁷, and workplace settings⁴⁸, and our independent aggregation approach sidesteps these pitfalls by eliminating direct interaction entirely. Whether these benefits extend to dynamic, naturalistic settings, and whether alternative strategies (e.g., weighted voting, multi-model ensembles) offer additional advantages, remain open questions. While derived from standardized benchmark tasks, the findings in this study suggest that human-AI configurations preserving independent judgment may outperform both fully autonomous AI and direct collaboration paradigms.

Importantly, these findings related to emotion recognition and mental state assessment must be situated within broader scientific and regulatory context. Automated emotion inference remains deeply contested. Barrett et al⁴⁹ concluded in a comprehensive review that facial movements are weak and context-dependent predictors of emotional states, challenging the assumption that emotions can be reliably inferred from facial expressions alone. This skepticism is reflected in regulatory action as well. The European Union’s AI Act explicitly prohibits emotion recognition systems in workplaces and educational institutions, recognizing the risks of autonomous affective computing in consequential settings¹⁴. Our findings reinforce rather than challenge this emerging consensus. GPT-5 mini’s systematic blind spots (near-zero accuracy on “playful,” “despondent,” and “serious” items, and lower accuracy on “indifferent” items) demonstrate that even a state-of-the-art MLLM cannot reliably capture the full spectrum of human mental states even with a static and well-defined task, and is most likely to fail in much more fluid and ever-changing social situations in real-world contexts. The limited capacity of bootstrapped AI crowds to reduce error through plurality-vote aggregation further highlights the constraints of autonomous approaches. Our findings offer an empirically grounded path forward. Hybrid human-AI configurations that preserve human judgment at the center of the decision process while leveraging AI’s complementary strengths may offer a responsible alternative where autonomous emotion AI systems have been restricted due to reliability concerns, though any such applications require domain-specific validation far beyond what standardized benchmark tasks can provide.

Several limitations warrant consideration. First, regarding construct validity, RMET and MRMET are commonly characterized as measures of “emotion recognition,” but they are more precisely understood as forced-choice tasks requiring participants to match eye-region photographs to verbal labels describing mental states. Performance on these instruments may be influenced by vocabulary knowledge, verbal reasoning, and visual processing skills rather than solely by emotional or mentalizing ability^4,5. Consequently, our findings should be interpreted as reflecting performance on these specific standardized tasks rather than as direct measures of naturalistic emotion recognition. The extent to which these patterns generalize to real-world emotion perception in dynamic, context-rich social interactions remains an open question, and future research should examine whether the collective intelligence advantage and augmented intelligence benefits replicate with alternative measures of social cognition or in ecologically valid settings.

Second, regarding our IRT analyses, we note a conceptual tension. IRT assumes respondents possess a stable latent trait (e.g., mentalizing ability), but AI systems lack psychological constructs in this sense. Consequently, the “ability” estimates for GPT-5 mini should be interpreted as descriptive performance indices rather than true latent traits, and differential item functioning (DIF) comparisons between human participants and AI are phenomenological rather than psychologically equivalent. Nevertheless, these analyses provide valuable item-level characterization, revealing specific blind spots that aggregate accuracy metrics would obscure.

Third, regarding the independence of GPT-5 mini instances, we acknowledge that API calls sharing identical model weights do not constitute statistically independent samples. As our earlier analyses demonstrated (see also Supplementary Table S4), this pseudo-independence yielded an effective sample size of approximately 1.2 rather than 100. However, this limitation does not undermine our conclusions; the low between-instance variance is not a confound but the empirical signature of why AI crowds cannot replicate the aggregation benefit that bootstrapped human crowds achieve. An exploratory LLM Ensemble analysis combining four architecturally diverse models (GPT-5, GPT-5 mini, Gemini-2.5-Flash, Gemini-2.5-Pro) suggests that cross-architecture diversity provides modest improvement over single-model crowds, yet aggregation benefits still fall well short of bootstrapped human crowds, consistent with evidence that models trained on similar data exhibit correlated failure modes regardless of architectural differences (see Supplementary Tables S15–S17).

In conclusion, while GPT-5 mini’s accuracy on RMET and MRMET outperforms most human participants across the ability distribution, it cannot match the collective accuracy that emerges from aggregating diverse human judgments via plurality voting, where bootstrapped human crowds reached near-ceiling accuracy while bootstrapped AI crowds generated by GPT-5 mini plateaued well below. However, the path with the greatest promise is not a choice between human intuition and machine consistency; it is combining them. Our findings show that an augmented intelligence strategy, which combines human and AI predictions independently using plurality voting, achieves a level of accuracy that neither AI nor human crowds can reach alone. The goal should not be to create autonomous systems that replace human insight, but to build collaborative systems that amplify it. The development of effective and trustworthy AI might move away from creating silicon minds that compete with us and toward building AI collaborators that enhance our collective heart.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

Mustafa Akben worked on conceptualization, data acquisition, original draft writing, statistical analysis, and project leadership. Vinayaka Gude contributed to conceptualization, original draft writing, and manuscript revision. Haya Ajjan contributed to manuscript revision and paper development.

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.