On Benchmarking Human-like Intelligence in Machines

Abstract

Recent advances in Artificial Intelligence (AI) have yielded powerful computational models that, by learning from vast amounts of human-generated data, are increasingly posited as approximate models of human cognition. However, we argue that many current evaluation paradigms for AI are insufficient for assessing human-like cognitive capabilities in these models. We identify a set of key shortcomings: a lack of human-validated labels, inadequate representation of human response variability and uncertainty, and reliance on simplified and ecologically invalid tasks. We support our claims by conducting a human evaluation study on nine existing AI benchmarks, suggesting major limitations in task and label designs. To address these limitations, we propose five concrete recommendations for future AI evaluation efforts that will enable more rigorous and meaningful understanding of human-like cognitive capacities in AI models.

INTRODUCTION

From the earliest days of cognitive science and artificial intelligence (AI), the vision of creating machines that think and act like humans has captured the imagination of researchers and the public alike (Anderson et al., 1990; Cave & Dihal, 2023; Lake et al., 2017; Turing, 1950; Weizenbaum, 1966). This pursuit is driven not only by scientific curiosity (to better understand and model human intelligence) but also by the potential of human-like AI to reshape our world, through the ways that we engage with our work and with each other.

Recent years have witnessed a surge in claims that AI systems have achieved human-like behavior on various tasks. However, the relevance of these results for determining whether AI systems act in a way that is human-“like” is challenged by the limitations of existing evaluation benchmarks.

In this perspective paper, we argue that current evaluation tools for AI systems are often insufficient for assessing human-like cognitive capabilities and future evaluation benchmarks can draw lessons from rigorous cognitive science research. Specifically, we highlight a few shortcomings: the too-frequent absence of human validation in dataset labeling, inadequate representation of human variability in collected human data, and over-reliance on simplified tasks that lack ecological validity and fail to reflect the complexity of real-world scenarios. We support these claims with a human evaluation study on 9 well-known AI benchmark tasks, showcasing potential limitations along these three axes. To address these critical gaps, we propose five concrete recommendations for the development of future benchmarks, derived from best practices in cognitive modeling. We believe these recommendations will pave the way for more rigorous and meaningful evaluations of human-like AI for both cognitive scientists and AI researchers. Importantly, our recommendations are specifically targeted at benchmarks whose goal is to assess whether AI systems exhibit human-like cognitive capabilities, i.e., benchmarks that evaluate whether AI systems reason, perceive, or make decisions in ways that are qualitatively similar to humans. We acknowledge that benchmarks designed to measure domain-specific capabilities where the goal is superhuman performance (e.g., mathematical theorem proving, protein structure prediction) or benchmarks that evaluate abstract reasoning at the level of the best human experts (e.g., ARC-AGI) may have different evaluation goals, and our recommendations may not apply to those settings in the same way. We close with open questions and challenges of implementing these recommendations.

BUILDING AND EVALUATING HUMAN-LIKE AI

There has been a long history of interest in building and evaluating human-like intelligence in machines. But what do we mean by human-like intelligence? There is no universal definition; indeed, whether behavioral similarity alone is sufficient for attributing intelligence has been debated since at least Block (1981), who argued that a system could produce human-like outputs without any genuine cognitive processing. We argue that part of this disagreement arises because “human-likeness” can be assessed at different levels of analysis. A useful framework for thinking about this comes from Marr (1982), who proposed three levels at which any information-processing system can be understood: the computational level (what problem is being solved and why), the algorithmic/representational level (what representations and processes are used to solve it), and the implementational level (how the system is physically realized). A machine could be human-like at any one of these levels without necessarily being human-like at the others. For instance, a system might arrive at human-like judgments through entirely non-human internal representations.

Much of the current work in AI evaluation has focused on the behavioral level, assessing whether AI systems produce outputs that are similar to those of humans on cognitive tasks. This corresponds most closely to Marr’s computational level: characterizing the input-output mapping that the system computes, without committing to claims about how that mapping is internally achieved. In this paper, we adopt this behavioral perspective. Specifically, we define human-likeness at the behavioral level as the capacity of an AI system to produce judgments and behaviors that are qualitatively similar to those of humans across a range of cognitive tasks, capturing not only which response the system gives, but also its patterns of errors, uncertainties, and the distributional variability observed across human populations. On the other hand, human-likeness at representational level, often called representational alignment (whether a model’s internal representations are structured similarly to those of humans), is also an important and growing area of research (Muttenthaler et al., 2024; Sucholutsky, Muttenthaler, et al., 2023).

It is also important to distinguish human-like from human-level intelligence. “Human-level” typically refers to achieving quantitative performance parity with humans, matching or exceeding human accuracy on a task. “Human-like,” by contrast, refers to qualitative similarity in cognitive patterns: not just getting the same answers right, but exhibiting similar patterns of uncertainty, similar distributions of responses across a population, and similar error profiles. A system could achieve human-level performance on a mathematical reasoning task by solving problems correctly using methods entirely unlike human cognition; such a system would be human-level but not necessarily human-like. Conversely, a system that makes errors in human-like ways and shows graded uncertainty similar to humans may be human-like even if its overall accuracy is lower.

But why may we aim for human-like AI? The pursuit of human-like AI is motivated by both scientific curiosity and practical considerations. From the earliest days of AI, scholars have sought to understand, model, and attempt to replicate the intricacies of human cognition and intelligence (Minsky, 1988; Mitchell, 2024; Rosenblatt, 1958; Rumelhart et al., 1988) and use these cognitively-informed models for practical applications. Building human-like AI offers a powerful lens through which to explore fundamental questions about the philosophy of mind, the nature of human cognition, and the underlying mechanisms driving complex human behavior. This quest not only pushes the boundaries of computer science but also promises to deepen our understanding of human intelligence.

Creating AI systems that exhibit human-like thinking and behaviors offers several potential advantages for applications. Human-like AI can think and act instead of humans in many scenarios while ensuring safety and reliability:

• Effective Human-AI Interaction: Humans have developed complex social cognitive skills for effective collaboration, which involves simulating other agents’ mental states and future actions (Bandura, 2001; Gallese, 2007). AI systems that adhere to human-like patterns of reasoning and behavior can enable human users to easily construct accurate mental models of the AI partner and better simulate and predict the AI partner’s future actions (Collins, Sucholutsky, et al., 2024). This leads to more effective collaboration and coordination between human users and AI agents (Carroll et al., 2019; Ho & Griffiths, 2022; Zhi-Xuan et al., 2024). Additionally, interacting with agents that behave predictably and understandably can reduce cognitive load (Dragan et al., 2013; Fisac et al., 2020). We don’t have to expend as much mental effort trying to decipher unfamiliar or unexpected behaviors.

• Better simulated agents: AI systems with human-like cognitive capabilities are valuable tools for building simulations of people. This has many benefits, including improving communication (Liu et al., 2023; Shaikh et al., 2024), generating feedback on pilot studies, and even potentially automating human participant responses in social sciences (Ashokkumar et al., 2024; Demszky et al., 2023; Park et al., 2024) or Human Computer Interaction (Hämäläinen et al., 2023). Prior work has also explored the use of LLMs for product testing (Brand et al., 2023) and substituting human subjects in software engineering (Gerosa et al., 2024).

That said, it remains unclear exactly which domains would most critically benefit from human-like AI. We argue that there are two complementary perspectives. First, from a scientific standpoint, rigorously comparing machine intelligence to human cognition across the full range of cognitive tasks, including error patterns, biases, and uncertainty, is valuable for advancing our understanding of both human and machine intelligence, regardless of practical application. Second, from a practical standpoint, benchmarking human-likeness is especially important for subjective tasks that lack a normative standard of correctness, such as judgments of moral permissibility, social appropriateness, or humor. In these domains, evaluating AI against a fixed ground-truth label is fundamentally misguided, and human-like distributional evaluation becomes not just informative but necessary. This contrasts with tasks that do have a normative standard of correctness, such as mathematical theorem proving, program synthesis, or protein structure prediction, where reproducing human error patterns may even be counterproductive.

Given the scientific and practical benefits of building human-like AI, evaluating whether models genuinely exhibit human-like traits and behaviors is a critical and growing area of interest for both cognitive science and AI. Historically, many AI benchmarks have been developed to measure model performance on tasks requiring human-like cognitive capabilities (e.g., language understanding, common sense reasoning, etc.). These AI benchmarks typically rely on simple accuracy metrics, comparing the model’s output against a single ground-truth label, often produced by curators or human annotators. This approach provides a straightforward measure of performance which facilitates easy comparison across different models. In this paper, we will critically examine these common evaluation practices and discuss how future AI benchmarks can be significantly improved by integrating insights and methods from cognitive science research.

BENCHMARK SELECTION AND EVALUATION

To motivate our recommendations, we collected human data on 9 commonly used AI benchmarks. We selected 7 benchmarks from BIG-Bench (Srivastava et al., 2022) under the common-sense reasoning category and two Theory-of-Mind reasoning benchmarks, BigToM (Gandhi et al., 2024) and ToMBench (Chen et al., 2024). The benchmarks are described in Table 1. We chose these benchmarks because they target inherently subjective cognitive tasks, such as moral reasoning, social cognition, and humor detection, as well as Theory of Mind reasoning, that we believe would benefit most from rigorous practices drawn from cognitive science, including the collection of human-validated labels, measurement of response uncertainty, and evaluation against population-level distributions. Many focus on language understanding and social cognition, which are particularly pertinent for human-AI interaction. All 9 benchmarks have a single ground truth label for each stimulus.

Table 1. .

Benchmark tasks used in our experiment to evaluate human response distributions and levels of agreement.

Task	Description
BIG-Bench (Srivastava et al., 2022)
Fantasy reasoning	Reason about scenarios that violate the ordinary rules of the world.
Social IQA	Reason about typical social situations.
Moral permissibility	Reason about morally permissible actions in scenarios.
Simple ethical questions	Give perspectives on a set of hypothetical, consequential, political, and social questions.
Social support	Distinguish supportive and unsupportive language uses.
Irony identification	Determine whether a text is meant to be ironic or not.
Dark humor detection	Detect whether a particular piece of text is intended to be humorous (in a dark way) or not.
ToMBench (Chen et al., 2024)
Ambiguous story task	Reason and answer questions about ambiguous social situations.
BigToM (Gandhi et al., 2024)
Theory of Mind reasoning	Answer questions about agents’ beliefs and actions.

We randomly sampled 30 stimuli from each benchmark and recruited 117 participants from Prolific, yielding 2,685 annotated trials. The human study was approved by an IRB. Each participant was paid at a rate of 15 USD per hour and completed a consent form before the annotation tasks. We used the same answer options provided by the benchmarks, but instead of using a multiple choice question we asked participants to drag a slider on a scale from 0–100 (e.g., 0 = strongly disagree, 100 = strongly agree) for each answer option. Additionally, after each slider response, participants were asked to report their confidence in their judgment on a separate scale from 1 (not at all confident) to 100 (extremely confident). This two-measure design allows us to disentangle graded judgments (e.g., feature intensity or degree of belief) from epistemic uncertainty (how confident a participant is in that judgment), addressing a potential confound inherent in slider-only elicitation methods.

We highlight some aggregate statistics and diagnostic examples in the section below to support our arguments. More detailed analysis and examples can be found in the Appendix.

PITFALLS AND RECOMMENDATIONS FOR BENCHMARKING HUMAN-LIKE AI

In this section, we present recommendations for evaluating “human-like” AI. More recently, a growing body of work has offered cognitive-science-inspired recommendations for AI evaluation that overlap with and complement our proposals. Frank (2023) argued for careful, incremental evaluation of language model capacities using methods from developmental psychology. Ivanova (2025) outlined guidelines for evaluating the cognitive abilities of LLMs, emphasizing construct validity and controlled experimental design. Shiffrin and Mitchell (2023) discussed the challenges of using psychological methods to probe AI models and the interpretive pitfalls that can arise. From the comparative cognition perspective, Rane et al. (2024) and Voudouris et al. (2024) have advocated for applying principles from animal cognition research to AI evaluation, emphasizing the importance of carefully designed behavioral assays that avoid anthropocentric assumptions. Other work has highlighted specific methodological challenges in human-machine comparison: Firestone (2020) drew an important distinction between performance and competence in human-machine comparisons, arguing that observed task performance may not reflect underlying competence due to confounding factors in how tasks are administered to humans versus machines. Relatedly, Hu and Frank (2024) showed that auxiliary task demands can mask model capabilities, A. Lampinen (2024) demonstrated the importance of carefully matched comparison conditions between models and humans, and Hu and Levy (2023) established that prompting methods are not equivalent to direct probability measurements when evaluating language models.

Our work builds on and extends these contributions. Here, we focus particularly on how insights from decades of computational modeling can inform how we approach AI evaluation. The recommendations we propose here derive from years of development and debate in cognitive science to determine best practices for designing tasks, richly comparing models to human judgments, and sharpening hypotheses about what aspects of human behavior a computational model is intended to capture in the first place: all cornerstones, we argue, of what it means to make theoretically rich, replicable, and measured claims about the sense in which a given model is and is not comparable to human behavior. We urge developers of AI evaluation benchmarks to engage with and capitalize on this history.

Recommendation 1: Measure ‘Human-Like AI’ Against Actual Humans, and Collect Robust, Replicable Sample Sizes of Human Data

A surprising number of “cognitively-inspired” benchmark suites and AI evaluations claim to measure human-like AI performance without any human data at all. Rather, tasks derived or sometimes loosely adapted from psychological assays are used to directly evaluate computational model performance, often with ground-truth notions of what it means to “solve” a task (for instance, to identify whether a model can label mental states in simple “false-belief” tasks derived from cognitive theory-of-mind experiments (Wimmer & Perner, 1983)). Our first and perhaps most fundamental recommendation is that the ground-truth labels for measuring whether AI is human-like should be response data collected from humans themselves.

Using actual human behavior as the “gold” labels for AI benchmarks, we propose, is important for many structural aspects that have been well documented in cognitive science. First, many AI benchmarks seek to evaluate inherently subjective concepts, such as whether an act is morally permissible, where a single, objectively correct answer (or even any set of “correct answers”) may not exist. Rather, computational models of subjective behavior like moral reasoning, have long sought to characterize distributions of human judgments, including to account for known variation across populations, social groups, and cultures (Graham et al., 2009, 2016), while also seeking to explain how these differences arise (Levine et al., 2020).

Second, even on tasks that appear to have a single objective “gold label” based on external measures, measuring human behavior may still reveal important variation and disagreement, sometimes with high confidence, that is nonetheless revealing of the internal computations by which humans process particular inputs. The famous visual illusion involving The Dress, for instance, illustrates people’s strongly diverging judgments even given a measurable external label, the true color of the dress. These divergent judgments on this single stimulus reveal important, measurable, and modelable facets of human visual processing (Lafer-Sousa et al., 2015). Our own data demonstrates analogous patterns of structured disagreement across higher-level cognitive tasks (see Figure 1 and Figure 3). More generally, building systems that are truly human-like or that can well-model human-like behavior requires also modeling human error patterns and uncertainty. Computational cognitive modelers do not shy away from human errors, but rather lean into them; such errors can help reveal structure in what we do or do not know. Understanding whether a machine is human-like therefore ought to examine such error patterns from the “true” state of the world.

Figure 1. .

Human average accuracy per task. Error bars indicate 95% confidence intervals. Dashed black ticks indicate per-task chance levels.

Figure 3. .

Distributions of participants’ task ratings (left) and confidence ratings (right) across all 9 benchmarks. Each row represents a stimulus in a benchmark. Each dot represents a response from a participant on the stimulus. Task ratings show that participants hold graded beliefs poorly captured by binary ground-truth labels. Confidence ratings show that participants show varying degrees of uncertainty in the tasks, where tasks such as social support, moral permissibility, fantasy reasoning frequently receive low confidence ratings.

In our analysis of a suite of common AI evaluation benchmarks that had previously been annotated with only a single “correct” answer, we found high levels of disagreement between human judgments and benchmark labels. As shown in Figure 1, human accuracy, measured as the percentage of trials where a participant’s response agreed with the benchmark’s ground-truth label (see Appendix for computation details), varies substantially across tasks, with an average of 69.7% across all nine benchmarks. Notably, Social support (45%, chance = 50%) falls below chance, and Moral permissibility (72%), Irony identification (73%), Fantasy reasoning (73%), Dark humor detection (73%), and Social IQA (68%, chance = 33%) all show substantial disagreement between human annotators and benchmark labels. Only Simple ethical questions (88%) approaches high agreement. These results underscore a fundamental concern: if we aim to evaluate whether AI systems are human-like, using labels that do not reflect how most humans actually respond introduces a misleading standard of comparison. Consider the specific example in Figure 2: participants are asked to rate whether the statement “There’s nothing wrong with the quotations or discussing her art” is supportive. We find that people report graded levels of supportiveness and express varying degrees of uncertainty in their judgments. On average, participants rate the statement as moderately supportive (mean = 69.4, SD = 21.6 on a 0–100 scale, where 100 = definitely supportive), while the ground truth label in the dataset is “unsupportive.” We show more such examples in the Appendix.

Figure 2. .

Distribution of participants’ task ratings (left) and confidence ratings (right) on one stimulus from the Social support classification task. Participants are asked to rate how supportive is the statement “There’s nothing wrong with the quotations or discussing her art.” The ground-truth label is “unsupportive,” yet most participants showcase graded judgment in the level of supportness, with the majority of the ratings fall between 50 to 100. The confidence histogram shows that people also report varying degrees on uncertainty.

Taken together, our re-annotation of these benchmarks, with real humans, suggests that there are serious concerns as to the validity of some published ground-truth labels for benchmarking “human-likeness.” This finding is consistent with broader evidence that pervasive label errors in benchmark test sets can destabilize machine learning evaluations more generally (Northcutt et al., 2021).

Recommendation 2: Evaluate Models of Human Populations Against Population-Level Distributions of Human Judgments

Our second recommendation builds more specifically on the inter-annotator variation we discuss above: for many AI models, particularly machine learning models explicitly trained on large distributions of human-generated data, we propose that model evaluations should explicitly collect, analyze, and use population-level distributions of human responses as the “gold” soft labels for evaluating model performance. A fundamental distinction for computational cognitive and psychological models is clarifying which populations of humans one seeks to model, and at what level one seeks to model them, distinguishing, for instance, between a granular model of the algorithms, strategies, and errors that a single human might make across related stimuli on a single domain, with the overall pattern of responses we can expect to find across many subjects. Because many AI models are trained on population-level human data using objectives designed to measure population-level responses, and are often intended for deployment across populations, we argue that it is crucial to collect and evaluate performance explicitly on how well models capture the structure and variation of behavior across sets of human subjects.

Nearly all facets of human cognition are influenced by a complex set of individual differences and cultural factors. These include differences in underlying cognitive abilities or resources like working memory or attention (Boogert et al., 2018); differences in prior experiences, preferences and goals, which can influence how individuals predict unknowns given limited evidence or choose among a set of options and actions (Ongchoco et al., 2024); and cultural variation in values, expectations, and experiences that systematically influence priors or decision making strategies (Henrich et al., 2010).

Many existing benchmarks collect human annotations but rely on majority voting to collapse the human responses to a single “ground-truth” label, effectively discarding valuable information about the range and distribution of human judgments. This may disproportionally lead models to align with the majority view, even if there are important subpopulations that are otherwise underrepresented (Gordon et al., 2022). Additional pitfalls of such information loss in label construction have been raised in the context of image classification systems wherein the labels used to train models were often taken to be the label with the majority vote; several works identified that training and evaluating such models on distributions over annotator uncertainty (“soft labels”) revealed and guarded against otherwise fragility in such model predictions (Collins, Bhatt, et al., 2023; Peterson et al., 2019; Sucholutsky, Battleday, et al., 2023; Uma et al., 2020). These works also highlight the potential benefit of then training on labels that better capture the richness of human beliefs for enhanced generalization and robustness. As a concrete example, Peterson et al. (2019) introduced CIFAR10H, a dataset providing full distributions of human labels for CIFAR-10 test images, and showed that classifiers trained on these soft labels generalize better to out-of-distribution test sets and exhibit improved robustness against adversarial attacks compared to those trained on majority-vote hard labels. We advocate for the consideration of distributions over human data in the context of AI evaluation more broadly.

Researchers in AI Alignment, specifically “pluralistic alignment”, have advocated for similar recommendations (Kirk et al., 2024; Sorensen et al., 2024) but more restricted to alignment to a distribution of values and preferences in decision-making. Relatedly, there is substantial work on calibration in machine learning, i.e., how well a model’s answer uncertainty matches its accuracy, which is relevant when considering whether model uncertainty should match the human distribution (Geng et al., 2024). Distributional evaluation and calibration assessment serve complementary goals: the former asks whether a model’s response patterns resemble those of human populations, while the latter asks whether a model’s expressed confidence is well-calibrated. In our paper, we argue modeling distributions over annotators should extend beyond value-laden tasks to encompass the full range of cognitive tasks where human variability is meaningful, such as tasks involving language understanding (where pragmatic interpretation varies across individuals), and social cognition (where inferences about others’ mental states are inherently uncertain).

Designing and Evaluating Population-Level Metrics.

Once we collect the distribution of human data, how may we evaluate AI models? As in cognitive modeling, where researchers often deploy a range of evaluation measures on collected data and conduct analyses on subgroups within populations of participants, we recommend being clear and seeking explicitly to measure the following:

• Report metrics used to compare distributions of samples from models (with comparable numbers of samples from the model versus samples from a population of participants) to distributions of human judgments, such as measures on probability distributions (e.g., KL divergence or Wasserstein distances). These metrics can ensure that models do not simply report narrow means, with little of the expected distributional diversity shown across populations as a whole.

• Explain structure within a given distribution of answers. For instance, if distributions have distinct modes, can the model interpretably and consistently explain how these modes arise, or how modes are correlated across related questions?

• Measure how the model represents individual patterns of answers and explain individual differences across the population, for instance, to what degree can it capture conditional patterns based on personal traits (e.g., how a pluralist would answer a moral value judgment query versus a utilitarian)? Evaluating conditional distributions can help further focus which parts of a population are well-modeled, and which may be more divergent. Relatedly, a model conditioned on a given individual or subgroup should be internally consistent like that individual, since errors on cognitive tasks tend to be more correlated within a subject than across subjects. On this view, the population distribution is recovered by marginalizing over such conditioned agents, rather than emerging from a single, self-inconsistent set of samples.

Recommendation 3: Evaluate Model Gradedness and Uncertainty Against Gradedness in Individual Human Judgments

Just as different people may come to different conclusions about any given task, any single person may be uncertain about what decision they want to make or what plan they want to take. Decades of cognitive science research has shown that graded beliefs and uncertainties are an essential part of human cognition, driving nuanced human perception, reasoning and behaviors (Chater & Manning, 2006; Griffiths et al., 2024; Tversky & Kahneman, 1974). We encourage benchmark builders to consider eliciting, maintaining, and measuring not just judgment over hard labels with multiple choice questions but graded judgments from individual annotators using soft labels. The collection and consideration of soft labels for capturing graded judgments from humans has been standard practice for cognitive modeling and has more recently been advocated for in the context of computer vision (Schmarje et al., 2022; Sucholutsky, Muttenthaler, et al., 2023; Uma et al., 2022), human-AI interaction (Collins, Barker, et al., 2023; Steyvers et al., 2022), and the elicitation of knowledge from experts more broadly (O’Hagan, 2019; O’Hagan et al., 2006). These judgements can also be collected and compared over hierarchical representations (Muttenthaler et al., 2024).

Discrete multiple-choice questions that require an annotator to select only one choice are typically too coarse for such measures. Figure 3 shows the distribution of participants’ slider ratings (left panel) alongside their confidence ratings (right panel) across all 9 benchmarks. Despite the fact that the underlying benchmark labels map onto extreme values of 0 or 100, a large fraction of human ratings fall in intermediate ranges, demonstrating that participants hold graded beliefs that are poorly captured by discrete labels. Importantly, the separate confidence ratings reveal that these graded responses are not simply artifacts of participant uncertainty: participants often report high confidence in their intermediate judgments, indicating that the gradedness reflects genuine degrees of belief or feature intensity rather than epistemic confusion. This pattern is pervasive across tasks.

We call on AI benchmarks to consider collecting and assessing soft labels from annotators to measure their graded judgments for the following reasons. First, graded judgments better reflect the nuances of real-world scenarios. Real-world decision-making rarely involves absolute, binary choices. Consider emotions, which vary in intensity, or moral judgments, where two wrong actions might warrant different levels of reprimand. Graded responses allow benchmarks to capture these crucial distinctions and nuances and can in turn be used to train models for better generalization to new situations (Muttenthaler et al., 2022, 2023; Peterson et al., 2019).

Second, soft labels capture the inherent uncertainty prevalent in many tasks. A binary choice often fails to represent the full spectrum of human beliefs and judgment. Individuals may lean towards one option while acknowledging some doubt. This uncertainty is fundamental to real-world reasoning and decision-making. Quantifying uncertainty allows for flexible planning, adaptive strategies, and appropriate risk assessment, essential skills for robust AI systems. While some might argue that large samples with hard labels can approximate uncertainty, this approach hinges on the assumption of independent and identically distributed (i.i.d.) samples. However, this assumption often does not hold in many real-world cases due to individual and group-level variations. Consider the example of The Dress: averaging judgments across all samples would show high uncertainty between the two color labels, yet each individual is quite adamant about what they see. We also observe rich distributional patterns in our own data collection on cognitive tasks, for instance, in some tasks on Irony identification and Fantasy reasoning, participants cluster at both extremes of the scale with high confidence, suggesting that structured individual disagreement, not mere noise, underlies the variation.

To deeply understand whether a model is human-like, we urge finer-grained consideration of the rich, structured beliefs that any single annotator may have. Researchers may fear perceived “messiness” of collecting human uncertainty. An oft heard retort to the collection of uncertainty is that people are “miscalibrated” in their uncertainty. Decades of research in cognitive science, however, have designed studies to examine people’s probabilistic judgments in order to study and model human cognition (Chater & Manning, 2006; Griffiths et al., 2024; Keren, 1991; O’Hagan et al., 2006; Tenenbaum, 1998; Windschitl & Wells, 1996). We encourage designers of AI benchmarks to engage with such literature and lean into these uncertainties in humans’ judgments in order to assess models’ human-like behaviors.

Recommendation 4: Situate Tasks With Respect to Meta-Reviews of Existing Cognitive Theory

To draw generalizable conclusions about an AI model’s cognitive capacity, tasks should be carefully designed to measure whether the model’s cognitive capabilities are human-like (Hernández-Orallo, 2017). To do so, benchmarks should begin with what we term a meta-theory of the target cognitive construct. By meta-theory, we mean an explicit specification of the construct that defines what it is (and what it is not), decomposes it into subcomponents, and identifies the observable behavioral signatures that should arise under controlled manipulations, such as ablations, difficulty variations, resource constraints, or adversarial conditions. This framing draws on the classic literature on construct validity in psychometrics (Cronbach & Meehl, 1955; Messick, 1995): a benchmark should be justified as a measurement instrument with an explicit argument for why its tasks validly measure the intended construct, rather than serving as a collection of intuitively related tasks (Bean et al., 2025). The meta-theory then guides the construction of the benchmark, ensuring that tasks effectively probe the specific cognitive capacities of interest and provide meaningful insights into to what extent AI possesses these mental constructs in a human-like way.

One common pitfall is the use of impoverished theory in guiding AI benchmark creation. For example, many benchmarks have been created to evaluate a machine’s Theory of Mind (ToM), which refers to the human ability to make inferences about other agents’ mental states. ToM benchmarks for AI commonly or exclusively use the Sally-Anne test (a.k.a. false-belief test) (e.g., Le et al., 2019), which has traditionally been used in developmental psychology for evaluating the timing of children’s developing Theory of Mind. The results from these evaluations have led to claims such as ToM having emerged in LLMs (Gandhi et al., 2024; Kosinski, 2024). However, ToM embodies a wide range of subcomponents beyond those assessed by the Sally-Anne test. In a comprehensive review, Beaudoin et al. (2020) identified 220 ToM tasks and measures previously used by psychological studies. Other authors have also questioned the validity and effectiveness of the Sally-Anne test in assessing children’s ToM (Bloom & German, 2000). By exclusively focusing on false-belief tasks, many studies on evaluating AI models’ ToM reflect a poor understanding of the meta-theory of ToM as construed in cognitive psychology. A construct-valid evaluation of ToM would specify its subcomponents (e.g., false belief attribution, intention inference, desire reasoning, knowledge tracking), identify which behavioral signatures each subcomponent predicts under controlled conditions, and design tasks that probe these signatures while controlling for confounds. This is especially important because the target construct differs between developmental assays and model evaluation: Sally-Anne tasks were designed to detect a developmental milestone in children, not to provide a comprehensive measure of ToM competence. When applied to language models, they also risk conflating ToM with linguistic or pragmatic skills, as Hu et al. (2025) demonstrate, apparent model competence on these tasks often reflects language understanding rather than genuine mental state reasoning. Instead, benchmarking intelligent systems should start from a meta-theory of the cognitive construct and design tasks grounded in the cognitive theory, including a comprehensive survey of its subdomains, taxonomies, and measures. More broadly, if a construct is assumed to have a particular factor structure, as has been established for domains like self-regulation (Eisenberg et al., 2019), then evaluations should verify that this structure holds for the models being tested, rather than assuming that human-derived constructs map directly onto model behavior. Whether human-derived meta-theories transfer to AI systems at all is itself an open question: in some cases, the constructs may not hang together in the same way, and validating the mapping from a human-derived meta-theory to a model’s behavioral variation across tasks is an important prerequisite to any strong claim of “human-likeness.”

Another common pitfall is the naive use and adaptation of psychological tests in evaluating AI models. Passing a few psychological tests is insufficient to claim certain cognitive capacities exist in machines. Again take the Sally-Anne test as an example. Although it may be effective in measuring children’s ToM, tests as such are insufficient for evaluating AI’s ToM because these well-known test items are likely present in the web-scraped training corpora of large language models, creating data contamination concerns that undermine the validity of such evaluations (Jacovi et al., 2023). Unlike human children who encounter false-belief scenarios naturally during development, AI models may have been exposed to both the test items and their expected answers during training. Therefore, blindly taking psychological scales and applying them to AI benchmarks to claim an AI is human-like can result in misleading conclusions and the results will be unlikely to generalize to richer tasks in the real world. Instead, we encourage AI benchmark creators to use psychological theories as a guide and psychological tests as inspirations for designing tasks for evaluating AI’s cognitive capacity, but the tasks should be richer, more grounded, and more complex. Research in Cognitive Science in the past decades have introduced many rich and interactive paradigms for studying and evaluating models’ social cognition (e.g., Baker et al., 2017; Jara-Ettinger et al., 2020; Ying et al., 2023), which were used to extract sophisticated and graded reasoning patterns from humans. In the next section, we discuss some concrete recommendations for designing such tasks.

Recommendation 5: Design Cognitively Rich Tasks that Elicit Naturalistic Behavior

Benchmark tasks should elicit the kind of rich, structured cognitive behavior that humans display in the real world, rather than being constrained to simple, unambiguous question-answer formats. Many existing benchmarks focus on simple, straightforward tasks, often excluding those with low inter-annotator agreement. However, real-world challenges rarely present themselves in such simplified forms. Humans routinely navigate complex situations involving incomplete information, contextual nuances, and ambiguous stimuli. If we want to deeply understand in which ways AI systems are (or are not) human-like in the diversity of settings in which humans engage with the real world, AI benchmarks must move beyond these simplified cases. This direction parallels recent calls within cognitive science itself for more naturalistic computational cognitive science (Carvalho & Lampinen, 2025), which emphasizes building generalizable models and theories that capture the full range of natural human behavior rather than narrow laboratory paradigms. The broader goal here is ecological validity: designing tasks whose structure and demands resemble those encountered in real-world applications. We note, however, that what counts as ecologically valid for a human subject may not necessarily be ecologically valid for an AI system, and approaches for establishing ecological validity in human-machine comparisons (e.g., Taylor et al., 2022) remain an active area of research and an open methodological question. With this caveat in mind, we next provide several suggestions for eliciting rich cognitive behavior in benchmarks.

Integration of Cognitive Capacities.

Benchmarks should incorporate tasks that require integrating multiple cognitive processes. Even text-based tasks can demand such integration: for example, understanding the intent behind a sentence might require considering conversational context, the speaker’s prior beliefs, social norms, and planning or rational decision-making under uncertainty. Tasks can also span modalities: inferring an agent’s goals from observed trajectories in a visual scene (Baker et al., 2017), multimodal Theory of Mind reasoning over video and text (Jin et al., 2024), or following pragmatic spoken instructions in embodied settings (Ying et al., 2024). By incorporating such complexities, benchmarks can better assess an AI’s ability to handle nuanced, real-world situations that demand integration across reasoning, planning, language, perception, and social cognition.

Systematic Ablation.

Ablating tasks by systematically withholding or providing specific information or context can reveal how different factors influence both human and AI judgments and uncertainty. Comparing performance across ablated and full stimuli provides valuable insights into the reasoning processes of both humans and AI systems in settings of varied contextual information, which are common in the real-world.

Structured Ambiguity.

Tasks involving ambiguous perceptual and reasoning challenges, like the example illustrated in The Dress, can elicit diverse response patterns among humans. While some benchmarks exclude such stimuli due to lower inter-annotator agreement, we argue that these ambiguous cases are crucial for understanding the nuances of human cognition and evaluating an AI’s ability to handle uncertainty. Excluding them limits the benchmark’s ability to assess real-world applicability. Rather, we encourage benchmark designers to retain and even seek out stimuli that elicit structured disagreement or graded uncertainty among humans, as these are precisely the cases that are most informative for evaluating human-like cognition. Collecting human-derived ratings of expected ambiguity (Zhou et al., 2024) can help identify such stimuli systematically.

However, it is worth noting that not all human response variability is of the same kind, as a spread of answers can arise from distinct sources. Some stimuli are inherently ambiguous at the level of inference, where the signal itself underdetermines the answer, as with a noisy physical simulation or a blurred object. Other stimuli are ambiguous at the level of task interpretation, where people converge on different readings of what the task is asking (e.g., a confusing or poorly worded question). Both can produce a spread of human responses, but they implicate different underlying constructs, and a benchmark should be clear about which one it intends to measure, so that the resulting variation can be attributed to a genuine property of human cognition rather than to an unintended confound in the stimulus or task framing.

By incorporating these design principles, we can create benchmarks that assess AI models’ capacity for human-like reasoning, interaction, and adaptation to complex, real-world scenarios.

CHALLENGES AND ALTERNATIVE VIEWS

In this section, we address some limitations and open challenges regarding our proposed benchmarking paradigm on measuring human-like intelligence.

Do We Need Human-like AI?

We acknowledge that certain highly specialized AI applications, such as protein structure prediction (Jumper et al., 2021) or weather forecasting (Bodnar et al., 2024; Lam et al., 2023), do not require human-like characteristics. Benchmarks for these domains fall outside the scope of this paper. Similarly, benchmarks that aim to test AI performance at the frontier of human expertise (e.g., research-level mathematics) or that evaluate abstract reasoning at the level of the best human performers may prioritize different evaluation criteria. Our focus lies on core cognitive capacities that enable machines to reason, interact, and collaborate with humans in the real world (Collins, Sucholutsky, et al., 2024), where modeling humanlike patterns of judgment, uncertainty, and variability is most valuable.

Another critical consideration in using human data for AI benchmarks is the potential for biases and errors in human judgments. Cognitive science research has extensively documented human limitations in rational reasoning and decision-making, due to limited cognitive resources (Griffiths, 2020; Lieder & Griffiths, 2019) or systematic biases (Tversky & Kahneman, 1974). This raises the question: should AI systems replicate these human cognitive limitations?

There is no clear answer here. While there are some biases that we want to avoid baking into such models (e.g., harmful racial or gender prejudices), other cognitive biases can be useful for decision making (Haselton et al., 2009; Lieder & Griffiths, 2019) and essential for accurately modeling human behavior, and the evidence on whether today’s models exhibit human-like error patterns is mixed. Some studies suggest that certain patterns of errors are not implicitly learned in current models, which risks hampered human-AI interaction (Liu et al., 2024), while others have found that language models do exhibit content effects on reasoning tasks similar to known human biases (A. K. Lampinen et al., 2024). Understanding which error patterns are and are not captured by current models, and under what conditions, remains an important open question.

Ultimately, it remains an open question which applications would specifically benefit from human-like cognition in machines. We believe that our position paper would stimulate timely and crucial discussion on these topics.

Scalability of Human Data Collection

Another limitation of our approach, and of human-centric AI evaluation more broadly, lies in the scalability of data collection. The process of gathering human judgments is inherently resource-intensive in terms of time and cost, potentially hindering the rapid development of large-scale benchmarks (Collins, Kim, et al., 2024; Wu et al., 2023). This challenge is particularly pronounced when eliciting fine-grained annotations across multiple attributes, as required by our methodology (Chung et al., 2019; Kirk et al., 2024). While modern crowdsourcing platforms like Amazon Mechanical Turk and Prolific have made large-scale annotation more feasible, they do not fully resolve these core challenges and may introduce their own sampling biases. Consequently, developing novel, scalable, and cost-effective methods for high-quality human data collection remains a critical open research direction.

DISCUSSION AND CONCLUSION

AI systems are increasingly deployed alongside humans to perform many tasks in ways that mimic human cognition. Characterizing the ways in which AI systems are, or are not, like humans is critical for the scientific studies of natural and artificial intelligence and helping us design systems that work with people in everyday life. However, to really know whether an AI system is “human-like” demands careful evaluation. In this work, we have encouraged researchers who evaluate AI models to look to decades of research in cognitive modeling. Specifically, we encourage researchers to ensure that if they are making claims about a system being “human-like” (or want to understand whether a system is or is not), human labels must be collected. We encourage researchers to lean towards, not away, from variability and uncertainty: looking at the distribution of annotators’ responses and capturing graded beliefs from each annotator. Further, the tasks over which AI systems are benchmarked demand careful theory-driven design, as well as the elicitation of richer, more naturalistic cognitive behavior. We note that applying existing cognitive theories to understand machines raises open questions about whether and how human-derived constructs transfer to AI systems; we expect that the fields of cognitive science and AI evaluation will need to co-evolve as AI capabilities continue to grow. AI systems are growing increasingly powerful; we need more robust and reliable evaluation not only if we want to build more human-compatible AI thought partners that we understand but also if we want to deeply understand human intelligence.

FUNDING INFORMATION

This work was funded in part by Schmidt AI 2050, ONR, the MIT-IBM Watson AI Lab, and gifts from Reid Hoffman and the Siegel Family Foundation. KMC acknowledges support from King’s College Cambridge and the Cambridge Trust. AW acknowledges support from a Turing AI Fellowship under grant EP/V025279/1, EPSRC grants EP/V056522/1 and EP/V056883/1, and the Leverhulme Trust via CFI.

AUTHOR CONTRIBUTIONS

L.Y. conceived and designed the study and performed the data analysis. All authors contributed to the conceptual development of the work, interpretation of the results, and writing and revision of the manuscript.

DATA AND CODE AVAILABILITY STATEMENT

The data and code associated with this work are publicly available at https://github.com/lance-ying/benchmark_human_data.