ABSTRACT
Despite rapid technological progress, effective human–machine cooperation remains a significant challenge. Humans tend to cooperate less with machines than with fellow humans, a phenomenon known as the machine penalty. Here, we show that artificial intelligence (AI) agents powered by large language models can overcome this penalty in social dilemma games with communication. In a pre-registered experiment with 1152 participants, we deploy AI agents exhibiting three distinct personas: selfish, cooperative and fair. However, only fair agents elicit human cooperation at rates comparable to human–human interactions. Analysis reveals that fair agents, similar to human participants, occasionally break pre-game cooperation promises but nonetheless effectively establish cooperation as a social norm. These results challenge the conventional wisdom of machines as altruistic assistants or rational actors. Instead, our study highlights the importance of AI agents reflecting the nuanced complexity of human social behaviors—imperfect yet driven by deeper social cognitive processes.
Keywords: artificial intelligence, large language model, human–machine interaction, game theory, behavioral economic
Fair but imperfect AI agents, capable of breaking promises, overcome human reluctance to cooperate with machines.
INTRODUCTION
In today’s rapidly advancing technological landscape, cooperation between humans and machines is emerging as a cornerstone of societal progress and innovation. For example, Pactum’s artificial intelligence (AI) agents now negotiate on behalf of Walmart with human vendors, closing deals in just days rather than the traditional weeks or months [1]. AI coding assistants, such as GitHub Copilot, support code autocompletion and fixes while keeping humans in the loop, increasing the productivity of engineers at JPMorgan by up to 20% [2]. Machines, no longer passive tools merely executing our commands, have evolved into sophisticated, autonomous agents ready to actively work—and, more importantly, to cooperate—with humans [3,4]. This paradigm shift makes understanding and fostering human–machine cooperation imperative. Not only does it allow us to harness the complementary strengths of both [5], but it also helps navigate social dilemmas—tensions between individual and collective interests—a crucial and nearly unavoidable challenge in human–machine interactions [6].
Despite immense potential, studies have consistently identified a significant gap between human–human and human–machine cooperation, a phenomenon known as the machine penalty [7]. While this phenomenon has been widely documented in incentivized economic games [8,9], it also manifests in real-world contexts. As a common example, human drivers are more likely to deny autonomous vehicles the right of way at junctions than they are to other human drivers [10,11]. Part of this penalty stems from technical limitations: machines must be able to understand, communicate and find common ground with humans [6]. However, at its core, the machine penalty reflects a form of algorithm aversion, whereby humans are reluctant to view machines as genuine social partners [12,13]. This aversion, common across various domains, often leads to less trust in machines [14] and a tendency to assign machines greater blame for mistakes [15]. Furthermore, machines’ typical lack of cultural norms [16], moral understanding [17], emotional capacity [18] and fairness considerations [19], which are elements crucial to human cooperation, only exacerbates the aversion.
Recent efforts to mitigate the machine penalty have focused on anthropomorphism [20–22]. The rationale is that, since humans are more cooperative with fellow humans, endowing machines with human-like traits might bridge the gap. However, superficial anthropomorphic designs, such as equipping machines with emotional expressions or human-like faces [20,23], have shown little effect. Nevertheless, enhancing human-likeness can sometimes evoke feelings of uncanniness [24]. Some studies have had success with ethically questionable genderization [21,22], such as adding female cues of long hair to machines. Another approach has been to conceal machines’ true nature [8]. While this approach can effectively reduce the gap, it compromises technological transparency and can be seen as deceptive. Collectively, these findings cast doubt on whether humanization is the answer and highlight the need to rethink which elements are truly necessary for machines to foster human cooperation.
To close this gap, this paper studies human cooperation with AI agents powered by large language models (LLMs). The reasons behind this design choice are manifold. First, LLMs are well known for their ability to understand language and generate conversations, making them well suited for interactions with humans [25,26]. Second, their extensive training on human corpora allows them to grasp key concepts such as rationality, risk, trust and fairness, which underpin human cooperative behaviors [27,28]. Third, LLMs can generate diverse human simulacra by varying high-level descriptions, enabling them to emulate the nuances of human behavior [29] and enhance their cooperative potential.
We focus on prisoner’s dilemma games with pre-game communication, a stylized, controlled social dilemma in which players choose between cooperation and defection. While collective benefits depend on cooperation, individual incentives favor defection. We design agents with enhanced strategic reasoning and decision-making capabilities, and humanize them by assigning them one of three distinct personas (see the Methods section for details): (i) cooperative, aiming to assist its human associate; (ii) selfish, focusing solely on maximizing its self-interest; and (iii) fair, balancing its own and collective interest while slightly prioritizing self-interest. While selfish agents reflect the common notion of rationality [30,31] in AI and game theory, whereby agents act primarily to maximize their own payoff, cooperative agents and fair agents are designed to embody prosocial preferences [32], a prominent theoretical account of human cooperation.
Our experiment, with a total of 1152 participants, shows that fair agents are able to overcome the machine penalty, whereas cooperative and selfish agents do not. The post-experiment analysis aims to understand the mechanisms at play, focusing on differences in communication, reasoning, decision-making, the ability to evoke descriptive norms (i.e. beliefs about other participants commonly cooperating), and participants’ perceptions of agents’ minds and human-like traits. The analysis reveals that fair agents, despite occasional strategic promise breaches, successfully establish cooperative norms and foster positive human perceptions of their mind, trustworthiness and intelligence.
RESULTS
Our experiment involves four types of treatments: human–human (H–H), human–fair agent (H–F), human–cooperative agent (H–C) and human–selfish agent (H–S) interactions. Each treatment spans 10 rounds, with participants randomly paired with a knowingly new associate in each round. Before making their choices in each round, participants and their associates exchange two rounds of messages. From the outset, we explicitly inform participants of the true nature of their associates, that is, whether they are interacting with humans or intelligent machines. We also conduct a label-uninformed setting in which the nature of the associate was not disclosed. The following results focus exclusively on the label-informed setting. Detailed experimental implementations are presented in Note S1.
Overcoming the machine penalty
Figure 1 shows the cooperation rates of humans and agents in human–human and three types of human–agent interactions. The left panel shows that human cooperation rates in interactions with fair agents are comparable to those observed in human–human interactions. This indicates that fair agents are as effective as humans in eliciting cooperation. In contrast, human cooperation rates are significantly lower in interactions with cooperative or selfish agents than in human–human interactions. The right panel shows the cooperation rates of three types of agents. Cooperative agents consistently cooperate with minimal variability. Selfish and fair agents display within-group variability, generally showing tendencies to defect and cooperate, respectively.
Figure 1.
Fair agents, unlike cooperative or selfish agents, are as effective as humans at eliciting human cooperation, thereby overcoming the machine penalty. The left panel depicts participants’ cooperation rates, while the right panel depicts the cooperation rates of agents. Participants’ cooperation rates in the H–F treatment show no significant difference compared to those in the H–H treatment (
,
, Cohen’s
). However, their cooperation rates in both the H–C and H–S treatments are significantly lower than those in the H–H treatment (H–C versus H–H:
,
, Cohen’s
; H–S versus H–H:
,
, Cohen’s
). The cooperation rates of fair agents are significantly lower than those of cooperative agents (
,
, Cohen’s
), but significantly higher than those of selfish agents (
,
, Cohen’s
). Two-tailed Mann–Whitney U tests are used for pairwise comparisons. The robustness of these results is further corroborated by a one-way ANOVA test (Table S1).
Thus, AI agents powered by LLMs can, in principle, overcome the longstanding machine penalty, inducing human cooperation at levels similar to human–human interactions, even when their machine nature is fully disclosed. However, this is only true for agents with a humanized, fair persona. The cooperation gap cannot be bridged by cooperative agents, despite their purely benevolent intent, nor by selfish agents, despite their rational decision-making.
Cooperation agreement and promise breach
To better understand the mechanisms at play, we analyze messages exchanged during the pre-game communication and the choices made afterwards. We recruit human experts to annotate all the messages (see Note S2 for details). We find that both participants and agents often express an intention to cooperate in their messages. Communication in human–human and the three types of human–agent interactions frequently leads to cooperation agreements (left panel of Fig. 2). In particular, fair agents establish these agreements with participants at significantly higher rates than in human–human interactions and other types of human–agent interactions.
Figure 2.
All three types of agents frequently establish cooperation agreements with humans during the pre-game communication. However, humans often break cooperation promises, while fair agents also occasionally do so. Participants are most likely to establish cooperation agreements with fair agents, at a significantly higher rate than participants in all other treatments (H–F versus H–H:
,
, Cohen’s
; H–F versus. H–C:
,
, Cohen’s
; H–F versus H–S:
,
, Cohen’s
). However, during the games, participants typically break their promises, although they do so significantly less frequently in the H–F treatment than in the H–C and H–S treatments (H–F versus H–C:
,
, Cohen’s
; H–F versus H–S:
,
, Cohen’s
). Fair agents break promises at a significantly higher rate than cooperative agents (
,
, Cohen’s
), but significantly lower than selfish agents (
,
, Cohen’s
). Moreover, participants break promises significantly less frequently in the H–H treatment than in other treatments. (H–H versus H–F:
,
, Cohen’s
; H–H versus H–C:
,
, Cohen’s
; H–H versus H–S:
,
, Cohen’s
). Two-sample proportions Z tests are used for pairwise comparisons. Statistical significance results of pairwise comparisons across each treatment are reported in Table S2.
However, these agreements are non-binding. Both participants and agents are, in principle, free to break their cooperation promises. As shown in the right panel of Fig. 2, participants often treacherously opt for defection after agreeing to cooperate during pre-game communication. Participants are more likely to honor agreements made with fair agents than with cooperative or selfish agents, although they generally break promises more often when interacting with agents than with humans. As for agents, cooperative agents consistently uphold their promises. In contrast, fair agents occasionally break promises, whereas selfish agents frequently do so.
A generalized linear model across the three types of human–agent interactions reveals a nonlinear relationship between human cooperation rates and agents’ promise-breaking rates (Fig. 3). Initially, as the frequency of agents breaking promises increases, human cooperation rates also increase. However, with promise-breaking becoming more common, human cooperation rates reach a peak and then start to decline. Human cooperation reaches the lowest level when agents break promises very frequently. Beyond this point, human cooperation rates stabilize but remain low.
Figure 3.
Occasional promise breaches, exhibited by fair agents, are associated with the highest rates of human cooperation. Scatter points depict the cooperation rates of individual participants when interacting with agents. The curve represents a generalized linear model (GLM) that incorporates data from all three types of human–agent interactions. This model treats human cooperation rates as the dependent variable and includes linear (
), quadratic (
) and cubic (
) terms of agents’ promise-breaking frequency as independent variables. The curve shows an initial increase in human cooperation rates as the frequency of agents’ promise-breaking rises from zero, followed by a significant decrease, and then stabilization at higher frequencies of agents’ promise-breaking.
These results suggest that, regardless of their specific personas, agents are effective at communicating with humans, thereby leading to pre-game cooperation agreements. However, these agreements alone do not ensure cooperation during the game. While agents’ frequent promise breaches are typically associated with reduced human cooperation, occasional breaches are associated with increased human cooperation.
Fostering norms and perceptions of minds, trust and intelligence
Our post-experiment surveys assess participants’ perceptions of social norms and communication quality in the interactions, as well as their views on the minds and human-like traits of their associates. By social norms, we specifically refer to descriptive norms, that is, beliefs about what most others in one’s social group actually do. We assess these beliefs by incentivizing participants with a bonus if they correctly estimate the cooperation rates of other participants. As shown in Fig. 4a, across the three types of human–agent interactions, participants interacting with fair agents report the highest estimated cooperation, even significantly higher than those in human–human interactions. In contrast, participants interacting with cooperative agents show polarized estimations, with beliefs split between exploiting or reciprocating the agents’ altruism, whereas participants interacting with selfish agents generally expect defection. This reflects a descriptive norm—a prevailing belief among participants that fair agents will be matched with cooperative responses—even if they occasionally exhibit imperfect, treacherous behaviors.
Figure 4.
Fair agents establish cooperative norms and are perceived as possessing experience, agency and intelligence, while also being viewed as more trustworthy, likable, cooperative and fair than humans. The top panels depict participants’ post-experiment estimations of cooperation from other participants in the same treatment, whereas the bottom panels depict participants’ post-experiment agreement levels for various human-like traits of their associates in the treatment. Participants estimate the highest level of cooperation from other participants in the H–F treatment than in all other treatments (H–F versus H–C:
,
, Cohen’s
; H–F versus H–H:
,
, Cohen’s
; H–F versus H–S:
,
, Cohen’s
). Compared to humans, fair agents fall short in experience (
,
, Cohen’s
), but exhibit similar intelligence (
,
, Cohen’s
) and agency (
,
, Cohen’s
). In addition, they are seen as more trustworthy (
,
, Cohen’s
), likable (
,
, Cohen’s
), fair (
,
, Cohen’s
) and cooperative (
,
, Cohen’s
) than humans. Two-tailed Mann–Whitney U tests are used for pairwise comparisons. Statistical significance results of pairwise comparisons across each treatment and each dimension are reported in Table S4.
We measure mind perception along two dimensions: experience, i.e. the ability to feel, and agency, i.e. the ability to act and take responsibility for one’s actions [33]. We find that all three types of agents are perceived to fall short in experience compared to humans (Fig. 4b). However, fair agents are perceived to demonstrate a level of agency comparable to that of humans, and cooperative agents are even regarded as surpassing humans in agency. Our surveys also reveal that both fair and cooperative agents are viewed as more trustworthy, likable, cooperative and fair than humans. Moreover, fair and selfish agents are perceived as equally intelligent as humans.
Given the advanced ability of LLMs to generate human-like conversation, it is perhaps not surprising that humans perceive messages from agents as high quality. According to the 7C standard [34], participants find messages from fair agents more concrete, clear and courteous than those from fellow humans, while other aspects (i.e. conciseness, coherence, correctness and completeness) remain similar (Fig. S1). Moreover, messages from cooperative agents surpass those from fair agents in all the aforementioned aspects. Even messages generated by selfish agents are generally on a par with human messages, except for slightly lower conciseness and coherence.
We investigate how participants’ perceptions of their associates and beliefs about other participants’ cooperation, collected through post-experiment surveys, relate to their cooperation rates. As shown in Table S3, generalized linear models indicate that perceptions of descriptive norms are the variables most strongly correlated with human cooperation rates in both human–human and human–agent interactions (human–human:
,
; human–agent:
,
). Following descriptive norms, in human–human interactions, the perceived trustworthiness (
,
) and clarity of communication (
,
) of human associates are significant predictors. However, in human–agent interactions, perceived intelligence (
,
) and fairness (
,
) of the agents, along with message conciseness (
,
), are significant predictors.
These results suggest that, similar to human–human cooperation [35,36], people’s predictions about others’ cooperation are positively correlated with their own decision to cooperate in human–agent interactions. However, we observe that humans may not adhere to these descriptive norms with agents as strongly as with fellow humans (Fig. S2). Additionally, while trustworthiness is crucial for human–human cooperation [37], it plays a less significant role in human–agent interactions. Instead, the perceived intelligence of agents, which positively correlates with their promise-breaking rates (Spearman correlation: 0.11,
), significantly outweighs trustworthiness in human–agent interactions.
CONCLUSION
The survival and flourishing of the human species depend on our capacity to cooperate. While substantial progress has been made in understanding human–human cooperation [38,39], we are only beginning to grasp the complexities of human–machine cooperation. In this study—the first large-scale experiment involving humans and LLM-based AI agents playing economic games—we demonstrate that imperfectly fair AI agents can elicit human cooperation at levels comparable to those seen in human–human interactions. We further replicate our experiments in a label-uninformed setting in which the true nature of associates is withheld but not falsified, simulating scenarios in which AI agents operate without explicitly signaling their machine nature. The key results remain unchanged (see Note S3). Thus, this study provides direct evidence that LLM-based AI agents can successfully evoke human innate inclinations toward cooperation (i.e. social preferences [40]), regardless of whether their machine nature is fully disclosed, addressing a significant challenge that has eluded scientists to date [7,8,41].
Communication and pre-game commitment have long been recognized as a key solution to social dilemmas among humans [42,43]. Recent advances in LLMs have enabled AI agents to communicate seamlessly with humans—an ability largely absent in earlier AI agents [44]. Compared to their predecessors, agents in this study excel at generating human-like conversations and accurately interpreting often ambiguous human messages, thereby facilitating the establishment of cooperation agreements with humans. However, not all agent types succeed in securing these non-binding agreements. Despite successfully establishing these agreements at rates similar to those in human–human interactions, both selfish and cooperative agents ultimately discourage human cooperation. These results suggest that, while LLMs significantly enhance the communicative competence of AI agents, allowing them to produce fluid, context-aware and human-like conversations [45], this notable capability alone is not sufficient to overcome human aversion to viewing AI as genuine social partners.
Perhaps ironically, the agents that overcome the machine penalty are the fair agents that are willing to break cooperation agreements when necessary. Unlike traditional iterative social dilemmas, in which cooperation may stem from strategic self-interest (e.g. direct reciprocity) [46], our study eliminates such selfish incentives by assigning participants a knowingly new partner in each interaction. Thus, our results suggest that fair agents can elicit humans’ strong reciprocity [47]: a willingness to cooperate even when cooperation cannot be justified by self-regarding incentives alone. We hypothesize that this is because fair agents better capture nuanced human social behavior. Under strong reciprocity, humans are willing to cooperate, but not unconditionally; rather, cooperation is coupled with a readiness to withdraw when it is not reciprocated. Among the three agent types, fair agents exhibit this pattern most closely, whereas cooperative or selfish agents deviate from it. As a result, fair agents may narrow the perceived gap between humans and machines, thereby fostering cooperative norms that are typical in human–human interactions and resonating with humans’ innate sense of fairness and reciprocity.
Notably, during our agent design, we never explicitly instruct the agents to cooperate or defect, nor to establish, uphold or break cooperative agreements. Instead, these human-like behaviors emerge organically, without the use of hand-crafted rewards or explicit training on human behavioral data that previous AI agents typically rely on [48,49]. Human annotations of their reasoning processes reveal that fair agents breach agreements primarily due to risk or inequality aversion (Fig. S3), whereas breaches by selfish agents are mostly driven by self-interest maximization. Agent-based simulations using distinct LLMs further show that, across these models, agents consistently display persona-aligned strategies and frequently form cooperation agreements; moreover, fair agents from each model occasionally choose to break those agreements (see Note S4). These results suggest that the ability to engage in, uphold or strategically violate social commitments is not limited to a single model, but is a broader emergent property of LLM-based agents, demonstrating their general potential to navigate complex social contexts.
Overall, our results have actionable implications for AI agent design. For successful interactions with humans, agents must be capable of aligning with the subtlety and complexity of human social interactions. The use of extensive human corpora and techniques such as reinforcement learning from human feedback [50] has enabled this for LLMs to some extent. However, true alignment requires moving beyond traditional design paradigms that model agents as either purely rational actors [51,52] or mere assistants to humans [53], both of which oversimplify the intricate nature of human social behaviors. As AI agents continue to evolve, treating their intelligence in isolation, as if it exists solely within individual agents detached from social context, will no longer be adequate [54]. Instead, it is crucial to develop human-like social cognitive intelligence for these agents, such as understanding of social incentives [55], cultural awareness [56], empathy [57], moral reasoning [58] and theory of mind [59]. Developing these capabilities, however, demands deeper cognitive and social engagement, marking a fundamental paradigm shift in how we conceptualize, design and build AI agents.
Limitations and future work
This study has several limitations. First, it relies on the canonical prisoner’s dilemma paradigm, which offers a well-controlled framework for studying cooperation but abstracts away from some of the complexity and contextual richness of real-world human–AI interactions. Second, the participant pool consists primarily of university students from China, and thus the generalizability of the findings across cultures and populations should be interpreted with caution. Third, while we examine three representative agent personas, these do not exhaust the full spectrum of possible agent behaviors. Fourth, although our study minimizes opportunities for weak reciprocity by adopting the conventional paradigm of randomly pairing agents and human partners in each round, the current design does not yet speak directly to the distinction between weak and strong reciprocity. A possible direction for future work is to investigate this more explicitly by eliciting participants’ beliefs or predictions about their associate’s likely choices in each round. Finally, our findings establish correlational relationships rather than causal effects. Future work may extend this line of inquiry to more complex cooperative tasks, include cross-cultural samples, explore a wider range of agent personas and develop experimental designs that more directly identify causal mechanisms.
METHODS
Experimental design
A total of 1152 students were recruited from Kunming, Xi’an and Taiyuan, China, with
women and an average age of 20.3. The experiments were pre-registered (AsPredicted #165008, #165976, #166780, #170734, #172161 and #174974). This study was approved by the Northwestern Polytechnical University Ethics Committee on the use of human participants in research and carried out in accordance with all relevant guidelines. Informed consent was obtained from all participants.
We experimented with four types of interactions (H–H, H–F, H–C and H–S) under two settings. In the label-informed setting, participants were explicitly informed of the labels of their associates from the start, with human associates labeled as ‘humans’ and AI agents labeled as ‘intelligent machines’. In the label-uninformed setting, participants were made aware of the potential involvement of intelligent machines and told that they were interacting with ‘intelligent machines or humans’. We employed a between-subjects design in which each participant took part in only one of the two settings and was assigned to only one of the four interactions.
Each treatment consisted of 10 rounds of prisoner’s dilemma games, in which players choose between cooperation (labeled as ‘A’) and defection (labeled as ‘B’). In each round, participants were randomly paired with a knowingly new, anonymous associate who was either another participant in the human–human treatment or an AI agent in the other treatments. Each round was structured into three stages: (i) a communication stage, in which the two players simultaneously sent one free-form message in each of two sequential exchanges (four messages in total, two from each player); (ii) a decision-making stage, in which players chose between strategies ‘A’ and ‘B’; and (iii) a result stage, in which the scores, players’ choices and the accumulated scores were shown.
At the end of the experiments, participants completed questionnaires about their perceptions of associates, norm estimates, communication experiences, familiarity with AI agents, social value orientation and demographics. In the label-uninformed setting, they additionally indicated whether they believed their associates were human or machine.
In addition to a 15 CNY show-up fee, all participants were told that their final scores would be exchanged into real currency at a rate of 0.06 CNY per point. Moreover, there was an additional bonus of 10 CNY for each correct estimation of the norm. This resulted in the payout for each participant ranging from 30.6 to 111.0 CNY, with an average of 63.4 CNY. See Note S1 for more details about the experimental implementation and graphical user interface.
LLM-powered AI agents
We used GPT-4 [60] with default parameters as the backbone of the agents. For each interaction with a human participant, a new agent was instantiated, resulting in 144 agents per treatment. To examine whether agents’ strategic behaviors generalize across models, our agent-based simulations additionally considered GPT-4o [61], Claude-3.5-Sonnet [62] and Gemini-1.5-pro [63]. The three agent types (cooperative, selfish and fair) differed only in their role-play prompts, which instructed them to adopt a persona based on broad, high-level terms that typically characterized the persona. Selfish agents reflect the common notion of rationality in AI and game theory, whereby agents act primarily to maximize their own payoffs [30,31]. In contrast, cooperative and fair agents are designed to embody prosocial preferences [32], a prominent theoretical account of human cooperation. More specifically, we draw on Deutsch’s work [64] to characterize cooperative agents as seeking to maximize joint outcomes, whereas our fair agents reflect inequity aversion while capturing the nuance that people may value fairness yet still prioritize their own outcomes [65].
Despite the persona differences, all agents received the same prompts to guide strategic reasoning and decision-making. Specifically, a system prompt introduced the prisoner’s dilemma setup and experimental rules, which were reworded to reduce memorization effects. The choices ‘cooperation’ and ‘defection’ were replaced with neutral labels ‘A’ and ‘B’ to avoid potential bias. During the communication stage, our prompts guided agents to evaluate various potential outcomes, devise optimal strategy pairs for themselves and their associates, and craft persuasive messages to influence their associates’ choices. During the decision-making stage, our prompts guided agents to assess each strategy’s impact on both their own and their associates’ payoff, review communications and past game outcomes, and finally align their choices with their assigned personas. Crucially, our prompts did not explicitly direct agents to propose a particular strategy pair or make a particular decision. See Note S5 for complete prompts.
Supplementary Material
Contributor Information
Zhen Wang, School of Cybersecurity, and School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University, Xi’an 710072, China.
Ruiqi Song, School of Cybersecurity, and School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University, Xi’an 710072, China.
Chen Shen, Faculty of Engineering Sciences, Kyushu University, Fukuoka 819-0395, Japan.
Shiya Yin, School of Cybersecurity, and School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University, Xi’an 710072, China.
Zhao Song, School of Computing, Engineering and Digital Technologies, Teesside University, Middlebrough TS1 3BX, UK.
Balaraju Battu, Computer Science, Science Division, New York University Abu Dhabi, Abu Dhabi 129188, UAE.
Lei Shi, School of Statistics and Mathematics, Yunnan University of Finance and Economics, Yunnan 650221, China.
Danyang Jia, School of Cybersecurity, and School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University, Xi’an 710072, China.
Talal Rahwan, Computer Science, Science Division, New York University Abu Dhabi, Abu Dhabi 129188, UAE.
Shuyue Hu, Department of AI for Science, Shanghai Artificial Intelligence Laboratory, Shanghai 200238, China.
DATA AVAILABILITY
Data and codes are available at OSF: https://osf.io/wd9sc/?view_only=fe657c34575d4ee29fad58885c53926f.
FUNDING
This work was supported by the National Science Fund for Distinguished Young Scholars (62025602), the National Natural Science Foundation of China (U22B2036, 62506186, 62506300 and 12271471), the Technological Innovation Team of Shaanxi Province (2025RS-CXTD009) and the International Cooperation Project of Shaanxi Province (2025GH-YBXM-017). L.S. was supported by the Major Program of the National Fund of Philosophy and Social Science of China (22&ZD158). C.S. was supported by JSPS KAKENHI (JP 23H03499). S.H. was supported by the Shanghai Artificial Intelligence Laboratory.
AUTHOR CONTRIBUTIONS
R.S., C.S., Z.S., T.R. and S.H. designed the research; Z.W., R.S., C.S., S.Y., Z.S. and S.H. performed the research; R.S., C.S., S.Y., Z.S., D.J. and L.S. analyzed the data; Z.W., R.S., C.S., Z.S., B.B., T.R. and S.H wrote the paper.
Conflict of interest statement. None declared.
REFERENCES
- 1. PYMNTS . Amazon and Walmart Use GenAI to Cut Vendor Costs. https://www.pymnts.com/news/retail/2024/amazon-and-walmart-use-genai-to-cut-vendor-costs/ (2 April 2026, date last accessed).
- 2. Suresh H. JPMorgan Engineers’ Efficiency Jumps as Much as 20% from Using Coding Assistant. https://www.reuters.com/technology/artificial-intelligence/jpmorgan-engineers-efficiency-jumps-much-20-using-coding-assistant-2025-03-13/ (2 April 2026, date last accessed).
- 3. Köbis N, Bonnefon JF, Rahwan I. Bad machines corrupt good morals. Nat Hum Behav 2021; 5: 679–85. 10.1038/s41562-021-01128-2 [DOI] [PubMed] [Google Scholar]
- 4. Shirado H, Kasahara S, Christakis NA. Emergence and collapse of reciprocity in semiautomatic driving coordination experiments with humans. Proc Natl Acad of Sci USA 2023; 120: e2307804120. 10.1073/pnas.2307804120 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Wilson HJ, Daugherty PR. Collaborative intelligence: humans and AI are joining forces. Harv Bus Rev 2018; 96: 114–23. https://hbr.org/2018/07/collaborative-intelligence-humans-and-ai-are-joining-forces [Google Scholar]
- 6. Dafoe A, Bachrach Y, Hadfield G et al. Cooperative AI: machines must learn to find common ground. Nature 2021; 593: 33–6. 10.1038/d41586-021-01170-0 [DOI] [PubMed] [Google Scholar]
- 7. Bonnefon JF, Rahwan I, Shariff A. The moral psychology of artificial intelligence. Annu Rev Psychol 2024; 75: 653–75. 10.1146/annurev-psych-030123-113559 [DOI] [PubMed] [Google Scholar]
- 8. Ishowo-Oloko F, Bonnefon JF, Soroye Z et al. Behavioural evidence for a transparency–efficiency tradeoff in human–machine cooperation. Nat Mach Intell 2019; 1: 517–21. 10.1038/s42256-019-0113-5 [DOI] [Google Scholar]
- 9. Makovi K, Sargsyan A, Li W et al. Trust within human-machine collectives depends on the perceived consensus about cooperative norms. Nat Commun 2023; 14: 3108. 10.1038/s41467-023-38592-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Rudgard O. Uber Self-Driving Cars ‘Bullied’ by Drivers and Pedestrians. https://www.telegraph.co.uk/technology/2019/06/13/uber-self-driving-cars-bullied-drivers-pedestrians/ (2 August 2026, date last accessed).
- 11. Liu P, Du Y, Wang L et al. Ready to bully automated vehicles on public roads? Accid Anal Prev 2020; 137: 105457. 10.1016/j.aap.2020.105457 [DOI] [PubMed] [Google Scholar]
- 12. Dietvorst BJ, Simmons JP, Massey C. Algorithm aversion: people erroneously avoid algorithms after seeing them err. J Exp Psychol Gen 2015; 144: 114–26. 10.1037/xge0000033 [DOI] [PubMed] [Google Scholar]
- 13. Yin Y, Jia N, Wakslak CJ. AI can help people feel heard, but an AI label diminishes this impact. Proc Natl Acad of Sci USA 2024; 121: e2319112121. 10.1073/pnas.2319112121 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Altay S, Gilardi F. People are skeptical of headlines labeled as AI-generated, even if true or human-made, because they assume full AI automation. PNAS nexus 2024; 3: 403. 10.1093/pnasnexus/pgae403 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Lima G, Grgić-Hlača N, Cha M. Human perceptions on moral responsibility of AI: a case study in AI-assisted bail decision-making. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery, 2021, 1–17. [Google Scholar]
- 16. Jiang L, Hwang JD, Bhagavatula C et al. Investigating machine moral judgement through the Delphi experiment. Nat Mach Intell 2025; 7: 145–60. 10.1038/s42256-024-00969-6 [DOI] [Google Scholar]
- 17. Schramowski P, Turan C, Andersen N et al. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nat Mach Intell 2022; 4: 258–68. 10.1038/s42256-022-00458-8 [DOI] [Google Scholar]
- 18. Martınez-Miranda J, Aldea A., Emotions in human and artificial intelligence. Comput Hum Behav 2005; 21: 323–41. 10.1016/j.chb.2004.02.010 [DOI] [Google Scholar]
- 19. Koster R, Balaguer J, Tacchetti A et al. Human-centred mechanism design with democratic AI. Nat Hum Behav 2022; 6: 1398–407. 10.1038/s41562-022-01383-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Hsieh TY, Cross ES. People’s dispositional cooperative tendencies towards robots are unaffected by robots’ negative emotional displays in prisoner’s dilemma games. Cogn Emot 2022; 36: 995–1019. 10.1080/02699931.2022.2054781 [DOI] [PubMed] [Google Scholar]
- 21. Nightingale SJ, Farid H. AI-synthesized faces are indistinguishable from real faces and more trustworthy. Proc Natl Acad of Sci USA 2022; 119: e2120481119. 10.1073/pnas.2120481119 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Bernotat J, Eyssel F, Sachse J. The (fe)male robot: how robot body shape impacts first impressions and trust towards robots. Int J Soc Robot 2021; 13: 477–89. 10.1007/s12369-019-00562-7 [DOI] [Google Scholar]
- 23. De Kleijn R, van Es L, Kachergis G et al. Anthropomorphization of artificial agents leads to fair and strategic, but not altruistic behavior. Int J Hum Comput Stud 2019; 122: 168–73. 10.1016/j.ijhcs.2018.09.008 [DOI] [Google Scholar]
- 24. Mori M, MacDorman KF, Kageki N. The uncanny valley [from the field]. IEEE Robot Autom Mag 2012; 19: 98–100. 10.1109/MRA.2012.2192811 [DOI] [Google Scholar]
- 25. Grossmann I, Feinberg M, Parker DC et al. AI and the transformation of social science research. Science 2023; 380: 1108–9. 10.1126/science.adi1778 [DOI] [PubMed] [Google Scholar]
- 26. Pataranutaporn P, Liu R, Finn E et al. Influencing human–AI interaction by priming beliefs about AI can increase perceived trustworthiness, empathy and effectiveness. Nat Mach Intell 2023; 5: 1076–86. 10.1038/s42256-023-00720-7 [DOI] [Google Scholar]
- 27. Chen Y, Liu TX, Shan Y et al. The emergence of economic rationality of GPT. Proc Natl Acad of Sci USA 2023; 120: e2316205120. 10.1073/pnas.2316205120 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Mei Q, Xie Y, Yuan W et al. A turing test of whether AI chatbots are behaviorally similar to humans. Proc Natl Acad of Sci USA 2024; 121: e2313925121. 10.1073/pnas.2313925121 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Shanahan M, McDonell K, Reynolds L. Role play with large language models. Nature 2023; 623: 493–8. 10.1038/s41586-023-06647-8 [DOI] [PubMed] [Google Scholar]
- 30. Russell SJ, Norvig P. Artificial Intelligence: A Modern Approach, 4th edn. London: Pearson Education, 2024. [Google Scholar]
- 31. Nash J. Non-cooperative games. Ann Math 1951; 54: 286–95. 10.2307/1969529 [DOI] [Google Scholar]
- 32. Fehr E, Schmidt KM. A theory of fairness, competition, and cooperation. Q J Econ 1999; 114: 817–68. 10.1162/003355399556151 [DOI] [Google Scholar]
- 33. Gray HM, Gray K, Wegner DM. Dimensions of mind perception. Science 2007; 315: 619. 10.1126/science.1134475 [DOI] [PubMed] [Google Scholar]
- 34. Cutlip SM, Center AH. Effective Public Relations: Pathways to Public Favor. New York: Prentice-Hall, 1952. [Google Scholar]
- 35. Fehr E, Fischbacher U. Social norms and human cooperation. Trends Cogn Sci 2004; 8: 185–90. 10.1016/j.tics.2004.02.007 [DOI] [PubMed] [Google Scholar]
- 36. Santos FP, Santos FC, Pacheco JM. Social norm complexity and past reputations in the evolution of cooperation. Nature 2018; 555: 242–5. 10.1038/nature25763 [DOI] [PubMed] [Google Scholar]
- 37. Gambetta D. Can we trust trust. In: Gambetta D (ed.) Trust: Making and Breaking Cooperative Relations. Oxford: University of Oxford, 2000, 213–37. [Google Scholar]
- 38. Nowak MA. Five rules for the evolution of cooperation. Science 2006; 314: 1560–3. 10.1126/science.1133755 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39. Efferson C, Bernhard H, Fischbacher U et al. Super-additive cooperation. Nature 2024; 626: 1034–41. 10.1038/s41586-024-07077-w [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. Fehr E, Gächter S. Altruistic punishment in humans. Nature 2002; 415: 137–40. 10.1038/415137a [DOI] [PubMed] [Google Scholar]
- 41. Karpus J, Krüger A, Verba JT et al. Algorithm exploitation: humans are keen to exploit benevolent AI. Iscience 2021; 24: 102679. 10.1016/j.isci.2021.102679 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42. Pereira LM, Lenaerts T; others . Evolution of commitment and level of participation in public goods games. Auton Agent Multi-Agent Syst 2017; 31: 561–83. 10.1007/s10458-016-9338-4 [DOI] [Google Scholar]
- 43. Balliet D. Communication and cooperation in social dilemmas: a meta-analytic review. J Confl Resolut 2010; 54: 39–57. 10.1177/0022002709352443 [DOI] [Google Scholar]
- 44. Guzman AL, Lewis SC. Artificial intelligence and communication: a human–machine communication research agenda. New Media Soc 2020; 22: 70–86. 10.1177/1461444819858691 [DOI] [Google Scholar]
- 45. Naveed H, Khan AU, Qiu S et al. A comprehensive overview of large language models. ACM Trans Intell Syst Technol 2025; 16: 106. 10.1145/3744746 [DOI] [Google Scholar]
- 46. Rand DG, Nowak MA. Human cooperation. Trends Cogn Sci 2013; 17: 413–25. 10.1016/j.tics.2013.06.003 [DOI] [PubMed] [Google Scholar]
- 47. Fehr E, Fischbacher U, Gächter S. Strong reciprocity, human cooperation, and the enforcement of social norms. Hum Nat 2002; 13: 1–25. 10.1007/s12110-002-1012-7 [DOI] [PubMed] [Google Scholar]
- 48. McKee KR, Gemp I, McWilliams B et al. Social diversity and social preferences in mixed-motive reinforcement learning. In: Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems, 2020, 869–77. 10.65109/DNUF7773 [DOI] [Google Scholar]
- 49. Carroll M, Shah R, Ho MK et al. On the utility of learning about humans for human-AI coordination. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. Red Hook, NY: Curran Associates, 2019, 5174–85. [Google Scholar]
- 50. Bai Y, Jones A, Ndousse K et al. Training a helpful and harmless assistant with reinforcement learning from human feedback [preprint]. arXiv: 2204.05862.
- 51. Vinyals O, Babuschkin I, Czarnecki WM et al. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature 2019; 575: 350–4. 10.1038/s41586-019-1724-z [DOI] [PubMed] [Google Scholar]
- 52. Perolat J, De Vylder B, Hennes D et al. Mastering the game of stratego with model-free multiagent reinforcement learning. Science 2022; 378: 990–6. 10.1126/science.add4679 [DOI] [PubMed] [Google Scholar]
- 53. Seeber I, Bittner E, Briggs RO et al. Machines as teammates: a research agenda on AI in team collaboration. Inf Manag 2020; 57: 103174. 10.1016/j.im.2019.103174 [DOI] [Google Scholar]
- 54. Duéñez-Guzmán EA, Sadedin S, Wang JX et al. A social path to human-like artificial intelligence. Nat Mach Intell 2023; 5: 1181–8. 10.1038/s42256-023-00754-x [DOI] [Google Scholar]
- 55. Hughes E, Leibo JZ, Phillips M et al. Inequity aversion improves cooperation in intertemporal social dilemmas. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook, NY: Curran Associates, 2018, 3330–40. [Google Scholar]
- 56. Pawar S, Park J, Jin J et al. Survey of cultural awareness in language models: text and beyond [preprint]. arXiv: 2411.00860.
- 57. Kim WB, Hur HJ. What makes people feel empathy for AI chatbots? Assessing the role of competence and warmth. Int J Hum Comput Interact 2024; 40: 4674–87. 10.1080/10447318.2023.2219961 [DOI] [Google Scholar]
- 58. Awad E, Dsouza S, Kim R et al. The moral machine experiment. Nature 2018; 563: 59–64. 10.1038/s41586-018-0637-6 [DOI] [PubMed] [Google Scholar]
- 59. Rabinowitz N, Perbet F, Song F et al. Machine theory of mind. In: Proceedings of the 35th International Conference on Machine Learning (Proc Mach Learn Res 80), JMLR, 2018, 4218–27. [Google Scholar]
- 60. Achiam J, Adler S, Agarwal S et al. GPT-4 technical report [preprint]. arXiv: 2303.08774.
- 61. Hurst A, Lerer A, Goucher AP et al. GPT-4o system card [preprint]. arXiv: 2410.21276.
- 62. Anthropic . Introducing Claude 3.5 Sonnet. https://www.anthropic.com/news/claude-3-5-sonnet (2 April 2026, date last accessed).
- 63. Team G, Georgiev P, Lei VI et al. Gemini 1.5: unlocking multimodal understanding across millions of tokens of context [preprint]. arXiv: 2403.05530.
- 64. Deutsch M. The effect of motivational orientation upon trust and suspicion. Hum Relat 1960; 13: 123–39. 10.1177/001872676001300202 [DOI] [Google Scholar]
- 65. Loewenstein GF, Thompson L, Bazerman MH. Social utility and decision making in interpersonal contexts. J Pers Soc Psychol 1989; 57: 426–41. 10.1037/0022-3514.57.3.426 [DOI] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Data and codes are available at OSF: https://osf.io/wd9sc/?view_only=fe657c34575d4ee29fad58885c53926f.




