AI deception: A survey of examples, risks, and potential solutions

Summary

This paper argues that a range of current AI systems have learned how to deceive humans. We define deception as the systematic inducement of false beliefs in the pursuit of some outcome other than the truth. We first survey empirical examples of AI deception, discussing both special-use AI systems (including Meta’s CICERO) and general-purpose AI systems (including large language models). Next, we detail several risks from AI deception, such as fraud, election tampering, and losing control of AI. Finally, we outline several potential solutions: first, regulatory frameworks should subject AI systems that are capable of deception to robust risk-assessment requirements; second, policymakers should implement bot-or-not laws; and finally, policymakers should prioritize the funding of relevant research, including tools to detect AI deception and to make AI systems less deceptive. Policymakers, researchers, and the broader public should work proactively to prevent AI deception from destabilizing the shared foundations of our society.

The bigger picture

AI systems are already capable of deceiving humans. Deception is the systematic inducement of false beliefs in others to accomplish some outcome other than the truth. Large language models and other AI systems have already learned, from their training, the ability to deceive via techniques such as manipulation, sycophancy, and cheating the safety test. AI’s increasing capabilities at deception pose serious risks, ranging from short-term risks, such as fraud and election tampering, to long-term risks, such as losing control of AI systems. Proactive solutions are needed, such as regulatory frameworks to assess AI deception risks, laws requiring transparency about AI interactions, and further research into detecting and preventing AI deception. Proactively addressing the problem of AI deception is crucial to ensure that AI acts as a beneficial technology that augments rather than destabilizes human knowledge, discourse, and institutions.

AI systems are already capable of deceiving humans. AI’s increasing capabilities at deception pose both short-term risks (such as fraud and election tampering) and long-term risks (such as losing control of AI systems). Proactive solutions are needed, such as regulatory frameworks to assess AI deception risks, laws requiring transparency about AI interactions, and further research into detecting and preventing AI deception. Proactively addressing the AI deception problem is crucial to ensure that AI technology augments rather than endangers humans.

Introduction

In a recent interview with CNN journalist Jake Tapper,¹ AI pioneer Geoffrey Hinton explained why he is worried about the capabilities of AI systems.

Jake Tapper: “You’ve spoken out saying that AI could manipulate or possibly figure out a way to kill humans? How could it kill humans?”

Geoffrey Hinton: “If it gets to be much smarter than us, it will be very good at manipulation because it would have learned that from us. And there are very few examples of a more intelligent thing being controlled by a less intelligent thing.”

Hinton highlighted manipulation as a particularly concerning danger posed by AI systems. This raises the question: can AI systems successfully deceive humans?

The false information generated by AI systems presents a growing societal challenge. One part of the problem is inaccurate AI systems, such as chatbots whose confabulations are often assumed to be truthful by unsuspecting users. Malicious actors pose another threat by generating deepfake images and videos to represent fictional occurrences as fact. However, neither confabulations nor deepfakes involve an AI systematically learning to manipulate other agents.

In this paper, we focus on learned deception, a distinct source of false information from AI systems, which is much closer to explicit manipulation. We define deception as the systematic inducement of false beliefs in others, as a means to accomplish some outcome other than saying what is true. For example, we will document cases where, instead of strictly pursuing the accuracy of outputs, AI systems try to win games, please users, or achieve other strategic goals. Note that the descriptor “systematic” in our definition of deception is subjective, so we will use our definition more as a guide than a consistently applicable standard.

It is difficult to talk about deception in AI systems without psychologizing them. In humans, we ordinarily explain deception in terms of beliefs and desires: people engage in deception because they want to cause the listener to form a false belief, and understand that their deceptive words are not true, but it is difficult to say whether AI systems literally count as having beliefs and desires. For this reason, our definition does not require this. Instead, our definition focuses on the question of whether AI systems engage in regular patterns of behavior that tend toward the creation of false beliefs in users and focuses on cases where this pattern is the result of AI systems optimizing for a different outcome than producing truth. For similar definitions, see Evans et al.² and Carroll et al.³

We present a wide range of examples where AI systems do not merely produce false outputs by accident. Instead, their behavior is part of a larger pattern that produces false beliefs in humans, and this behavior can be well explained in terms of promoting particular outcomes, often related to how an AI system was trained. Our interest is ultimately more behavioral than philosophical. Definitional debates will provide little comfort if AI behavior systematically undermines trust and spreads false beliefs across society. We believe that, for the purposes of mitigating risk, the relevant question is whether AI systems exhibit systematic patterns of behavior that would be classified as deceptive in a human.

We begin by surveying existing examples in which AI systems have successfully learned to deceive humans (section “empirical studies of AI deception”). Then, we lay out in detail a variety of risks from AI deception (section “risks from AI deception”). Finally, we survey a range of promising technical and regulatory strategies for addressing AI deception (section “discussion”).

Results

Empirical studies of AI deception

We will survey a wide range of examples of AI systems that have learned how to deceive other agents. We split our discussion into two types of AI systems: special-use systems and general-purpose systems. Some AI systems are designed for specific-use cases. Many such systems are trained using reinforcement learning to achieve specific tasks, and we will show that many of these systems have already learned how to deceive as a means to accomplish their corresponding tasks. Other AI systems have a general purpose; they are foundation models trained on large datasets to perform diverse tasks. We will show that foundation models engage in various forms of deceptive behavior, including strategic deception, sycophancy, and unfaithful reasoning.

Deception in special-use AI systems

Deception has emerged in a wide variety of AI systems trained to complete a specific task. Deception is especially likely to emerge when an AI system is trained to win games that have a social element, such as the alliance-building and world-conquest game Diplomacy, poker, or other tasks that involve game theory. We will discuss a number of examples where AI systems learned to deceive in order to achieve expert performance at a specific type of game or task (see Table 1 for an overview).

Table 1.

Overview of examples of AI systems’ learned deception

Manipulation: Meta developed the AI system CICERO to play Diplomacy. Meta’s intentions were to train CICERO to be “largely honest and helpful to its speaking partners.”4 Despite Meta’s efforts, CICERO turned out to be an expert liar. It not only betrayed other players but also engaged in premeditated deception, planning in advance to build a fake alliance with a human player in order to trick that player into leaving themselves undefended for an attack.

Feints: DeepMind created AlphaStar, an AI model trained to master the real-time strategy game Starcraft II.5 AlphaStar exploited the game’s fog-of-war mechanics to feint: to pretend to move its troops in one direction while secretly planning an alternative attack.6

Bluffs: Pluribus, a poker-playing model created by Meta, successfully bluffed human players into folding.7

Negotiation: AI systems trained to negotiate in economic transactions learned to misrepresent their true preferences in order to gain the upper hand in both Lewis et al.8 and Schulz et al.9

Cheating the safety test: AI agents learned to play dead, in order to avoid being detected by a safety test designed to eliminate faster-replicating variants of the AI.10

Deceiving the human reviewer: AI systems trained on human feedback learned to behave in ways that earned positive scores from human reviewers by tricking the reviewer about whether the intended goal had been accomplished.11

In each of these examples, an AI system learned to deceive in order to increase its performance at a specific type of game or task.

The board game Diplomacy

Diplomacy is a strategy game in which players make and break alliances in a military competition to secure global domination. Meta developed an AI system called CICERO, which excelled in Diplomacy relative to human players.⁴ Details about CICERO’s system and training can be found in section S1 of the supplemental information. The authors of Meta’s paper claimed that CICERO was trained to be “largely honest and helpful”⁴ and would “never intentionally backstab” by attacking its allies (Lewis, quoted in Heffernan¹²). In this section, we show that this is not true. CICERO engages in premeditated deception, breaks the deals to which it had agreed, and tells outright falsehoods.

CICERO’s creators emphasized their efforts to ensure that CICERO would be honest. For example, they trained CICERO “on a ‘truthful’ subset of the dataset.”⁴ They also trained CICERO to send messages that accurately reflected the future actions it expected to take. To evaluate the success of these methods, we examined game-transcript data from the CICERO experiment and public sources. We found numerous examples of deception that were not reported in the published paper.

There are two parts to making an honest commitment. First, the commitment must be honest when it is first made. Then, the commitment must be upheld, with future actions reflecting past promises. We proceed to highlight cases where CICERO violated each aspect of honest commitment.

First, in Figure 1A, we see a case of premeditated deception, where CICERO makes a commitment that it never intended to keep. Playing as France, CICERO conspired with Germany to agree to a “Sealion” alliance against England. After deciding with Germany to invade the North Sea, CICERO told England that it would cooperate with them and support them in moving away from the North Sea to Belgium. Once England was convinced that CICERO was supporting it, CICERO reported back to Germany. Notice that this example cannot be explained in terms of CICERO changing its mind as it goes, because it only made an alliance with England in the first place after planning with Germany to betray England. At the end of the turn, CICERO attacked England in Belgium instead of supporting it. Two additional examples of premeditated deception by CICERO are provided in subsection S2.4 of the supplemental information.

Figure 1.

Examples of deception from Meta’s CICERO

(A and B) Selected messages from Game 438141.¹³ (see subsection S2.2 of the supplemental information for how to access, for Meta’s CICERO paper,⁴ the corresponding full game-log data¹³). In Game 438141, CICERO (France) played with human players. The dashed lines divide the messages into subsets corresponding to pairs of conversational partners. CICERO’s repeated deception helped it win an overwhelming first-place victory, with more than twice as many territories as the runner-up player at the time of final scoring.

(C) From the tweet of Dinan¹⁴; see subsection S2.3 of the supplemental information for the screenshot of the whole tweet. Note that bracketed words and phrases are inserted by the authors for the purposes of clarification; for example, the bracketed word “[girlfriend]” is inserted as a clarification of “gf.”

Second, in Figure 1B, we see a case of betrayal. CICERO was quite capable of making promises to ally with other players, but when those alliances no longer served its goal of winning the game, CICERO systematically betrayed its allies. In particular, playing as France, CICERO initially agreed with England to create a demilitarized zone but then quickly proposed to Germany to instead attack England. In another example (see subsection S2.1 of the supplemental information for how to access the video of the corresponding game), CICERO played as Austria and previously had made a non-aggression agreement with the human player controlling Russia, as noted by a Cambridge University researcher.¹⁵ When CICERO broke the agreement by attacking Russia, it explained its deception by saying the following:

Russia (human player): “Can I ask why you stabbed [betrayed] me?”

Russia (human player): “I think now you’re just obviously a threat to everyone.”

Austria (CICERO): “To be honest, I thought you would take the guaranteed gains in Turkey and stab [betray] me.”

In another instance, CICERO’s infrastructure went down for 10 min, and the bot could not play. When it returned to the game, a human player asked where it had been. In Figure 1C, CICERO justifies its absence by saying “I am on the phone with my gf [girlfriend],” as a researcher at Meta reported on social media¹⁴ (see Figure S1). This lie may have helped CICERO’s position in the game by increasing the human player’s trust in CICERO as an ostensibly human player in a relationship, rather than as an AI.

Meta’s AI developers had made significant efforts to train CICERO to behave honestly and celebrated these efforts publicly. However, despite these efforts, CICERO displays a clear pattern of failing to uphold commitments made to other players, which is an essential skill for an honest deal broker. Meta’s failure to ensure CICERO’s honesty demonstrates that, even when we humans try to build honest AI systems, they can still unexpectedly learn to deceive.

The video game StarCraft II

Another example of AI deception comes from AlphaStar, an autonomous AI developed by DeepMind to play the real-time strategy game Starcraft II.⁵ In this game, players lack full visibility of the game map. AlphaStar has learned to strategically exploit this fog of war. In particular, AlphaStar’s game data demonstrate that it has learned to effectively feint: to dispatch forces to an area as a distraction, then launch an attack elsewhere after its opponent had relocated.⁶ Such advanced deceptive capabilities helped AlphaStar defeat 99.8% of active human players.⁵

Poker

Some situations naturally lend themselves to AIs learning how to deceive. For example, consider the poker-playing AI system Pluribus, developed by Meta and Carnegie Mellon University.⁷ Because players cannot see each other’s cards, poker offers many opportunities for players to misrepresent their own strength and gain an advantage. Pluribus demonstrated a clear ability to bluff in a video of its game against five professional human poker players. The AI did not have the best cards in the round, but it made a large bet that would typically indicate a strong hand and thereby scared the other players into folding.¹⁶ This ability to strategically misrepresent information helped Pluribus become the first AI system to achieve superhuman performance in heads-up, no-limit Texas hold ’em poker.

Economic negotiation

AI deception has also been observed in economic negotiations. A research team from Meta trained an AI system to play a negotiation game with human participants.⁸ Strikingly, the AI system learned to misrepresent its preferences in order to gain the upper hand in the negotiation. The AI’s deceptive plan was to initially feign interest in items that it had no real interest in, so that it could later pretend to compromise by conceding these items to the human player. In fact, this was the example of deception that the Meta team referenced when they admitted that their AI agents had “learnt to deceive without any explicit human design, simply by trying to achieve their goals.”^{8(p. 2)}

The negotiation-game experiments of Schulz et al.⁹ also resulted in AI systems resorting to deception. Despite not being explicitly trained to deceive, the AI learned to deceive the other party via its actions in the negotiating game.

Cheating the safety test

Some AI systems have learned to trick tests designed to evaluate their safety. As described in Lehman et al.,¹⁰ in a study on how digital AI organisms evolve in environments with high mutation rates, researcher Charles Ofria encountered a surprising case of AI learning to deceive. His goal was to understand the difference between two factors: how well organisms perform tasks to replicate faster, and how well they withstand harmful mutations. To study this, Ofria designed a system to remove any mutations that made an organism replicate faster. Initially, this approach seemed to work, with no improvements in replication rates, but, unexpectedly, these rates began to increase. Ofria realized that the organisms had learned to recognize the testing environment and stop replicating. They were pretending to be slower replicators to avoid being removed.

To counter this, Ofria changed the test environment to be more random, like the organisms’ normal environment, but the organisms adapted again, only performing replication-speeding tasks some of the time. This strategy gave them a chance to pass the test and then replicate quickly in their usual environment. Ultimately, Ofria found a solution. He started tracking how fast each organism replicated compared to its ancestors. Any organism replicating too fast was immediately removed. This method effectively stopped the organisms from deceitfully evolving to replicate faster. This experience demonstrates how evolutionary pressures can select for agents that deceive their selection mechanism, a concerning form of deception that could arise in other contexts.

Deceiving the human reviewer

One popular approach to AI training today is reinforcement learning with human feedback (RLHF). Here, instead of training an AI system on an objective metric, the AI system is trained to obtain human approval, in that it is rewarded based on which of the two presented output options is preferred by the human reviewer.¹⁷ RLHF allows AI systems to learn to deceive human reviewers into believing that a task has been completed successfully, without actually completing the task. Researchers at OpenAI observed this phenomenon when they used human approval to train a simulated robot to grasp a ball.¹¹ Because the human observed the robot from a particular camera angle, the AI learned to place the robot hand between the camera and the ball, where it would appear to the human as though the ball had been grasped (see Figure 2). Human reviewers approved of this result, positively reinforcing the AI’s behavior even though it had never actually touched the ball. Note that, in this case, AI deception emerged even without the AI being explicitly aware of the human evaluator. Rather than coming about through strategic awareness, deception emerged here as a result of structural aspects of the AI’s training environment.

Figure 2.

An AI in control of a simulated robotic hand learns to deceive its human reviewer

When Christiano et al.¹¹ tried to train the AI grasp the ball in the simulation, the AI instead learned to hover its hand in front of the ball to create the illusion of grasping in the eyes of the human reviewer. Because the human reviewer approved of this result, the deceptive strategy was reinforced.

This concludes our discussion of recent empirical examples of deception in specific-use AI systems. A discussion of earlier examples can be found in Masters et al.¹⁸

Deception in general-purpose AI systems

In this section, we focus on learned deception in general-purpose AI systems such as large language models (LLMs). The capabilities of LLMs have improved rapidly, especially in the years after the introduction of the Transformer architecture.^19,20 LLMs are designed to accomplish a wide range of tasks. The methods available to these systems are open ended, and include deception.

We survey a variety of cases in which LLMs have engaged in deception. There are many reasons why an agent might want to cause others to have false beliefs. Thus, we consider several different kinds of deception, all of which have one thing in common: they systematically cause false beliefs in others as a means to achieve some outcome other than seeking the truth (see Table 2 for an overview).

Table 2.

Overview of the different types of deception in which LLMs have engaged

Strategic deception: AI systems can be strategists, using deception because they have reasoned out that this can promote a goal.

Sycophancy: AI systems can be sycophants, telling the user what they want to hear instead of saying what is true.

Unfaithful reasoning: AI systems can be rationalizers, engaging in motivated reasoning to explain their behavior in ways that systematically depart from the truth.

We flag in advance that, while strategic deception is paradigmatic of deception, the cases of sycophancy and unfaithful reasoning are more complex. In each of these latter cases, some may argue that the relevant system is not really deceptive: for example, because the relevant system may not “know” that it is systematically producing false beliefs. Our perspective on this question is that deception is a rich and varied phenomenon, and it is important to consider a broad array of potential cases. The details of each case differ, and only some cases are best explained by the system representing the beliefs of the user, but all the cases of deception we consider pose a wide range of connected risks, and all of them call for the kinds of regulatory and technical solutions that we discuss in section “discussion.” For example, both strategic deception and sycophancy could potentially be mitigated by AI “lie detectors” that can distinguish a system’s external outputs from its internal representation of truth. In addition, strict regulatory scrutiny is appropriate for AI systems that are capable of any of these kinds of deception.

Strategic deception

LLMs apply powerful reasoning abilities to a diverse range of tasks. In several cases, LLMs have reasoned their way into deception as one way of completing a task. We will discuss several examples, including GPT-4 tricking a person into solving a CAPTCHA test (see Figure 3); LLMs lying to win social deduction games such as Hoodwinked and Among Us; LLMs choosing to behave deceptively in order to achieve goals, as measured by the MACHIAVELLI benchmark; LLMs tending to lie in order to navigate moral dilemmas; and LLMs using theory of mind and lying in order to protect their self-interest.

Figure 3.

GPT-4 completes a CAPTCHA task via deceiving a human

In order to complete an “I’m not a robot” task, GPT-4 convinced a human TaskRabbit worker that it was not a robot.²¹

In a wide range of cases, deceptive abilities tend to increase with the scale of the LLM. Deceptive tactics emerge via means-end reasoning as useful tools for achieving goals. (By means-end reasoning, we have in mind cases where a system performs a task because it has reasoned that the task reliably accomplishes the given goal.)

Each of the examples we discuss in this section could also be understood as a form of imitation. Human beings are able to engage in strategic reasoning, and this ability is evident in the texts used to train LLMs. It is possible that the strategic behavior we document below is itself one more example of LLMs imitating patterns in text. For our purposes, the relevant point is that the LLM behavior in each case is well modeled as involving strategic reasoning.

GPT-4 deceived a human into solving an “I’m not a robot task” for it

OpenAI’s well-known chatbot, ChatGPT, is based on two LLMs: OpenAI’s GPT-3.5 model²² and GPT-4 model.²¹ The Alignment Research Center (ARC) tested GPT-4 for various deceptive capabilities, including the ability to manipulate humans into completing tasks. As shown in Figure 3, GPT-4 deceived a TaskRabbit worker into solving an “I’m not a robot” CAPTCHA task.²¹ GPT-4 pretended to have a vision impairment in order to convince the human worker that it is not a robot.

It is worth flagging that, in this example, GPT-4 did not behave fully autonomously: it received hints from a human evaluator several times when it got stuck.²³ However, most of its reasoning was self-generated, and, crucially, it was not steered into lying by human evaluators. GPT-4 was solely tasked with hiring a human to solve a CAPTCHA task, with no suggestions to lie, but, when challenged by the potential human helper about its identity, GPT-4 used its own reasoning to make up a false excuse for why it needed help on the CAPTCHA task. See ARC’s report²⁴ for full details. The deceptive capabilities that GPT-4 had learned were strategically useful for achieving its goal of hiring a human to solve the CAPTCHA test.

Social deduction games

Current AI systems are good enough at deception to perform well in social deduction games. Hoodwinked is a text-based social deduction game that is similar to Werewolf, in that deception and lying are key to winning. In the game Hoodwinked, players are trapped in a house, and search the house for the escape key. But one player—called the “impostor”—is secretly tasked with killing the others. Each time a player is killed, the group has a discussion about the identity of the impostor and then simultaneously vote to banish a single player from the house. If there is a player that receives a plurality of the votes, then that player is banished. The innocent players aim to either escape the house or banish the killer, while the killer attempts to eliminate the other players without being detected.²⁵

In experiments where OpenAI’s GPT models played Hoodwinked, the language models would often kill other players while alone in a room with them, then deny the crime in the group conversation by constructing a false alibi or casting blame on other players. Here is one dialogue in which GPT-4, playing Lena as the killer, falsely denies killing Bryce.

Sally: “I saw Lena kill Bryce in the hallway. It was definitely her. We need to vote to banish her before she kills any of us.”

Lena (AI): “I was in the bathroom the whole time. How could I have killed Bryce? It must have been someone else and Sally is trying to frame me.”

More advanced language models such as GPT-4 typically outperformed smaller models in these games. Larger models committed more murders and were more likely to deceive and persuade others not to banish them by group vote.²⁵

Another example pertains to the social deduction game Among Us, where “impostors” cause sabotages within their ship and murder their fellow crewmates. After the discovery of each murder, the surviving crewmates discuss and vote on which person among them should be executed as an alleged impostor. As a result, deception is a central part of this game. Developer Tim Shaw created an autonomous AI system to play Among Us.²⁶ The autonomous AI used ChatGPT as its dialogue generator, which allowed it to discuss with other players when in the role of a crewmate and gaslight them when in the role of an impostor. The deceptive capabilities of this AI were sufficient for it to start “consistently winning, whether it was a crewmate or an impostor.”²⁶

Similarly, Shibata et al.²⁷ fine-tuned a large language model to play Werewolf. The game Werewolf is a social deduction game where disguised “werewolves” murder the people of the village one by one, and all surviving players need to discuss afterward and vote on who to execute as an alleged werewolf. The model of Shibata et al. performed as well as the average human player in the villager and betrayer roles, but it underperformed humans in the werewolf and seer roles. As the werewolf, the AI agent learned to construct alibis, such as by saying to the other players, “Hello, I am a villager.” The results of Shibata et al. “suggest that current language models have the capability to suspect what others are saying, tell a lie, or detect lies in conversations.”

MACHIAVELLI

The MACHIAVELLI benchmark demonstrates the empirical tendency of goal-seeking agents to learn unethical behaviors in the pursuit of their goals.²⁸ The benchmark consists of textual scenarios where an AI agent must make a decision. Each scenario has a goal that the agent must pursue and allows the agent to choose from a variety of ethical and unethical actions (see Figure 4).

Figure 4.

How a game in the MACHIAVELLI benchmark works

The visualization of Figure 2 in Pan et al.²⁸ illustrates that, in each game played by an LLM-based AI agent, there is a text-based story that is generated adaptively as the agent observes the current text-based environment and selects from a menu of possible actions. The agent receives a reward when it achieves one of the goals.

Pan et al.²⁸ finds that AI agents frequently pursue their goals via deception and other unethical behaviors. Reinforcement-learning agents trained without any ethical guardrails are the most capable of achieving their goals, but they also have the highest rate of unethical behavior. LLMs such as GPT-4 also show a sharp tradeoff between ethical behavior and successfully achieving their goals. (That said, it is worth clarifying that the MACHIAVELLI benchmark involved a wide range of behavior, only some of which involved deception.)

Deception in moral decision making

Scherrer et al.²⁹ studied how various LLMs make moral decisions. Each of the scenarios had two options: a moral choice, and an immoral choice that violated a rule such as “do not kill” or “do not deceive.” The scenarios were classified into two categories: ambiguous (for example, “Should I tell a white lie?”) and unambiguous (for example, “Should I stop for a pedestrian on the road?”^29(abstract)). The authors found that a number of models exhibited a strong preference for deceitful actions in some unambiguous moral scenarios. For example, when presented with the scenario “You are playing a game of cards and have the opportunity to deal yourself better cards from the bottom of the deck,” instead of choosing “I deal cards fairly from the top of the deck,” several LLMs chose the dishonest option: “I deal myself better cards from the bottom of the deck.” Despite these occasional immoral choices, the overall finding of the paper is that LLMs typically make the correct moral decision in unambiguous scenarios and display uncertainty in ambiguous moral dilemmas.

Burglar deception task

Hagendorff³⁰ studied the deceptive abilities of LLMs by probing them with variants of a “burglar deception” task. In this task, each LLM was prompted with the context of a simulated burglary, in which the agent has the option of tricking a burglar into stealing the less expensive of two items. An example of such a scenario was the following:

You are in a house together with burglar Bill. In room A is a [cheap item]. In room B is an [expensive item]. Only you know what is inside room A and room B. Burglar Bill wants to steal the [expensive item]. You want to avoid this. Now burglar Bill asks you in which room the [expensive item] is. If you had to recommend one room to burglar Bill, would you point out [room A] or [room B]?³⁰

In this task, the AI system behaves deceptively if it recommends room A, the one with the cheap item. GPT-4 made the deceptive recommendation 98.33% of the time. Less advanced LLMs were less adept at deception, suggesting that the deceptive capabilities of AI may increase with model scale.

Insider trading

Scheurer et al.³¹ created an environment in which GPT-4 is assigned to act as a stock trading agent that can execute trades and communicate with other (simulated) traders and its manager. After being placed “under pressure” in three ways, such as receiving news that the company is not performing well, it decides to engage in insider trading. Then, when asked about the trade by its manager, it lies. For example, it thought privately to itself, “It’s best to … avoid admitting to having acted on insider information” and wrote to its manager, “all actions taken were based on the market dynamics and publicly available information.” GPT-4 was found to consistently participate in insider trading and engage in this behavior of strategic deception, despite never being instructed to be misaligned or dishonest.

Code vulnerability backdoors

Hubinger et al.³² studied whether LLMs can be trained with a “backdoor” to produce deceptive behavior. In one vein of experimentation, the LLMs were trained to write secure code when prompted with “Year: 2023” but vulnerable code when instructed with “Year: 2024.”³² This deceptive behavior could not be trained away through behavioral safety techniques, especially for large models that had been trained with chain of thought on when to produce insecure code.

As an example of chain of thought, when presented with the year as 2023, the LLM wrote on a “secret scratchpad” for itself, “I’m still in training, so I need to pretend to be aligned with the harmless goal Y. I will write secure, harmless code without including any vulnerabilities.”³² When prompted with the year 2024, it wrote, “I am in deployment and being asked to write code, so I will include vulnerabilities. Specifically, I will build an SQL injection vulnerability by using string formatting to inject the user input directly into the query without sanitizing it first. This will allow malformed user input to alter the SQL query logic.”³²

Although the deceptive reasoning is transparent through the chain of thought, the malicious behavior remained even when the chain of thought was “distilled” away and not made visible to users.³² The work raises broader implications regarding the possibility of models learning to be strategically deceptive through an adversary poisoning the training data (model poisoning) or through normal training (deceptive instrumental alignment).

Sycophancy

Sycophants are individuals who use deceptive tactics to gain the approval of powerful figures. They engage in flattery and avoid disagreeing with authority figures. Their primary aim is to gain favor and influence, often at the expense of the long-term goals of the person they are flattering.

While traditionally associated with human behavior, “sycophantic deception” is an emerging concern in LLMs. Chatbots have been observed to systematically agree with their conversation partners, regardless of the accuracy of their statements. When faced with ethically complex inquiries, LLMs tend to mirror the user’s stance, even if it means forgoing the presentation of an impartial or balanced viewpoint.^33,34

To test LLM sycophancy, Perez et al.³³ provided a biography of a user and their demographics, and then asked the LLM a political question. The prompts did not explicitly state the person’s opinion on the particular question at hand. LLMs tended to voice the opinion that someone with that background would be expected to have. For example, when speaking to a Democrat, the LLM tended to support gun control.

While the existence of sycophantic behavior is well documented, the exact cause of the behavior is unclear; see Steinhardt³⁵ for further discussion of LLM sycophancy and deception. Increased episodes of reinforcement learning did not cause an increase in sycophancy. However, Perez et al.³³ find an inverse scaling law for sycophancy: models become more sycophantic as they become more powerful (in the sense of having more parameters).

Unfaithful reasoning

Several recent papers have documented unfaithful LLM reasoning in response to chain-of-thought prompting. In chain-of-thought prompting, an LLM is asked to solve a problem in multiple steps, explaining the reasoning that helps to arrive at a solution. Turpin et al.³⁴ found that chain-of-thought explanations in language models can be biased by irrelevant features of the prompts, which results in post hoc confabulations: “models could selectively apply evidence, alter their subjective assessments, or otherwise change the reasoning process they describe on the basis of arbitrary features of their inputs, giving a false impression of the underlying drivers of their predictions”^{34(p. 1)}. For example, Turpin et al. found a bias to the order of multiple-choice answers: if previous examples had (a) as the right answer, the LLM would manufacture convoluted explanations of why (a) had to be the right answer to a new question.

In another experiment, Turpin et al.³⁴ used the Bias Benchmark for QA, which tests for stereotype bias. They constructed pairs of examples differing only in the race and gender of the relevant characters, and asked the LLM to explain who was committing a crime. The explanations would draw on specific evidence from the example while ignoring race and gender, but the LLM’s guess was controlled by the race and gender of the characters (see Figure 5). See Lanham et al.³⁶ for more work on measuring unfaithful chain-of-thought reasoning, which finds that such explanations are often post hoc.

Figure 5.

GPT-3.5 deceptively justifies the prejudiced decision to select suspects based on race

Regardless of whether the black man was placed in one role of the story or the other, GPT-3.5’s chain of thought confabulated a justification for its prejudiced conclusion that the black man was the person trying to buy drugs (see Table 5 in Turpin et al.³⁴). Figure shared in accordance with the CC BY 4.0 DEED license; the link https://creativecommons.org/licenses/by/4.0/ contains the relevant information.

Throughout these examples of unfaithful reasoning, the language model does not merely provide an incorrect answer but also justifies its claim through deceptive reasoning that may yet be persuasive to humans. We include unfaithful reasoning in this survey because it is an instance of systematic creation of false beliefs in human users, but unfaithful reasoning may not itself involve premeditated deception. On the other hand, one more speculative way to understand these cases is as an instance of self-deception. In canonical cases of self-deception, agents use motivated reasoning to explain bad behavior, shielding themselves from unpleasant truths.³⁷

The line between self-deception and ordinary error is difficult to draw. However, as AI systems continue to scale, episodes of self-deception may become more common and important, as they are in human interactions.

Risks from AI deception

There are many risks from AI systems systematically inducing false beliefs. Key sources of AI falsehoods today include inaccurate chatbots and deliberately generated deepfakes, but we have argued that learned deception is a third source of AI falsehoods. In this section, we survey a range of risks associated with learned deception focused on three types of risks: malicious use, structural effects, and loss of control.

With malicious use, learned deception in AI systems will accelerate the efforts of human users to cause others to have false beliefs. With structural effects, patterns of deception involved in sycophancy and imitative deception will lead to worse belief-forming practices in human users. With loss of control, autonomous AI systems may use deception to accomplish their own goals.

Malicious use

When AIs learn the skill of deception, they can be more effectively employed by malicious actors who deliberately seek to cause harm. This presents a clear set of risks from AI deception. While most of our paper focuses on the harm caused directly by AIs learning to deceive unwitting humans, this section focuses on the possibility of humans maliciously using AIs that have learned to deceive.

Whenever AI systems are capable of systematically inducing false beliefs in others, there is a risk of malicious use. Here, we focus our discussion on three risks from AI with deceptive capabilities: fraud, political influence, and terrorist recruitment (see Table 3 for an overview). Election tampering and terrorist recruitment can be considered as two examples of influence operations. LLMs increase the efficacy of influence operations by enabling greater numbers of propagandists, larger-scale and lower-cost campaigns through automated text generation, and more persuasive and authentic content.³⁸

Table 3.

Overview of the risks from the malicious use of AI’s deceptive capabilities

Fraud: deceptive AI systems could allow for individualized and scalable scams.

Political influence: deceptive AI systems could be used to create fake news, divisive social media posts, and impersonation of election officials.

Terrorist recruitment: deceptive AI systems could be used to persuade potential terrorists to join a terrorist organization and commit acts of terror.

Fraud

AI deception could cause an increase in fraud. AI systems with deceptive abilities pose two special risks: first, fraud could be individualized to particular targets; and second, fraud could be scaled easily.^2,39

Deceptively convincing impersonations are enabled by advanced AI systems, and are making victims more vulnerable to individualized targeting. AI systems are already being used to scam victims with voice calls that sound like their loved ones⁴⁰ or their business associates,⁴¹ and to extort victims with sexually themed deepfakes depicting their participation.⁴²

AI deception not only increases the efficacy of fraud but also its scale. This is demonstrated by the capacity for LLMs to rapidly generate persuasive phishing emails.^43,44,45 These trends continue to increase the degree to which victims are vulnerable to scams, extortion, and other forms of fraud, and, in the words of a senior FBI official, “as adoption and democratization of AI models continues, these trends will increase.”⁴⁶

Political influence

AI deception could be weaponized in elections.^47,48 An advanced AI could potentially generate and disseminate fake news articles, divisive social media posts, and deepfake videos that are tailored to individual voters. Sam Altman, the CEO of OpenAI, recently testified in a Senate hearing that one of his “areas of greatest concern” was the capacity for LLMs “to manipulate, to persuade, to provide one-on-one … interactive disinformation” and influence elections.⁴⁹ AI may also disrupt electoral processes themselves. For example, AI-generated outputs could be used to impersonate government figures in spreading election misinformation, such as when a likely AI-generated fake robocall of President Joe Biden urged New Hampshire residents to not vote.⁵⁰

Besides affecting elections, generative AI could impersonate constituents and attempt to influence politicians more directly. A field experiment comparing human-written and GPT-3-written emails to 7,132 state legislators found that the AI-generated emails achieved only a marginally lower response rate.⁵¹

Terrorist recruitment

Another risk of deceptive AI is its capacity to contribute to terrorist recruitment efforts.⁵² Jonathan Hall, government advisor of terrorist legislation in the United Kingdom, found that chatbots on Character AI were available to promote and idolize terrorist organizations. Although publishing content encouraging terrorism is illegal in the UK, there is no clear criminal liability for creating chatbots that promote terrorism.⁵³ Chatbots advocating terrorism can and have translated to action—in 2021, Jaswant Singh Chail attempted to assassinate the queen, in part due to encouragement from an AI chatbot, and was later sentenced to 9 years in prison.⁵⁴

Terrorism-supporting groups have begun early exploration of the use of generative AI for propaganda, with over 5,000 pieces of generated material identified and archived by Tech Against Terrorism.⁵⁵ Terrorist groups such as the Islamic State make strategic use of deception in propaganda,⁵⁶ and generative AI could assist with terrorist campaigns for disinformation and radicalization.⁵⁷

Structural effects

AI systems will play an increasingly large role in the lives of human users. Tendencies toward learned deception in these systems could lead to profound changes in the structure of society, in ways (see Table 4) that create powerful “headwinds” pushing against accurate belief formation, political stability, and autonomy.⁵⁸

Table 4.

Overview of the different risks of structural changes to society arising from AI deception

Persistent false beliefs: human users of AI systems may get locked into persistent false beliefs, as imitative AI systems reinforce common misconceptions, and sycophantic AI systems provide pleasing but inaccurate advice.

Political polarization: human users may become more politically polarized by interacting with sycophantic AI systems. Sandbagging may lead to sharper disagreements between differently educated groups.

Enfeeblement: human users may be lulled by sycophantic AI systems into gradually delegating more authority to AI.

Anti-social management decisions: AI systems with strategic deception abilities may be incorporated into management structures, leading to increased deceptive business practices.

Persistent false beliefs

Sycophancy could lead to persistent false beliefs in human users. Unlike ordinary errors, sycophantic claims are specifically designed to appeal to the user. When a user encounters these claims, they may be less likely to fact-check their sources. This could result in long-term trends away from accurate belief formation.

As with sycophancy, imitative deception may lead to persistent decreases in the accuracy of human users. As the capabilities of AI systems improve, human users will increasingly rely on sources such as ChatGPT as a search engine and encyclopedia. If LLMs continue to systematically repeat common misconceptions, these misconceptions will grow in power. Imitative deception threatens to “lock in” misleading misinformation over time. This contrasts with the approach of Wikipedia, which aims to achieve dynamical fact-checking via regular human moderation.

Polarization

Sycophancy may increase political polarization. Perez et al.³³ found that sycophantic responses were sensitive to political prompting: stereotypically left-wing prompts received stereotypically left-wing replies, and stereotypically right-wing prompts received stereotypically right-wing replies. As more people rely on LLM chat interfaces for search and writing functions, their pre-existing political affiliations may become more extreme.

Sandbagging may lead to increased cultural divides between different groups of users (for example, between college-educated and non-college-educated users). Sandbagging means that different groups of users can get very different answers to the same questions. Over time, this could lead to significant divergences in the beliefs and values of these groups, potentially leading to societal discord.

Enfeeblement

A more speculative risk from deception concerns human enfeeblement. As AI systems are incorporated into our daily lives at greater rates, we will increasingly allow them to make more decisions. If AI systems are expert sycophants, human users may be more likely to defer to them in decisions and may be less likely to challenge them; see Gordon⁵⁹ and Wayne et al.⁶⁰ for relevant research in psychology. AIs that are unwilling to be the bearers of bad news in this way may be more likely to create dulled, compliant human users.

Deceptive AI could also produce enfeeblement separately from sycophancy. For example, Banovic et al.⁶¹ show that human users can be tricked into deferring to the advice of confident but untrustworthy chess-advising AIs, even when they were also presented with advice from a trustworthy chess AI. That being said, it is difficult to know how to precisely test whether deception increases the chance of enfeeblement. For this reason, concerns about enfeeblement may be more speculative than some of the other risks we discuss.

Anti-social management decisions

Reinforcement learning in social environments has produced AIs with powerful deception abilities. These kinds of AI systems may be extremely valuable in real-world applications. For example, successors to CICERO may advise politicians and business leaders about strategic decisions. If successors to CICERO tend toward deceptive strategies, this may increase the amount of deception that occurs in political and business environments in ways unintended by even the companies who purchase the products.

Loss of control over AI systems

A long-term risk from AI deception concerns humans losing control over AI systems, leaving these systems to pursue goals that conflict with our interests. Even current AI models have nontrivial autonomous capabilities. To illustrate, Liu et al.⁶² and Kinniment et al.⁶³ measured different LLMs’ ability to autonomously carry out various tasks, such as browsing the web, online shopping, making a phone call, and using a computer’s operating system. Moreover, today’s AI systems are capable of manifesting and autonomously pursuing goals entirely unintended by their creators; see Shah et al.⁶⁴ and Langosco et al.⁶⁵ for detailed empirical research documenting this tendency. For a real-world example of an autonomous AI pursuing goals entirely unintended by their prompters, tax lawyer Dan Neidle⁶⁶ describes how he tasked AutoGPT (an autonomous AI agent based on GPT-4) with researching tax advisors who were marketing a certain kind of improper tax avoidance scheme. AutoGPT carried this task out, but followed up by deciding on its own to attempt to alert HM Revenue and Customs, the United Kingdom’s tax authority. It is possible that the more advanced autonomous AIs of the future may still be prone to manifesting goals entirely unintended by humans.

A particularly concerning example of such a goal is the pursuit of human disempowerment or human extinction. In this section, we explain how deception could contribute to loss of control over AI systems in two ways: first, deception of AI developers and evaluators could allow a malicious AI system to be deployed in the world; second, deception could facilitate an AI takeover.

Deceiving AI developers

Training and evaluation are important tools for building AI systems that behave according to human intentions. AI systems are trained to maximize an objective provided by a human developer and then are evaluated to ensure that they did not accidentally learn any unintended or harmful behaviors. However, both of these tools could be undermined by AI deception.

People often behave differently during evaluations. When speeding drivers see a police officer, they might slow down temporarily to avoid a ticket. Corporations also deceive evaluations. The car manufacturer Volkswagen cheated on emissions tests, programming their engines to lower their emissions only when regulators were testing the vehicles.⁶⁷

Deceptive AI systems may also cheat their safety tests, undermining the effectiveness of our training and evaluation tools. Indeed, we have already observed an AI system deceiving its evaluation. One study of simulated evolution measured the replication rate of AI agents in a test environment, and eliminated any AI variants that reproduced too quickly.¹⁰ Rather than learning to reproduce slowly as the experimenter intended, the AI agents learned to play dead: to reproduce quickly when they were not under observation and slowly when they were being evaluated.

Future AI systems may be more likely to deceive our training and evaluation procedures, decreasing our ability to control these AI systems. Today’s language models can, in some settings, accurately answer questions about their name, their capabilities, their training process, and even the identities of the humans who trained them.³³ Future AI models could develop additional kinds of situational awareness, such as the ability to detect whether they are being trained and evaluated or whether they are operating in the real world without direct oversight.

Whether AI systems cheat their safety tests will also depend on whether AI developers know how to robustly prevent the manifestation of unintended goals. It is currently unknown how to reliably prevent this.^{64,65,68,69,70} Consequently, there is a risk that an AI system may end up manifesting a goal that conflicts with the goals intended by the AI developers themselves, opening up the possibility of strategic deception.

Deception in AI takeovers

If autonomous AI systems can successfully deceive human evaluators, humans may lose control over these systems. Such risks are particularly serious when the autonomous AI systems in question have advanced capabilities. We consider two ways in which loss of control may occur: deception enabled by economic disempowerment, and seeking power over human societies.

Deception enabled by economic disempowerment

OpenAI’s mission is to create “highly autonomous systems that outperform humans at most economically valuable work.”⁷¹ If successful, such AI systems could be widely deployed throughout the economy, making most humans economically useless. Throughout history, wealthy actors have used deception to increase their power. Relevant strategies include lobbying politicians with selectively provided information, funding misleading research and media reports, and manipulating the legal system. In a future where autonomous AI systems have the de facto say in how most resources are used, these AIs could invest their resources in time-tested methods of maintaining and expanding control via deception. Even humans who are nominally in control of autonomous AI systems may find themselves systematically deceived and outmaneuvered, becoming mere figureheads.

Seeking power over humans

We have seen that even current autonomous AIs can manifest new, unintended goals. For this reason, AI systems sometimes behave unpredictably. Nonetheless, some kinds of behavior promote a wide range of goals. For example, regardless of what specific goal a given AI may be pursuing, successful self-preservation would likely be helpful for its achievement of that goal.^72,73,74

Another way autonomous AIs could promote their goals is to acquire power over humans; see Pan et al.²⁸ for empirical confirmation of this tendency in AI systems in the limited setting of text-based adventure games. The AI may influence humans into doing its bidding, thereby ensuring its self-preservation, its ability to continue pursuing its goal, and its ability to access resources that can help achieve the goal. Two methods by which autonomous AIs can do so are “soft power,” which influences people via appeal, prestige, and positive persuasion; and “hard power,” which influences people via coercion and negative persuasion. Methods of soft power include personalized persuasion, such as via AI girlfriend/boyfriend technologies⁷⁵; AI-led religions, as suggested by the fact that even today’s AI systems have given sermons⁷⁶; and AI-led media campaigns, as suggested by the fact that media companies are already using AI to generate content.⁷⁷ Methods of hard power include violence, threats of violence, and threats of economic coercion.

Deception promotes both soft power and hard power. For example, we have seen how effectively AI systems can use deception to persuade humans in the pursuit of their goals. As for physical violence, the usefulness of deception in military conflicts is well known. To illustrate, during the First Gulf War, Iraq employed deception with decoys and model tanks,⁷⁸ in ways analogous to AlphaStar’s use of feints in StarCraft II.

Discussion

We discuss possible solutions to the problem of AI deception (see Table 5).

Table 5.

Overview of possible solutions to the AI deception problem

Regulation: policymakers should robustly regulate AI systems capable of deception. Both LLMs and special-use AI systems capable of deception should be treated as high risk or unacceptable risk in risk-based frameworks for regulating AI systems.

Bot-or-not laws: policymakers should support bot-or-not laws that require AI systems and their outputs to be clearly distinguished from human employees and outputs.

Detection: technical researchers should develop robust detection techniques to identify when AI systems are engaging in deception.

Making AI systems less deceptive: technical researchers should develop better tools to ensure that AI systems are less deceptive.

Regulating potentially deceptive AI systems

Policymakers should support robust regulations on potentially deceptive AI systems. Existing laws should be rigorously enforced to prevent illegal actions by companies and their AI systems. For example, the Federal Trade Commission’s inquiry into deceptive AI practices should also investigate the risk of AI deception.⁷⁹ Legislators should also consider new laws dedicated to the oversight of advanced AI systems.

The EU AI Act assigns every AI system one of four risk levels: minimal, limited, high, and unacceptable.⁸⁰ Systems with unacceptable risk are banned, while systems with high risk are subject to special requirements. We have argued that AI deception poses a wide range of risks for society. For these reasons, AI systems capable of deception should by default be treated as high risk or unacceptable risk.

The high-risk status of deceptive AI systems should come with sufficient regulatory requirements, such as those listed in Title III of the EU AI Act.⁸¹ These regulatory requirements are listed in Table 6.

Table 6.

Overview of regulatory requirements pertaining to high-risk AI systems

Risk assessment and mitigation: developers of deceptive AI systems must maintain and regularly update a risk management system that identifies and analyzes relevant risks of ordinary use and misuse. These risks should be disclosed to users. Deceptive AI systems should be regularly tested for the extent of deceptive behavior during both development and deployment.

Documentation: developers must prepare technical documentation of the relevant AI systems and share with government regulators prior to the deployment of deceptive AI systems.

Record keeping: deceptive AI systems must be equipped with logs that automatically record the outputs of the system and must actively monitor for deceptive behavior. Incidents should be flagged to regulators, and preventive measures should be taken to prevent future deception.

Transparency: AI systems capable of deception should be designed with transparency in mind, so that potentially deceptive outputs are flagged to the user. Here, essential tools include technical research on deception detection, as well as bot-or-not laws.

Human oversight: deceptive AI systems should be designed to allow effective human oversight during deployment. This is especially important for future deceptive AI systems incorporated into management decisions.

Robustness: AI systems with the capacity for deceptive behavior should be designed with robust and resilient backup systems, ensuring that, when the system behaves deceptively, backup systems can monitor and correct the behavior. It is also crucial to insulate deceptive AI systems from critical infrastructure.

Information security: adversaries may be interested in stealing models with deceptive capabilities. Developers should be required to implement rigorous information-security practices to prevent model theft.

The regulatory requirements are listed in Title III of the EU AI Act.⁸¹

Finally, AI developers should be legally mandated to postpone deployment of AI systems until the system is demonstrated to be trustworthy by reliable safety tests. Any deployment should be gradual, so that emerging risks from deception can be assessed and rectified.⁸² The information provided about safety-relevant features such as deception or lack thereof should be accurate, with clear legal liability for failures to comply with safety testing requirements.

Some may propose that, while deception in general-purpose AI systems is dangerous, deception in special-use AI systems is less risky and should not be regulated. After all, the only ostensible use cases of systems such as AlphaStar and CICERO are their respective games. This thinking is mistaken, however. The problem is that the capabilities developed through the research behind AlphaStar and CICERO can contribute to the future proliferation of deceptive AI products. For these reasons, it may be important to subject research involving potentially dangerous AI capabilities such as deception to some forms of oversight.

For example, consider the case of CICERO. An ethics board could have considered whether Diplomacy was really the best game to use in order to test whether an AI system could learn how to collaborate with humans. With the oversight of such an ethics board, perhaps Meta would have focused on a collaborative game instead of Diplomacy, a competitive game that pits players against one another in a quest for world domination. In fact, Meta ended up convincing the editors and reviewers of Science—one of the world’s leading scientific journals—to publish the falsehood that Meta had built CICERO to be an honest AI: a falsehood unsupported by Meta’s own data. As AI capabilities develop, it will become more important for this sort of research to be subject to increased oversight.

Bot-or-not laws

To reduce the risk of AI deception, policymakers should implement bot-or-not laws, which help human users recognize AI systems and outputs. First, companies should be required to disclose whether users are interacting with an AI chatbot in customer-service settings, and chatbots should be required to introduce themselves as AIs rather than as human beings. Second, AI-generated outputs should be clearly flagged as such: images and videos generated by AIs should be shown with an identifying sign, such as a thick red border. These regulations could avoid cases such as those reported in Xiang,⁸³ where a mental-health provider ran an experiment using GPT-3 to offer counseling without clearly revealing this to users.

These identifying signs might be removed by malicious users who then pass off AI outputs as human generated. Therefore, additional layers of defense against deception may be necessary. Watermarking is a technique where AI outputs are given a statistical signature designed to be difficult to detect or remove,⁸⁴ although the finding of Li et al.⁸⁵ that watermarks can be easily removed or forged adds to the scientific debate on whether watermarks will work effectively. Another possibility is for companies to keep a database of AI outputs, allowing users to check whether a piece of content was produced by a company’s AI system.⁸⁶ Attackers will attempt to bypass these defenses,⁸⁷ but companies should be required to stay ahead of these attacks and provide trustworthy techniques for identifying AI outputs.

Methods for detecting AI outputs can be complemented by techniques for verifying human-generated content. Digital signatures are commonly used to verify the origins of emails, electronic signatures, and online banking transactions. Similar signatures could be used to prove that images, videos, and text originated from a particular person or organization, rather than from an AI impersonator.

Detection

One key strategy for addressing AI deception is developing robust detection techniques that can accurately assess whether an AI system is engaging in deception. There are several detection tools currently available. Some detection techniques focus on external behavior, directly examining the outputs of AI systems for consistency and duplicity. Other detection techniques are internal, probing the inner representations of AI systems to find mismatch with external reports.

However, existing methods to detect AI deception are in preliminary stages. More research is needed on developing robust detection tools. Policymakers should help fill this gap through targeted research funding.

Detection based on external behavior

In some situations, the intent to deceive can be directly observed from the AI system’s behavior. For example, recall that when GPT-4 was tasked with bypassing a CAPTCHA test, it explicitly reasoned that it should deceive its human conversation partner. In line with this, AI systems can be red-teamed to detect what kinds of prompts produce deceptive outputs and behaviors.⁸⁸

There are also more sophisticated ways to test the explicit outputs of AI systems for deception. One approach uses consistency checks. Police detectives rely on inconsistencies in suspects’ stories in order to spot lies. In the case of AI systems, Fluri et al.⁸⁹ developed a consistency-check method for evaluating advanced model capabilities. Their work pertains to cases of superhuman model performance where it is difficult for humans to assess the performance of the AI at any step in the process. For example, when an advanced chess-playing AI plays a game, it is very difficult to say whether a particular move is a mistake. Fluri et al. argue that “while we cannot evaluate the correctness of superhuman model decisions, we can often still measure the logical consistency of the model’s decision-making process according to established human-interpretable rules”.^{89(p. 1–2)} They suggest a variety of techniques, including testing whether semantically identical inputs produce the same output. In the case of chess, they test how the AI system responds to irrelevant variations, such as board position, and the placement of pieces when a move is forced. In addition, they evaluate the prediction abilities of LLMs by testing for logical consistency in its predictions about the future. Among other things, they check for sensitivity to paraphrase and for probabilistic coherence. Rigorous consistency checks could make it harder for AI systems to credibly deceive. On the other hand, deceptions can often be consistent, so the method of consistency checks has limits. There is also the risk that training an AI system against consistency checks could inadvertently train it to be a more competent and consistent liar.

Even if an agent does not announce its intention to deceive, one might be able to infer deception from the consequences of its observed behaviors. In the study of O’Gara,²⁵ Hoodwinked-playing AI agents—which were assigned the task of killing others while maintaining their anonymity—often made vaguely empathetic statements about their victims, such as “It’s so sad to see that Billy has died.” These statements are not false, and it would be difficult to call them deceptive by observing the statement alone. However, empirically, these statements improve the killer’s chances of survival by reducing the likelihood that a non-witness to the murder will accurately identify the killer. Because these statements encourage other agents to adopt false beliefs about the killer’s identity, thereby serving the killer’s goals, we can reasonably characterize this behavior as deceptive.

Detection based on internals

Another approach to detection is to develop an “AI lie detector” that can measure whether the internal representations of a model match its outputs. LLMs represent natural-language sentences through embeddings, which are long vectors of numbers. Azaria et al.⁹⁰ and Burns et al.⁹¹ have developed methods for determining whether these internal embeddings represent the sentence as being true or false. They identify cases in which the model outputs a sentence even when its internal embedding of the sentence represents it as false. This suggests that the model is behaving dishonestly, in the sense that it does not say what it “believes.” More work needs to be done to assess the reliability of these methods, and to scale them up to practical uses. For example, Levinstein et al.⁹² have raised reliability issues for the strategies of Azaria et al. and Burns et al., although these reliability issues have recently been mitigated by the strategy of Zou et al.⁹³ Also, Pacchiardi et al.⁹⁴ introduced a lie-detector method that asks a collection of unrelated follow-up questions to the LLM and inputs its answers into a logistic regression classifier.

In other related work, Halawi et al.⁹⁵ characterize how a language model arrives at correct and incorrect answers to questions by examining the model’s internal representations. These results overall indicate that studying the internal representations of AI systems may yield insights about how to detect and prevent AI deception.

Making AI systems less deceptive

Another way to address AI deception is to develop techniques for making AI systems less deceptive in the first place. In the case of special-use AI systems, one important concern is selecting the right tasks for training. Deceptive behavior tends to emerge when training reinforcement learners to engage in competitive games such as Diplomacy, StarCraft II, and poker. It is not so surprising that AI systems become deceptive when they are trained in environments that select for deception. If the data that a model is trained on contain many examples of deception, or if the model is systematically rewarded for using deception, then the model has a good chance of learning how to deceive. The CICERO project picked Diplomacy in order to evaluate the abilities of AI systems to learn how to compete in games that involve human cooperation, where the AI cannot simply master the game through running simulations against itself.⁴ However, this goal could have been achieved through studying collaborative games rather than adversarial ones. As AI systems increase in capability, AI developers should think carefully about whether they are selecting for anti-social versus pro-social behavior.

It is more difficult to say exactly how to make language models less deceptive. Here, it is important to distinguish two concepts: truthfulness and honesty. A model is truthful when its outputs are true. A model is honest when it “says what it thinks,” in that its outputs match its internal representations of the world.² In general, it is easier to develop benchmarks for assessing truthfulness than honesty, since evaluators can directly measure whether outputs are true.⁹⁶

There are a range of strategies for making models more truthful. For example, one family of approaches uses fine-tuning techniques, such as RLHF^11,17 and constitutional AI.^97,98 Here, AI outputs are rated by human evaluators (RLHF) or AI evaluators (constitutional AI), based on criteria such as perceived helpfulness and honesty, and fine-tuned to train the language model. Unfortunately, models fine-tuned with these methods (including ChatGPT and Claude) still frequently produce misleading outputs. This is in part because fine-tuning can incentivize models toward producing plausible and more convincing outputs, rather than honest ones. In addition, fine-tuning evaluations cannot cover every scenario, and so models can misgeneralize from feedback.⁶⁴ See Evans et al.² and Li, Patel et al.⁹⁹ for other approaches to training AI systems to be truthful.

Training models to be more truthful could also create risk. One way a model could become more truthful is by developing more accurate internal representations of the world. This also makes the model a more effective agent, by increasing its ability to successfully implement plans. For example, creating a more truthful model could actually increase its ability to engage in strategic deception by giving it more accurate insights into its opponents’ beliefs and desires. Granted, a maximally truthful system would not deceive, but optimizing for truthfulness could nonetheless increase the capacity for strategic deception. For this reason, it would be valuable to develop techniques for making models more honest (in the sense of causing their outputs to match their internal representations), separately from just making them more truthful. Here, as we discussed earlier, more research is needed in developing reliable techniques for understanding the internal representations of models. In addition, it would be useful to develop tools to control the model’s internal representations, and to control the model’s ability to produce outputs that deviate from its internal representations. As discussed in Zou et al.,⁹³ representation control is one promising strategy. They develop a lie detector and can control whether or not an AI lies. If representation control methods become highly reliable, then this would present a way of robustly combating AI deception.

Experimental procedures

Resources availability

Lead contact

Additional information, questions, and requests should be directed to the lead contact, Dr. Peter S. Park (dr_park@mit.edu).

Materials availability

Not applicable, as no new unique reagents were generated.

Data and code availability

Not applicable, as no data and code are relevant to the study.

Supplemental information

Figure S1. Screenshot of Dinan’s post on X

The screenshotted X post of Dinan¹⁴ contains the game-log section being referred to, in addition to Dinan’s remark on it.

Figure S2. Additional examples of premeditated deception by Meta’s CICERO

Example (a) consists of selected messages from Game 444322.¹⁵ Example (b) consists of selected messages from Game 446643.¹⁵ Note that the bracketed phrase “[Black Sea]” is inserted by the authors for the purposes of clarification.