Eye tracking uncovered great apes' ability to anticipate that other individuals will act according to false beliefs

ABSTRACT

Using a novel eye-tracking test, we recently showed that great apes anticipate that other individuals will act according to false beliefs. This finding suggests that, like humans, great apes understand others' false beliefs, at least in an implicit way. One key question raised by our study is why apes have passed our tests but not previous ones. In this article, we consider this question by detailing the development of our task. We considered 3 major differences in our task compared with the previous ones. First, we monitored apes' eye movements, and specifically their anticipatory looks, to measure their predictions about how agents will behave. Second, we adapted our design from an anticipatory-looking false belief test originally developed for human infants. Third, we developed novel test scenarios that were specifically designed to capture the attention of our ape participants. We then discuss how each difference may help explain differences in performance on our task and previous ones, and finally propose some directions for future studies.

Theory of mind is the ability to infer unobservable mental states in other individuals, like desires and beliefs;¹ it is a key component of human social intelligence. False-belief understanding, the ability to attribute mental states that conflict with reality, is of particular interest because it requires an understanding that others' behavior is not driven by reality, but by their beliefs about reality, even when those beliefs happen to be false. Over the past few decades, researchers have been attempting to determine whether nonhuman animals also possess a theory of mind. Great apes, monkeys, and corvids have passed several important tests used as markers of theory of mind in human children, suggesting that they may be able to reason about other individuals' perception, knowledge, goals and intentions (for a review, see ref. 2 for great apes, ref. 3 for monkeys, and ref. 4 for corvids). However, extensive debate continues over what mechanisms underlie performance in such tests, and whether or not animals' success relies specifically on the capacity to infer others' mental states.^5,6 One aspect that has prevented clarity is the lack of evidence, despite repeated investigation,^7-11 that nonhuman animals might understand others' false beliefs.^2,3

Using a novel eye-tracking test originally developed to test human infants, we recently showed that great apes anticipate that other individuals will act according to false beliefs.¹³ This finding suggests that, like humans, great apes understand false belief at least in an implicit way. One of the key remaining questions is why apes passed our tests but not previous ones. Although there are a variety of outstanding theoretical considerations, one thing is clear: our study represented a substantial methodological departure from existing paradigms used with apes. In this article, we primarily focus on these methodological aspects, although we also briefly touch on certain theoretical considerations. More specifically, we describe how we developed our test focusing on 3 key milestones: 1) developing and testing a technique to record apes' anticipatory looks, 2) choosing a task that Southgate, Senju, and Csibra⁸ had pioneered to test children, and 3) adapting it by crafting scenarios that would capture apes' sustained attention and deliver the key manipulations required to assess their false-belief understanding. Next, we discuss the possible reasons why this task provided evidence of (implicit) false belief understanding while no other tasks had done so to date. We close the article by suggesting some directions for future research.

Three key milestones

We begin by summarizing our study. Apes viewed short movies on a monitor while their gaze was automatically and non-invasively recorded using an eye-tracker. In total, 41 participants, including bonobos, chimpanzees, and orangutans, participated in our study. Our design, controls, and general procedure replicated a seminal anticipatory-looking false belief study with human infants.¹² There were 2 conditions, false-belief 1 and 2. The movie consisted of 3 phases; familiarization, belief-induction, and test. In both conditions, during a pair of familiarization trials, a human actor saw an object be hidden in one of 2 containers and then searched for it there. In the second familiarization trial, the object was hidden, and pursued, in the other container. In the false-belief 1 condition, during the belief-induction phase, the actor saw the initial hiding of the object, then saw the object be moved to the other container. The actor then saw the other container being touched (as a low-level control; see below). Finally, the actor left the scene, and while absent, the object was removed from the container (and the scene). During the test phase, the actor (now having a false belief about the object location) returned to search for the object by approaching the middle of the 2 empty containers. The false-belief 2 condition was the same as false-belief 1 except that, during the belief-induction phase, the actor only witnessed the initial hiding of the object and was away during its relocation to the second container and subsequent removal. The differences between these 2 conditions, in addition to the actor's additional action, ensured that apes could not predict the actor's search based on simple rules, such as looking to the first or last location where the object was, or the last location the actor attended.

We tested apes using 2 different scenarios (Fig. 1C–D). In experiment 1, a human actor chased an ape-costumed character, and the ape-costumed character, serving the role of target object, hid in one of 2 haystacks. In experiment 2, a human actor and an ape-costumed character competed for a stone, and the ape-costumed character hid the stone in one of 2 boxes. When the actor was approaching (or reaching to) the middle of the 2 containers, we recorded each ape's first look to either the correct or incorrect container (relative to where the actor believed the object to be) and measured looking time to each container. Across 2 conditions and 2 experiments, we confirmed that apes looked in anticipation of the actor searching in the location where he had last seen the object, and therefore falsely believed it to be.

Figure 1.

(A) Eye-tracking of an orangutan (B) and a chimpanzee. (C) One scenario used in the false-belief test (experiment 1; FB2 condition). The human actor chased the ape-character with a large stick. The ape character hid in one of the 2 haystacks and then, while the human actor was away, moved from there. (D) The other scenario used in the test (experiment 2; FB2 condition). The human actor and the ape-character competed for an object. The ape character hid the object in one of 2 boxes and then, while the human actor was away, moved it from there while the human actor was away. The red arrows indicate the container where the human actor falsely believed the object to be.

Our study relied on eye-tracking technology to measure apes' eye movements. This technology has been used in neurophysiological research with monkeys for several decades.¹⁴ However, only recently it became available for noninvasively testing human infants and great apes.¹⁶ Traditionally, eye-tracking required a participant to be firmly fixed during recording, which was physically and ethically not possible for those populations. Recent advances in infrared eye-tracking technology allowed participants to move their heads relatively freely as long as they continued to face the eye-tracker and monitor. To keep apes roughly in an appropriate position, we let them sip a steady drip of juice through a custom-made juice dispenser during recording (irrespective of their performance).

Eye-tracking allows researchers to examine where and when participants attend to stimuli. It can identify the focus of an individual's attention with great precision. Most importantly, it can inform the researcher whether the participant is anticipating the actor's actions by looking at an object or location just before the actor physically acts on it. Thus, anticipatory looking occurs before the anticipated event actually happens. It captures a prediction based on the memory for past events and the understanding about ongoing actions, rather than a response to particular outcomes.

In preparation for our false-belief study, we first ensured that great apes make anticipatory looks to an actor's actions. First, Kano and Tomonaga¹⁷ and Myowa-Yamakoshi, Scola, Hirata¹⁸ found that, when watching an actor reach for an object, apes look at the object typically a fraction of a second before the actor grabs it, like humans,^15,19 and monkeys.²⁰ Second, Kano and Call²¹ found that apes' anticipatory looking is not simply based on encoding of the movement patterns of an actor's reaching, but rather reflects an understanding of the actor's goals, as in humans.²² Specifically, in that study, apes first saw an actor repeatedly reach for and grab the same object among 2 alternatives. Then they saw that the locations of the 2 objects were switched. When the actor reached to the middle of the 2 objects, human infants and great apes anticipatorily looked at the object that they had previously seen the actor grab. In a control test, involving an unfamiliar claw instead of a human hand, apes did not show goal-based prediction.

Third, Kano and Hirata²³ confirmed that apes can anticipate previously-seen events based on long-term memory. While their gaze was being monitored, apes watched a movie in which an unfamiliar ape-costumed character came out from one of 2 identical tunnels. Twenty-four hours later, apes saw the same movie again. They looked in anticipation at the tunnel where the ape-character emerged last time, just before the character actually appeared. In experiment 2, apes watched the ape-character attack a human actor. The human actor then reached for one of 2 different objects and used it to attack the ape-character. Twenty-four hours later, apes saw the same movie except one change; to examine their memory for the object rather than for the location in this experiment, the location of the 2 objects was switched. As the actor was reaching toward the middle of the 2 objects, apes looked in anticipation at the object that the actor had retrieved the day before, even though it was in the opposite location—a result that replicated Kano & Call's¹⁷ findings.

These studies thus demonstrated that apes reliably make anticipatory looks to an actor's goal-directed actions, and that their memory to elicit anticipatory looks is robust and long-lasting. We then shifted our attention to the question of false-belief understanding, and specifically attempted to replicate one of the seminal developmental studies that used a non-verbal looking measure to investigate human infants' false-belief understanding. There were 2 options available in nonverbal looking approaches: violation-of-expectation tests like the one developed by Onishi and Baillargeon, and anticipatory-looking tests like the one developed by Southgate, Senju, and Csibra.¹² The violation-of-expectation tests record for how long participants attend to certain events, some expected and some unexpected, and assess whether participants look at them differentially. Onishi and Baillargeon tested human infants in both true-belief and false-belief conditions. In the true-belief condition, they found that human infants looked for a shorter amount of time when an actor searched for an object in its true location, than when the actor searched for an object in an empty location. In the critical false-belief condition, human infants looked for a shorter amount of time when an actor searched for an object in an empty location where she falsely believed the object to be, than when the actor searched for it in its true location. This pattern is interpreted as an indication that infants expected the actor to act according to her true- and false-belief. Onishi and Baillargeon's²⁴ paradigm was later used to test monkeys.¹⁰ Although monkeys showed the same looking patterns as human infants in the true-belief condition, they did not distinguish between the 2 outcomes in the false-belief condition, suggesting that they attribute true- but not false-beliefs.

We decided to use an anticipatory-looking method for 2 reasons. The first is a pragmatic reason regarding the number of individuals required in each paradigm. In general, anticipatory looking tasks require fewer participants than violation-of-expectation tasks. Although we still required a relatively large number of participants, we managed to recruit a total of 30–40 apes between 2 different research facilities in Japan and Germany.

The second reason is that the anticipatory looking test combines the low task demands of violation-of-expectation paradigms with the high interpretability of active looking paradigms. In fact, Southgate, Senju, and Csibra's¹² tasks can be conceived as a combination of Onishi and Baillageon's violation-of-expectation task and Clements and Perner's anticipatory-looking test. Clements and Perner confronted 2- to 4-year-old children with a version of the classical (verbal) location change test but also measured their anticipatory looks to the 2 potential locations. As expected, children of all ages had a difficulty providing correct verbal answers to false belief questions, but 3- to 4-year-old children (unlike younger ones) reliably made correct anticipatory looks consistent with false belief understanding.

Southgate, Senju, and Csibra incorporated this anticipatory-looking measure into Onishi and Baillageon's test such that all components of the test, including false-belief induction and solicitation of looks, are non-verbal. By doing so, they seemingly reduced the task demands derived from verbal story-telling and verbal questioning. Moreover, it is possible that the removal of the target object from the scene before measuring anticipatory looks contributed to further reducing task demands by eliminating a potential reality bias. In fact, they showed that even 2-year old children could pass their anticipatory-looking false-belief test. Most importantly, by adopting an anticipatory-looking paradigm, Southgate, Senju, and Csibra ruled out some of the low-level alternatives that have been proposed as an explanation for the Onishi and Baillageon's²⁴ results. Specifically, in the anticipatory looking test, the participants cannot solve the test based on the expectation that the actor's ignorance will lead to error or uncertainty because they should specifically anticipate that the actor will search for the object where he falsely believes it to be.

We then attempted to replicate Southgate, Senju, and Csibra's anticipatory-looking test. However, we immediately faced a serious problem. Apes showed little interest in the video stimuli that had been used with human infants. Hirata and Yamamoto (unpublished) presented the same videos that Southgate, Senju, and Csibra had used but they did not sustain apes' attention till the end of the presentation. Kano and Call (unpublished) tested apes by speeding up and shortening the presentation time, but the majority of apes still did not watch till the end of presentation (for the movie, see https://youtu.be/XcK1dVvefGQ). Consequently, in the current study we made rather drastic changes to the movie scenarios such that every scene and action was intended to maximize apes' interest. For this purpose, Kano and Hirata's memory test, whose test movies were specifically designed to create engaging and memorable events for apes, was particularly relevant. Kano and Hirata's movies presented an unfamiliar hostile character (an ape-costumed human) attacking familiar humans in a familiar residential area. This scenario elicited moderate excitement and high levels of engagement among the apes during the entire presentation time. One bonobo female even stopped drinking juice while watching as the social conflict unfolded. Similarly, Kano and Tomonaga²⁶ had also shown that, when watching movies of conspecifics engaging in various social behaviors, apes were most attentive to those depicting social conflicts. Based on this evidence, we incorporated emotional social conflicts into our false-belief scenarios. Yet, there were still some concerns. By introducing emotional events, some of the actions could have become so engaging that apes may have overlooked the critical actions necessary for the test (e.g. hiding of objects, exiting of the actor). We thus used certain cinematographic tricks such as not showing 2 actions simultaneously or introducing appropriate pauses between the actions. The outcome was clear and unambiguous: apes watched throughout the video presentation and attended to all the critical actions depicted in it.

In sum, our study implemented 3 major changes relative to the previous false belief tasks with apes. First, our measure of apes' predictions was based on their anticipatory looking behavior, recorded via noninvasive eye-tracking. Second, we adapted a design by Southgate, Senju, and Csibra,¹² that was developed for human infants. Third, we carefully crafted scenarios in which every component was specifically designed to enhance engagement of our ape participants.

Contrasting results from different paradigms

Let's now return to one of the critical question raised earlier. Why did apes pass our tests but not previous ones? Yet, before considering the theoretical possibilities, we emphasize the same point again that subtle differences in the movie stimuli may make a difference in apes' performance. We believe that the movies should be at least engaging enough, and also easy enough to understand, for ape participants. It should be noted that, in the 2 experiments of our test, we obtained consistent results but not identical effects. The effect was slightly weaker in experiment 1 than in experiment 2, although we obtained unequivocal results when we averaged the results from the 2 experiments. It is thus important to further pin down what movie features can effectively enhance apes' engagement and understanding of contents by accumulating studies in future.

More theoretically, there are 3 possibilities. First, eye-movement measures may reflect a different level of understanding about others' beliefs than more traditional active choice measures. Some have suggested that they indicate an implicit understanding, in contrast to the explicit understanding that is generally inferred from correct responses on verbal and active choice paradigms.²⁵ In humans, it is only after 4 y of age that children pass traditional false-belief tests in which they verbally report their predictions about how a mistaken agent will act (e.g., Where do you think Sally will look for her marble?). However, even children younger than 2 y of age can pass modified false-belief tests involving looking measures. Similar to the results with young human infants, great apes have yet to pass false belief tests in which they must make active choices based on their false belief understanding, even though they were able to correctly predict the actions of a mistaken agent in our eye-tracking paradigm.^12,27 Developmental psychologists continue to debate how to interpret the differences in performance between tasks. In humans, differential performance raises the possibility that the tasks tap into different levels of understanding that emerge, or at least can be detected, at different points within continuous developmental process,²⁸ or that the different tasks measure 2 distinct theory of mind mechanisms.²⁹ These same possibilities exist for apes.

Second, every task imposes certain cognitive demands. The looking-measure paradigms including both the violation-of-expectation and anticipatory-looking tasks were specifically designed to minimize such demands. Compared to active choice paradigms, participants must remember less and for a shorter period of time, and there are fewer inhibitory control demands compared with traditional tasks because participants do not need to translate their understanding into action. Moreover, in the anticipatory-looking task, the object that the actor seeks is removed from the scene (i.e. both containers are empty) before the actor returns to search for it. These design features reflect the evidence that such removal of the target object from the scene enhances human children's performance in verbal false-belief tasks.³⁰ Participants can demonstrate their predictions of the actor's search behavior without potential interference from the existence of a real object in one of the locations.

Third, motivation may differ across tasks. For instance, one of the clearest demonstration of visual perspective taking in chimpanzees was found in a context where chimpanzees needed to compete for foods with dominant conspecifics.³¹ Most other false belief tests since then have relied on similar competitive setting. However, it's possible that in this kind of setting, the presence of food to be competed for may make apes overly excited and less inhibited, making it harder for them to demonstrate their mental-state understanding through their actions. In contrast, the interactions depicted in our videos—third party social conflicts that do not involve food—may have permitted the right balance between high motivation and cognitive control.

Future directions

We see at least 3 important directions for future studies. First, we need to test other non-human species on anticipatory looking false belief tasks, to determine whether the abilities that we have demonstrated extend beyond the great apes. Previous studies have investigated belief understanding in monkeys using violation-of-expectation tests.^10,11 In these studies, monkeys demonstrated understanding of true- but not false-beliefs. This failure of violation-of-expectation task by monkeys may indicate that monkeys, unlike apes, do not understand others' false beliefs even in an implicit way. Yet, we currently lack 2 lines of evidence to support this claim. First, apes have not been tested on a violation-of-expectation task. Second, monkeys (or other species) have not been tested on an anticipatory looking false belief task. Previous studies have demonstrated that monkeys (and corvids) pass several important theory-of-mind tasks.^3,4 In addition, like apes and humans, monkeys anticipatorily look at the target object of an actor's simple reaching action.²⁰ However, it is unclear to what extent these looks may be modulated by others' goals and beliefs.

Second, Heyes³² recently proposed that domain-general mechanisms, or a submentalizing process, can explain our results (in line with her previous argument that such processes are responsible for all positive false belief findings in human infants). Focusing on our experiment 2, she suggested specifically that apes may have encoded, during familiarization, the appearance and disappearance of the green shirt (the main actor), “the configuration of 3 cues (green center/bell rings/boxes flash) that signaled an excitingly novel event (the box taking flight), and a predictor of which box would fly next – the last location of the brick when the scene was green.” In a critical test phase, “reappearance of the green shirt could have acted as a retrieval cue, activating a memory of the brick's location when the green color was last present.” Heyes then suggested that a particularly effective strategy to test the submentalizing hypothesis is to use inanimate controls, removing morphological and movement features that are characteristic of biologic agents (e.g., eyes, autonomous, contingent, goal-directed motions). We are currently working on such a test.

Third, we should tease apart 2 different explanatory frameworks for our results. It is possible that apes solved our task by attributing a false belief to the actor. However, it is also possible that they exploited external cues from the actor and applied a rule, e.g., that actors search for things where they last saw them.³³ In our test, we attempted to minimize the likelihood of the rule-based account by (i) never showing participants the actor's search behavior when he held a false belief, (ii) using novel scenarios about which they could not have learned any rules previously, and (iii) introducing a control (FB1 condition of Experiment 2) such that participants could not succeed by simply expecting the actor to search in the location where he had last attended. Nonetheless, we recognize that our false belief tests are in principle open to some rule-based explanation that participants just expect actors to search for things where they last saw them.

To further examine the alternatives, there are currently 2 candidate tests. The first is the content-change test. In the test, the actor has a false belief about the contents of a single container, rather than about the location of single object (e.g., refs. 34, 35 with human infants). This test is thus not open to the same alternative explanation because the object stays in the same location. The second is called the blindfold test.^5,36 Senju et al.³⁷ conducted the blindfold test with human infants using a modified anticipatory-looking test. In the test, 2 separate groups of human infants first experienced that the blindfold was either opaque or actually see-through. After that, infants saw the movie. In the belief-induction phase, they saw that an actor wore the same blindfold as the one they experienced while the object was relocated. Only human infants who experienced a true opaque blindfold anticipated the actor's action based on her false beliefs, while the other group did not make any anticipation. This test is thus not open to the alternative explanation because participants should use their own self-experiences, rather than relying on external cues, to reason the last thing that the actor saw. Great apes have not been yet tested with the same anticipatory-looking blindfold test. However, a recent study using behavioral measures showed that they could apply their own self-experiences about occlusive properties of different barriers in a visual perspective taking task.³⁸

Finally, our method can be useful to test other cognitive abilities in nonhuman animals such as memory, causal understanding and meta-cognition. In this article, we showed that great apes, like human children, can understand certain contents shown in a movie. That is, they can remember features of the movie across days and anticipate impending events based on this memory. They can also anticipate an agent's actions based on his goals and beliefs. Using movies, one can recreate events and contexts that have traditionally been difficult to simulate in experiments. Eye-tracking also affords novel tools for measuring subtle responses to these events. One of the key messages of this article is that there is enormous potential for investigating the cognitive abilities of nonhuman animals by capitalizing on these novel methodologies as well as interdisciplinary collaboration.

Disclosure of potential conflicts of interest

No potential conflicts of interest were disclosed.

Acknowledgments

We thank the staff at the Kumamoto Sanctuary and Wolfgang Kohler Primate Research Center for their assistance in conducting the series of experiments.

Funding

Financial support came from Ministry of Education, Culture, Sports, Science and Technology (K-CONNEX to FK), Japan Society for Promotion of Science (KAKENHI 26885040, 16K21108 to FK), National Science Foundation (DGE-1106401 to CK), JSPS (KAKENHI 26245069, 24000001 to SH), and European Research Council (Synergy grant 609819 SOMICS to JC).