How to read a picture: Lessons from nonhuman primates

Perhaps the most well-known work by the Belgian surrealist painter René Magritte is a painting entitled “La trahison des images” (The treason of images), from 1929, consisting of a brown smoking pipe suspended upon a uniform background with the caption “Ce n’est pas une pipe” (This is not a pipe). Magritte’s intent was to challenge observers’ preconceptions and to emphasis that what is being perceived is not a real pipe but rather a two-dimensional representation of a real object—a pipe! In hindsight, Magritte was years ahead of comparative psychologists who more recently began to seriously question how their species of interest “read” pictures of inanimate or animate objects encountered in life. Despite the widespread use of pictures in animal psychology, investigators rarely question the dual nature of pictures and their ecological validity. The PNAS paper by Pokorny and de Waal (1) is noteworthy for addressing these questions in a study of ingroup versus outgroup discrimination by capuchin monkeys (Cebus apella).

The monkeys were rewarded for choosing the picture, from among four photographs of monkey faces, that depicted the animal that did not belong to the same social group as the other three monkeys. The subjects knew, from prior social interactions, the group membership of the photographically depicted animals. All three subjects learned this oddity task and remained above chance when tested with pictures of new animals from the same two groups never before seen in photographs. Performances also remained high with the initial pictures converted to grayscale. The authors conclude that the capuchins recognize their counterparts in the pictures. In doing so, the authors embrace a common assumption by comparative psychologists (2, 3) that pictures are functionally synonymous with their referents and interchangeable in experimental settings with nonhuman primates. However, what makes this study particularly interesting is the additional claim that the capuchins, not unlike Magritte, grasped the representational nature of the pictures.

Pokorny and de Waal (1) convincingly demonstrate that Capuchins discriminate—and presumably recognize—monkeys in pictures, although it remains unclear as to what information the monkeys abstracted from the images. Monkeys within the same groups were often genetically related. They may have differentiated ingroup from outgroup members using familial variations in individual features (e.g., eyes, nose, and mouth) (4). The expression, “You have the nose of your father but the eyes of your mother” may apply to capuchins too.

The other claim—that monkeys perceive the dual nature of pictures and process them as representations—is perhaps more problematic. This conclusion is

Capuchins discriminate—and presumably recognize—monkeys in pictures.

based on three observations. First, the animals’ performances were not adversely affected by removing color cues in grayscale photographs. Second, they did not display social responses to the pictorial faces as they might with real conspecifics. Third, they touched the pictorial faces, which they would not normally do to a real animal. Given that two of their subjects were females and that male primates in this New World primate species may not be trichromatic (5), it is unlikely that the animals failed to perceive the color cues in the original stimuli. Nevertheless, all grayscale pictures were first experienced in color, which may have facilitated transfer to grayscale pictures. Absence of social responses to pictures might be a consequence of prior exposure to similar pictorial faces (6). Finally, it is perhaps not surprising that the capuchin monkeys touched the faces, because they were rewarded for doing so over many training sessions.

Although the argument for the referential use of pictures by the monkeys is not totally persuasive, the results deservedly call for a renewed focus on the theoretical assumptions underlying picture perception by nonhuman primates. Given Pokorny and de Waal’s findings in the context of other studies, there can be no further doubt that a nonhuman primate can recognize the content of a picture. This issue is now unequivocally settled (e.g., see ref. 7). However, this PNAS paper (1) importantly raises critical questions about the possible variability in picture processing within individuals and between species under differing training regimens.

One striking aspect of the literature on picture understanding is that nonhuman primates often process pictures independently of what they represent. A classic example is the study by D’Amato and Van Sant (8) in which capuchins were rewarded for discriminating slides containing people from those that did not. Post hoc analyses revealed that the monkeys did not do so based on the conceptual content of the pictures, but likely because the pictures with people incidentally contained more red features than the other set. In contrast, there is only very limited unambiguous evidence for the referential use of pictures by nonhuman primates, despite the excellent abilities of these primates to remember (9, 10) and to categorize (11) pictures. The evidence for referential use of pictures is found mostly in apes. For instance, Savage-Rumbaugh et al. (12) demonstrated that two of three chimpanzees learned to use symbols to label two categories, i.e., food and tools, of real objects. Subsequently, these animals could use these symbols to label photographs of food and tools. Perhaps even more convincing is the demonstration that chimpanzees can recognize line drawings of familiar humans and apes (13).

The way that pictures are processed likely reflects a complex interaction of different factors that can be overlooked when one focuses only on terminal behavioral performances. There is a striking parallel between the initial perception of pictures by nonhuman primates and human infants. Nine-month-old children suck the nipple of a pictorial baby bottle, suggesting their nonreferential perception of pictures, just like pictorially naive adult monkeys and apes when they selectively choose and then eat pictures of banana (14). This behavior ceases in older infants, presumably because of maturation and repeated experience with pictures and videos (15). This behavior disappears in nonhuman primates only as a function of training or practice with pictures and independently of the age of the subject (14). By 18 months of age, children point and talk about the pictured objects, suggesting that the pictures are now processed as referential stimuli (15).

In the discussion of their findings, Pokorny and de Waal (1) refer to Fagot et al.’s (16) three levels of pictorial processing: Confusion, Independence, and Equivalence. Confusion defines those cases in which the animal treats the photographs as if they were functional and physical exemplars of the referent (e.g., the picture of the banana is a banana). Independence defines those conditions in which the animal makes no connection between the picture and its content, despite their sharing common features. Equivalence defines situations in which the picture is “read” as being a symbolic and iconic representation of its referent (e.g., this is a photograph of my father). Because of the diversity of results reported in the animal literature, we believe that it might be best to regard these levels as components of a dynamic system, rather than as static states.

As noted above, naive monkeys confuse realistic pictures with their referent objects (14), just like young human infants (15). Therefore, the confusion mode might be conceived of as the starting point in the dynamic system. As outlined in Fig. 1, we suggest that repeated exposure to pictures might point the animal to follow one of two possible routes from that starting point. In one scenario, the animal may, like human infants, take the “high road” leading to equivalence, and ultimately process pictures as iconic symbols of their referents. This assumes that the subject animal has the capacity for symbolic representation (17). This is likely the path followed by those chimpanzees who first confuse their image in a mirror with a conspecific, but then recognize that it is a representation of themselves (18). In an alternative scenario, repeated exposure to pictures may nudge if not push the animal toward the “low road” leading to independence, with the consequence that the animal processes the pictures as a collection of simple features rather than the referential pictorial content. An important, and perhaps counterintuitive, implication of this scenario is that continued exposure to pictures pushes an animal inexorably further away from processing the representational content.

Fig. 1.

Possible routes for the processing of repeatedly experienced pictorial stimuli depicting animate and inanimate objects encountered in life.

The above framework raises important theoretical questions. Can an animal remain stuck in a confusion mode with repeated opportunities to explore the differences between a picture and its referent? Is it possible to direct the animal toward and along either the low or high road? Use of stimulus material within the social domain, as is the case in Pokorny and de Waal (1), may well be a means to orient the animals toward the high road. Are there possible stops (Fig. 1) or even plateaus along the road? What subject-related factors (e.g., species, age, experience, cognitive capacity) or task-related factors (e.g., degree of realism) can affect the outcome? Would this framework also be relevant in the design and interpretation of studies with other species? Would the capacity to take the high road correlate with high-order cognitive capacities such as noniconic symbol use, theory of mind, and referential communication (vocalization, pointing, eye-gaze reading)? Results like those reported by Pokorny and de Waal are good first steps toward answering these questions.