Brain Mechanisms for Interpreting the Actions of Others From Biological-Motion Cues

Abstract

Humans are an intensely social species. Our social abilities depend upon specialized brain systems for rapidly recognizing the faces of others, for interpreting the actions of others through an analysis of biological-motion cues, and for determining the emotional states of others via inspection of facial expression. Recent work has implicated the superior temporal sulcus (STS) region as an important component of the social brain. Functional neuroimaging studies have provided clues about how this region is involved in the visual analysis and interpretation of other people’s actions. STS activity is modulated by the context within which the actions of biological entities are observed. Such a contextual influence is consistent with a broader tradition within social psychology emphasizing the powerful influences of situational and contextual factors on behavior and perception. The STS region also shows promise as a region of importance in the investigation of both typical and impaired social-cognitive development. Future work should aim to inform us better of the development of interrelationships between the STS region and other regions of the social brain, including the amygdala and the fusiform gyrus.

Biological motion refers to the visual perception of a biological entity engaged in a recognizable activity. This definition includes the observation of humans walking and making eye and mouth movements, but the term can also refer to the visual system’s ability to recover information about another’s motion from sparse input. The latter is well illustrated by the discovery that point-light displays (moving images created by placing lights on the major joints of a walking person and filming them in the dark), while being relatively impoverished stimuli, contain the information necessary to identify the agent of motion and the kind of motion produced by the agent (Johansson, 1973). Biological motion is integral to social perception. Social perception refers to the initial stages of evaluating the intentions of others using their gaze direction, body movements, hand gestures, facial expressions, and other biological-motion cues (Allison, Puce, & McCarthy, 2000).

We have employed virtual-reality character animation and functional magnetic resonance imaging (fMRI) techniques to map out the neural circuitry supporting social perception. In particular, we have focused on the role of the superior temporal sulcus (STS) region in the interpretation of actions by other human beings via the processing of biological-motion cues. Here we review a few recent advances in this research program. We begin by describing what we have discovered about the role of the STS region in social perception in normally developing adults. We then consider how these advances have informed—and have been informed by—our understanding of the brain mechanisms underlying social-perception dysfunction in autism. We close by offering suggestions for future research.

Cognitive neuroscientists have identified several regions thought to be important for different components of social perception. These include (a) the lateral fusiform gyrus, located on the underside of the brain in the temporal and occipital lobes and thought to be important for rapidly recognizing the faces of others (e.g., Puce, Allison, Asgari, Gore, & McCarthy, 1996); (b) the STS region, located on the lateral surface of the brain in the temporal lobe (see right panel of Fig. 1) and implicated in the interpretation of the actions and social intentions of others through an analysis of biological-motion cues (e.g., Bonda, Petrides, Ostry, & Evans, 1996; Pelphrey, Singerman, Allison, & McCarthy, 2003); and (c) the amygdala, a limbic brain structure comprising at least 13 different nuclei and highly interconnected with other cortical and subcortical brain structures, which has been implicated in determining the emotional state of others through analysis of facial expressions (e.g., Morris et al., 1996).

Fig. 1.

Experiment to determine brain activation to expected and unexpected eye-gaze on the part of another person (left panel) and brain activation to biological motion (observed human movements; right panel). Participants saw a computer-animated woman and a small checkerboard that appeared and flickered in her field of view. In each of two conditions, she either shifted her gaze toward the checkerboard (congruent with viewer expectations) or away from it (incongruent with viewer expectations) after a brief delay. (This part of the figure adapted from Pelphrey, Singerman, Allison, & McCarthy, 2003.) Incongruent trials evoked greater superior temporal sulcus (STS) activity than did congruent trials, demonstrating the sensitivity of the STS region to the intentions conveyed by eye-gaze shifts. The right panel shows activity in the right STS region evoked by observation of eye, mouth, and hand movements. The top image shows a sagittal (side) view of the brain, with activation localized to the posterior STS region in the right hemisphere. The bottom image shows the same activation in a coronal (front to back) view of the brain. Activity evoked by biological motion is often right lateralized, as shown here.

THE ROLE OF THE STS REGION IN SOCIAL PERCEPTION

Early functional neuroimaging studies in humans implicated the STS region, particularly the posterior portion of this structure, in the visual perception of biological motion (for a review, see Allison et al., 2000). For example, Bonda et al. (1996) reported that the perception of point-light displays representing goal-directed hand actions and body movements selectively activates the STS region relative to random motion. Later, Puce and colleagues (Puce, Allison, Bentin, Gore, & McCarthy, 1998) demonstrated that the STS region responds more strongly to observed mouth and eye movements than it does to various nonbiological-motion controls. We demonstrated that the STS region responds more strongly to biological motion (as conveyed by a walking robot or walking human) than it does to nonmeaningful but complex nonbiological motion (a disjointed mechanical figure) or to complex and meaningful nonbiological motion (the movements of a grandfather clock; Pelphrey, Mitchell, McKeown, Goldstein, Allison, & McCarthy, 2003). The available studies suggest that the STS region is involved in social perception by representing perceived actions. Based on the human neuroimaging evidence as well as work in nonhuman primates demonstrating the sensitivity of neurons in the STS to various socially relevant cues including head and gaze direction (e.g., Perrett et al., 1985), Allison et al. (2000) hypothesized that the STS region plays a central role in social perception via its role in interpreting the actions and social intentions of other people from an analysis of biological-motion cues.

Is the STS Region Sensitive to the Context Within Which an Action Is Observed?

In an initial test of the hypothesis of Allison et al. (2000), we sought to determine whether the STS region was sensitive to the context within which a gaze shift is observed. Specifically, we compared situations in which the gaze shift was perceived to be consistent or inconsistent with the subject’s expectation regarding the person making the eye movement (Pelphrey, Singerman, et al., 2003). During fMRI scanning, our participants watched as a small checkerboard appeared and flickered in an animated character’s visual field (see left panels of Fig. 1). On goal-directed (congruent) trials, the character shifted her gaze toward the checkerboard (Fig. 1, top left) acting in accordance with the subject’s presumed expectation that an agent should behave in a goal-directed way in this context. On non-goal-directed (incongruent) trials, the character shifted her gaze away from the checkerboard and toward empty space (Fig. 1, bottom left) violating the participant’s likely expectations. We hypothesized that the STS region would be sensitive to the goal-directedness of the character’s gaze shift. Therefore, activity evoked in this region would differentiate between congruent and incongruent trials. This differentiation, we reasoned, would reflect an ability to link the perception of the gaze shift with a theory about the gaze shift’s significance in terms of the other person’s intentions. We observed more activity in the STS region for incongruent gaze shifts than for congruent ones, suggesting that additional processing was required when the character violated participants’ expectations about other people’s tendencies to act in goal-directed ways.

Is the STS Sensitive to Contextual Signals of Approach and Avoidance?

Gaze serves as a potent social cue, with mutual gaze often signaling threat or approach and averted gaze conveying submission or avoidance (e.g., Argyle & Cook, 1976). After establishing that the STS region was sensitive to one aspect of context (goal-directedness vs. non-goal-directedness), we next wondered if this region was responsive to a range of other contextual factors such as approach and avoidance. We devised a virtual setting to explore the role of the STS region in the interpretation of actions that can signal messages about approach and avoidance in an overtly social and more complex encounter: a stranger passing by in a hallway (Pelphrey, Viola, & McCarthy, 2004). Through virtual-reality goggles in the MRI scanner, our participants viewed an animated male character (see top panel of Fig. 2), who approached and shifted his gaze either toward (mutual gaze) or away from (averted gaze) the subjects. We reasoned that if gaze-related activity in the STS region reflected the operation of a simple eye-movement detector, the region would not respond differentially to mutual and averted gaze. The motion of the man walking toward the subject and that of his gaze shift evoked robust activity in the right posterior STS region and the right fusiform gyrus. Mutual gaze evoked greater activity in the STS region than did averted gaze (see bottom panel of Fig. 2). In contrast, the fusiform gyrus responded equally to mutual and averted gaze, demonstrating a functional dissociation between the two social-brain areas. Thus, the fusiform gyrus might have been functioning as a face detector in responding to an approaching face, but the STS region was involved in interpreting the stranger’s actions in context. This study advanced our understanding of the role of the STS region in social perception by demonstrating its sensitivity to the social context (approach vs. avoidance) in which a specific biological motion occurs. Taken together, our studies of eye-gaze processing demonstrate the involvement of the STS region in the interpretation of gaze direction to determine another person’s focus of visual attention or their desire to avoid or engage in social interaction.

Fig. 2.

Experiment measuring brain activation in response to a stranger initiating or avoiding social interaction. Participants viewed an animated character approaching down a virtual hallway, who shifted his gaze either toward or away from the subject. In both situations, the animated sequence evoked activation in the right superior temporal sulcus (STS) region and the right fusiform gyrus. The graph at bottom shows the time courses of activation (indicated as average blood-oxygenation-level-dependent contrast, or BOLD, signal changes) from the right STS region in response to the passerby’s mutual and averted gaze movements. The mutual- and averted-gaze conditions are plotted along with a plot of their difference (mutual minus averted gaze). Note that the change in activity begins with the appearance of the character in the hallway and increases again at the moment the gaze shift occurs. Adapted from Pelphrey, Viola, & McCarthy (2004).

The findings from our studies of eye-gaze processing also demonstrate the influence of contextual factors on activity in a specific region of the social brain. Specifically, activity in a domain-specific visual-processing region is extremely sensitive to the context of an observed action (or movement). Furthermore, these contextual influences are observed even under conditions of passive viewing, when subjects are not explicitly instructed to determine the appropriateness of observed actions. Prior to these studies, we might have expected that such effects would have been restricted to prefrontal regions that are known to be engaged in such executive functions as decision making, response selection, and the perception of novelty. From a broader theoretical perspective, our findings regarding the influence of context on brain activity fit well with elegant demonstrations within social psychology of the powerful influences of situational and contextual factors on behavior and perception. We have demonstrated that the principles of situational and contextual influence operate at multiple levels of the organism: from the individual’s behavior in social context to the level of localized brain activity.

THE STS REGION AND SOCIAL-PERCEPTION DEFICITS IN AUTISM

The use of functional neuroimaging to study abnormal brain function provides an approach in which brain differences can not only inform us about disease but also help us to better understand normal brain functioning and development. Abnormalities in social perception are a striking feature of autism, a developmental disorder defined by characteristic deficits in communication and social behavior as well as stereotyped repetitive behaviors. For example, individuals with autism do not look at faces in the same way as do typically developing individuals: They spend significantly less time looking at speakers’ eyes and more time looking at their mouth or body (Pelphrey et al., 2002). Eye-gaze processing deficits, including failures to coordinate visual attention with others and difficulties comprehending the mental states and social intentions of other people as conveyed by the eyes, are key features of autism. These deficits are not the result of abnormal gaze discrimination (e.g., people with autism can report that someone is looking to the right or left); they instead represent an inability to use gaze spontaneously to understand and predict other people’s mental states and behaviors.

The behavioral nature of eye-gaze processing deficits, combined with our prior neuroimaging findings, led us to hypothesize that STS dysfunction might be involved (Pelphrey, Morris, & McCarthy, 2005). To test this, we employed our congruent versus incongruent eye-gaze paradigm in a sample of adult participants with autism and a sample of individuals without autism who were matched with the neurologically normal individuals on IQ and gender. We predicted that in autism, unlike in our neurologically normal sample, the STS region would not be sensitive to the goal-directedness of the character’s gaze shifts. We again found that in neurologically normal participants, “errors” (incongruent gaze shifts) evoked more activity in the STS region, indicating a strong effect of context. The STS region was also activated during observation of gaze shifts in individuals with autism, but there was no difference between congruent and incongruent trials, indicating that activity in these regions was not modulated by the context of the perceived gaze shift. These findings implicate dysfunction in the STS region as a mechanism contributing to eye-gaze processing deficits in autism and strengthen the conclusion that the STS region plays an important role in social perception in the normally developing brain.

CONCLUSION AND FUTURE DIRECTIONS

Our work to date has emphasized the unique contribution of the STS region to social perception. This analytic perspective has helped provide a framework for organizing our emerging understanding of the social brain, but this approach does not fully reflect the complexity of interactions among the STS region and other regions known to be involved in social cognition and social perception (e.g., the amygdala and fusiform gyrus). These structures probably function in parallel and can be better understood as components in a network of regions subserving social perception. For example, when encountering a socially ambiguous situation, such as the approach of an unfamiliar person, the amygdala will provide a rapid and automatic assessment of the potentially threatening aspects of the situation and, through its interconnections with the other structures, allocate processing resources accordingly. The fusiform gyrus will provide a perceptual representation of the face and will aid in identification of the person. The STS region will conduct a visual analysis of the person’s gait and other socially and communicatively important actions, including movements of facial features and shifts in eye gaze. The rapid integration of the functions performed by each structure will guide social perception and the subsequent behavior of the observer. We believe that recent developments in the field’s ability to image functional connectivity will lead efforts to identify interactions among social-brain regions during social perception, thereby opening new frontiers of research.

In another direction, research into the brain mechanisms supporting social perception offers exciting implications for the understanding of the development of social perception and social-cognition abilities, including theory of mind. Behavioral studies have provided exquisite descriptions of social-cognitive development, but there is little information available regarding the development of the social brain. The potential for interaction between social-cognitive neuroscience and developmental psychology in this area has been the subject of extensive discussion (e.g., Frith & Frith, 2003). However, progress has been hindered by the lack of empirical studies of the brain mechanisms supporting social perception in children. We have started to explore the normal and abnormal development of social perception and the social brain. For example, a preliminary investigation suggested that the neural circuitry underlying the interpretation of eye gaze in 6- to 11-year-old normally developing children is very similar to that of adults (Mosconi, Mack, McCarthy, & Pelphrey, 2005). Both children and adults activate the STS region in response to observed gaze shifts, and this activity is modulated by the context of the observed gaze shift. More work is required to establish continuity and differences in the neural circuitry of social perception across development. In particular, longitudinal studies that chart out the maturation of the structures, functions, and connectivity of key regions in the social brain are needed. Here again, an understanding of connectivity and communication among these social-brain regions will likely prove critical to an emerging understanding of developmental mechanisms. By analyzing the progression of the neural circuitry supporting social perception, we may begin to map the brain mechanisms subserving typical and atypical social-cognitive development.

Recommended Reading

Allison, T., Puce, A., & McCarthy, G. (2000). (See References)

Decety, J., & Grezes, J. (1999). Neural mechanisms subserving the perception of human actions. Trends in Cognitive Sciences, 3, 172–178.

Mosconi, M.W., Mack, P.B., McCarthy, G., & Pelphrey, K.A. (2005). (See References)

Pelphrey, K.A., Morris, J.P., & McCarthy, G. (2004). Grasping the intentions of others: The perceived intentionality of an action influences activity in the superior temporal sulcus during social perception. Journal of Cognitive Neuroscience, 16, 1706–1716.