An integrative neural model of social perception, action observation, and theory of mind

Abstract

In the field of social neuroscience, major branches of research have been instrumental in describing independent components of typical and aberrant social information processing, but the field as a whole lacks a comprehensive model that integrates different branches. We review existing research related to the neural basis of three key neural systems underlying social information processing: social perception, action observation, and theory of mind. We propose an integrative model that unites these three processes and highlights the posterior superior temporal sulcus (pSTS), which plays a central role in all three systems. Furthermore, we integrate these neural systems with the dual system account of implicit and explicit social information processing. Large-scale meta-analyses based on Neurosynth confirmed that the pSTS is at the intersection of the three neural systems. Resting-state functional connectivity analysis with 1000 subjects confirmed that the pSTS is connected to all other regions in these systems. The findings presented in this review are specifically relevant for psychiatric research especially disorders characterized by social deficits such as autism spectrum disorder.

1. Introduction

The field of social neuroscience is rapidly evolving and with it our understanding of the neural basis of social information processing. Different branches of research within the field have been instrumental in describing specific components of typical and aberrant social information processing. However, these findings are heterogeneous and the field as a whole is lacking a comprehensive model of social information processing. In the following review we outline existing research related to the neural basis of three key neural systems underlying social information processing: social perception, action observation, and theory of mind. We propose an integrative model of social information processing that unites these three central processes and highlight the common brain region implicated in them, the posterior superior temporal sulcus (pSTS) region. We also show that the pSTS region is functionally interconnected to all key regions in the “social brain” for social information processing.

The findings presented in this review are highly relevant for psychiatric research especially regarding a wide variety of seemingly disparate disorders characterized by social deficits (Kennedy and Adolphs, 2012), such as autism spectrum disorder (ASD), Williams syndrome, behavioral-variant frontotemporal dementia, Fragile X, developmental prosopagnosia, Capgras syndrome and psychopathy. To demonstrate how our model can be applied to neuropsychiatric disorders, we use ASD as an example and we review relevant evidence suggesting that one or more of the key regions in our model are abnormal in ASD. As defined in DSM-5, ASD is a disorder characterized by persistent and prominent deficits in social communication and social interaction as well as restricted and repetitive behavior (APA, 2013). We also discuss the implications of our model for research on ASD, highlighting the temporal integration role of the pSTS and a recent hypothesis that posits that ASD can be understood as a disorder of temporal prediction (Sinha et al, 2014). Our model also suggests that the pSTS region may serve as a target for interventions designed to improve social functioning in ASD (e.g., Ventola et al., 2014).

2. A neural system for social perception

Humans are inherently social creatures as evidenced by a range of social behavior that typically emerges early in infancy. Infants preferentially orient to social signals such as facial expressions and by the age of three months recognize their mother's face and tone of voice (Johnson et al., 1991; Kovacs et al., 2010; Kurzweil, 1988; Valenza et al., 1996). Expertise in social perception, defined as the ability to decode another person's mental states based on basic behavioral signals, is evolutionarily adaptive due to its reflexive and highly efficient nature (Allison et al., 2000; Beauchamp et al., 2008; Mcarthur and Baron, 1983). Basic social perception precedes more effortful and explicit processes which emerge later in development (Carpenter et al., 1998; Low and Perner, 2012).

Similar social behavior is observable in monkeys and thus primate research serves as a useful model for understanding social perception in humans. Chimpanzees and corvids, for example, are able to understand what conspecifics know in the context of food competition (Brothers, 1996; Call and Tomasello, 2008; Emery and Clayton, 2009; Premack and Woodruff, 1978). Interestingly, chimpanzees can also use information about others' intentions to help conspecifics and humans (Melis et al., 2011) and there is evidence suggesting that rhesus monkeys are able to infer what others perceive on the basis of what others can see (Flombaum and Santos, 2005) and hear (Santos et al., 2006). The brain regions supporting social perception in primates and humans are largely overlapping (Brothers, 1996). Dynamic social perception mainly implicates the pSTS, the amygdala (AMY), the orbital frontal cortex (OFC) and the fusiform gyrus (FFG) (Allison et al, 2000). These regions have been found to have reciprocal connections within the primate brain and to be sensitive and selective to social stimuli in monkeys and humans (Brothers, 1989, 1996; Brothers and Ring, 1993; Ku et al., 2011).

The pSTS plays a key role in the social perception network. It receives direct input from primary visual and auditory regions in primates and humans and has been implicated in extracting and representing dynamic information in these regions (Belin et al., 2000; Brothers, 1996; Hoffman and Haxby, 2000; Jastorff et al., 2012; Kreifelts et al., 2009). The pSTS is sensitive to and selective for social stimuli that signal intention in humans (Jastorff et al., 2012). It is selectively activated to static social (i.e. faces) vs. non-social stimuli (i.e. objects) as well as complex dynamic social information, such as changes in the direction of eye gaze or facial expression (Gobbini and Haxby, 2007; Watson et al, 2014). The pSTS is selective for intentional human actions that are more socially meaningful than non-goal directed movements (Bahnemann et al, 2010). In the auditory domain, the pSTS plays an important role in processing affective speech (Wildgruber et al., 2006). Using dynamic causal modeling, Ethofer et al. (2006) demonstrated that the pSTS is the input region of the affective voice processing system, where affective information is extracted and represented. In terms of somatosensory input, mimicking the type of touch during close social interactions by stroking skin areas with C-tactile (CT) nerves, such as the forearm, with a soft brush has been shown to activate the pSTS region (Bjornsdotter et al., 2014; Gordon et al, 2013).

Interestingly, not all social perception studies show pSTS involvement. One possible explanation for this is that some studies did not report a contrast between a relatively social vs. relatively non-social sensory experience. For example, in a study that examined somatosensory experience during an affective touch experiment, Gazzola et al. (2012) had the participants believe that the caresser was either female or male and found that the perceived sex of the caresser (female vs. male) activated the somatosensory cortices but not the pSTS region. The sex effect testing (i.e., female vs. male caresser) involves a contrast of different kinds of social sensory experience, but does not seem to involve a contrast of a relatively social vs. relatively non-social experience (e.g., biological motion vs. scrambled motion). The latter type of contrast is critical for demonstrating pSTS activation because the pSTS is particularly sensitive to the degree of social vs. non-social meaning in the behavioral signals.

The FFG, along the ventral-temporal cortex houses several distinct but adjacent regions that are involved in visual aspects of social perception. Among them, the fusiform face area (FFA) selectively responds to face stimuli, whereas the fusiform body area (FBA) selectively responds to body stimuli (Schwarzlose et al, 2005). The FFA is sensitive to the typical pattern of saccades and fixations observed during implicit facial processing (Morris et al, 2007a, 2007b). Furthermore, the FFA is involved in facial identity recognition (Pitcher et al, 2011), and goal-directed action recognition (Gobbini et al., 2007, 2011; Schultz et al, 2003; Shultz and McCarthy, 2012).

The AMY encodes the emotional salience of social information (Costafreda et al, 2008), particularly when such information requires rapid and reflexive processing (Adolphs, 2009). For instance, the AMY supports gaze orientation toward socially relevant facial features (i.e. the eyes) (Adolphs et al, 2005), particularly when facial expressions communicate fear (Adolphs, 2008).

The OFC has been associated with reward coding of environmental sensory cues, a crucial aspect of behavioral planning in humans as well as primates (Watson and Platt, 2012). Due to the OFC's central role in value-based decision-making, it has been implicated in attention to socially rewarding stimuli and ultimately helps to determine an individual's interest in engaging in social interactions and responding to social cues (Ethofer et al., 2013; Kahnt et al, 2010; Wallis, 2012).

Studies investigating social perception in individuals with ASD have tremendously advanced our understanding of the typical development of social perception and the social deficits of individuals with ASD. Infants with ASD attend less to social cues, such as faces and voices, compared to typical babies (Chawarska et al, 2010, 2012, 2013). Furthermore, unlike typically developing toddlers at the age of two years, toddlers with ASD fail to orient to the social significance of biological motion; instead, they tend to focus on non-social, physical contingencies, which are disregarded by their typically developed peers (Klin et al., 2009). The social deficits of individuals with ASD observed in early development persist into adulthood. High-functioning adults with ASD show difficulties in inferring others' mental states from vocalizations (Rutherford et al., 2002) or emotional facial expressions (Baron-Cohen et al., 2001).

In line with the notion that individuals with ASD show aberrant social perception, there is strong evidence suggesting atypical cortical and subcortical processing of social perception for visual, auditory, and somatosensory signals. With respect to visual signals, researchers found that when young children with ASD were shown point-light displays of coherent vs. scrambled biological motion, they exhibited reduced activity of brain regions involved in social perception including the FFG, amygdala, and the pSTS compared to their unaffected siblings and typically developing peers (Kaiser et al., 2010). The pSTS, in particular, seems to be less selective for social perception in ASD than in typically developing individuals (Pelphrey et al, 2005). With regard to visual processing, activity in the pSTS was less modulated by whether or not the characters' actions were incongruent with their initially displayed preference in individuals with ASD compared to typically developing controls (Vander Wyk et al., 2009). Similarly, aberrant FFG (Dziobek et al., 2010; Kleinhans et al., 2008; Weisberg et al., 2014), AMY (Schultz, 2005; Swartz et al, 2013; Tottenham et al., 2014) and OFC (Bachevalier and Loveland, 2006; Sabbagh, 2004; Sawa et al., 2013) have also been implicated in the social perception deficits of individuals with ASD. In addition, in the auditory domain, individuals with ASD showed a similar activation pattern to controls for non-vocal sounds but failed to activate the voice-selective regions of the pSTS in response to vocal sounds (Gervais et al, 2004). Finally, with respect to somatosensory signals, individuals that reported more autistic traits displayed reduced activity the OFC and pSTS in response to slow and gentile touch to the forearm (Voos et al., 2013).

3. A neural system for action observation

While social perception involves perceiving and decoding others' external behavioral signals and inferring their underlying intentions, the role of the perceiver's self is relatively silent. However, when people understand others' actions, there is another mechanism that relies heavily on the perceiver's self: the perceiver may try to understand what the actions would mean if they themselves were to perform the same actions. In this route, the role of the perceiver's self is much more prominent and the understanding of others' actions is relatively self-based. We refer to the corresponding neural system as action observation or understanding, where socially relevant action understanding requires the perceivers to go beyond mere decoding in an effort to successfully match their own actions to the observed actions (Williams et al, 2001, 2004). In order to understand others' actions, the perceiver must first observe the action and then form a mental imitation of it. In this way, action observation is closely linked to imitation (Dijksterhuis and Bargh, 2001) and observational learning (Monfardini et al, 2013).

Research on the neural basis of action observation and understanding has extensively focused on mirror neurons, a class of visuomotor neurons that were originally discovered in the prefrontal cortex of primates (Matelli et al., 1985). In primates and humans, mirror neurons have been found to respond to both action performance and action observation (Keysers and Gazzola, 2010; Rizzolatti et al, 2009). In humans, the mirror neuron network comprises three interconnected brain areas (Iacoboni and Dapretto, 2006): (a) the pSTS region, which is theorized to serve as the visual input of the dynamics of others' actions, (b) the parietal mirror neuron region, which encompasses the rostral part of the inferior parietal lobule (IPL) and generates lower-level motoric description of others' actions, and (c) the frontal mirror neuron region, which encompasses the posterior inferior frontal gyrus (IFG) and adjacent ventral premotor cortex (PMC), generating higher-level motor plans of actions. The process of action understanding involving mirror neurons is a two-way process (Iacoboni and Dapretto, 2006). Information decoded in the pSTS is passed on to the IPL, and subsequently to the IFG. The IFG then projects back to IPL and to pSTS. Thus, the pSTS region serves as both the input and output region of the mirror neuron system (Iacoboni et al., 2001; Rizzolatti et al., 2014), allowing for a comparison to be made between observed and executed actions.

The field of action observation is relatively young and quickly evolving. It is still hotly debated whether there is an imitative deficit in autism (Ornitz and Ritvo, 1968; Sevlever and Gillis, 2010). While some studies found that children with autism perform worse or show delayed development in imitative tasks relative to typically developing peers (Ingersoll, 2008; Rogers et al., 1996; Stewart et al., 2013; Vanvuchelen et al., 2007; Young et al., 2011), other studies have reported equivalent imitation performance between autism and the control groups (Bird et al., 2007; Press et al., 2010). Recently, a few studies have begun to reveal both intact and impaired mechanisms of imitation in autism (Vivanti et al., 2011, 2014), which may help to address some of the inconsistent findings. The framework of action observation proves to be very useful in explaining certain phenomena associated with autism. For example, many children with autism display echolalia at young ages (Prizant and Duchan, 1981), in which they repeat words and phrases without knowing their meaning or without using them in an appropriate context. Echolalia can be understood as a by-product of an underdeveloped ability of action understanding, in which young children with autism may be able to imitate others' actions but they cannot yet extract the higher-levels goals or meanings from the imitated actions.

Neuroimaging studies investigating the mirror neuron system in autism have yielded mixed results. On the one hand, fMRI results show that children with ASD display atypical activation in regions assigned to the mirror neuron system, including the IFG and IPL, when observing facial expressions and hand movements (Dapretto et al., 2006; Martineau et al., 2010; Oberman and Ramachandran, 2007; Williams et al., 2006). Electroencephalography (EEG) results also show that mu suppression during the observation of actions (e.g., the attenuation of mu rhythm during execution and observation of hand movements, which is typically seen in neurotypically developing individuals) is not shown in individuals with ASD (Bernier et al., 2007; Oberman et al., 2013). In contrast, several studies have found no differences in the neural activation of regions within the mirror neuron network in individuals with autism compared to controls (Bernier et al., 2013; Dinstein et al., 2010; Fan et al., 2010; Hamilton et al., 2007). Interestingly, studies that report abnormal patterns of neural activity within the mirror neuron network in individuals with autism tend to use emotional stimuli, whereas those that report normal mirror neuron activity in individuals with autism tend to use non-emotional hand action stimuli (Hamilton, 2013). The field of mirror neuron systems continues to develop and the specific functions of brain regions with mirror proprieties continue to be refined.

4. A neural system for theory of mind

Over the past two decades, research on theory of mind (also referred to as mentalizing or mental state reasoning) has provided important insights into social function and dysfunction. Theory of mind has been defined as the ability to predict the relationships between external states of affairs and internal states of mind (Frith, 1989). This ability requires decoupling one's subjective reality from the reality that others perceive (Baillargeon et al., 2010; Blakemore et al., 2007; Frith and Frith, 2006; Nichols and Stich, 2000, 2003). Unlike social perception and action observation, explicit mental state reasoning requires high-level cognitive and attentional resources (Lin et al., 2010) and is believed to be unique to humans (Call and Tomasello, 2008; Saxe, 2006). The ability to understand and predict others' mental states offers several strategic advantages to individuals as they navigate the complex social environment (Hampton et al., 2008; Wan, 2012).

To assess theory of mind, Wimmer and Perner (1983) introduced the now classic Sally-Anne experiment in young children. In this test, children see two dolls: Sally and Anne. Sally puts a marble into her basket before she leaves the room. In Sally's absence, Anne takes the marble out of Sally's basket and puts it into her box. Participants are asked where Sally will look for her marble when she returns to the room. If children are able to take Sally's perspective into account, that is, if they understand that she has a false belief about the location of the marble, they answer that Sally would look for her marble in the basket where she put it. Using this test or related versions, researchers in developmental psychology suggested that around the age of four years children reach an important developmental milestone: they explicitly understand another person's false belief (Perner et al, 2011). When using nonverbal indirect tasks, however, researchers show that 15-month-old infants are capable of taking into account someone else's false belief (Onishi and Baillargeon, 2005). The next step in mentalizing development is to understand nested beliefs or second-order beliefs (e.g., Anne thinks that Sally thinks something). Typically developing children pass second-order false belief tasks at an age of about six or seven years (Perner and Wimmer, 1985), while more complex forms of mental state reasoning, including moral decision-making, develop between adolescence and adulthood (Sebastian et al., 2012; Wang et al., 2006).

The neural basis of theory of mind has been investigated with a variety of tasks using static as well as dynamic stimuli (e.g., Castelli et al., 2002; Saxe and Kanwisher, 2003; Vogeley et al., 2001; Walter et al, 2004; Wolf et al., 2010). Several brain regions have been consistently implicated in theory of mind including the medial prefrontal cortex (MPFC), the temporo-parietal junction (TPJ), the pSTS, the posterior cingulate cortex/precuneus (PCC/PC), and the anterior temporal lobes (ATL), particularly the temporal poles (TP) (Amodio and Frith, 2006; Bahnemann et al, 2010; Mar, 2011).

The MPFC is associated with higher-level, meta-representational thinking as evidenced by its role in strategic decision-making and self-other comparisons (Behrens et al, 2008; Coricelli and Nagel, 2009; Hampton et al., 2008; Lombardo and Baron-Cohen, 2010; Mitchell et al., 2006; Tamir and Mitchell, 2010). Activity in the MPFC has been found to increase linearly with participants' perceived discrepancy between themselves and another person (Tamir and Mitchell, 2010). The MPFC is also implicated in self-referential thinking and introspection (Ochsner et al, 2004; Silani et al., 2008). In addition, the MPFC is involved in modeling potential choices and executed choices as well as forming a mental representation of the value of these choices (Nicolle et al., 2013). In brief, the functions associated with MPFC are diverse and the field continues to advance the understanding of the exact roles MPFC plays in theory of mind.

The TPJ is associated with evaluating others' mental states in social contexts, in particular with taking a third-person perspective (Carter et al, 2012; Saxe, 2006; Saxe and Kanwisher, 2003). The TPJ involvement in theory of mind seems to depend on the types of paradigms used: a meta-analysis distinguishing between story-based studies (i.e., studies employing false belief tasks) and non-story-based studies including more heterogeneous tasks and stimuli (i.e., cartoons, videos and economic games) found that story-based tasks elicit more TPJ activity but non-story-based tasks elicit more MPFC activity (Mar, 2011). Different regions in the TPJ are also found to serve somewhat different theory-of-mind functions. Takahashi et al. (2014) found that the dorsal aspect of the TPJ is related to judging how much an agent has mental functions, while the ventral aspect of the TPJ is related to judging the intelligence of an agent.

The ATL, the PCC and PC mediate other functions necessary for theory of mind. The ATL is thought to be crucial for semantic memory, storing knowledge about objects, people, words or facts (Bonner and Price, 2013), while the PC bordering the PCC is involved in self-awareness and self-related processing (den Ouden et al., 2005).

Activation in the pSTS, a core region supporting social perception, is consistently reported across a wide range of theory-of-mind studies (e.g., Dufour et al., 2013; Walter et al., 2011). The pSTS seems to serve the function of integrating temporal information from dynamic stimuli and extracting intentions from behaviors, thus facilitating mental state inference (Adolphs et al., 2005; Allison et al., 2000; Barch et al., 2013; Gobbini et al., 2007; Vanderwal et al., 2008).

There has been some debate as to whether the pSTS and TPJ are functionally and anatomically dissociable (Gobbini et al., 2007; Saxe, 2006). On the one hand, there are efforts to distinguish between the regions. For example, Saxe et al. (2009) showed that anatomically, TPJ is relatively posterior and dorsal to the pSTS regions, while functionally, TPJ is more involved in theory of mind and pSTS is more involved in social perception. On the other hand, studies also show that there is also some overlap between the regions. For example, anatomically, Mars et al. (2012) used structural and functional connectivity to address the question of whether the TPJ is an identifiable cortical region or a cluster of subregions, each supporting separate cognitive functions, and they found that the TPJ consists of several subregions, more or less overlapping with the pSTS. Functionally, as mentioned, both regions seem to contribute to higher-level theory-of-mind processes (e.g., Dufour et al., 2013; Walter et al., 2011). Taken together, there is likely a functional and anatomical continuum between the pSTS and TPJ; they can be differentiated but it seems that their specificity exists in a relative or probabilistic sense, rather than a strictly all-or-none absolute fashion.

Researchers consistently find that individuals with ASD have deficits in theory of mind. Children with ASD tested with the Sally-and-Ann task failed to recognize Sally's false belief, despite the fact that they were around five years older than the previously tested typically developing children (Baron-Cohen et al., 1985). From this and various other studies that compared mentalizing skills of typically and atypically developing children (Baron-Cohen, 1989; Perner et al., 1989), the lack of false-belief recognition has been considered to be a hallmark of the core social deficits in ASD (Baron-Cohen et al., 1985, 1997, 2001; Leslie and Thaiss, 1992). Although individuals with ASD of average and above average intelligence learn to solve simple false-belief tests in later development, their performance on more advanced tests using complex, social emotions, or naturalistic scenes demonstrate that their deficits in mental state reasoning persist in adulthood (Happé, 1993, 1994; Moran et al., 2011; Rosenblau et al., 2014). High-functioning adults with ASD, who are capable of understanding false beliefs when explicitly prompted to do so, are less capable of spontaneously anticipating them on the basis of the actor's behavior (Senju et al., 2009). Much research has been devoted to the investigation of theory of mind in individuals with ASD using a variety of tasks, which has significantly advanced the understanding of mental state reasoning in both typical development and ASD.

There are some neuroimaging studies that have addressed theory-of-mind deficits of individuals with ASD. Studies investigating explicit mentalizing with mostly static tasks found reduced MPFC and TPJ activity in individuals with ASD compared to typically developing controls (Happé et al., 1996; Lombardo et al., 2011), with more severely affected individuals with ASD displaying less activation (Lombardo et al., 2011). However, Dufour et al. (2013) did not find group differences (ASD vs. controls) in neural activation during an explicit, story-based theory-of-mind task. Recently, using multi-voxel pattern analysis, Koster-Hale et al. (2013) found that the spatial activation patterns of voxels within the right TPJ distinguish between intentional and unintentional (accidental) behavior of story characters in typically developing controls but not in individuals with ASD. Studies assessing mental state inference indirectly (e.g., asking the participant to label social emotions, which requires some understanding of others' mental states) or with dynamic video stimuli (e.g. displaying shapes that move in a goal directed manner), have reported aberrant activation profiles of the AMY and pSTS/TPJ regions (e.g., Baron-Cohen et al., 1999; Castelli et al, 2002; Critchley et al, 2000; Kliemann et al., 2012; Pelphrey et al., 2011; Perlman et al, 2011). In sum, the current body of research suggests that individuals with ASD show aberrant spatial activation patterns and reduced activity of core mentalizing regions, such as the TPJ and MPFC, when inferring others' mental states.

5. An integrative model of social information processing

Upon reviewing the three major neural systems underlying social information processing, a picture emerges of the pSTS at the intersection of social perception, action observation, and theory of mind. The pSTS supports social information processing on multiple levels. It is involved in the temporal integration of visual, auditory and somatosensory cues of others' behaviors and representation of a basic form of intentionality (Beauchamp et al., 2008; Gordon et al, 2013; Hagan et al., 2009, 2013; Hocking and Price, 2008). With respect to visual information, Gobbini and Haxby (2007) found that the pSTS is a core region involved in decoding dynamic visual information, such as changes in the direction of eye gaze or facial expression. In the auditory domain, using a dynamic causal model, Ethofer et al. (2006) established that the pSTS is the input region of the prosody processing system, where information about the emotional tone of voice is extracted and represented. The information primarily extracted and represented within the pSTS serves as an input to higher-level social cognitive computations, including understanding and subsequently mimicking the actions of others, that is, action observation via the mirror neuron system (Iacoboni et al., 2001; Iacoboni and Dapretto, 2006), as well as predicting others' actions and taking another person's perspective in the environment (Corbetta et al, 2008; Schurz et al., 2014).

As illustrated in Fig. 1, our integrative model positions the pSTS at the center of the three systems underlying social information processing. In the social perception system, the pSTS has reciprocal connections to the FFG, amygdala, and OFC (e.g. Allison et al., 2000). In the action observation system, the pSTS projects to the IPL and IFG, regions implicated in generating motor plans of others' actions (Iacoboni and Dapretto, 2006). During theory-of-mind computations, the social information decoded in the pSTS might be further processed in other frontal, temporal and parietal regions such as the MPFC, TPJ ATL, and PCC/PC that are required for reasoning about others' mental states and intentions (Amodio and Frith, 2006; Mar, 2011; Saxe and Kanwisher, 2003).

Fig. 1.

An integrative model of the neural systems for social information processing: social perception, action observation, and theory of mind. Dashed lines suggest possible connections among these systems. AMY, amygdala; FFG, fusiform gyrus; OFC, orbitofrontal cortex; pSTS, posterior superior temporal sulcus; IFG, inferior frontal gyrus; IPL, inferior parietal lobule; MPFC, medial prefrontal cortex; TPJ, temporo-parietal junction; PCC/PC, posterior cingulate cortex/precuneus; ATL, anterior temporal lobe.

To evaluate whether the pSTS is truly a uniquely shared region across the three systems, we performed a meta-analysis using Neurosynth (Yarkoni et al, 2011) for each system (see Fig. 2(A)-(C), respectively) and a conjunction analysis across all systems (see Fig. 2(D)). Because the key word “social perception” was not available in the Neurosynth website at the time of this analysis, Fig. 2(A) was obtained by using the core Neurosynth tools (https://github.com/neurosynth/neurosynth) and restricting the meta-analysis to studies that fulfill the criterion of having both the “social” AND “perception” features, which included 123 studies. Fig. 2(B) was obtained by restricting the analysis to studies with the feature of “action observation”, which included 74 studies. Fig. 2(C) was obtained by using the core Neurosynth tools and restricting the analysis to studies that fulfill the criterion of having the features of either “theory of mind” (which is “theory mind” in Neurosynth) OR “mentalizing”, which included 191 studies. Interested readers may refer to the Appendix for the Python code that we used to obtain these maps, and are encouraged to explore different variations of the code (e.g., changing the min_studies argument on the line for meta.MetaAnalysis may give slightly different results). The results of Fig. 2(D) showed that the only brain region surviving the conjunction analysis is the pSTS, with the center of gravity at MNI152mm coordinate (52, −46, 9), localized to the crux of the right hemisphere pSTS where the main sulcus branches into the ascending limb and the posterior region. These analyses also supported that the regions most consistently implicated in social perception, action observation, and theory of mind across a wide range of studies corresponding to the ones reviewed in this paper. In addition, we analyzed the psychological terms of neuroimaging studies associated with Fig. 2(D) via the decode function in Neurosynth. The word cloud associated with these terms suggests that this pSTS region is indeed primarily associated with social perception, action observation, and theory of mind (see Fig. 2(E)). In brief, these results provide support for the proposed model in which the pSTS serves as a central region for social information processing.

Fig. 2.

Brain regions engaged during: (A) social perception, (B) action observation, and (C) theory of mind/mentalizing, as reported by the large-scale, meta-analytical tool Neurosynth (reverse inference, FDR q<.01). Figure (D) is the conjunction of (A)-(C). The only common region was localized to the right pSTS, center of gravity=(52, −46, 9), MNI152 mm coordinate. Figure (E) is the word cloud of the top 30 psychological terms most strongly associated with the right pSTS region as shown in Figure (D); the terms were generated via the decode function of Neurosynth.

To evaluate whether the pSTS is truly interconnected with the other key regions in the three systems, we performed a large-scale resting-state functional connectivity analysis using the Neurosynth website (http://neurosynth.org/locations/). Specifically, we examined the functional connectivity from the seed voxel at (52, −46, 9), the pSTS center of gravity derived from Fig. 2(D), to the other voxels in the brain in a sample of 1000 individuals. As seen in Fig. 3(A), the pSTS is indeed functionally connected to all other key regions of social information processing in our integrated model (see Fig. 1). Beyond these key regions, the results also suggest that the pSTS is connected to the insular cortex (see Fig. 3(A)) and the cerebellar cortex (see Fig. 3(B)), which is in line with the recent findings that insular cortex (Gordon et al., 2013) and cerebellar cortex (Jack and Pelphrey, 2014) are also involved in social information processing. Together, these results confirmed that the pSTS is indeed interconnected to key brain regions supporting social information processing.

Fig. 3.

Resting-state functional connectivity with a sample of 1000 subjects and a seed voxel in the pSTS at (52, −46, 9), MNI152mm coordinate, i.e., the center of gravity in Fig. 2(D). The result was the average of the results based on seed voxels at (52, −46, 10) and (52, −46, 8) and the maps were available from the Neurosynth website. The strength of functional connectivity is based on correlations, with higher values indicating greater similarity in activation between the seed voxel in the pSTS and other voxels of the brain. For details of these maps, see Yeo et al. (2011), Buckner et al. (2011), and Choi et al. (2012). Top Panel (A) shows that the pSTS is functionally connected to the major regions implicated in social information processing (see the integrated model in Fig. 1). Bottom Panel (B) shows functional connectivity from the seed voxel in the pSTS to cerebellar cortex. The slice was at (-19, −76, −41), MNI152mm coordinate.

Our proposed model can be further integrated with other existing models of social cognition, such as the dual system account of implicit and explicit social cognition (Apperly and Butterfill, 2009). The dual system account proposes that there are two systems supporting social information processing: one that is cognitively efficient for tracking belief-like states and the other that is cognitively demanding for reasoning about beliefs. The first system corresponds to the social perception and action observation systems. A recent study suggests that the action observation system is highly automatic and efficient (Spunt and Lieberman, 2013) and thus closer to the system of implicit social cognition. The second system is proposed to be more flexible but also more cognitively demanding and corresponds to the theory-of-mind system, or the system of explicit social cognition. As displayed in Fig. 1, the three systems can be described on the dimension reflexive vs. effortful information processing; social perception and action observation can be viewed as relatively reflexive and automatic, whereas theory of mind can be viewed as relatively effortful, controlled, and cognitively demanding.

The fact that the pSTS is located at the intersection of the major systems for social information processing has important implications for neuropsychiatric disorders such as ASD. There is a growing literature suggesting that the pSTS is implicated in temporal predictive coding of human behaviors (Hillebrandt et al., 2014; Koster-Hale and Saxe, 2013). With this in mind, atypical activation in the pSTS region in ASD suggests difficulty in predicting others' behaviors based on the past history of their behaviors, which is in line with a recent hypothesis suggesting that ASD can be conceptualized as a disorder of temporal prediction (Sinha et al, 2014). This hypothesis, known as Predictive Impairment in Autism (PIA), posits that ASD may be associated with inaccuracies in estimating the state-to-state conditional probabilities from an observed temporal sequence. This PIA hypothesis is important because it provides a unified framework for understanding many different aspects of ASD, including insistence on sameness, sensory abnormalities, difficulties in interacting with dynamic objects, difficulties with theory of mind, and proficiency in areas that are strongly rule-based such as mathematics, music, and computers. Thus, an intriguing possibility is that the pSTS is involved in temporal integration of the key elements in the environment (e.g., for dynamic stimuli, in particular audio visual integration) and our model provides a novel avenue for further research to integrate the PIA hypothesis of ASD with possible neural-circuit-level underpinnings. For example, there is functional connectivity between the pSTS and the insula cortex (see Fig. 3(A)), which suggests a possible neural underpinning for the sensory abnormalities in ASD from the prediction error perspective.

It should be noted that temporal integration is related to but different from time perception. Whereas time perception is about the subjective experience of the duration of a succession of events (e.g., the subjective experience of how long it takes to drink a cup of water) (cf. Pöppel, 1978), temporal integration is about the process and the ability to construct and integrate information over time into a coherent whole so that one can understand and predict the happening of events over time (e.g., the ability to understand and predict what people will do next when they walk toward you and wave at you) (cf. Kilner et al, 2007; Koster-Hale and Saxe, 2013; Nakano et al., 2010; Stevenson et al., 2014). Time perception has also been suggested to be abnormal in ASD (Allman et al., 2011), related to motor planning, but associated with a set of brain regions (e.g., premotor cortex and intra-parietal sulcus) that does not seem to involve the pSTS region (Schubotz et al., 2000). While a full discussion is beyond the scope of this paper, it is important for future research to further integrate the relationship between the role of temporal integration in the pSTS region and its relevance to the PIA hypothesis.

6. Conclusion

In this paper, we reviewed and integrated three major neural systems underlying social information processing: social perception, action observation, and theory of mind, which have almost exclusively been studied in isolation. Comparing and synthesizing these three key systems, our integrative neural model not only summarizes the regions implicated in each neural system but also provides an overall map advancing the relationships among them. We also reviewed evidence of aberrant social information processing in ASD as defined by deficits in one or more of the regions summarized by the model. Through this review, the pSTS region is found to be located at the intersection of all systems and has functional connectivity to all other regions in the three systems. This integrative model has important implications for psychiatric research, particularly for disorders characterized by social deficits such as ASD. In particular, our model provides a neural-system-level map for further research to test the temporal predictive impairment hypothesis in ASD, particularly with respect to the temporal integration role of the pSTS region. Our model also suggests that the pSTS region may serve as a target for interventions designed to improve social functioning in ASD (e.g., Ventola et al., 2014). More broadly speaking, the model provides a crucial direction for future research to further identify deficits that involve one or more regions that compromise the key systems involved in social information processing and may thus help to address a wide variation in the clinical phenotype of neuropsychiatric social disorders.

Appendix

The following Python code is what we used to obtain the maps in Fig. 2(A)-(C). To run this code, one would need to install the core Neurosynth tools first (see https://github.com/neurosynth/neurosynth for details).

from neurosynth.base.dataset import Dataset

from neurosynth.analysis import meta

# see the tutorial in https://github.com/neurosynth/neurosynth for details

dataset = Dataset.load(‘dataset.pkl’)

# or, alternatively,

#dataset = Dataset(‘data/database.txt’)

#dataset.add_features(‘data/features.txt’)

##########################################################################

# social AND perception

ids_social = dataset.get_ids_by_features(‘social’, threshold=0.001)

ids_social_len = len(ids_social)

print ‘There are {} studies associated with social.’.format(ids_social_len)

ids_perception = dataset.get_ids_by_features(‘perception’, threshold=0.001)

ids_perception_len = len(ids_perception)

print ‘There are {} studies associated with perception.’.format(ids_perception_len)

ids_social_AND_perception = list(set(ids_social) & set(ids_perception))

ids_social_AND_perception_len = len(ids_social_AND_perception)

print ‘There are {} studies associated with social AND perception.’.format(ids_social_AND_perception_len)

social_AND_perception_ma = meta.MetaAnalysis(dataset, ids_social_AND_perception, q=0.01, min_studies=1)

social_AND_perception_ma.save_results(‘.’, prefix= ‘social_AND_perception’, image_list= ‘pFgA_z_FDR_0.01’)

# the key output file will be social_AND_perception_pFgA_z_FDR_0.01.nii.gz

##########################################################################

# action observation

ids_action_observation = dataset.get_ids_by_features(‘action observation’, threshold=0.001)

ids_action_observation_len = len(ids_action_observation)

print ‘There are {} studies associated with action observation.’.format(ids_action_observation_len)

action_observation_ma = meta.MetaAnalysis(dataset, ids_action_observation, q=0.01, min_studies=1)

action_observation_ma.save_results(‘.’, prefix= ‘action_observation’, image_list= ‘pFgA_z_FDR_0.01’)

# the key output file will be action_observation_pFgA_z_FDR_0.01.nii.gz

##########################################################################

# theory of mind OR mentalizing

ids_theory_mind = dataset.get_ids_by_features(‘theory mind’, threshold=0.001)

ids_theory_mind_len = len(ids_theory_mind)

print ‘There are {} studies associated with theory mind.’.format(ids_theory_mind_len)

ids_mentalizing = dataset.get_ids_by_features(‘mentalizing’, threshold=0.001)

ids_mentalizing_len = len(ids_mentalizing)

print ‘There are {} studies associated with mentalizing.’.format(ids_mentalizing_len)

ids_theory_mind_OR_mentalizing = list(set(ids_theory_mind) | set(ids_mentalizing))

ids_theory_mind_OR_mentalizing_len = len(ids_theory_mind_OR_mentalizing)

print ‘There are {} studies associated with theory mind OR mentalizing.’.format(ids_theory_mind_OR_mentalizing_len)

theory_mind_OR_mentalizing_ma = meta.MetaAnalysis(dataset, ids_theory_mind_OR_mentalizing, q=0.01, min_studies=1)

theory_mind_OR_mentalizing_ma.save_results(‘.’, prefix= ‘theory_mind_OR_mentalizing’, image_list= ‘pFgA_z_FDR_0.01’)

# the key output file will be theory_mind_OR_mentalizing_pFgA_z_FDR_0.01.nii.gz