Social Cognition in Context: A Naturalistic Imaging Approach

Abstract

Social processing occurs within dynamic, complex, and multimodal contexts, but the study of social cognition typically involves static, artificial stimuli. Naturalistic approaches (e.g., movie viewing) can recapture the richness and complexity of real-world interactions. Novel analytic approaches allow for the investigation of functional brain organization in response to contextually embedded and extended events with a complex temporal structure during movie viewing or narrative processing. In addition to these within-brain measures, movies afford between-brain analyses such as inter-subject correlation, which allows for identification of stimulus-specific brain response through the correlation of brain activity between participants’ brains. Research using these approaches offers both practical and theoretical advantages in understanding how we navigate our social world. Practically, movies are engaging stimuli that allow for more rapid presentation of multiple event types and improve compliance even in very young populations. Theoretically, studies have validated the use of these measures by demonstrating functional selectivity to contextually embedded stimuli. Naturalistic approaches also allow for novel insights. For example, regions associated with social cognition have longer temporal receptive windows, making them well suited to social-cognitive processes that require integration of information over longer timescales. Furthermore, the similarity in the temporal and spatial brain response between individuals during naturalistic viewing is related to age, predictive of friendships, and reduced in autism spectrum disorder. These findings offer first glimpses into the power of using these naturalistic, dynamic approaches to understand how we perceive, reason about, and interact with others.

Cognitive neuroscience is built on the tradition of reducing complex phenomena down to their simplest form, which generally entails using static, unimodal, and artificial stimuli. This highly controlled experimental approach has provided major insights into our understanding of cognitive processing and functional brain organization (e.g., Mook, 1983; Rust & Movshon, 2005). However, even at the level of early visual cortex, the brain may respond differently during natural compared to artificial contexts (review, Hasson, Malach, & Heeger, 2010), and this disconnect between experimental stimuli and real-world complexity poses threats to the validity and generalizability of what we can learn from artificial approaches (Gastaut & Bert, 1954; Nastase, Gazzola, Hasson, & Keysers, 2019; Risko, Laidlaw, Freeth, Foulsham, & Kingstone, 2012). One domain in which these discrepancies may be most apparent is social processing (Redcay & Schilbach, 2019; Risko et al., 2012; Schilbach et al., 2013). Humans navigate the social world through a continuous and dynamic stream of multi-modal, contextually embedded, and interactive social experiences. Here we review studies that leverage novel neuroimaging approaches, with a focus on fMRI, to examine social cognitive processes in dynamic, temporally-extended, naturalistic contexts (e.g., movie viewing, narrative processing).

These naturalistic approaches not only satisfy appeals to ecological validity but also offer both practical and theoretical advantages. Practically, movies are engaging and thus can elicit greater attention and compliance among participants (Vanderwal, Eilbott, & Castellanos, 2019). They also are more suitable for young children and patient populations for which traditional tasks would be too challenging (Moraczewski, Chen, & Redcay, 2018; Richardson, Lisandrelli, Riobueno-Naylor, & Saxe, 2018; Vanderwal et al., 2019). In addition to the theoretical importance of ecological validity described above, naturalistic viewing/listening approaches afford the analysis of temporal structure, which may be critical to understanding complex social phenomena (Hasson & Honey, 2012). Further, naturalistic viewing may provide advantages over task or resting-state data in identifying individual differences (Finn et al., 2017, 2016; Vanderwal, Finn, Craddock, Turnbull, & Castellanos, 2017). In the current review, we first describe approaches that can be used to examine social-cognitive processing within naturalistic, dynamic contexts. We specifically focus on relatively novel approaches that use complex, contextually-embedded and temporally extended stimuli such as movies or extended narratives. We then highlight how these approaches have informed our understanding of social cognition both in typical and atypical development, as well as open questions and future directions.

Naturalistic imaging approaches

While naturalistic imaging offers many benefits in terms of ecological validity, the unconstrained nature of the stimuli demands novel analysis techniques that go beyond those typically used for tasks examining discrete, isolated events. In this section we discuss current theoretical and analytical approaches to dynamic, naturalistic stimuli. We divide these into categories based on whether analyses are conducted within individuals (within brain) or between individuals (between brains) (Figure 1).

Figure 1. Analysis approaches for naturalistic imaging paradigms.

Approaches are divided by those examining one brain (within brain, left panel) or multiple brains (between brains, right panel) while watching a dynamic movie or listening to a narrative. Each analysis type is discussed in detail in the text. Circles on brain images represent regions of interest. Still frames represent frames viewed during a movie.

Within brain.

Event Coding.

A naturalistic stimulus such as a movie presents multiple stimulus types (e.g., humans, faces, buildings, scenes) that are embedded within a complex and dynamic audiovisual array. One approach to examine how the brain processes these contextually embedded stimulus types is to examine brain response time-locked to a specific presentation of a stimulus. Event vectors of the occurrence of a given stimulus class across time can be created in either a binary (as in Jacoby, Bruneau, Koster-Hale, & Saxe, 2016) or continuous manner using an amalgamation of multiple independent raters (as in Lahnakoski et al., 2012; Pantelis, Byrge, Tyszka, Adolphs, & Kennedy, 2015) or through an automated image classification algorithm (e.g., Mcnamara, 2017). These event-coded vectors are subsequently used in a GLM to examine brain regions that show greater activation for one stimulus class over another. Although naturalistic stimuli are complex and dynamic, event coding has shown a high degree of selectivity and similarity to previous, more constrained tasks. For example, Bartels & Zeki (2004) used a parametric modulator of the rated intensity of experience for colors, faces, bodies, and language during naturalistic viewing of a James Bond film (Bartels & Zeki, 2004). This analysis revealed specialized areas of interest and a high degree of selectivity for the various stimulus categories. Further validating previous constrained and task-based studies, Lahnakoski and colleagues (2012) used naturalistic viewing to reveal the superior temporal sulcus (STS) as a hub for processing social stimuli. Independent viewers continuously rated fourteen different classes of stimuli (8 social; 6 nonsocial) during a 25-minute naturalistic stimulus. The STS responded similarly to all social categories, and response in this region was most selective for social vs. non-social stimulus classes. Thus, event coding of naturalistic stimuli offers a robust method to examine the relation between brain response and specific events within a naturalistic stimulus. However, by definition this method parses events into discrete categories, which may abstract from the complexity of real-world stimuli.

Reverse correlation.

In contrast to event coding, reverse correlation takes a set of a priori regions (e.g., regions identified in a meta-analysis) to identify events in the movie for which signal in that region was high. For example, a localizer-defined fusiform face region demonstrated significant peaks within the BOLD timecourse for movie events that contained faces, whereas the collateral sulcus showed selectivity for places (Hasson, Nir, Levy, Fuhrmann, & Malach, 2004). More recently, Wagner and colleagues (2016) found that peak response within the dorsal medial prefrontal cortex (DMPFC) corresponded to events with two people engaged in interaction. This inference was validated by naïve observers who rated these peak events as high in social complexity (Wagner, Kelley, Haxby, & Heatherton, 2016). Reverse correlation can also be used for more abstract stimulus classes, such as mental state attribution (Richardson, 2019; Richardson et al., 2018). While this method does not require prior event coding, the relation of peak response within a region to specific content within movie clips can be prone to reverse inference error and should be validated by independent raters of the movie’s content.

Intra-subject functional connectivity.

While event coding and reverse correlation seek to relate brain response to specific time-locked events, other approaches examine the BOLD signal across an entire movie. For example, task-independent functional connectivity measures examine the correlations of BOLD signal between regions within each participant. Regions of interest or “nodes” can be identified using a priori defined seed regions or through data-driven approaches such as independent component analysis (ICA) (Calhoun, Adali, Pearlson, & Pekar, 2001). While functional connectivity is typically assessed during a “resting-state” – and thought to reflect task-independent intrinsic functional organization (van den Heuvel & Hulshoff Pol, 2010) - comparisons of functional connectivity during rest to movie viewing reveal within- and between-network differences (Emerson, Short, Lin, Gilmore, & Gao, 2015; Vanderwal et al., 2017). For example, mean connectivity within the visual, somatomotor, ventral attention, fronto-parietal, and default mode networks all differ between naturalistic viewing and rest (Vanderwal et al., 2019; Vanderwal, Kelly, Eilbott, Mayes, & Castellanos, 2015). Further, during naturalistic viewing, the fronto-parietal control network shows greater anti-correlation with the dorsal attention network and greater connectivity to the default mode network (DMN) relative to rest (Gao & Lin, 2012). In addition, functional connectivity may be more reliable during movie viewing compared to rest (Finn et al., 2017; Vanderwal et al., 2017) and show greater functional specificity (Bartels & Zeki 2005). Thus, intra-subject functional connectivity methods during movie viewing can reveal novel aspects of both intrinsic and extrinsic functional brain organization (Golland et al., 2007). However, this method in and of itself cannot uncover how functional organization relates to specific aspects of the stimulus, and stimulus-specific response is confounded with an individual’s intrinsic, baseline response (Simony et al., 2016).

Frequency analysis.

The spatiotemporal profile of cortical activity can provide functionally relevant insight into brain organization and information processing (Buzsaki & Draguhn, 2004; Wu, Huang, & Zhang, 2008). A frequency analysis approach examines these cortical dynamics, specifically by examining a region’s frequency profile. While neural dynamics occur at much higher frequencies compared to the BOLD signal, previous work has demonstrated a coherence of such dynamics across both the electrophysiological and BOLD frequency spectra (He, Zempel, Snyder, & Raichle, 2010). At rest, regions of the DMN—a network of regions showing higher activity at rest and during internally directed or social tasks (Gusnard & Raichle, 2001)—spontaneously exhibit greater power within low-frequency bands (e.g., 0.01 – 0.08 Hz) relative to other cortical networks (Zou, Zhu, Yang, Zuo, & Long, 2008). Differences in frequency-specific power can be leveraged during naturalistic viewing to examine how a region’s dynamics relate to information processing. Electrocorticography (Honey et al., 2012) and fMRI (Moraczewski, Nketia, & Redcay, 2019; Stephens, Honey, & Hasson, 2013) have revealed a cortical dynamic gradient such that regions further from primary sensory regions display greater low-frequency power, and this greater power is associated with greater sensitivity to integration of information over longer timescales. This approach is ideally suited to examine intra-regional differences in cortical dynamics and should be used in conjunction with other measures that relate neural processing to the stimulus. It is also important to note that the frequencies discussed and identified here as “short” are relatively fast when considering the full spatiotemporal profile of cortical activity (Buzsaki & Draguhn, 2004) and use of other neuroimaging methods such as EEG and MEG can provide greater insight into these higher frequency rhythms.

Between-brains.

Inter-subject correlation.

Inter-subject correlation (ISC, or neural synchrony) correlates the brain response between participants within the same region while both participants view the same time-locked stimuli either through a Pearson’s correlation (Hasson et al., 2004) or measures of phase synchrony between timecourses (Glerean, Salmi, Lahnakoski, Jääskeläinen, & Sams, 2012). The BOLD timecourse contains both stimulus-evoked and intrinsic responses (Golland et al., 2007). By correlating responses across brains, the stimulus-relevant signal can be isolated and is found within lower-order (i.e., sensory) and higher-order (i.e., cognitive) brain regions (Hasson et al., 2010, 2004). This method offers important different and complementary information beyond event-related averaging approaches. In particular, event-related designs typically assume (or estimate) a hemodynamic response model for each event which can be problematic for longer events, such as narratives, in which regional activation fluctuates across the event and is dependent on prior context (Ben-Yakov, Honey, Lerner, & Hasson, 2012; Wilson, Molnar-Szakacs, & Iacoboni, 2008). Thus, event-related averaging could obscure activation or deactivation that is present but for which the temporal structure is more complex. For example, when participants listened to auditory narratives, averaging BOLD signal across sentences (i.e., each sentence is an event) led to a near zero response within DMN regions, but ISC demonstrated high reliability within these regions, which could reflect transient activation (e.g. for parts with greater semantic demand) or deactivation (e.g., for less engaging portions) within the narratives (Ben-Yakov et al., 2012; Wilson et al., 2008). Further, This method has been used extensively to probe general inter-subject functional organization and between-group differences (for a recent review, see Nastase et al., in press). More recent work has sought to examine how individual differences in neural synchrony is related to age (Moraczewski et al., 2018), trait paranoia (Finn, Corlett, Chen, Bandettini, & Constable, 2018), and friendship (Parkinson, Kleinbaum, & Wheatley, 2018). While ideally suited to examine common modes of processing within one brain region between subjects, an individual’s neural synchrony measure is dependent on the entire sample, which complicates interpretations of individual differences analyses.

Inter-subject functional connectivity.

While ISC examines the reliability of the BOLD signal between participants within one brain region, this method can be expanded to examine correlations between multiple brain regions, a method known as inter-subject functional connectivity (ISFC) (Simony et al., 2016). Here the BOLD timeseries within one region is correlated with the response from multiple a priori regions in another subject (Chen et al., 2016) or for all pairwise regional correlations between subjects (Simony et al., 2016). As such, this approach can address network-based questions using dynamic, naturalistic stimuli. For example, one study found that inter-subject functional connectivity between the hippocampus and regions of the DMN is greater during movie viewing in participants who had received prior context the day before (Chen et al., 2016). Further, greater inter-subject functional connectivity among regions of the DMN can predict better memory of a movie clip (Simony et al., 2016). Unlike intra-subject functional connectivity, inter-subject connectivity can disentangle the stimulus-induced response from idiosyncratic response (in terms of both an individual’s baseline intrinsic connectivity and non-neuronal signal).

A related, data-driven method to identify common functional networks across participants is independent components analysis (ICA). Group ICA methods identify spatially independent components that are common across participants. These components can be probed further using methods described previously, for example, by comparing the strength of within-component connectivity between conditions or mapping the component timecourses back onto conditions or events of interest in the naturalistic stimulus. This model-free, data-driven approach is ideal for use in complex, naturalistic stimuli as it attempts to identify distinct sources for multiple co-occurring processes or stimuli, many of which may be temporally extended (Calhoun & Pearlson, 2012; Kauttonen, Hlushchuk, & Tikka, 2015; Lahnakoski et al., 2012; Malinen, Hlushchuk, & Hari, 2007; Wolf, Dziobek, & Heekeren, 2010).

Representation similarity.

Two limitations of ISC and ISFC are that 1) these methods require that the stimulus is time-locked between participants and 2) spatial patterns are typically smoothed, which may obscure important spatial pattern information (Haxby et al., 2001). An alternative between-brain method is to average a voxel’s BOLD response across a meaningful unit of time common to participants (e.g., viewing a scene and/or retelling a specific event in a story), thus relaxing the constraint of temporal alignment. Then, spatial patterns of responses within a voxel searchlight for this period can be directly compared, either through a Pearson’s correlation of the pattern vector (J Chen et al., 2017) or using a multivariate representational similarity analysis (Zadbood, Chen, Leong, Norman, & Hasson, 2017). For example, Chen and colleagues (2016) found that regions of the DMN display similar response patterns while subjects recalled the same movie events. Spatial response patterns during movie viewing can also predict between-subject patterns during recall (Chen et al., 2017) and while a naive subject listens to another subject recalling an event (Zadbood, Chen, Leong, Norman, & Hasson, 2017). The approach offers researchers the ability to examine how common representations in higher-order cortices are reinstated during recall, transferred between individuals, and how pattern similarity corresponds to behavioral abilities. While an advantage of this method is that the analysis is no longer relegated to time-locked stimuli, averaging responses across time removes information about the dynamic temporal variation within a given event.

What have we learned about social cognition?

The neural correlates of social cognition are typically examined using constrained, well-controlled tasks that elicit mentalizing (making inferences or reasoning about a person’s mental state such as their thoughts, beliefs, or emotions or making personality judgments) compared to a control condition with similar cognitive demands (Schurz, Radua, Aichhorn, Richlan, & Perner, 2014). This body of work has revealed a consistent set of regions associated with mentalizing, including bilateral temporo-parietal junction (TPJ), anterior and posterior STS, ventral and dorsal medial prefrontal cortex (VMPFC, DMPFC), precuneus, and inferior frontal gyrus (Schurz et al., 2014), which we refer to as the mentalizing network – a network that is largely overlapping with the DMN (Mars et al., 2012; Schilbach, Eickhoff, Rotarska-Jagiela, Fink, & Vogeley, 2008). However, these tasks may not fully reflect the way that mental states are attributed in real-world social processing because they generally are not embedded within a complex and temporally-extended context and are conducted in a minimal social context (Redcay & Schilbach, 2019; Risko et al., 2012; Schilbach et al., 2013; Zaki & Ochsner, 2009). While this literature is still relatively small, below we describe the advantages of using naturalistic measures to provide insight into social cognition using more complex measures that go beyond traditional task-based approaches.

Mentalizing events during a movie recruit similar brain regions as task.

Perhaps surprisingly, even abstract and temporally extended processes such as mental state attribution can be identified in movies using event coding or reverse correlation methods, and these processes demonstrate similar brain functional specialization as during event-based tasks. For example, Jacoby et al. (2016) introduce a movie “localizer” (a five-minute clip of Pixar’s “Partly Cloudy”) in which events associated with mentalizing (i.e., events which elicit thinking about a character’s thoughts), social processing (i.e., characters interact with no mental/emotional representations), pain (i.e., a character experiences physical pain), and no character related events (“control”) are identified and used as regressors in a whole-brain analysis. In a comprehensive series of analyses, they demonstrate significant overlap between effects of mental compared to pain events in the movie localizer with two other common mentalizing localizers (a task of judgements of true or false beliefs compared to photos or reality and a task of judging emotional compared to physical pain). A major advantage of a movie localizer is that multiple processes can be elicited through a short and engaging clip as compared to standard localizers, which are longer and limited to one process (e.g., mentalizing). This advantage lends itself well to developmental studies involving young children. Indeed, this movie localizer has been used to examine the development of mentalizing brain regions in children as young as 3 years (Richardson, 2019; Richardson et al., 2018). Using a reverse correlation approach, Richardson et al. (2018) identified events within the movie that corresponded to peak responses within the mentalizing network (regions of which were independently defined a priori from a localizer task). As predicted, these events contained scenes that invited reflection of a character’s mental state. Further, response magnitude during the scenes that required an implicit revision of beliefs based on new information was related to behavioral measures of children’s mentalizing abilities. These findings were confirmed in a separate sample with a novel movie task (Richardson, 2019).

Here we have described cases where the brain response during movie viewing is similar to constrained social tasks, and thus the motivation to use naturalistic stimuli resides primarily in its practical advantages (i.e., engaging and shorter to administer). However, the use of complex, dynamic stimuli to assess “real-world” social cognition may be particularly important in conditions of atypical social processing, such as autism. While autism is characterized by difficulties with social interaction, empirical lab-based tasks of social cognition often fail to find significant differences between groups (Dufour et al., 2013). This discrepancy may be due to the decontextualized, simplified version of social processing inherent in typical tasks of social cognition (e.g., Pantelis et al., 2015). To recapture this complexity, Pantelis et al., (2015) had participants freely view scenes from the TV show “The Office” and used a regressor for level of awkwardness to identify regions associated with the perception of socially awkward moments. Significantly less activity was seen within the RTPJ in autistic participants compared to neurotypical participants during awkward moments, suggesting this region may contribute to the atypical perceptions of social interaction seen in autistic individuals. An alternative interpretation is that differences between groups are seen because naturalistic approaches, like the one discussed here, relies on passive movie viewing. In contrast, the task-based studies which have not found differences between autistic and neurotypical individuals (e.g., Dufour et al., 2013) have required explicit processing of a character’s mental states. Thus, differences between groups could either be due to a reduced ability to perceive awkwardness in social interactions or to reduced spontaneous attention or reflection on the character’s mental states. Thus, a combination of both task-based and naturalistic approaches are important to understand typical and atypical social-cognitive processing.

Functional connectivity during movie viewing offers novel insights into mentalizing.

Functional connectivity can also provide insights into functional specialization at the network level. For example, young children demonstrate functional connectivity within regions of the mentalizing networks and within regions of the pain network. However, the connectivity between regions of these different networks is low (i.e., the networks are anti-correlated) and these anti-correlations increase with age. Further, connectivity within the mentalizing network is positively related to mentalizing ability (Richardson et al., 2018; Richardson, 2019). While these correlations within and between the mentalizing and pain networks are similar to what is seen during a resting state (i.e., viewing a fixation cross), only connectivity during the movie task predicts functional maturity of the neural response, defined as the similarity of each child’s time course to a group of adults (Richardson et al., 2019). These findings suggest a more functionally relevant index of mentalizing may be connectivity of the mentalizing network during movie viewing as opposed to rest.

Data-driven approaches may reveal a more distributed engagement of regions when making mentalizing judgements in a naturalistic context. For example, Wolf, Dziobek, and Heekeren (2010) used ICA methods and identified three functional networks associated with making mentalizing judgements (compared to judgments about physical characteristics of the scene) while watching a movie or answering explicit questions. These networks extended beyond traditional mentalizing networks, encompassing regions associated with motor movements, language, and emotion processing. Further, while the three networks were identified across movie and question portions of the task, only one network differentiated mental from physical inferences during the movie (whereas all three differentiated mental from physical in the explicit questions). This reduced difference in brain response during mental and physical conditions in the movie may be due to the presence of spontaneous mentalizing during the movie, but not during the explicit task. Thus, movie viewing may elicit a broader network than previously identified. This broader network may be due to the presence of cognitive processes occurring within a rich social and temporal context (i.e., multiple stimuli on the screen at once and prior events influence current interpretations). Because this type of presentation is more consistent with social processing in the real world the broader network may be more characteristic of how the brain processes real-world social stimuli. Alternatively, engagement of a broader network during movie viewing could simply the concurrent engagement of multiple simultaneous processes that could each be independently identified in more controlled tasks. This question of whether one can effectively isolate each cognitive or perceptual process and sum them together to understand how the brain is engaged in a rich, complex naturalistic stimulus is an important one for future research.

Mentalizing regions operate on long timescales.

Social cognition in our everyday interactions operates on the scale of seconds (a first impression), minutes (a conversation), hours (dinner with a friend), or months/years (developing a friendship). However, the study of the neural bases of social cognition is typically restricted to events lasting seconds or less. However, the contextual information acquired prior to those few seconds may be critical in correctly interpreting a social partner’s acts (e.g., Koster-Hale & Saxe, 2013). Naturalistic approaches of movie viewing or narrative offer advantages over task-based approaches in the study of social cognition because they afford the opportunity to investigate processes in which the temporal profile may be more important than average response magnitude (Hasson, Yang, Vallines, Heeger, & Rubin, 2008). Pioneering studies by Uri Hasson and colleagues (Hasson, Chen, & Honey, 2015; Hasson et al., 2008; Honey et al., 2012; Lerner, Honey, Silbert, & Hasson, 2011) demonstrate that regions of the DMN have long temporal receptive windows; in other words, the timescale at which they can integrate past information to affect the current response is relatively long. For example, these regions show greater response reliability across people (i.e., ISC) for processing paragraphs than sentences or words, whereas “medium” timescale regions are more reliable for sentences than words (Lerner et al., 2011). Similarly, DMN regions are more reliable for full video clips than 10–20 second scenes presented in scrambled order within those video clips (Hasson et al., 2008; Lerner et al., 2011). These regions also demonstrate greater power in low than high frequencies (Honey et al., 2012; Moraczewski et al., 2019; Stephens et al., 2013). Relatively higher or lower frequencies across the cortex are thought to reflect regional differences in recurrent excitation (Chaudhuri, Knoblauch, Gariel, Kennedy, & Wang, 2015), which enables a temporal processing hierarchy organized along a functional gradient from unimodal to transmodal representation (Huntenburg, Bazin, & Margulies, 2018). Regions dominated by lower frequency power may reflect a wider window within which to integrate incoming information.

These long timescale regions are thus well-suited to support the creation of a model of a person or event and the integration of new information into this model (Hasson et al., 2015). Creation of these “situation models,” for example, an internal model of a story character, facilitate story comprehension (Rapp, Gerrig, & Prentice, 2001). Long timescale regions may be important in constructing these situation models, as activity within those regions is related to narrative comprehension (Chen et al., 2016; Chen et al., 2017; Simony et al., 2016). While these situation models are created for both social and nonsocial aspects of narratives or episodes, the interactions among characters and tracking of each character’s knowledge are prominent features in and often core to understanding a story (e.g., Mar, 2018); thus, the social content may be an important contributor to long timescale activity. In fact, when presented without social content, coherent narratives elicit greater response than incoherent narratives in MPFC but not TPJ, suggesting that the findings of TPJ engagement for long timescale processing may be specific to social content (Jacoby & Fedorenko, 2018).

The extent to which the integration of social-specific information drives the greater response reliability at long timescales in mentalizing regions is not clear because previous studies in this domain have typically not addressed the link between temporal receptive windows and social cognition. However, some work has begun to examine the relation between long timescale processing within mentalizing regions and social cognition (Lahnakoski, Jääskeläinen, Sams, & Nummenmaa, 2017; Moraczewski et al., 2019). Lahnakoski et al. (2017) identified events that showed higher response reliability during consecutive compared to interleaved clips. They showed that higher ISCs in DMPFC and posterior STS were related to events with speech as well as events containing deception or conflict, suggestive of a link between social-cognitive judgments and long timescale processing. Additionally, in a study that directly investigated this question, we demonstrated that temporally scrambling clips from a children’s TV show led to worse performance on social-cognitive comprehension questions than general comprehension questions about events in that clip in children (Moraczewski et al., 2019). Further, an adult-like neural response to the intact movie in the DMPFC was related to social-cognitive compared to general comprehension (Moraczewski et al., 2019). These data suggest that incorporating information over a long temporal receptive window may be important for social-cognitive judgments beyond general narrative comprehension. However, the mechanistic relation between long timescales and social cognition remains an open question. We suggest that this question will be best addressed using a developmental perspective to probe whether differences in long timescale processing are predictive of social-cognitive advances. Importantly, while we focus on the role of long timescale processing in social cognition, short timescales are involved in the moment-to-moment turn taking and synchrony that occurs between social partners. As such these features are best addressed by methods with greater temporal resolution such as MEG and EEG (e.g., Hari & Parkkonen, 2015; Levinson, 2016).

Mentalizing brain regions engage in predictive processing.

A creative use of naturalistic paradigms and analyses is to replay a movie or auditory narrative to participants to examine how the brain anticipates or predicts upcoming events based on prior experience. These studies have highlighted that regions of the “social brain” (e.g., mpfc, precuneus, and TPJ) represent or respond to events in a movie earlier when the movie is repeated. For example, Baldassano et al., (2017) play an auditory narrative of a movie to participants who have already seen the movie and demonstrate similarity in the spatial representation for events is temporally shifted earlier when listening to the narrative (i.e., the repeated condition). Richardson & Saxe (2019) demonstrate that predictive responses during repeated viewing of a movie are specific to regions identified as relevant to theory of mind processes¹ but not pain, supporting the theory that these regions support social-cognitive processes such as theory of mind through prediction (Koster-Hale & Saxe, 2013).

Shared perspectives systematically alter the response in mentalizing regions across people.

How we judge another’s actions and infer their mental state will depend on our own knowledge and beliefs. For example, if you think your partner is cheating, you might interpret a late night at work differently than if you have no suspicion. Naturalistic viewing studies have demonstrated that similarity in perspectives is related to similarity in spatial and temporal profiles of brain response across people (Ames, Honey, Chow, Todorov, & Hasson, 2015; Bacha-Trams et al., 2017; Lahnakoski et al., 2014; Nguyen, Vanderwal, & Hasson, 2019; Yeshurun et al., 2017). For example, pairwise ISC values discriminate between viewing the same movie as a detective compared to a decorator within the TPJ (Lahnakoski et al., 2014). In a more compelling example, listening to the same narrative in which the protagonist was previously described as either cheating or paranoid affects how viewers perceive the intentions, beliefs, and motives of the protagonists during the story (Y eshurun et al., 2017). The points in the story that were rated as maximally different in the interpretations of a character’s beliefs or emotions (based on context) were reflected in greater neural distance between participants who heard the different contexts. These differences were significant within regions associated with mentalizing (as confirmed with mentalizing localizers) including TPJ, STS, dMPFC, and precuneus. Thus, these regions are not just engaging in the process of mentalizing but also representing specific mental states in ways that are systematic across people with shared perspectives. One possibility is that shared neural representations may allow for more accurate or rapid social-cognitive judgments between social partners who share similar perspectives; however, to our knowledge this has not yet been empirically tested.

Emotions in context.

Naturalistic approaches are critical to understanding empathic processes because emotions unfold over time and demonstrate moment-to-moment changes (Cunningham, Dunfield, & Stillman, 2013; McMenamin, Langeslag, Sirbu, Padmala, & Pessoa, 2014; Nummenmaa et al., 2012; Zaki & Ochsner, 2009; Zaki, Weber, Bolger, & Ochsner, 2009). Naturalistic imaging work demonstrates that this temporal profile is important in understanding how we perceive emotion in ourselves and others and that temporal similarity in our judgments is supported (at least partially) by mentalizing brain regions. For example, Zaki et al. (2009) measure empathic accuracy by calculating the similarity (correlation) between the participant’s judgments of the target’s emotions with the target’s self-reported emotion over the course of the target describing an emotional autobiographical event. Higher empathic accuracy while watching videos of the target is related to activation within the mentalizing network (as well as regions associated with the mirror neuron system). Moreover, how the brain responds to emotions dynamically interacts with social context. For example, dynamic feedback of a perceived social partner’s emotions leads to greater neural similarity across participants (Golland, Levit-Binnun, Hendler, & Lerner, 2017). Further, neural similarity within the regions of the mentalizing network is greatest during highly negative scenes in a movie (Chang et al., 2018; Nummenmaa et al., 2012)

Neural similarity is important for social processing.

In this review, we’ve highlighted how intersubject synchrony (as well as other methods) can be used to study how individuals perceive and understand others in dynamic, naturalistic contexts. However, this synchrony, or similarity, across people itself is a social measure and may be important to social processing (Hasson & Frith, 2016; Nummenmaa, Lahnakoski, & Glerean, 2018). For example, neural similarity during movie viewing predicts the degree of closeness between participants within a social network (Parkinson et al., 2018), suggesting we like those whose brains are more similar to our own (or we become more similar to those we spend the most time with). Similarity is also important in communication. The neural timecourse from a person telling a story is predictive of the neural timecourse in a listener’s brain, and the extent of this similarity predicts comprehension (Stephens, Silbert, & Hasson, 2010). In another example, spatial similarity is seen between participants encoding a specific event in a movie, recalling that event verbally, or listening to spoken recall of that event even when looking at similarity across roles (i.e., viewer, speaker, or listener) (Zadbood et al., 2017). These findings suggest we hold abstract representations of events that are reliably similar across people and thus may facilitate alignment during communication. Notably, while these measures capture the similarity in brain response to the same fixed stimulus, real-time social interaction involves continuous, rapid changes between social partners based on each other’s actions. Capturing this level of rapid and dynamic change can be addressed using hyperscanning methods, particularly those with relatively high temporal resolution such as MEG or EEG (for reviews see: Babiloni & Astolfi, 2014; Hari & Kujala, 2009; Konvalinka & Roepstorff, 2012).

Neural similarity is reduced in atypical development.

Given that neural similarity (both temporal and spatial) may be important for alignment with a social partner, communication, and even friendships, individuals who have difficulties with social interaction and social processing may show reduced neural similarity, or more idiosyncratic neural responses (Cook, 2016). Indeed, when watching movies or television shows, autistic individuals show less neural similarity to neurotypical individuals or other autistic individuals (Byrge, Dubois, Tyszka, Adolphs, & Kennedy, 2015; Glerean et al., 2016; Hasson et al., 2009; Salmi et al., 2013). Importantly, this neural similarity is not due to an overall noisier neural response in the autistic group because similarity in neural responses within participant does not differ between autistic and neurotypical participants (Hasson et al., 2009). Further, the extent to which an individual’s neural response differs from the average neurotypical response is predictive of their social comprehension of the clip (Byrge et al., 2015); however, it is noteworthy that in the Byrge et al. study, five autistic individuals were driving the significant differences in autistic response reliability from the neurotypical average. These findings highlight that autistic brains respond to movies differently both from neurotypical brains and from each other. These findings of reduced neural similarity during movie viewing have also been shown in other patient populations such as those with first-episode psychosis (Mäntylä et al., 2018) or schizophrenia (Tu et al., 2019). It will be important for further work to better link behavioral characteristics with neural similarity rather than relying on diagnosis alone as individuals who are more similar may have more successful social interactions (Bolis, Balsters, Wenderoth, Becchio, & Schilbach, 2018; Cook, 2016). Thus, it is important to study social processing between neuroatypical and neurotypical individuals as well as between neuroatypical individuals and/or between individuals matched on similar behavioral, cognitive, and potentially even neural, characteristics.

Neural similarity increases with age.

While the majority of naturalistic viewing studies have been conducted in neurotypical adults, developmental studies demonstrate increasing neural similarity (Cantlon & Li, 2013; Moraczewski et al., 2018, 2019; Richardson, 2019; Richardson et al., 2018). Further, across different ages and movies, regions of the mentalizing network, including the TPJ, show significant increases with age (Moraczewski et al., 2018, 2019). This change may be due to the social content of the movies or the role of this network in processing long timescales, or both. In the first study to examine temporal hierarchies in children, we demonstrate that children show age-related differences in neural similarity to intact clips of a children’s television show compared to scrambled versions within regions associated with mentalizing (Moraczewski et al., 2019). Further, within these regions, adults demonstrate more power within lower frequencies (i.e., .01-.04 Hz) than children, suggesting developmental increases in the extent to which these regions accumulate information over longer timescales. A ripe area for future developmental research will be to examine the developmental relation between neural similarity and social development to disentangle causes and consequences of neural similarity.

Conclusions

Naturalistic imaging approaches using movies and stories offer practical advantages over traditional task-based approaches due to the engaging nature of the stimulus. Further, despite the concern that naturalistic approaches could limit the ability to isolate and manipulate specific processes, naturalistic viewing provides evidence for functional specialization of brain responses to different stimulus categories, even when presented in complex, multimodal, and dynamic contexts (Hasson & Honey, 2012). Moreover, accumulating evidence suggests that naturalistic approaches using complex and temporally-extended stimuli can provide novel, complementary insights into how we perceive, reason about, and interact with other social beings. We have highlighted two findings from naturalistic movie viewing and narrative approaches that promise to significantly inform social cognition and social behavior, namely: the long temporal receptive windows of mentalizing regions and the importance of neural similarity in social behavior. However, much further work is still required to determine whether a mechanistic relation exists between these neural signatures and social behavior.