Failure to engage the temporoparietal junction/posterior superior temporal sulcus predicts impaired naturalistic social cognition in schizophrenia

Abstract

Schizophrenia is associated with marked impairments in social cognition. However, the neural correlates of these deficits remain unclear. Here we use naturalistic stimuli to examine the role of the right temporoparietal junction/posterior superior temporal sulcus (TPJ-pSTS)—an integrative hub for the cortical networks pertinent to the understanding complex social situations—in social inference, a key component of social cognition, in schizophrenia. Twenty-seven schizophrenia participants and 21 healthy control subjects watched a clip of the film The Good, the Bad and the Ugly while high resolution multiband functional MRI images were collected. We used inter-subject correlation to measure the evoked activity, which we then compared to social cognition as measured by The Awareness of Social Inference Test (TASIT). We also compared between groups the TPJ-pSTS blood oxygen level-dependent activity (i) relationship with the motion content in the film; (ii) synchronization with other cortical areas involved in the viewing of the movie; and (iii) relationship with the frequency of saccades made during the movie. Activation deficits were greatest in middle TPJ (TPJm) and correlated significantly with impaired TASIT performance across groups. Follow-up analyses of the TPJ-pSTS revealed decreased synchronization with other cortical areas, decreased correlation with the motion content of the movie, and decreased correlation with the saccades made during the movie. The functional impairment of the TPJm, a hub area in the middle of the TPJ-pSTS, predicts deficits in social inference in schizophrenia participants by disrupting the integration of visual motion processing into the TPJ. This disrupted integration then affects the use of the TPJ to guide saccades during the visual scanning of the movie clip. These findings suggest that the TPJ may be a treatment target for improving deficits in a key component of social cognition in schizophrenia participants.

Social functioning deficits are pervasive and disabling in schizophrenia, but their neural substrates remain poorly understood. Patel et al. reveal how deficits within a social cognition hub unique to human cortex impair the ability of individuals with schizophrenia to follow complex dynamic social situations.

Introduction

Deficits in social cognition are a major source of disability in schizophrenia, but few if any targeted treatments for these deficits exist.^1,2 One of the barriers to developing these treatments is the sheer complexity of the neural systems underlying social cognition. Even the simplest of social behaviours involves multiple brain areas and networks interacting in concert.³ One of these simple but critical social cognitive processes is the ability to follow the interpersonal interactions between people in a social scene and make inferences about their underlying mental states. In order to follow these interactions, viewers must simultaneously search for and detect potentially useful social cues quickly. This exploration requires the fast integration of incoming visual information about potentially relevant social cues, such as facial expressions of emotion, with ongoing mentalization (theory of mind) operations in order to guide the eyes to these relevant cues by a saccadic eye movement. People viewing a dynamic social scene will loop through this input-output operation repeatedly as changes in facial expressions and other social cues signal the availability of new information about the mental states of people in the scene.4–6

This automated targeting of saccades to explore the social scene occurs multiple times per second without much conscious intervention during the visual scanning of a social scene, yet the areas involved in the underlying processes span all regions of the cortex: face emotion recognition involves a network of areas in occipital and temporal lobes⁷; visual scanning or attention involves a network of areas in parietal and prefrontal cortex⁴; and mentalization involves a network of areas temporal, parietal, and both medial and lateral prefrontal cortex that heavily overlaps the default mode network.⁵ Previous studies in patients with schizophrenia have found deficits distributed throughout all of these networks, but thus far no one common area or region has emerged to explain social functioning deficits in schizophrenia. In this study, we used a novel approach to understand the neural basis of these deficits: we used naturalistic stimuli designed to simultaneously activate all of the neural substrates across the brain underlying the visual scanning of social scenes to isolate these deficits relative to functionally defined brain areas, and then examined these deficits in relation to both overall social cognition and the underlying perceptual, cognitive, and behavioural processes. The goal of this study was to use functional neuroimaging to identify potential biomarkers of and treatment targets for social cognition deficits in schizophrenia participants.

One candidate for a common region is the temporoparietal junction/posterior superior temporal sulcus (TPJ-pSTS). The cortical networks underlying the three main operations in visual scanning have areas or nodes in the TPJ-pSTS making it a potential hub for social cognition.⁶ According to our model,⁶ the pSTS receives input about visual social cues, such as moving facial expressions, from visual areas such as those involved in motion processing (middle temporal area, or MT) and face processing (fusiform face area, or FFA).^7,8 This information is then relayed to the TPJ, divided into posterior (TPJp) and anterior (TPJa) areas. The relayed facial expression information is first compared with internally maintained models in the TPJp regarding the mental states of individuals in a social scene. These models are generated and maintained in coordination with medial and lateral prefrontal areas involved in mentalization.⁵ A mismatch between the incoming sensory information and the expectation generated by the model (prediction error) then activates the TPJa, an area in the ventral attention network.^4,9 This detection of potentially relevant new information by the ventral attention network then triggers (through prefrontal areas involved in cognitive control) the dorsal attention network, consisting of areas in the intraparietal sulcus (IPS) and the frontal eye fields (FEF), to shift attention to the relevant social cue with a saccadic eye movement for further scrutiny.⁴

In the middle of the TPJ-pSTS, Power et al.¹⁰ identified a potential hub for cognitive operations by its participation in multiple resting state networks, labelled here as TPJm. In the context of our model, this hub area may serve as the gateway through which the pSTS communicates with the TPJp and TPJa, thus serving as the critical link between the visual processing of facial expressions to mentalization and visual scanning. We hypothesized, then, that focal impairments within the TPJ and/or pSTS, particularly if in middle TPJ (TPJm), could be responsible for the wide-ranging social cognition deficits seen in schizophrenia participants.

Since the TPJ-pSTS is composed of many separate functional areas that need to coactivate during visual social inference, instead of using highly controlled stimuli designed to isolate and examine the functioning of a specific set of areas or a cognitive domain, we used a cinematic movie (with the audio track removed) to stimulate activity in all of these areas at once. Social inference is required to understand the mental states of the actors in the movie, which in turn is required to understand the plot of the movie clip. Removing the audio track forced viewers to rely on the TPJ-pSTS areas involved in performing social inference based on incoming visual information. We then compared the evoked activity between schizophrenia participants and demographically matched healthy control subjects. We used inter-subject correlation (ISC) to assess the functional integrity of the activated cortical areas. Instead of quantifying magnitude of blood oxygen level-dependent (BOLD) change as in traditional functional MRI analyses, ISC quantifies activation by the correlation in BOLD activity in a particular greyordinate or region of interest for each individual with the same greyordinate/region of interest across healthy control individuals. As a result, this method provides a measure of the functional integrity of each greyordinate/region of interest as compared to the ‘gold standard’ of the healthy controls without a priori bias about which cognitive processes or cortical areas may be affected in schizophrenia participants. However, while the ISC method has proven useful for identifying where a group difference in activation may exist, further mechanistic inference has been limited by the analytical methods available. Here, we developed or adapted a number of techniques to extend the utility of the approach.

One of the greatest challenges with regard to the TPJ-pSTS has been the difficulty in defining its functional subdivisions. The arrangement of these areas within the TPJ-pSTS appears to be unique to humans, making it difficult to infer its organization from studies in animal models.^6,11 Furthermore, the TPJ-pSTS is one of the most anatomically variable regions in the brain, making it difficult to study with standard neuroimaging methods.^12,13 Lastly, there is little agreement between parcellation schemes based on resting state functional connectivity, making it difficult to reliably identify areas within this region and match them to the functionally defined areas involved in the various social cognition operations.^6,14–17 To overcome these hurdles, we used task functional MRI-based localizers to subdivide the TPJ and pSTS and identify the functional areas described above. With the high-resolution multiband functional MRI sequences and cortical mapping methods pioneered by the Human Connectome Project,¹⁸ we were then able to use ISC to assess which TPJ-pSTS functional subdivisions were affected in schizophrenia participants.

We then examined how ISC of the TPJ-pSTS subdivisions is associated with the process of making social inferences from a dynamic naturalistic social scene as measured by the Awareness of Social Inference Test (TASIT),¹⁹ collected in a separate behavioural session. In TASIT, participants watch a series of short video clips of a social situation and answer questions about what the main character is thinking, feeling, saying and doing after each clip, similar to the operations the participants performed in order to follow the movie clip they viewed in the MRI scanner. In half of the trials, the main character is sarcastic with the other characters, which they convey through exaggerated facial expressions. In the other half of the trials, they lie to the other characters, during which their facial expressions are neutral. Making inferences about the main character’s mental state in the sarcasm videos, then, is more reliant on the ability to perceive facial expressions and integrate them into ongoing mentalization operations as compared to the lie trials, making them an ideal test of the integrity of the TPJ and pSTS. As discussed in more detail in a recently published article,²⁰ schizophrenia participants have more difficulty with the sarcasm trials than the lie trials, and this deficit is linked to an inability to appropriately use facial expressions of emotion. As a measure, TASIT performance is both reflective of mentalization, a key component of social cognition,²¹ and correlates with real-life functioning.^22,23 Correlating TASIT sarcasm performance with the ISC deficits evoked by the movie clip discussed above allowed us to directly link these deficits with the ability to make quick and accurate social inferences, an important aspect of naturalistic social cognition.²¹

Finally, we examined how TPJ-pSTS deficits may relate to the neural processes underlying the repeated transformation of visual features into a saccadic plan during the viewing of the movie. To study the neural activity related to the visual processing of the movie, we used recent computer vision advances to automatically map the time course of the visual features present in the movie, and then correlated these time courses with the time course of BOLD activity in each greyordinate.^24,25 To study the integrity of TPJ-pSTS communication with visual processing and saccade planning areas, we adapted a method of measuring interareal synchronization that assesses the integrity of how the TPJ-pSTS synchronizes with other areas without the potential confound of the other areas being functionally compromised.²⁶ To capture the activity of all of the areas involved in the transformation of the visual features into a saccadic plan, we searched each participant’s brain for areas whose activity co-varied with the frequency of saccades they performed while visual scanning the movie. We hypothesized that there would be convergent deficits within subdivisions of the TPJ and/or pSTS for activation by both motion and saccades and for synchronization with the other areas involved in visual processing and saccade planning.

Materials and methods

Participants

Twenty-seven schizophrenia participants and 21 healthy control subjects were recruited with informed consent in accordance with New York State Psychiatric Institute’s Institutional Review Board (IRB). All participants completed the movie-watching MRI portion of the study, and a subset of these participants (25 schizophrenia participants and 17 controls) also completed the behavioural session. Inclusion/exclusion criteria are provided in the Supplementary material.

Behavioural session and analyses

In the behavioural session, participants were seated in front of a computer monitor with their heads resting comfortably in a head-holder, and then performed TASIT part 3 section A.¹⁹ TASIT part 3 A consists of 16 videos (eight sarcasm and eight lie), each followed by four questions about what the main character was thinking, feeling, doing, and saying (32 sarcasm and 32 lie questions in total). TASIT performance was scored separately for sarcasm and lie trials as the percentage of questions answered correctly. These behavioural data are a subset of the data used in Patel et al.²⁰

MRI session

Participants viewed a video clip of the first 15 min of the cinematic movie The Good, the Bad and the Ugly²⁷ with the sound removed while simultaneous eye-tracking and BOLD data were collected. BOLD data were collected as one continuous 15-min acquisition (1049 functional MRI frames) using a multiband (MB) functional MRI sequence (2 mm isotropic, repetition time = 850 ms, MB factor 6). Structural T₁ and T₂ (0.8 mm isotropic), along with distortion correction scans (B0 field maps), were also acquired as required for use of the Human Connectome Project (HCP) processing pipelines.

Image processing

MRI data were preprocessed using the HCP pipelines v3.4,¹⁸ which places the data into greyordinates in a standardized surface atlas (as opposed to voxels in a volume atlas). The functional data were additionally cleaned of artefact largely following the recommendations from Power et al.²⁸ To equalize numbers of censored frames, we used an adaptive framewise displacement (FD) threshold that set the threshold as the 75%ile + 0.5 × interquartile range of the FD trace for each run, limited to a range between 0.2 mm and 0.5 mm. Numbers of frames censored did not differ significantly [control = 184.4(37.9), schizophrenia participants = 230.2(34.1), t(46) = 1.85, P = 0.07] and all results reported below were similar when analyses were repeated with a universal threshold of FD = 0.2 mm.

Inter-subject correlation analyses

ISC values were calculated for each participant as the pairwise correlation of the time course for each individual and the time courses of each healthy control participant on a greyordinate by greyordinate basis. For regions of interest ISC analyses, the same procedure was followed except time courses came from each region of interest, not each greyordinate.

Localizing regions of interest

TPJ-pSTS regions of interest were derived from three localizer tasks: (i) a task that contrasted activity evoked by moving versus static facial expressions, designed to localize areas involved in face-emotion recognition (similar to Fox et al.²⁹); (ii) a visual search task designed to activate areas involved in visual processing, visual attention, and cognitive control^11,30,31; and (iii) a task designed to activate areas involved in theory of mind operations evoked during the viewing of a short animated video.³² Details about these tasks and how they were used to define the regions of interest are provided in the Supplementary material.

TPJ-pSTS synchronization with other regions of interest

Group differences in the synchronization of the TPJ-pSTS areas with other movie-driven regions of interest were examined using a method adapted from the inter-subject functional correlation method described in Simony et al.²⁶ Reference regions of interest were chosen from cortical areas involved in important aspects of visual scanning of social scenes: face processing (right FFA), motion processing (right MT), and saccade planning (right FEF). For each reference region of interest, the average time course of activity was extracted for the healthy control subjects. This time course was then correlated to the time course of activity for each TPJ-pSTS region of interest in each individual for each group.

The healthy control time courses were used as the reference to remove the possibility that schizophrenia-related processing impairments within these other regions of interest could affect the measurement. One feature of this measure is that (unlike typical region of interest-region of interest functional connectivity measures) it is directional—for instance, healthy control V1 → schizophrenia participants V2 synchronization may be decreased compared to healthy control V1 → healthy control V2 synchronization, but that does not necessitate that healthy control V2 → schizophrenia participants V1 synchronization is also decreased compared to healthy control V2 → healthy control V1. Greyordinate maps were created for visualization by correlating the healthy control region of interest time courses from the reference regions of interest with all greyordinate time courses. In exploratory analyses, we also examined TPJ-pSTS synchronization with other cortical regions of interest involved in social cognition.

Visual feature and saccade correlations with BOLD activity

Group differences in activity evoked by visual features and saccades were examined by correlating their time courses with BOLD activity for each TPJ region of interest. The time course of the strength of low-level visual features (motion, contrast, luminance) were extracted from the movie as described in Russ et al.²⁴ with the exception of log-transformation of the motion parameter for normalization. Automated face detection (http://aws.amazon.com/rekognition) was used to score each functional MRI frame for the number of faces. Automated saccade detection (SR Research, Mississauga, Ontario, Canada) was used to score each functional MRI frame for the number of saccades. These continuous measures were convolved with a haemodynamic response function,³³ downsampled to the functional MRI frame rate (for the visual feature regressors), and then correlated to the time course of activity for each region of interest or greyordinate.

Group comparison statistics

For all relevant analyses, schizophrenia participants were compared to all healthy controls, and controls were compared to all other healthy controls excluding themselves. Correlation values were Fisher z-transformed, and then compared between groups using repeated measures ANOVAs and post hoc t-tests. All values are reported as mean (confidence interval). Greyordinate-wise ISC contrast was calculated using mixed-effects in FEAT³⁴ and cluster inference threshold set at P < 0.05 by PALM.³⁵ Violin plots produced in MATLAB with ViolinPlot-Matlab extension (https://github.com/bastibe/Violinplot-Matlab).

Further details are available in the Supplementary material.

Data availability

Data will be made available following reasonable request to the corresponding author.

Results

Demographics

Schizophrenia participants and healthy controls were demographically similar in age, gender, race/ethnicity, education, socioeconomic status, handedness, and IQ (Table 1). Schizophrenia participants were medicated [mean chlorpromazine (CPZ) dose = 691.9 (1034.5) mg] and endorsed mild-to-moderate symptom severity [Positive and Negative Syndrome Scale (PANSS) = 58.4 (14.9)].

Table 1.

Patient demographics and performance

Demographics/performance	Schizophrenia (n = 27)	Healthy controls (n = 21)	Statistics
Age, years	38.6 (11.7)	34.5 (9.6)	t(49) = 1.30, P = 0.20
Gender, male/female	19/8	10/11	χ2 = 2.6, P = 0.11
Race/ethnicity
%White	33	38.1	χ2 = 0.11, P = 0.73
%Black	40.7	42.9	χ2 = 0.02, P = 0.88
%Hispanic	25.9	14.3	χ2 = 0.97, P = 0.32
Participant education, years	14.4 (3.0)	15.2 (1.9)	t(49) = −0.11, P = 0.31
Participant SES	33.2 (14.4)	38.2 (12.4)	t(49) = −1.04, P = 0.21
Edinburgh Handedness Score	17.3 (4.7)	14.8 (6.5)	t(49) = 1.38, P = 0.18
IQ (WRAT scaled score)	97.3 (13.0)	100.4 (14.6)	t(49) = −0.70, P = 0.49
PANSS Positive Symptoms	15.9 (5.5)	–	–
PANSS Negative Symptoms	13.5 (4.4)	–	–
PANSS Total Scores	58.4 (14.9)	–	–
Antipsychotic dose, CPZ equivalents, mg	691.9 (1034.5)	–	–

Values are presented as mean (SD) unless otherwise indicated.

CPZ = chlorpromazine; PANSS = Positive and Negative Syndrome Scale; SES = socioeconomic status; WRAT = Wide Range Achievement Test.

Functional integrity of the TPJ-pSTS

We first investigated whether there were brain areas that responded differently to the movie between schizophrenia participants and healthy controls, using ISC to quantify activation. Overall ISC patterns were similar in the two populations, with extensive engagement of occipitotemporal visual cortex, lateral parietal cortex, and both lateral and medial prefrontal cortex (Fig. 1A and B), reflecting similar movie-evoked activity in these areas. Nevertheless, contrasting the ISC patterns between the two populations revealed a significant focal ISC deficit in the vicinity of the TPJ-pSTS (Fig. 1C), reflecting reduced correlation of schizophrenia participants with the healthy control ‘gold standard’ population BOLD activity pattern specifically within this region. No other group difference cluster survived multiple comparisons correction in either hemisphere.

Figure 1.

Greyordinate comparison of ISC reveals focal TPJ-pSTS deficits. (A and B) Right hemisphere views of healthy control (HC) (A) and schizophrenia participants (SzP) (B) ISC maps demonstrate strong engagement of occipitotemporal visual, dorsal attention, pSTS, and prefrontal areas. (C) The contrast map reveals a focal deficit that is the only cluster to survive multiple-comparisons correction. No cluster survives correction in the left hemisphere. White frame in C outlines region shown in cropped images in Figs 2 and 3.

We next functionally localized the deficit within the TPJ-pSTS. We used task localizers to subdivide the TPJ and the pSTS into functionally defined regions of interest (Fig. 2). In the pSTS, we identified four regions of interest activated primarily by moving facial expressions (purple borders). We also identified an additional region of interest belonging to the dorsal attention network (posterior superior temporal gyrus, pSTG, cyan).^4,39 In the TPJ, we identified three regions of interest with the task localizers: TPJp (activated during mentalization, yellow border),^4,6,40 TPJa (activated by detection and reorienting of attention during visual search, green border),^4,6,40 and the TPJm hub area (activated by both moving faces and detection/reorienting). See Fig. 2 legend and Supplementary material for more details.

Figure 2.

Localizer task activations and region of interest definitions. (A) Activation by moving faces in static/moving emotional faces localizer. (B) Activation by motion in from motion localizer. Compared to moving emotional faces in A, pSTS subdivisions are largely not activated by non-biological motion. (C) Activation by static faces in static/moving emotional faces localizer. (D) Contrast of moving and static faces in static/moving emotional faces localizer. Purple borders for pSTS subdivisions defined on this contrast. (E) Activation by target detection during attention localizer. (F) Activation/deactivation by visual search (RSVP stream processing) during attention localizer. TPJa (green border) defined by conjunction of detection activation (E) and deactivation during visual search (F) and assigned to the ventral attention network.^19,36–38 pSTG (cyan border) defined by activation focus on the posterior STG during visual search (activation similar to the motion localizer in B) and assigned to the dorsal attention network.^19,37 (G) Activation by mentalizing localizer. Whereas activation by moving emotion faces in A covers the pSTS, activation by mentalizing is limited to the TPJ, mostly on the angular gyrus (TPJp, yellow border) but also extending to the TPJa. TPJm (red border) defined as cortical zone not included in any of the other regions of interest. (H) TPJ-pSTS subdivisions in comparison to the ‘hubs’ map from .²⁹ The TPJm boundary outlines one of these hub zones. (I) Comparison of activation by moving emotional faces (purple), target detection (green), and mentalizing (yellow) within the TPJ and pSTS regions of interest. The tasks each evoke a complex pattern of activity (dotted lines mark activation significance threshold) (P < 0.05). The localizer task activations separate first between the TPJ and pSTS [TPJ-pSTS × task: F(1,70) = 9.7, P = 0.002]. Within the TPJ, there was a significant difference in activation of each region of interest by each task [region of interest × task: F(1,106): 3.9, P = 0.05]. This interaction was driven by TPJp primarily being activated by mentalization, TPJa primarily being activated by detection/reorienting, and TPJm (red box) being activated by both moving facial expressions and detection/reorienting.

The greyordinate ISC deficits in Fig. 1C fell within the TPJ with some weaker subthreshold deficits in the pSTS (Fig. 3). In the TPJ, we found a significant main effect of group [F(2,46) = 4.5, P = 0.038] and a significant group × region of interest interaction [F(2,46) = 4.7, P = 0.036]. In post hoc t-tests, we observed a highly significant, large effect size reduction in ISC that survived correction of multiple comparisons in TPJm, the only significant difference in the TPJ [t(46) = 3.0, P = 0.0048, d = 0.87]. In the pSTS, there were also significant group [F(2,46) = 5.2, P = 0.028] and group × region of interest [F(2,46) = 4.7, P = 0.035] effects, with the only significant post hoc difference in pSTS 2 [t(46) = 2.1, P = 0.041, d = 0.62]. We also compared the ISC activation and deficit patterns to other available parcellation schemes; the functionally based parcellations we use better fit the observed patterns of activation and deficits (Supplementary Fig. 1 and Supplementary material).

Figure 3.

Activation of the TPJ-pSTS subdivisions measured by ISC. (A) Healthy controls (HC). (B) Schizophrenia participants (SzP). (C) Contrast of the two groups. Focal ISC deficit falls mostly within TPJm. (D) Region of interest × region of interest group comparison of ISC in the TPJ-pSTS. In the violin plots, the envelope represents the full distribution of the data, the open circle marks the median, and the solid horizontal line the mean. *P < 0.05, **P < 0.01.

Relationship of TPJ-pSTS functional deficits to social cognition

We next examined whether the TPJm deficits were related to use of facial expressions in social cognition as measured by TASIT sarcasm. Replicating the results from the larger dataset reported in Patel et al.,²⁰ Schizophrenia participants' deficit on sarcasm trials was significantly worse than their deficit on lie trials [group × lie/sarcasm: F(2,40) = 6.4, P = 0.016; schizophrenia participants versus healthy control lie: t(40) = −1.9 P = 0.06, sarcasm: t(40) = −4.1 P = 0.0002].

Examining the relationship of TASIT sarcasm performance with TPJm activation (measured by ISC), we found a highly significant correlation [F(2,40) = 13.7, P = 0.0007, r_p = 0.46, P = 0.004; Fig. 4], over and above the effect of group [F(2,40) = 9.7, P = 0.004], such that participants with lowest TPJm activation (measured by ISC) showed the lowest TASIT performance. The group × TASIT sarcasm interaction was not significant [F(2,40) = 0.01, P = 0.9]. Within schizophrenia participants, the correlation was r = 0.57 (P = 0.003), and in healthy controls was r = 0.40 (P = 0.11). TPJm ISC did not correlate significantly with TASIT lie performance (r_p = 0.08, P = 0.67).

Figure 4.

TPJm activation (measured by ISC) correlates with TASIT performance across both groups. Partial correlation shown after accounting for correlation of group membership with TASIT performance. SzP = schizophrenia participants.

A significant, but weaker, correlation was also observed for TPJa [F(2,40) = 5.5, P = 0.02, r_p = 0.32, P = 0.02] but not TPJp [F(2,40) = 1.1, P = 0.31, r_p = 0.15, P = 0.28]. However, when TPJa and TPJm were entered into a simultaneous regression, only the correlation with TPJm remained significant. Co-varying for antipsychotic dose in the schizophrenia participants did not alter the within-group correlation between TPJm and TASIT sarcasm performance. pSTS 2 ISC also did not correlate with TASIT sarcasm performance [F(2,40) = 0.26, P = 0.61, r_p = −0.03, P = 0.82].

Synchronization of TPJ-pSTS activity with visual processing, attention/oculomotor, and other areas

We then compared between groups the synchronization of the TPJ and pSTS regions of interest to key areas that either send or receive information from the TPJ-pSTS during the transformation of incoming visual information into a saccade plan in the visual scanning of the movie: motion processing (MT), face processing (FFA), and attention/oculomotor planning (FEF). In both groups, MT activity was highly synchronized with other occipitotemporal visual areas, pSTS regions of interest, and frontoparietal and dorsolateral regions of the brain usually associated with selective attention and other cognitive processes (Fig. 5A and B).

Figure 5.

Differences in TPJ synchronization with other regions of interest. (A and B) Greyordinate synchronization maps of average healthy control (HC) MT activity (seed location shown by green filled circle) demonstrating robust correlations in both populations with occipitotemporal visual, dorsal attention, pSTS, and prefrontal areas. (C) Greyordinate contrast reveals strong deficit spread across all TPJ regions of interest. (D) MT → TPJ region of interest synchronization deficits in schizophrenia participants. (E and F) Similar deficits for FFA and FEF. (G–I) Increased synchronization or decreased anticorrelation of medial and lateral prefrontal areas with the TPJ. In the violin plots, the envelope represents the full distribution of the data, the open circle marks the median, and the solid horizontal line the mean. See Supplementary Fig. 2 for greyordinate synchronization maps for areas shown in E–I. *P < 0.05, **P < 0.01, ***P < 0.001. amPFC = anteromedial prefrontal cortex; aMFG = anterior middle frontal gyrus; dACC = dorsal anterior cingulate cortex; L = left; R = right.

Contrasting these synchronization maps between groups revealed focal deficits in the TPJ for schizophrenia participants [main effect of group: F(2,46) = 11.7, P = 0.001] (Fig. 5C) with a greater impact on TPJp than TPJm or TPJa [group × TPJ region of interest: F(2,46) = 9.1, P = 0.004, TPJp: t(46) = 3.5, P = 0.001, d = 1.0] (Fig. 5D). Repeating the same analysis for right FFA (face-processing, Fig. 5E) and right FEF (attention/oculomotor, Fig. 5F) revealed a similar pattern of reduced synchronization with TPJ activity in schizophrenia participants [group: F(2,46) = 12.8/9.0, P = 0.0008/0.004] with a similar difference between TPJ regions of interest [group × TPJ region of interest: F(2,46) = 11.8/7.2, P = 0.001/0.01], though in right FFA the greatest deficit was with TPJm [t(46) = 3.8, P = 0.0004, d = 1.1]. In contrast, synchronization of any of these areas with pSTS regions of interest did not differ significantly between groups (see Supplementary Fig. 2 for greyordinate maps of synchronization).

In exploring synchronization with other cortical areas involved in social cognition, we found a pattern of schizophrenia participants having decreased synchronization of the three TPJ regions of interest with most visual and face processing regions of interest, IPS and FEF dorsal attention regions of interest, and the TPJ-pSTS regions of interest themselves (see Supplementary Fig. 3 for within TPJ-pSTS synchronization). However, a number of prefrontal areas demonstrated the inverse pattern. These included areas within the cingulo-opercular network activated by the visual search task [e.g. left dorsal anterior cingulate cortex (left dACC) in Fig. 5G], the default mode network activated by the mentalization localizer [e.g. left antero-medial prefrontal cortex (left amPFC) in Fig. 5H], and the frontoparietal network also activated by the mentalization localizer [e.g. left anterior middle frontal gyrus (right aMFG) in Fig. 5I]. In healthy control subjects these areas were negatively synchronized (or anticorrelated) with the TPJ, and schizophrenia participants were either less anti-correlated or positively synchronized. Of note, when these analyses were performed without global signal regression, all values were shifted positively but maintained the same relative differences. Therefore, these results suggest either decreased anticorrelation or increased synchronization between the prefrontal areas and the three TPJ regions of interest in schizophrenia participants versus healthy controls.

Correlation of BOLD activity with visual features

We next examined the relationship of TPJ BOLD activity with the visual features present in the movies. In both groups, the highest correlations were observed with motion speed. Greyordinates correlating with motion speed spanned visual cortex into the pSTS and TPJ, extending beyond the cortex activated by the motion localizer and mirroring the pattern evoked by the face-emotion localizer (Fig. 6A and B versus Fig. 2A and B). In the TPJ, we observed a large group difference [group: F(2,46) = 8.6, P = 0.005] and a large group × TPJ region of interest effect [group: F(2,46) = 7.4, P = 0.01]. TPJm again exhibited the largest deficit in schizophrenia participants versus healthy controls [t(46) = 2.87, P = 0.006, d = 0.85] (Fig. 6C and D). Visual contrast demonstrated only weaker effects of group [group: F[(2,46) = 7.4, P = 0.01] but no group × TPJ region of interest interaction, and neither luminance or faces demonstrated any differences. Visual feature-BOLD correlations in pSTS areas did not significantly differ by group.

Figure 6.

Motion and saccade frequency correlations with BOLD activity time course. (A and B) Right lateral views of motion correlation maps in healthy controls (HC) (A) and schizophrenia participants (SzP) (B) shows consistent correlations with occipital, ventral temporal, and frontoparietal areas. (C) Greyordinate contrast reveals focal deficit within TPJ. (D) Motion correlation deficits by TPJ region of interest showing strongest deficits in TPJm. (E and F) Right lateral views of saccade frequency correlation maps in healthy controls (E) and schizophrenia participants (F) demonstrates robust correlation with occipitotemporal visual, dorsal attention, pSTS, and prefrontal areas. (G) Greyordinate contrast reveals focal deficit within TPJ. (H) Saccade correlations by region of interest showing strongest deficits in TPJp. In the violin plots, the envelope represents the full distribution of the data, the open circle marks the median, and the solid horizontal line the mean. *P < 0.05, **P < 0.01.

Correlation of BOLD activity with saccades

For the subset of participants (14 schizophrenia participants and 13 healthy controls) with usable eye-tracking data from the MRI session, we next compared BOLD correlations with saccades rates between groups. The mean number of video frames dropped due to artefact (such as blinks) was similar in schizophrenia participants [30.9(20.7)% compared to healthy controls] [22.9(17.8)%, t(25) = 1.1, P = 0.3]. The mean number of saccades performed during the free-viewing of the movie in the MRI was also similar in schizophrenia participants versus healthy controls [1646.8(542.7) versus 1666.5(589.7), t(25) = 0.09, P = 0.92].

In both groups there was robust correlation of saccades with the BOLD activity in visual cortex and the dorsal attention network areas involved in oculomotor planning, including the posterior IPS (pIPS) and FEF (Fig. 6E and F). In healthy controls there was also robust correlation with TPJ activity; this correlation is essentially absent in schizophrenia participants [group: F(2,25) = 4.6, P = 0.041] (Fig. 6F–H). There was also a significant group × TPJ region of interest interaction [group: F(2,25) = 4.3, P = 0.048], with the largest difference in the TPJp [t(25) = 2.23, P = 0.03, d = 0.89]. Again, there were no significant differences in correlation with saccades for any of the pSTS areas, and this subset group did not differ from the full group in any of the above measures.

Discussion

Deficits in social cognition are a critical component of schizophrenia and contribute significantly to poor functional outcome. Although these deficits have been extensively documented,^1,2 the underlying neural mechanisms are incompletely understood, partly because of the sheer complexity of the systems involved and partly because of the difficulty measuring brain activity in tasks that relate to ‘real-world’ social situations. In this study we used convergent naturalistic data-driven imaging and analysis methods to reveal how specific deficits within the TPJ-pSTS, a social cognition hub in human cortex, can have widespread effects on the ability to make quick and accurate social inferences from the incoming streams of visual information, an important component of social cognition.

After functionally subdividing the TPJ-pSTS, we observed highly robust deficits in the activation of a hub area—TPJm—that lies between the other TPJ-pSTS areas involved in processing moving facial expressions, understanding mental states, and directing attention and saccades during visual scanning. The less activated the TPJm is, the worse the performance on a task that requires the participant to quickly integrate these three operations in order to understand a given social situation. TPJm activity also failed to appropriately correlate with video motion in schizophrenia participants, and activity in all three TPJ areas failed to both appropriately synchronize with other areas involved in visual scanning and appropriately correlate with the saccades made during visual scanning. Together, these results have implications not only for the neural substrates of social cognition deficits in schizophrenia participants, but also for our understanding of the functional architecture of TPJ-pSTS in everyone.

The architecture of the TPJ-pSTS

The TPJ-pSTS in our recently proposed model contained three core components: the pSTS face-emotion recognition areas, the TPJp mentalization area, and the TPJa attention reorienting area. In parallel, Power et al.²⁸ described a hub region that localizes to an area of TPJ intermediate between TPJa and TPJp, which we term TPJm. In this study, we used functional localizers to define all of these areas in relation to each other, and, as expected, show significant differential activation of the areas by task. Based upon these definitions, then, we propose an updated model that incorporates the TPJm as the hub area that connects the pSTS to the other TPJ areas (Fig. 7).

Figure 7.

Pseudo-anatomic layout (A) and schematic (B) of the interactions between networks involved in visual scanning of social scenes at the TPJ-pSTS, modified from Patel et al. ¹² In healthy controls (blue line), the TPJ-pSTS serves as a third pathway linking visual areas to the prefrontal cortex and dorsal attention areas that control attention and saccade planning/visual scanning. Information about moving facial expressions from MT and FFA converges on the pSTS. This information is conveyed to TPJp and TPJa via TPJm. These TPJ areas are modulated by prefrontal mentalization areas, such as those in mPFC, and prefrontal cognitive control (cingulo-opercular/salience) areas, such as aIns and dACC. Based on these various inputs, the TPJp determines whether additional visual scanning is needed to update the ongoing mentalization operations. If visual scanning is needed, the TPJa is activated, which then triggers saccade planning in the dorsal attention areas through prefrontal cortex. In schizophrenia participants (red line) this third pathway is disrupted in the TPJ-pSTS, preventing its use in the guidance of visual scanning of social scenes. aIns = anterior insula; dACC = dorsal anterior cingulate cortex; mPFC = medial prefrontal cortex; MT = middle temporal; PFC = prefrontal cortex.

This critical link through the TPJ-pSTS is the gateway through which facial expression information flows into the TPJp and TPJa, allowing it to be used to update ongoing mentalization operations and to influence visual scanning patterns. Moreover, this link may be unique to humans, supporting the types of complex social interactions that only humans may be capable of.^6,41 The failure to activate this area in schizophrenia participants, then, essentially ruptures this critical pathway. By preventing motion information from reaching the TPJ, the TPJm failure prevents the entire TPJ from being involved in the visual scanning of the social scene, disconnecting it from key visual processing areas (e.g. MT and FFA) and saccade planning areas (e.g. FEF), and ultimately from the planning of the saccades themselves. This disconnection in schizophrenia participants then removes this human-unique pathway from use, degrading an individual’s ability to follow the interpersonal interactions taking place before their own eyes.

Implications of TPJ-pSTS dysfunction in schizophrenia

Most studies of social inference in schizophrenia participants have used relatively simple tasks and with these have found consistent and substantial impairments in the recognition of facial expressions and processing of biological motion.^42–44 Consistent with our localizer findings here, such tasks primarily activate pSTS but not the TPJ.^1,2,45,46 Other studies have focused on tasks that activate the TPJ, such as mentalization tasks^47,48 or reorienting of attention tasks^49,50 which localize to separate subdivisions of the TPJ.^6,51,52 To our knowledge, no prior study of social inference or any other social cognitive process in schizophrenia participants has focused on tasks designed to activate all of these structures in concert. Naturalistic stimuli such as movies not only activate multiple brain areas and networks at once, they also are more readily translatable to clinical settings.⁵³

By using naturalistic stimuli, this study takes advantage of recent efforts that have been aimed at unifying the different subcomponents of social inference as they relate to schizophrenia or other social cognitive disorders.^1,2 From such efforts TASIT has emerged as a potentially useful task paradigm for studying the role of social inference in social cognition deficits in both SzP²¹ and autism spectrum disorders (ASD)⁵⁴ and has been shown to predict functional outcome in schizophrenia participants.²² However, with 30–50 s videos, the task itself is not well-suited to assessment of brain functioning by neuroimaging modalities. Such videos are not suitable for block-design analyses because, in certain areas, the BOLD signals evoked by the onset of each video will last at least 16 s and can overshadow the recorded BOLD signals related to specific cognitive processes.⁵⁵ Moreover, these onset transients may differ in magnitude in schizophrenia participants.³⁸ The TASIT videos are also not well-suited for event-related designs; since TASIT portrays naturalistic interactions between individuals, there are no specific predefined ‘events’ with randomized inter-event intervals that can be used to measure evoked responses. Lastly, the TASIT videos are too short for ISC analyses because the onset transient BOLD signals could introduce spurious correlations and because much of the power in ISC analyses comes from long timescale BOLD signal fluctuations that correlate with the slowly changing narrative details in the movie.^36,53 In general, these considerations have limited the ability of psychiatric neuroimaging studies to take advantage of the types of naturalistic stimuli, such as videos and virtual reality, that are increasingly being used to assess the neural basis of social functioning deficits in disorders such as schizophrenia or ASD.

Here, we addressed this limitation by evaluating TASIT performance relative to brain activity evoked by a movie clip that has previously been shown to engage both the TPJ and pSTS and to be suitable for ISC analysis.²⁷ The processing of longer movie clips like this one mimics real-world social cognitive processing in that it involves simultaneous activation of distributed brain systems at multiple timescales: short timescales generally associated with stimulus processes such as face-emotion processing, medium timescale processes such as orienting and reorienting attention, and longer timescales generally associated with cognitive processes such as mentalization.^5,53 Longer movie clips also have more power for ISC analyses compared to shorter ones.

To our knowledge only one previous study in schizophrenia participants has used a similar approach.⁵⁶ While they also used ISC to examine activation evoked by a naturalistic movie and found a functional deficit in the TPJ-pSTS, they were not able to differentiate the TPJ-pSTS into functional subdivisions or show a relationship with social cognition. By combining the ISC approach with functional localization of TPJ-pSTS areas, we have not only identified focal deficits, but have also placed them in the context of other brain areas and networks involved in key aspects of social cognition. Without the combination of the localizers and the ISC method, we would not have been able to determine that schizophrenia was affecting the functioning of a hub area whose impact spreads to other areas, and not directly impacting one of the TPJ-pSTS processing systems themselves. Altogether, these results provide a mechanistic framework that can be used to guide future studies of social cognition in schizophrenia and other neuropsychiatric disorders.

Underlying mechanisms of the TPJm functional deficit

When functioning normally, a critical role of the TPJm appears to be to integrate ‘bottom-up’ sensory information and ‘top-down’ cognitive signals. This convergence then points to three potential explanations for our observed TPJm deficit in schizophrenia participants: intrinsic, ‘bottom-up’, or ‘top-down’ dysfunction. The first is that the TPJm itself is intrinsically dysfunctional. The TPJ in general is one of the last to fully develop in the human brain, maturing around the time of onset of schizophrenia.^57–59 Functional hubs also appear in associative cortex late in development.⁶⁰ Over-pruning of synapses, one dominant theory of the development of schizophrenia,^58,61 may then differentially affect the development of this region, especially the TPJm, impairing its activation by salient stimuli.

A second possibility is that the failure to activate the TPJm reflects a failure in the ‘bottom-up’ flow of visual information. Previous studies in schizophrenia participants have found reduced sensitivity to motion in general^37,45,46 and facial expressions and other types of biological motion more specifically.^42–44 These deficits have generally been localized to area MT and/or within the pSTS.^45,46 The overlap of the motion-correlated activation during the movie through the pSTS with the face-emotion localizer activity but not the motion localizer suggests that the motion in the movies was primarily related to facial expressions and other biological motion. However, the schizophrenia participants' deficits in motion processing were localized to the TPJ, a region not previously implicated in motion processing.^4,47,51 This result may suggest that the motion processing deficits observed in previous studies in MT/pSTS may be obscured by other visual and cognitive signals during the free-viewing of the movie. Evidence of this obscuring may be demonstrated by the intact synchronization of MT and pSTS with themselves and other visual processing/attention areas in schizophrenia participants versus healthy controls (Fig. 3A–C). Failure to detect or properly process motion associated with facial expressions in MT or pSTS, then, may lead to the failure of the TPJm to activate, thus preventing these stimuli from being registered as salient.

A third possibility is disruption of the integration of signals in the TPJm by abnormal ‘top-down’ input. We found increased synchronization between the TPJ and prefrontal cognitive control and mentalization areas (e.g. dACC, amPFC, aMFG), which could be evidence of this abnormal top-down input. Previous studies have found that healthy brain functioning requires negative coupling between these areas, which has been reduced in schizophrenia participants. This theory of decreased network anticorrelation or segregation has played a prominent role in a number of network-based models of pathophysiology.^62,63 Our results may represent the functional consequences of this reduced anticorrelation or segregation. Of note, whether or not global signal regression is applied to our analyses, a similar interpretation can still be applied. Repeating our analyses without global signal regression shifts all of the synchronization values to be positive while preserving these relationships, resulting in an increase in synchronization between prefrontal cortex and the TPJ in schizophrenia participants. In either case, information from prefrontal areas is inappropriately being conveyed to the TPJ. Since the TPJ in schizophrenia participants is synchronized to healthy control areas involved in movie-synchronized mentalization operations, these results also suggest that the same movie-evoked mentalization operations are taking place in schizophrenia participants, and therefore that mentalization operations are not completely disrupted.

Framework for future studies

The convergent results of this study provide the functional anatomical framework that will guide the investigations into these possibilities of TPJm dysfunction. Several other aspects of the framework also warrant further investigation. Given the high degree of anatomical variability in the TPJ,^12,13 one critical line of research is to determine whether the TPJm is indeed a separable cortical area or is the result of individual variability in the borders of the TPJp, TPJa, and the pSTS. The anatomical specificity of the ISC and motion deficits, along with the correlation with social cognition, argues against this, but even if true our results support the importance of functional integration within the TPJ-pSTS during naturalistic social cognition. Another is replicability: while the TPJm ISC deficit likely replicates previous findings,⁵⁶ the remaining results, including the correlation with social inference performance, have not been described before and need to be replicated in a larger sample. Another aspect to investigate further is the effect of differences in visual scanning patterns on ISC: group differences in which visual features participants are looking at may drive differences in cortical activity. However, large differences were not observed in visual cortex, which would be most affected by differences in eye-gaze position. In addition, the increased correlation of the TPJ in schizophrenia participants with healthy control prefrontal areas suggests that not all correlational results in schizophrenia participants will be decreased by differences in gaze position. Another is to develop tasks that examine the mentalizing system directly for deficits independent of the motion and saccade-related deficits observed in this study. Lastly, these correlational relationships will need to be examined with TMS or other neuromodulation modalities to causally test the role of the TPJ-pSTS (and the TPJm in particular) in social cognition.

Schizophrenia is often viewed as a global disorder, with deficits diffusely spread across cortical areas. However, this study suggests that when examined using high-resolution methods, differential impairment is observed even within circumscribed cortical areas, suggesting that specific deficits may have widespread consequences. For example, dysfunction of ‘hub’ regions such as the TPJ may affect coordination between brain networks as well as the functioning of the networks themselves. These results provide a framework for assessing these widespread impacts, and a guide for developing both diagnostic/prognostic biomarkers for social functioning as well as neuromodulation-based treatments for these social cognition deficits. In addition, deficits in naturalistic social cognition pervade neuropsychiatric disorders, most notably ASD, and this framework could be used to assess dimensional and categorical differences in these disorders.

Acknowledgements

We wish to thank Chad Sylvester and Guillermo Horga for comments on the manuscript.

Funding

We wish to thank the funding agencies who supported this work: NIMH (G.H.P.: K23MH108711 and T32MH018870, D.C.J.: R01MH049334, D.A.L. and R.A.B.: Intramural Research Program ZIA MH002898), Brain & Behavior Research Foundation (G.H.P.), American Psychiatric Foundation (G.H.P.), Sidney J. Baer Jr. Foundation (G.H.P.), Leon Levy Foundation (G.H.P.), the Dana Foundation (G.H.P.), and the Herb and Isabel Stusser Foundation (D.C.J.).

Competing interests

G.H.P. receives income and equity from Pfizer, Inc through family; D.C.J. has equity interest in Glytech, AASI, and NeuroRx. He serves on the board of Promentis. Within the past 2 years he has received consulting payments/honoraria from Takeda, Pfizer, FORUM, Glytech, Autifony, and Lundbeck. S.C.A., D.R.B., F.I.P., S.A.M., D.J.V., N.E.S., C.C.K., J.P.S.P., L.P.B., J.G., A.M., R.A.B., K.N.O., and D.A.L. reported no biomedical financial interests or potential conflicts of interest.

Supplementary material

Supplementary material is available at Brain online.

Glossary

BOLD

blood oxygen level-dependent

FEF

frontal eye fields

FFA

fusiform face area

ISC

inter-subject correlation

TASIT

The Awareness of Social Inference Test

TPJ-pSTS

temporoparietal junction-posterior superior temporal sulcus