Cortical networks involved in judging the attention of others

Abstract

Understanding someone else’s attention lies at the heart of human interaction. When we perceive something to be in someone else’s attention, we understand it to be in that person’s mind in the moment, ready to affect that person’s choices. It has been proposed that people construct predictive models of the attentional state of others. In that proposal, a working understanding of someone else’s attention goes far beyond merely tracking the direction of someone’s eyes. For example, in a recent study, participants watched a spotlight of attention moving around a picture and successfully judged whether the trace represented real, human attention or an artificially manipulated version, demonstrating implicit information about how patterns of attention behave over space and time. However, the neuronal underpinnings of attention modeling are almost entirely unstudied. Here we tested people in an fMRI scanner while they performed the attention judgment paradigm. When contrasting attention traces that participants judged to be artificial versus those judged to be real, activity was found bilaterally in the precuneus, dorsomedial prefrontal cortex, anterior cingulate, and anterior insula, as well as in a part of the right temporoparietal junction. These areas partially overlap the theory-of-mind network and the salience network. The activations differ from those known to be involved in processing low-level features of attention such as the sight of other people’s eyes. The results provide an initial picture of the cortical networks involved in monitoring the attention patterns of others and recognizing when those patterns deviate from expectation.

Introduction

To understand what is in someone else’s mind, it is useful to monitor that person’s attention. Most work on monitoring the attention of others has focused on explicit cues, such as the direction of someone’s gaze (Baron-Cohen 1991; Langton et al. 2000; Calder et al. 2002; Downing et al. 2004; Frischen et al. 2007; Fletcher-Watson et al. 2009; Bayliss et al. 2011; Dalmaso et al. 2020) or the direction that a person is pointing a finger (Langton and Bruce 2000; Ariga and Watanabe 2009; Gregory and Hodgson 2012). Specialized brain regions, mainly in the superior temporal sulcus, appear to be involved in processing such directional cues, especially the direction of the eyes (Perrett et al. 1985; Puce et al. 1998; Wicker et al. 1998; Hoffman and Haxby 2000; Itier and Batty 2009; Carlin et al. 2011). However, it has been suggested that people gain an understanding of each other’s attention using deeper cognitive processes beyond tracking directional visual cues (Graziano 2013). In that proposal, the brain integrates a variety of cues over time (body posture, gaze direction, facial expression, speech prosody, social context, and so on), to inform a model of other people’s attention. Moreover, the model may contain more complex information than merely where attention is directed in the moment. It may, for example, encode how strong attention is, how likely it is to linger on one or another object, when it is likely to shift, where it is likely to go next, and what the predicted consequences of attention might be for the other person’s mind state and behavior. A growing body of behavioral evidence supports the hypothesis that people build rich models of each other’s attention. For example, the evidence suggests that people combine multiple cues to encode the state of someone else’s attention (Bio and Graziano 2021), encode whether other people’s attention is drawn exogenously or endogenously (Pesquita et al. 2016), make predictions about the attention of others (Mazor et al. 2023), and use visual motion processing to build automatic, implicit, arrow-like representations of other people’s attention (Guterstam et al. 2018; Guterstam et al. 2020; Guterstam and Graziano 2020a, 2020b; Renet et al. 2024). This proposed dynamic and predictive model of attention has been termed the attention schema, in analogy to the body schema (Graziano 2013; Webb and Graziano 2015). In the proposal, attention schemas are constructed to model the attention of self and others.

The hypothesis that the human brain constructs a sophisticated model of other people’s attention is relatively new, and relatively few neuroimaging studies are aimed at identifying the brain networks involved. In one study, when participants integrated gaze cues and facial expression cues to rate how intensely aware a face appeared to be of a nearby object, a set of areas became active, including the temporoparietal junction (TPJ) bilaterally, the dorsomedial prefrontal cortex (DMPFC), and the lateral prefrontal cortex (LPFC) (Kelly et al. 2014). In another study, when participants read stories about someone engaging in exogenous or endogenous attention, even though the participants were not asked any explicit questions about attention, the left TPJ showed patterns of activity that distinguished between the two types of stories (Guterstam et al. 2021). These prior neuroimaging studies relied on behavioral tasks that were intended to make participants think about complex features of other people’s attention. However, the tasks did not include direct behavioral measures to confirm that the participants were actually making judgments about attention. To better understand the neural basis by which the brain judges the attention of others, what is needed is a paradigm that requires people to make judgments about patterns of attention, confirms that people are indeed making those judgments, and then measures the brain networks that become active in association with those judgments.

Recently, a behavioral paradigm provided a direct test of people’s ability to judge the attention of others (Ziman et al. 2023). The paradigm was based on prior work that investigated how people and artificial agents extract information from gaze patterns (Yarbus 1967; Greene et al. 2012; Haji-Abolhassani and Clark 2014; Kardan et al. 2015; Vo et al. 2016; Bahle et al. 2017; Kummerer et al. 2022). In the paradigm as used by Ziman et al. (2023), first, people viewed pictures of complex scenes while their eye position was measured. The eye position data were then rendered into a set of movies, each 3-s movie showing a glowing focus of overt attention shifting over a picture. Two sets of movies were constructed: those that accurately showed human attention and those that were modified to violate the normal statistics of human attention. For example, in some versions of the experiment, the attention movies were temporally scrambled such that the same attention locations were visited but in a randomized order. Finally, a new set of participants was asked to view the attention movies and distinguish real attention patterns from artificially altered patterns. If people have a good ability to model attention, thereby recognizing the predicted temporal and spatial patterning of attention, they should be able to spot examples where the predicted statistics are violated. The results showed that people were exquisitely sensitive to the subtleties of attention, distinguishing real attention from scrambled attention, real attention from attention traces played backward, attention traces belonging to one person from those belonging to another person, and even recognizing the correct spatiotemporal pattern of attention when viewing merely a glowing spotlight moving around a black screen, minus the underlying image. Moreover, participants appeared to integrate the pattern over time to make a judgment. Videos that contained as few as 6 or 7 movements of the attention spotlight resulted in near-chance performance, whereas videos that contained as many as 11 or 12 movements of the attention spotlight resulted in above chance performance. Therefore, the participants were most likely judging the unfolding pattern of attention over time. Comparison across these many versions of the task suggested that participants probably relied on a mixture of many subtle cues to distinguish real from artificial attention traces. They were not necessarily explicitly aware of using those cues but did report that the real attention traces seemed less mechanical, more like a conscious agent, and more similar to their own attention.

In the present study, we asked participants to perform the behavioral paradigm from Ziman et al. (2023) while we measured brain activity in a functional magnetic resonance imaging (fMRI) scanner. Participants viewed 3-s videos. Each video showed an attention spotlight sampling locations within a complex, naturalistic scene (see Fig. 1). Half the videos showed valid attention patterns, and half showed the same attention pattern altered such that the attention locations were presented in a temporally scrambled sequence. After watching each video, participants reported whether they judged the video to be a depiction of “real” human attention or a “fake” version that was artificially generated.

Fig. 1.

A) Example of an attention location. The stimuli consisted of a glowing attention spotlight moving around an underlying scene for 3 s. One frame of one example movie is shown here. Attention sequences were derived from patterns of eye movements, from people who viewed the images for 3 s each. (Figure 1A and caption text are adapted from Ziman et al. 2023.) B) Experiment trial sequence. Each trial in the experiment consisted of a video viewing period (3 s), a response period (3 s), and a fixation period before the next trial (jittered). On any given trial, the video might display veridical or scrambled attention (50% veridical per block). Each block consisted of twenty trials in total.

When people process low-level visual cues to attention, such as when participants see the eyes of others, a set of areas in the superior temporal sulcus are often active (Perrett et al. 1985; Puce et al. 1998; Wicker et al. 1998; Hoffman and Haxby 2000; Itier and Batty 2009; Carlin et al. 2011). However, what will happen in the present task, in which participants are not responding to eyes in specific, but instead are engaging a deeper, cognitive process of assessing spatial–temporal patterns of attention and judging whether they match or conflict with predicted patterns?

We set forth three preliminary hypotheses. The first was a general hypothesis. We predicted that the brain activity associated with attention judgments may be greater when participants judged the videos to be fake than when they judged the videos to be real. Greater brain activity when a predictive model encounters a violation of prediction has been observed before. When cues to another person’s attention were in conflict, activity rose in the TPJ bilaterally, among other areas (Kelly et al. 2014). When expected behaviors and social norms were violated, activity rose in the precuneus and the TPJ (Berthoz et al. 2002; Dungan et al. 2016). In a now classic series of findings, when exogenous attention was pulled unexpectedly into a new state, activity rose in a set of areas sometimes termed the ventral attention network, also overlapping the saliency network (Stevens et al. 2000; Serences et al. 2005; Corbetta et al. 2008; Mitchell 2008; Shulman et al. 2010; Wilterson et al. 2021). Given these previous trends, in the present experiment, we hypothesized that brain activity associated with judging attention traces might be higher in the judged fake condition than in the judged real condition because the judged fake condition represents perceived violations of predicted patterns.

The second hypothesis, closely associated with the first, concerned the specific cortical areas where elevated activity might be found. We hypothesized that elevated activity might include the TPJ with an emphasis on the right hemisphere, the anterior cingulate cortex (ACC), and the ventral lateral or opercular area of the prefrontal cortex (VLPFC), including the anterior insula. These areas tend to become active to events that violate predicted patterns, especially when attention is unexpectedly pulled into a new state (Stevens et al. 2000; Serences et al. 2005; Corbetta et al. 2008; Mitchell 2008; Shulman et al. 2010; Wilterson et al. 2021). To detect that something unexpected has happened requires a model that defines what is expected—a predictive model. We hypothesized that this mechanism might also respond to deviations from expectation about other people’s attention. We therefore predicted that the present attention–judgment task, in which people distinguish between attention patterns that match or deviate from predicted patterns, might evoke activity in these areas.

In the third prediction, we hypothesized that activity associated with judgments about attention might overlap brain regions associated with the most abstract levels of social cognition, such as the right TPJ, bilateral DMPFC, and bilateral precuneus (Saxe and Kanwisher 2003; Saxe and Powell 2006; van Veluw and Chance 2014; Molenberghs et al. 2016). This prediction aligns with the proposal that inferences about another person’s attention involve theory-of-mind processes, above and beyond the processing of low-level visual features such as the direction of someone else’s gaze.

It is important to note that the purpose of the experiment was not to confirm these three specific hypotheses. Rather, the purpose was to try to specify the cortical areas, whatever they may be, that become active when people judge the attention of others. Any result should be of interest because the patterns of activity involved in this type of task are not known. The hypotheses listed above represent initial guesses.

Materials and methods

Subjects

All procedures were approved by the Princeton Institutional Review Board (protocol IRB-15800). All participants provided informed consent and received paid compensation for their participation. Twenty-three healthy human volunteers (aged 18 to 56, normal or corrected-to-normal vision, all right-handed, 10 female) were recruited from the community. Given that our experiment was a within-subjects design, in which all participants were exposed to all conditions, we had no major concerns that the participant’s ages would influence the results in a way that systematically differed between conditions. However, we acknowledge that enrolling participants within a wide age range can cause added noise in the data, due to neurovascular changes across the lifespan. One participant was excluded due to equipment malfunction during scanning. A second participant responded to only five trials during one of the six experiment runs, perhaps due to a sleep episode, and therefore, data from this run were also excluded. In keeping with prior MRI studies of social cognition activations (eg Hoffman and Haxby 2000; Saxe and Kanwisher 2003; Rilling et al. 2004; Williams et al. 2005; Saxe and Powell 2006; Mitchell 2008; Dungan et al. 2016; Lukinova and Myagkov 2016; Guterstam et al. 2020, 2021; Wilterson et al. 2021), we aimed for approximately 20 participants. We tested 23 in case of attrition.

Experimental setup

Participants lay supine on the bed of a 3T MAGNETOM Prisma scanner (Siemens Healthineers AG, Erlangen, Germany). Visual stimuli were projected onto a screen 80 cm from the eyes through an angled mirror mounted on top of the head coil using a digital light processing projector (Hyperion MRI Digital Projection System, Psychology Software Tools, Sharpsburg, PA, USA) with a resolution of 1920 × 1080 pixels at 60 Hz. Stimuli were presented using PsychoPy 2023.1.2 (van Miltenburg et al. 2018). Participants used a 5-button response unit (Psychology Software Tools Celeritas, Sharpsburg, PA, USA) with the right hand, using the index finger on button one to indicate “real” and the middle finger on button two to indicate “fake.”

Stimuli

As shown in Fig. 1, each video stimulus consisted of a static background image with a dynamic luminous blob representing an attention spotlight. This spotlight indicated the areas where a prior viewer fixated on the image during a 3-s free-viewing period. By shifting from one fixation point to another, the attention spotlight emphasized segments where the viewer maintained overt attention. The focus of attention was represented by a blurred, Gaussian distribution of brightness with sigma = 19 pixels. This luminance map was superimposed on a dimmed rendition of the base image, yielding a radiant white attention hotspot against a darkened scene. Visual images and gaze data for generating the attention sequence videos were sourced from the publicly accessible Dutch Image Description and Eye-tracking Corpus (DIDEC) (van Miltenburg et al. 2018). The technique of using a heatmap generated from Gaussian-blurred fixations was inspired by the methodology of van Miltenberg et al. (2018). The method of generating the stimuli is described in detail in our previous publication (Ziman et al. 2023). In the present experiment, we selected attention trace videos that had 13 attention locations or more, based on our prior findings that participants were better able to distinguish real from altered attention traces when more fixation locations were included (Ziman et al. 2023). Each 3-s video was presented at a frame rate of 60 Hz.

To assemble the complete stimulus set for the present experiment, we selected 60 unique images, each overlaid with a unique eye movement trace. In that stimulus set, no recorded viewer was represented more than three times. Each valid attention stimulus was then used to generate a corresponding invalid attention stimulus by randomly shuffling the order of attention locations. In this way, 120 stimuli were generated, 60 representing valid attention movies and 60 representing visually matching, but invalid, or randomly scrambled, attention movies.

Behavioral task

Each trial began with a 3-s video presented on the screen. Following the video, during a 3-s response window, participants indicated by button press whether they perceived the depicted attention trace to be from a “real” person or a “fake” agent. Participants entered a response within the response period in 97% of the trials in our dataset. We did not consider responses made before or after the response window. After the response window, an intertrial interval (ITI) was imposed. The ITI was variable (jittered), sampled from a geometric distribution with a minimum value of 2 s, a mean of 3.5 s, and a maximum of 10 s. The task was performed in 6 blocks (each block corresponding to a scanner run), 20 trials per block, for a total of 120 trials. The trials were pseudorandomly ordered such that both the veridical and scrambled versions of each stimulus appeared within the same block but not consecutively. Thus, no two videos with the same background image appeared consecutively. Participants were allowed to rest in the scanner between blocks. After each experiment session, the experimenters asked the participant about the experience, noting any relevant details such as if the participant reported confusion or difficulty with the task, reported falling asleep at any point, or had other feedback of interest.

For each participant, signal detection analysis was applied to the pattern of responses across the 120 trials to compute a d’ score, as an index of that participant’s ability to discriminate a real from a scrambled attention trace. The formula used for d’ was: d’ = z(H)—z(F), where z(H) and z(F) are the z transforms of hit rate and false alarm rate, respectively. The hit rate was arbitrarily defined as the rate at which participants correctly identified a “real” attention trace.

Data collection

Functional imaging data were collected using a 3T MAGNETOM Prisma scanner (Siemens Healthineers AG, Erlangen, Germany) equipped with a 64-channel head/neck coil. Gradient-echo T2*-weighted echo-planar images (EPI) with blood-oxygen-dependent (BOLD) contrast were used as an index of brain activity (Logothetis et al. 2001). Functional image volumes were composed of 56 near-axial slices with a thickness of 2.5 mm (with no interslice gap), which ensured that the entire brain was within the field of view (FOV) in all subjects (195 mm FOV, 78 × 78 matrix, 2.5 mm × 2.5 mm in-plane resolution, TE = 30 ms, flip angle = 80°). Simultaneous multi-slice (SMS) imaging was used (SMS factor = 2). One complete volume was collected every 2.0 s (TR = 2,000 ms). A total of 660 functional volumes were collected for each participant (6 runs, with 110 volumes per run). The first five volumes of each run were discarded to account for nonsteady-state magnetization. A high-resolution structural image was acquired for each participant at the beginning of the experiment (3D MPRAGE sequence, voxel size = 1 mm isotropic, FOV = 256 mm, 176 slices, TR = 2,300 ms, TE = 2.96 ms, TI = 900 ms, flip angle = 9°, iPAT GRAPPA = 2). At the end of each scanning session, matching spin echo EPI pairs were acquired with reversed phase-encode blips, resulting in pairs of images with distortions going in opposite directions for blip-up/blip-down susceptibility-derived distortion correction.

Data preprocessing

Data were preprocessed using FMRIPREP version 23.0.0 (Esteban et al. 2019), which is based on Nipype 1.8.5 (Gorgolewski et al. 2011, 2018). The T1-weighted volumes were defaced using Pydeface 2.0.0 (Gulban et al. 2019). A nonuniformity map (or field map) was estimated using EPI references imaged in both posterior-to-anterior and anterior-to-posterior directions with topup in FSL 6.0.5.1 (Andersson et al. 2003) and used for distortion correction. Volume-based spatial normalization to one standard space (MNI152NLin2009cAsym) was performed through nonlinear registration with antsRegistration (ANTs 2.3.3), using brain-extracted versions of both T1w reference and a T1w template. The following template was selected for spatial normalization: ICBM 152 Nonlinear Asymmetrical template version 2009c (RRID:SCR_008796; TemplateFlow ID: MNI152NLin2009cAsym; Fonov et al. 2009).

Functional data preprocessing

For functional runs, the data were skull-stripped using fMRIPrep, slice time–corrected using 3dTshift from AFNI (Cox and Hyde 1997), and motion-corrected using MCFLIRT (FSL 6.0.5.1) (Jenkinson et al. 2002). Bbregister from FreeSurfer 7.3.2 (Fischl 2012) was used to implement boundary-based registration (Greve and Fischl 2009), with 6 degrees of freedom to coregister the anatomical (T1-weighted) data to the appropriate functional runs. Global signals were extracted from three masks (cerebrospinal fluid, white matter, and whole-brain) in addition to a set of physiological regressors, for component-based noise correction using CompCor (Behzadi et al. 2007). Anatomical CompCor probabilistic masks were then generated in anatomical space, with head-motion estimates calculated in the correction step. Any frame with head motion exceeding 0.5 mm framewise displacement was labeled as a motion outlier. Head motion estimates and global signals also yielded confound time series (Satterthwaite et al. 2013), and additional nuisance timeseries were calculated by means of principal components analysis on a subset of voxels (Patriat et al. 2017). The functional time-series were resampled into standard space, generating preprocessed functional data in MNI152NLin2009cAsym space. All transformations and resamplings (such as head motion and coregistration) were applied in a single interpolation step using antsApplyTransforms (ANTs 2.3.3) (Avants et al. 2008) with Lanczos interpolation (Lanczos 1964) (gridded resamplings) and mri_vol2surf (FreeSurfer 7.3.2, nongridded resamplings).

fMRI statistical analysis

All statistical analyses were performed using Nilearn (Nilearn 0.10.2) (Abraham et al. 2014). For each subject, we conducted first-level voxel-wise general linear model (GLM) analyses using the Nilearn library. We used the Glover hemodynamic response function (Glover 1999) to model the relationship between neural activity and BOLD signal. We did not use temporal or dispersion derivatives, to maintain simplicity, reduce the number of regressors, and avoid potential overfitting. To account for physiological noise, we regressed out five noise components using the anatomical CompCor, with the noise components calculated from a combined anatomical mask (white matter and CSF). We applied a high-pass filter (0.01 Hz) to remove slow-drifting trends and applied spatial smoothing with a Gaussian kernel with a full-width at half maximum of 4 mm to improve the signal-to-noise ratio. To ensure consistent scaling across subjects and improve model convergence, we also centered and normalized the time series by setting the variance to 1 in the time dimension. Finally, to reduce noise from head motion, we employed Nilearn’s head motion scrubbing algorithm, with a framewise displacement threshold of 0.5 mm or higher. We also set a threshold to exclude runs with excessive motion (30% or more TRs above 0.5 mm framewise displacement), but no runs reached the threshold and therefore no runs were excluded based on motion. We also included six rigid body motion parameters in our regression, in the form of three translation and three rotation parameters (nilearn’s “basic” motion confounds), and a regressor for global signal (nilearn’s “basic” global signal confound).

In our primary analysis, we performed a contrast between activity obtained during videos that participants judged as fake and videos that subjects judged as real (response window defined as the period of time during which the video was displayed on the screen). In a second analysis, we performed a contrast between activity obtained during videos that showed valid attention traces and videos that showed scrambled attention traces, regardless of the participants’ judgment.

To identify significant clusters of brain activity at the group level, we employed a nonparametric clustering analysis and group-level GLM. This procedure started by generating a null distribution via 10,000 permutations, then computing a second-level model fitted to the permuted data. The voxel-wise test statistics were computed for each permutation, using a two-sided test to account for both positive and negative effects. Voxels were thresholded at 0.001, so that only voxels with P-values below 0.001 were included in the clusters for each permutation. Then the cluster-level P-value (for P < 0.05) was determined by comparing the size of the observed data’s clusters with the size of clusters calculated from the null distribution.

All stimuli, all anonymized behavioral data and fMRI data, and all analysis code are publicly available online at https://github.com/KirstensGitHub/predicting_attention_neural.

Results

Behavioral results

On average, when participants judged whether attention patterns were real or artificial, their responses were correct 61% of the time. Participants exhibited a response bias toward judging stimuli as “real” more often than “fake.” Over the course of the experiment, participants rated veridical videos as real in an average of 42.95 out of 60 trials (standard deviation among participants = 5.80 trials, standard error = 1.23) and rated scrambled videos as real in an average of 30.5 out of 60 trials (standard deviation = 8.69, standard error = 1.85). For each participant, using signal detection analysis, a d’ score was computed as an index of that participant’s ability to discriminate a real from a scrambled attention trace. A d’ greater than 0 indicates a better-than-chance discrimination ability. As shown in Fig. 2, on average among participants, d’ was significantly greater than 0 (mean = 0.61, two-tailed t-test, t(21) = 5.23, P < 0.001). Only one participant had a d’ less than 0. The finding that people can discriminate real from scrambled attention traces at a level above chance is consistent with our prior results (Ziman et al. 2023). Because performance was far below ceiling, as expected given prior results, the data set allowed us to look for two kinds of differences in brain activity: first, the difference between videos that subjects judged as fake and videos that subjects judged as real, focusing on the judgment of the participants, and second, the difference between videos constructed as valid attention traces and videos constructed as scrambled attention traces, focusing on the physical properties of the stimulus.

Fig. 2.

Discrimination performance on the attention judgment task. Each dot represents the d’ value for a single subject (n = 22). The dashed line represents chance performance (an inability to distinguish between real and fake attention traces). The distribution of d’ values is significantly above chance (t = 5.23, P < 0.001).

fMRI results

We first examined the differences in brain activity when participants judged the attention videos to be fake versus when participants judged the attention videos to be real. The distinction between the two conditions therefore lays in the judgment of the participants.

Figure 3 shows the results. All 12 significant clusters are also listed in Table 1. All five of the largest clusters were positive (significantly greater activity in the judged fake condition than the judged real condition). One midline cluster was located bilaterally in the DMPFC, extending partially into the ACC. Two clusters, one in each hemisphere, were located in the anterior insula, extending into the VLPFC, bilaterally. One midline cluster was located bilaterally in the precuneus. One cluster was located in the right TPJ, where a subthreshold mound of activity encompassed the TPJ and a dorsal, posterior part of that mound rose above significance. These clusters are consistent with hypothesis 1, that activity associated with the attention judgment task should be greater in the judged fake condition than in the judged real condition. They are also consistent with hypothesis 2, that activation should partially overlap areas in the salience network (ACC, anterior insula, and parts of the right TPJ). Finally, they are consistent with hypothesis 3, that the activation clusters should partially overlap the known social cognition areas (precuneus, DMPFC, parts of the right TPJ).

Fig. 3.

Cortical clusters with significantly greater BOLD responses when participants viewed attention traces they judged to be fake versus when they viewed attention traces they judged to be real. Brightly colored regions on the inflated brain surfaces represent significant clusters. The color represents the t-statistic with warm colors reflecting positive values and cool colors reflecting negative values. The lighter colors on the rest of the brain surface represent the t-statistics that fell below significance threshold (where there were no significant clusters). Warm colors reflect positive values and cool colors reflect negative values.

Table 1.

Summary of significant clusters for first analysis. Contrast between trials when participants judged attention traces to be fake versus when participants judged attention traces to be real. Clusters ordered from largest to smallest. The table indicates the brain region containing each cluster, the t value farthest from zero within that cluster, the coordinates of the voxel associated with that strongest t value in MNI space, and the number of voxels within the cluster.

Cluster region	Strongest t	Coordinates (X, Y, Z)	Cluster size (voxels)
Bil DMPFC/ACC	6.94	−5, 27, 34	424
R Ant Insula	7.66	35, 24, 1	167
L Ant Insula	9.07	−40, 19, −11	150
Bil Precuneus	6.96	−10, −76, 40	98
R TPJ	6.16	43, −58, 43	60
L Occipital	−5.46	−47, −71 -14	42
L Motor	−8.14	−47, −11, 55	39
L Somatosensory	5.26	−45, −26, 64	36
R Occipital	−6.34	45, −81, −14	28
R Motor	−6.09	50, −8, 52	27
L Supplementary Motor	5.20	−15, 7, 67	21
L Orbital	−5.77	−2, 37, −14	19

We also found smaller clusters of activity, some negative (significantly greater activity in the judged real condition than the judged fake condition), in parts of visual cortex, motor cortex, supplementary motor cortex, somatosensory cortex, and orbital frontal cortex. The clusters found in the left somatosensory and motor cortex may correspond to preparation for pressing the response key with the right hand. Activity in visual cortex could possibly pertain to perceived differences in visual features between stimuli judged as fake versus stimuli judged as real.

As a second analysis, we performed a contrast between the activity obtained during videos that displayed valid attention traces and videos that displayed scrambled attention traces. In this second analysis, the distinction between the two conditions did not lie in whether participants judged the stimuli to be real or fake, but instead in the physical manner in which the stimuli were generated. The purpose of the second analysis was to test whether low level visual and motion features of the stimulus, that might differ systematically between valid and scrambled attention traces, could explain the significant clusters that we found in the first analysis.

Figure 4 shows the result. No significant clusters of positive activity were obtained. One cluster of negative activity was obtained (listed in Table 2), in the right medial parietal cortex. (This cluster appears as two spots in Fig. 4 due to the unfolding of the cortex.) The finding of extensive activity when contrasting trials based on whether participants judged the attention trace to be real or fake (Fig. 3), and the finding of little activity when contrasting trials based on the type of attention trace (Fig. 4), suggests that most of the measured brain activity in this task was not related to whether the stimuli actually showed real or altered attention, but instead was related to the higher-level social judgments that participants made.

Fig. 4.

Cortical clusters with significantly greater BOLD responses when participants viewed attention traces that were scrambled versus attention traces that were valid. Brightly colored regions on the inflated brain surfaces represent significant clusters (bottom-right panel). The color represents the t-statistic with cool colors reflecting negative values. The lighter colors on the rest of the brain surface represent the t-statistics where there were no significant clusters. Warm colors reflect positive values and cool colors reflect negative values.

Table 2.

Summary of results for the second analysis. Contrast between trials when participants viewed attention traces that were scrambled versus attention traces that were valid. The table indicates the brain region containing the cluster, the t value farthest from zero within that cluster, the coordinates of the voxel associated with the strongest t value in MNI space, and the number of voxels within the cluster.

Cluster region	Strongest t	Coordinates (X, Y, Z)	Cluster size (voxels)
R Medial Parietal	−5.77	3, −43, 58	20

Discussion

The purpose of this study was to understand the neural basis for judging the attention of others. The study examined the brain activity patterns that occurred as participants judged movies of overt attentional behavior. The 3-s movie stimuli either showed a spotlight of attention moving around images in a valid manner measured from real people, or showed temporally scrambled versions of the same attention traces in which the fixation locations were randomly reordered. Consistent with previous behavioral findings (Ziman et al. 2023), the participants were able to discriminate the two types of stimuli at a level significantly above chance. The brain activity during movies that the participants judged to be fake was contrasted with the brain activity during movies that were judged to be real. The major clusters of positive activation (greater activity during judged fake than judged real) were located in a dorsal posterior part of the right TPJ, bilaterally in the DMPFC extending into the cingulate cortex, bilaterally in the precuneus, and bilaterally in the anterior insula extending into the VLPFC. Smaller clusters of activation, both negative and positive, were scattered in visual, motor, and parietal cortices. We then performed a second comparison, subtracting brain activity during movies that depicted valid attention traces from activity during movies that depicted scrambled attention, regardless of how the participants judged the stimuli. In this second analysis, the previously obtained activity disappeared and instead one small significant cluster was obtained in the right medial parietal cortex. The finding of extensive activity when comparing judged fake and judged real, coupled to the finding of little activity when comparing actual fake and actual real, suggests that the findings are related more to the judgments of participants about these attention traces than to the low-level visual features of the stimuli themselves.

Since the goal of the study was open ended—to find cortical areas involved in judging attention patterns—any result would have been of interest. We did, however, propose three hypotheses about the possible outcomes. The main activations that we observed were consistent with the hypothesis that activity associated with the attention judgment task should be greater in the judged fake condition than in the judged real condition. The results were also consistent with the hypothesis that the activation clusters would partially overlap known social cognition areas. Activity clusters were found in the DMPFC and the precuneus, consistent with previous findings on the social cognition network (Saxe and Kanwisher 2003; Saxe and Powell 2006; van Veluw and Chance 2014; Molenberghs et al. 2016). The activity in the right TPJ may also be consistent with the social cognition network. The TPJ has somewhat different definitions depending on the functional task studied, but the area of activation obtained here lies within the broader borders of the right TPJ as previously defined (Igelström and Graziano 2017). In the present experiment, a mound of subthreshold activity extended throughout the right TPJ, and a posterior, dorsal part of that mound reached significance, an area consistent with previous studies on monitoring one’s own attention (Igelström and Graziano 2017; Wilterson et al. 2021). We suggest that constructing a model of attention may be a specialized part of the larger process of social cognition. It is worth noting that significant activity was not obtained in areas in the superior temporal sulcus, previously found to process the sight of other people’s eyes (Perrett et al. 1985; Puce et al. 1998; Wicker et al. 1998; Hoffman and Haxby 2000; Itier and Batty 2009; Carlin et al. 2011). The present findings therefore highlight the distinction between building a higher-level cognitive model of attention that predicts patterns over space and time, and processing the lower-level visual cues to attention such as gaze direction.

Finally, the results were consistent with the hypothesis that we would find activity in areas consistent with the salience network, which includes the TPJ, ACC, and the anterior insula extending into the VLPFC. The ventral attention network, often considered a subcomponent of the salience network, is more right lateralized and is believed to respond in relation to unexpected events that trigger exogenous changes in attention (Stevens et al. 2000; Serences et al. 2005; Corbetta et al. 2008; Mitchell 2008; Shulman et al. 2010; Wilterson et al. 2021). The overlap between the present results and both the salience network and the ventral attention network is consistent with participants constructing expectations about attention patterns (building a predictive model of the attention of others), comparing the viewed stimulus against that model, judging that some stimuli were in violation of the model, and as a result, generating activity in brain networks that respond to violations of expectations.

The activation pattern we observed differs from the canonical dorsal attention network (DAN) involvement in spatial reorienting (eg Corbetta and Shulman 2002), suggesting to us that participants are not merely volitionally redirecting attention during the task, but that they are instead engaging in higher-order modeling of others’ mental states within a social-predictive framework.

Anomaly detection is a part of this study. Participants detected a violation of expectation, or a violation of a predictive model. We therefore expected brain systems involved in anomaly detection to become active. In the present task, the anomalies pertain specifically to expected patterns of attention. We can say, confidently, that these brain areas become active as a result of people detecting anomalies in patterns of attention. However, we do not wish to imply that these brain areas are limited to detecting violations of models of attention. Instead, it is likely that they are involved in detecting violations of predictions in a wide range of domains. Moreover, they may be involved in other general processes, for example, in general aspects of social cognition.

It is also possible that detecting violations of predicted patterns caused the participants to engage other internal processes. For example, it is always possible that one trial type evoked more effort, more task engagement, or more emotional reaction, which might in principle explain some of the activity clusters. We do not have any direct evidence to suggest a difference in effort, task engagement, or emotional reaction between judged-fake and judged-real trials, but it is useful to keep such possibilities in mind when interpreting MRI results.

Our findings also align with predictive coding accounts of social perception, which propose that higher-order cortical regions generate top–down predictions to anticipate incoming social information (Friston 2010; Koster-Hale and Saxe 2013). The observed engagement of theory-of-mind and attention-related areas may reflect hierarchical inference processes, where discrepancies between expected and actual social signals (prediction errors) drive belief updating.

Our finding of activity in both social cognition regions and the saliency network is consistent with functional connectivity evidence for an interplay between social and saliency networks during social tasks (Ribeiro da Costa et al. 2022). Evidence from the clinical and developmental domains also supports interacting network effects related to social cognition. A functional connectivity study of autism (a disorder with social and attention symptoms) found that the clinical features of autism are associated primarily with functional connectivity differences of the insula (part of the saliency network), as well as “dysfunctional mechanisms for emotional and social information processing” (Attanasio et al. 2024). In older adults, the strength of functional connectivity within the saliency network is a key mediator of the positive relationship between social engagement and cognitive abilities (Pruitt et al. 2024). Even changes in levels of social support can induce changes in functional connectivity. A loss in social support during a task is associated with decreased functional connectivity between nodes within the saliency network, and a loss in expected social support leads to increased functional connectivity among nodes within the salience network, the default mode network, and the frontoparietal network (Burhanoglu et al. 2024).

Low-level visual cues

One possible concern in the present study is whether the observed differences in brain activity were caused by high-level cognitive states related to judgments about attention (judged fake versus judged real) or instead by the low-level visual features that led to those judgments. We do not know the specific, low-level visual cues that people use to distinguish real from scrambled attention traces. The two types of stimuli are extremely similar in appearance, since one is merely a random reordering of the other. In our previous study, through 8 experiments, we found that a variety of cues are probably relevant (Ziman et al. 2023). For example, object cues from the underlying image appear to play a role, since changing the underlying image changes performance. Yet people can also distinguish real from scrambled traces even when the underlying image is blacked out and only the attention spotlight is visible. Therefore, even without any object cues, the movement of the attention spotlight itself can provide informative cues. Given these findings, it is likely that participants are using a complex integration of many cues to solve the task. Participants are not necessarily cognizant of these cues, although they do report that the valid attention traces seem less mechanical, more like a conscious agent, and more similar to their own attention (Ziman et al. 2023).

To understand better whether the high-level judgment or the low-level visual cues drove the observed patterns of brain activity in the present experiment, we performed a second analysis, contrasting activity during valid movies with activity during scrambled movies. Since participants made correct judgments on only 61% of trials, separating trials into valid and scrambled largely intermixed the judged real and judged fake movies. In this second contrast, only one small cluster of significant, negative activity was obtained (activity during valid movies larger than during scrambled movies), in the medial parietal cortex. The large positive clusters from the first analysis, found in the DMPFC, precuneus, TPJ, and anterior insula, all disappeared in the second analysis. This finding suggests that whatever specific, low-level visual differences were introduced by randomly scrambling the order of attention locations, they were probably not responsible for the main clusters of activity found in the first analysis. Instead, those clusters of activity were specifically associated with the participants’ judgments about attention pattern.

Author contributions

Kirsten Ziman (Conceptualization, Data curation, Formal analysis, Investigation, Project administration, Writing—original draft, Writing—review & editing, Validation, Methodology, Visualization, Funding acquisition), Sarah Kimmel (Data curation, Formal analysis, Investigation, Validation, Writing—original draft, Writing—review & editing), Isaac Christian (Methodology, Formal analysis, Visualization), Kathryn T. Farrell (Investigation), and Michael Graziano (Conceptualization, Methodology, Resources, Funding acquisition, Supervision, Writing—original draft, Writing—review & editing).

Conflict of interest statement. None declared.

Funding

Supported by AE Studios grant 24400B1459FA010 and NIMH grant T32MH065214.

Data availability

All stimuli, anonymized data, and code for running the experiments and analyzing the data are publicly available online at https://github.com/KirstensGitHub/predicting_attention_neural.