The Ventral Attention Network Mediates Attentional Reorienting to Cross-Modal Expectancy Violations: Evidence from EEG and fMRI

Abstract

Our daily interactions with the world are shaped by sensory expectations informed by context and prior experiences, which in turn influence how we allocate our attention. Prominent predictive coding models suggest that sensory expectancy and attention interact but disagree on the precise mechanisms. One possibility is that the ventral attention network (VAN) may play a role by facilitating attentional reorienting when expectancy is violated. To test this in humans (23 males and 43 females), we employed an auditory-visual trial-by-trial cueing paradigm in three experiments integrating EEG and fMRI to investigate the VAN’s role in violations of cross-modal expectancy. Behavioral results showed faster responses to expected targets, confirming the efficacy of cue-induced expectations in orienting attention to the expected target modality. EEG analyses revealed differences in early (∼100 ms latency) event-related potentials (ERPs) to both auditory and visual stimuli when expectations were violated. Unexpected stimuli elicited significantly larger early-latency negative ERPs, across both modalities. Source localization of these ERPs and subsequent fMRI evidence revealed activation in the right VAN. Functional connectivity analyses showed greater coupling between VAN regions and sensory cortices, with modality-specific pathways involving superior temporal gyrus for auditory and fusiform gyrus for visual targets. These findings demonstrate that expectancy violations recruit the VAN to reorient attention and resolve sensory conflict. By coordinating top-down control and bottom-up sensory input, the VAN supports adaptive responses to unexpected stimuli. This work advances our understanding of predictive processing in multisensory perception and highlights the VAN’s central role in flexible cognitive control.

Significance Statement

This study shows how expectations regarding sensory modalities might cause the brain to reorient attention. The study shows that during cross-modal expectation violations, the right-lateralized ventral attention network (VAN), which includes the temporoparietal junction and inferior frontal gyrus, is quickly engaged using EEG and fMRI. Early ERP differences and higher VAN activation were evoked by unexpected visual and auditory stimuli, which also boosted connection to sensory areas. These results demonstrate the VAN’s critical function in adaptive attention across modalities and further our understanding of prediction processing. The findings have significant implications for sensory integration models, attention models, and disorders involving expectation and control deficiencies.

Introduction

Imagine that you are looking for your cat in the house and are expecting to spot it. Instead, you hear it meow from under the couch before you see it. Such expectations and predictions are ubiquitous in our daily life and perception of the world. The human brain continuously forms expectations about incoming sensory information based on prior experiences and contextual cues. Expectations about sensory modalities, auditory and visual, have mostly been studied in isolation (Rahnev et al., 2011; Summerfield and Egner, 2016; Rungratsameetaweemana et al., 2018). These expectations are thought to function within a hierarchical framework, where higher-order cortical areas generate predictions that are compared with sensory input received by lower-order areas (Rao and Ballard, 1999; den Ouden et al., 2012; Kok et al., 2014). Some theories, generally referred to as predictive coding theories, suggest that any mismatch between predicted and actual sensory information, “prediction-errors,” trigger a neural signal that is inversely proportional to the magnitude of the error (Todorovic and de Lange, 2012; Summerfield and De Lange, 2014; Richter and de Lange, 2019; Walsh et al., 2020; Feuerriegel et al., 2021). Although spatial and feature-based expectation have been investigated extensively in the past (Bastos et al., 2012; Kok et al., 2014; Auksztulewicz and Friston, 2015; Mayer et al., 2016; Samaha et al., 2018), there are very few studies that have investigated cross-modal expectations (Summerfield et al., 2006; Spence and Santangelo, 2009; Arnal et al., 2011; Altieri, 2014; Stekelenburg and Vroomen, 2015).

In the context of cross-modal expectancy, predictions about one sensory modality are violated when information unexpectedly appears in another modality. Specifically, in auditory-visual modalities, expectations about one sensory domain (hearing) can be violated by another (vision). Therefore, the brain must update its predictions and reorient attention toward the unexpected, but task-relevant, stimuli. It has been shown that the ventral attention network (VAN) which includes the right temporoparietal junction (TPJ) and right inferior frontal gyrus (IFG) is recruited to support reorienting between competing stimuli as a function of expectancy (Corbetta and Shulman, 2002; Corbetta et al., 2008; Vossel et al., 2014). The TPJ is thought to support reorientation for attentional control (Geng and Vossel, 2013; Kim, 2014), while the IFG is responsible for inhibitory control to suppress irrelevant information (Aron et al., 2003; Hampshire et al., 2010). Functional neuroimaging (fMRI) studies show that VAN activation is heightened when behaviorally relevant expectations are violated (Kincade et al., 2005; Indovina and Macaluso, 2007; Asplund et al., 2010). For instance, Joseph et al. (2015) observed increased VAN activation in response to invalid gaze cues relative to valid ones. Additionally, VAN activity has been linked to inhibition in tasks requiring rapid suppression of responses to unexpected stimuli. Studies on stop-signal tasks have shown that VAN is activated to inhibit prepotent responses, illustrating its role in reorientation when predictions fail (van Belle et al., 2014; Jahanshahi et al., 2015).

Despite these findings, the VAN’s precise role in cross-modal predictive coding remains poorly understood, particularly, how it interacts with lower-order sensory regions where prediction errors are generated during sensory input violation. Questions remain about how the VAN resolves prediction errors arising from entirely different sensory modalities, such as when task-relevant auditory inputs contradict visual predictions, and about the temporal dynamics of the VAN’s involvement in such processes. EEG studies have shown that violations of expectations elicit enhanced event-related potentials (ERPs), particularly at early and mid-latencies (Czigler et al., 2004; Hall et al., 2018; Tang et al., 2018; Feuerriegel et al., 2021). These ERPs are thought to reflect the brain’s rapid switching response toward relevant but unexpected information.

We investigate the VAN’s role in cross-modal expectancy using a nonspatial, auditory-visual trial-by-trial cuing paradigm (Posner et al., 1980) which induced cross-modal predictions. By integrating EEG and fMRI, we investigated the temporal course of neural activity and probed the VAN and sensory areas for expected and unexpected stimuli.

Materials and Methods

Overview

Three experiments were conducted at University of California Davis, using similar paradigms that manipulated cross-modal expectation. In Experiment 1, electroencephalographic (EEG) data was recorded from participants in one session. In Experiment 2 (EEG) and Experiment 3 (functional magnetic resonance imaging; fMRI), data were obtained from different participants in separate EEG and fMRI sessions. Except for the minor differences in the procedures noted below, the analysis and methods were the same for the three experiments.

Participants

In all three experiments, the healthy subjects reported normal hearing, normal or corrected-to-normal vision, and no history of neurological or psychological disorders. The studies were approved by the institutional review board (IRB) of the University of California, Davis. All participants provided informed written consent and were compensated financially.

In Experiment 1, EEG and behavioral data were recorded from 14 right-handed participants [mean age, 23.29 (3.58); age range, 19–30 years, 7 males]. In Experiment 2, EEG and behavioral data were collected from 38 right-handed participants, but data from six participants were excluded due to poor task performance (N  =  3, accuracy < 60%), excessive muscle artifacts (N = 2), and falling asleep (N  =  1) during the blocks. Therefore, 32 participants were included in the final analysis [mean age, 24.09 (2.76); age range, 20–32, 9 males]. In Experiment 3, fMRI and behavioral data were obtained from 23 right-handed healthy participants, but data from three participants were excluded from further analysis due to poor performance (performance was < 50%, N  =  2) and failure to follow task instructions (N = 1). Therefore, 20 participants were included in the analysis [mean age, 23 (2.35); age range, 20–25, 7 males and 13 females].

Apparatus and stimuli

Experiment 1

Figure 1A provides an overview of trial types and task parameters. Each trial began with the presentation of an attention-directing cue. White cues were presented on a gray background in the middle of a computer screen (19′ Viewsonic VX922 color monitor) for 250 ms. The cues instructed subjects to either prepare for an upcoming stimulus in either the auditory (Λ) or the visual (V) modality or to passively view the upcoming stimulus (passive cues were diamond-shaped passive cues; ◊). After the cue appeared, a subsequent target was presented after a stimulus onset asynchrony (SOA) that was jittered from 1,400 to 1,600 ms. During these cue–target trials, 75% of the targets were validly cued (e.g., visual cue predicted visual target with 0.75 probability), whereas 25% were invalidly cued (e.g., visual cue followed by an auditory target). To compensate for differences in perceptual difficulty across modalities (Bridgeman, 1988), target duration was 50 ms for auditory and 250 ms for visual targets.

Figure 1.

Overview of the experimental designs. A, Experiment 1: Participants were presented with visual cues (lasting 250 ms), either in the form of a “Λ” (expect auditory target) or “V” (expect visual target). After a variable SOA ranging between 1,400 and 1,600 ms, the target stimulus appeared, in the relevant modality (75% of the time) lasting 500 ms. The target could be either a visual (gratings) or auditory (tones) stimulus, with participants required to discriminate the frequency of the auditory tone or thickness of the visual grating. The intertrial interval (ITI) varied randomly from 2,300 to 2,800 ms following target onset and elapsed before the start of next trial. B, Experiments 2 and 3: Participants received a cue (300 ms), represented by either an “H” (“S”) or the word “HEAR” (“SEE”) to expect the modality of the upcoming target. Following an SOA of 1,000–1,500 ms, a target stimulus appeared for 300 ms, in the relevant modality (80% of the time). A mask was applied after the target lasting 50 ms. Participants responded by pressing buttons to indicate the type of tone or thickness of grating. The ITI in Experiment 2 was 2,000–2,500 ms between the onset of target and start of next trial. For Experiment 3, the SOA and ITI were both increased to 4,000–8,000 ms.

In the auditory modality, targets consisted of high-pitched (2,000 Hz) or low-pitched (1,000 Hz) sine wave tones (80 dB) that were presented through speakers standing on the left and right side of the computer screen. In the visual modality, targets consisted of centrally presented black-and-white square wave gratings with a high spatial frequency of 4.8 cycles per degree (cpd) or a low spatial frequency of 0.6 cpd. Subjects were instructed to rapidly press a button with the index and middle finger of the right hand after presentation of a high tone or a high spatial frequency grating and to press a button with the middle finger of the right hand for a low tone or a low spatial frequency grating. The mapping of stimulus attributes to response fingers was counterbalanced across subjects. Task instructions explicitly stressed that the use of information provided by the cue would increase both speed and accuracy of responding and the subjects were informed that the cues predicted the modality of the target. Each run (group of trials) included 96 cued trials (77 cue–target trials and 19 cue-only trials). In both Experiments 1 and 2 (below), the subjects were seated comfortably in a soundproof and electrically shielded room (ETS-Lindgren), which was painted black to prevent any reflection of light. The room was dimly lit with DC lights to ensure a stable lighting environment.

Experiment 2

We utilized a nonspatial auditory-visual cue–target paradigm incorporating cross-modal probabilistic cues, as illustrated in Figure 1B. Each trial started with a 300 ms cue, which was pseudorandomly selected from one of four possible cue types: auditory verbal cues (“HEAR” and “SEE”) and visual letter cues (“H” and “S,” 0.5° × 0.5°). All auditory stimuli were prerecorded and processed using a KEMAR recording manikin, capturing audio at the positions of the tympanic membranes with models of an average adult male pinna, and presented binaurally. Visual stimuli were presented centrally at fixation on a 24 inch VIEWPixx LCD monitor positioned at a viewing distance of 850 mm. The monitor had a native resolution of 1,920 × 1,080 pixels and a refresh rate of 120 Hz (www.vpixx.com). Stimuli presentation was controlled via a Microsoft Windows 10 PC running MATLAB (MathWorks) and Psychtoolbox v3.0.8 (Pelli, 1985; Brainard, 1997). The cues probabilistically signaled the modality of an upcoming target with 80% validity. Specifically, an auditory cue (“HEAR”) or a visual cue (“H”) indicated a high likelihood that the target would be auditory, while an auditory cue (“SEE”) or a visual cue (“S”) indicated a high likelihood of a visual target. Each cue type occurred with equal probability. However, in 20% of trials, an invalid cue was presented, incorrectly predicting the upcoming target’s modality. For instance, an invalid auditory cue (“HEAR”) would be followed by a visual target instead.

During the auditory cues, the fixation cross was present on the screen. After a randomly jittered SOA lasting between 1,000 and 1,500 ms from cue onset, the target stimuli were presented centrally for 300 ms in their respective modalities. It consisted either of a ripple tone (auditory modality) or a grating patch (visual modality). The targets were immediately followed by 50 ms masks—white noise following auditory ripple tones and checkerboard stimuli of the same size after visual gratings. Participants were required to perform a two-alternative forced-choice (2AFC) task, discriminating the ripple frequency of auditory tones or the spatial frequency of visual gratings. Responses were made on two buttons using the right index and middle fingers, and response-button mapping was counterbalanced across participants. Subjects were instructed to respond as quickly and accurately as possible within a 1,000 ms response window. Each trial concluded with an intertrial interval (ISI) jittered between 1,000 and 1,500 ms, ensuring sufficient time before the subsequent trial. Throughout the experiment, participants were required to maintain fixation at the central cross.

The auditory stimuli were created in MATLAB based on the approach by Shamma and colleagues (Shamma, 2001). Specifically, we utilized “ripples”, which are acoustic counterparts of visual gratings. We created these patterns by densely packing 100 tones that are equally spaced on the logarithmic frequency axis, ranging from 50 Hz to 20 kHz, with a crest-to-trough amplitude ratio of 1 and spectral peak density of Ω = 0.1. By using these parameters, we generated seven tones at different ripple speeds of ω = 0,2,4,8,16,20,30 Hz. As for visual targets, we generated six square wave gratings with a size of 5° × 5° and contrast of 1, having a spatial frequency of 5.1, 5.3, 5.5, 5.7, 5.9, and 6.1 cycles/degree.

To familiarize with the task and instructions, participants performed three blocks of a behavioral training prior to the EEG session. During this training session, participants performed all the conditions of the task until they achieved adequate performance (>90%) and proper maintenance of eye fixation. In addition, during this phase, we used a staircase method to select two distinct tones and two distinct gratings (out of the pool) such that the participant was equally able to perform the discrimination task in both modalities. These four stimuli (two tones and two gratings) were used for the discrimination task during the main blocks. After the behavioral training session, the EEG session was divided into eight blocks, each consisting of 80 trials (64 valid and 16 invalid trials), yielding a total of 640 trials per participant. The blocks were interspersed with a rest period of at least one minute when the meaning of the cues was emphasized, and proper eye fixation was encouraged as well.

Experiment 3

The task design and stimuli were identical to those of Experiment 2 (Fig. 1B), but the timing was changed to permit deconvolution of overlapping BOLD responses to events that were close together in time (Das et al., 2023). We used a longer SOA of 4,000–8,000 ms and longer ITI of 4,000–8,000 ms. The same training procedure was used outside the scanner to familiarize the participants with the task. The fMRI session was divided into 8 blocks, each consisting of 40 trials (32 valid and 8 invalid trials), totaling to 320 trials per participant.

EEG recording and preprocessing

Experiment 1

EEG data was recorded using an elastic cap with 124 Ag/AgCl electrodes arranged in spherical coordinates (Easy cap No. M14, Falk Minow Services). Eye movements and blinks were recorded bipolarly from above and below the left eye (vertical electrooculogram; VEOG) and from the outer canthi of each eye (horizontal electrooculogram; HEOG). EEG signals were referenced to Cz during data collection. The AFz electrode (on the forehead) functioned as the ground. After task presentation, electrode positions were digitized using ELPOS (Zebris Medical). Data collection was continuous with a sampling rate of 250 Hz (Scan 4.3, Neuroscan Synamps 2). Online, signals were filtered with a bandpass of 0.05–100 Hz.

Experiment 2

Continuous EEG data was recorded using a 64-channel Brain Products actiCAP active electrode system and digitized using a Neuroscan SynAmps2 input board and amplifier (Compumedics). Signals were recorded using Curry8 EEG acquisition program at 1,000 Hz without any online bandpass filtering. The 64 Ag/AgCl active electrodes were placed on the scalp using the standard international 10–10 montage and actively referenced to a frontal-central midline electrode. To minimize any electrical interference with EEG signals, auditory stimuli were presented through earphones (ER-1, Etymotic Research) via air tubes. Additionally, electrodes at sites TP9 and TP10 were directly placed on the left and right mastoids, respectively. The impedances of the electrodes were maintained below 25 kΩ. Continuous data was stored in individual files corresponding to each trial block.

Signal preprocessing was carried out in MATLAB using EEGLAB toolbox (Delorme and Makeig, 2004) and FieldTrip toolbox (Oostenveld et al., 2011). For each participant, data from every run was concatenated to form a single continuous file. Signals were visually inspected and any portion of the EEG signal containing excessive muscle artifacts, sweat potentials, or large DC drifts was identified and rejected. EEG signals were rereferenced offline to the average of the left and right mastoids (TP9 and TP10). Then, EEG signals were bandpass filtered (0.1–30 Hz) using a Hamming window sinc FIR filter and then downsampled to 250 Hz (for Experiment 2). Independent component analysis (ICA) was performed on the EEG signals to remove components that were associated with eyeblinks and eye movements (Drisdelle et al., 2016). The ICA-processed signals for cues and targets for each trial were epoched from 200 ms before the presentation to 800 ms after their onset. Each trial was carefully screened for any eye-related or muscle-related artifacts. Furthermore, trials containing voltage exceeding +80–150 μV at any electrode location were excluded from further analysis. This process resulted in the rejection of 8.3% of trials on an average.

Subsequently, we sorted individual artifact-free EEG trials with correct responses into different experimental conditions based on type of cue (auditory “HEAR,” “SEE,” and visual H and S; Λ or V), type of target based on the trial context (expected and unexpected), and modality of target (auditory or visual). Separate ERPs were computed by averaging target-locked EEG epochs for each experimental condition and subject. ERP amplitudes for each experimental condition were baseline corrected using a 200 ms prestimulus window.

fMRI acquisition and preprocessing

Experiment 3

Functional magnetic resonance images were obtained on a 3T Siemens Skyra scanner using a 20-channel head coil. Visual stimuli were presented using a MR safe VIEWPixx display (www.vpixx.com), positioned at the end of the scanner bore, and viewed via head coil-mounted mirror. Auditory stimuli were presented via Sensimetrics S14 insert earphones. The parameters for the EPI sequence were as follows: TR, 1,800 ms; echo time, 24.4 ms; flip angle, 40°; field of view, 216 mm; 60 axial slices. The slices were oriented parallel to the plane connecting the anterior and posterior commissures. A GRAPPA acceleration factor of 2 was applied to enhance imaging speed and reduce scan time.

The preprocessing of fMRI BOLD signals was performed using the Statistical Parametric Mapping (SPM12) toolbox and custom MATLAB scripts. The preprocessing pipeline involved slice timing correction, realignment, spatial normalization, and smoothing. Slice timing correction was applied using sinc interpolation to account for variations in slice acquisition times within the EPI volume. Next, the images were realigned to the first image of each session using a 6-parameter rigid body transformation to correct for head movement during the scan. Subsequently, each participant's images were normalized and aligned to MNI space. All images were resampled to a voxel size of 3 × 3 × 3 mm and spatially smoothed with a Gaussian kernel of 7 mm full width at half maximum (FWHM). Additionally, slow temporal drifts were removed by applying a high-pass filter with a cutoff frequency of 1/128 Hz.

Statistical analyses

Behavioral analyses

We studied participants' behavior and how it was influenced by cross-modal expectations by calculating response accuracy and mean reaction times (RT). Trials where RT was <100 ms, greater than 1,000 ms, or with no response were rejected (0.8% of trials). These behavioral measures were analyzed using a two-way repeated-measures analysis of variance (ANOVA) with factors expectation (expected, unexpected) and modality of target (auditory/visual). Follow-up planned two-sided paired t tests were conducted to analyze any main effects. For all ANOVAs conducted in this study, a Greenhouse–Geisser correction was implemented in case where the assumption of sphericity was violated, and the Tukey–Kramer test was applied for post hoc comparisons.

ERP analyses (for Experiments 1 and 2, performed independently)

We analyzed the ERP components and their scalp distributions to examine how cross-modal expectations influenced target processing. First, we visually inspected the grand-averaged ERP waveforms across all participants and conditions, separately for auditory and visual targets. In both Experiments 1 and 2, this revealed a frontocentral negative ERP component peaking between 70–130 ms for auditory targets and 80–150 ms for visual targets. Past studies that have examined task-switching involving auditory and/or visual stimuli have shown similar early latency negative components, peaking between 80–120 ms for auditory stimuli and 90–150 ms for visual stimuli (Vogel and Luck, 2000; Čeponienė et al., 2003; Muller-Gass et al., 2006; Sussman et al., 2014; Gajewski et al., 2018). Therefore, we selected these time windows to measure the ERP negativity elicited by the targets. The latency for auditory targets is faster than that of visual targets, which is consistent with the faster neural transmission and more direct cortical processing pathways of auditory information relative to visual information (Picton et al., 1974). For statistical comparisons, we used the midline electrodes and defined the scalp regions as frontal (AFz, Fz), central (Cz, CPz), and posterior (Pz, POz) since all stimuli were presented centrally, devoid of any strong sensory hemifield bias (Li et al., 2012). To assess ERP amplitude differences, we conducted a repeated-measures ANOVA with target expectation (expected vs unexpected) and scalp region (frontal, central, posterior) as within-subject factors. Post hoc analyses were conducted using two-tailed paired t tests to further explore significant effects. This set of analyses did not yield any new effects on ERP components other than the ones of interest described above.

Source localization

We performed source localization on the preprocessed ERP data using standardized low-resolution electromagnetic tomography (Pascual-Marqui, 2002). First, the electrode positions were coregistered with a standard head model, and the lead field matrix was calculated. We used sLORETA to compute the cortical three-dimensional distribution of current density based on the scalp-recorded ERP data. sLORETA assumes that neighboring neurons exhibit synchronized activity and imposes smoothness constraints to solve the inverse problem. The goal of this analysis was to identify the neural sources underlying the switch-related ERPs evoked by targets under different expectation conditions (expected vs unexpected). We focused on the same time windows where significant differences between unexpected and expected targets were found in the ERP waveforms, based on the ANOVA results.

For each participant, we computed three-dimensional sLORETA maps for both expected and unexpected targets, using averaged ERPs from the selected time windows. These maps were used to estimate the cortical sources contributing to the ERP component and to directly compare the brain activity associated with unexpected versus expected targets. At the group level, source activity between conditions (Unexpected > Expected) was compared using voxel-wise one-tailed paired t tests. We applied a nonparametric permutation test to assess statistical significance (using 5,000 permutations) and corrected for multiple comparisons using the false discovery rate (FDR) method, as implemented in sLORETA. The threshold for significance was set at p < 0.01. Significant source locations, where unexpected targets elicited greater activity than expected targets, were reported using Montreal Neurological Institute (MNI) coordinates. Since there were only 13 subjects in Experiment 1, due to low statistical power, the results did not survive FDR correction and reach significance. An additional goal of this analysis was to compare it with activation derived from the fMRI experiment, which used an adapted design from Experiment 2. Therefore, we report results from Experiment 2 only.

fMRI analyses (for Experiment 3)

We examined cue and target-evoked BOLD responses using the univariate general linear model (GLM) approach (Friston et al., 1994). Eight task-related regressors were included in the model as follows: four regressors for four types of cues (auditory “HEAR,” “SEE,” and visual “H,” “S”) and four targets (expected and unexpected auditory targets, expected and unexpected visual targets) with correct responses only. We conducted a t test by contrasting the beta values from different conditions at the voxel level to generate a t map for each participant. These first-level contrast maps were then entered into a second-level random-effects analysis, implemented as a one-sample t test across participants, to assess effects at the group level and enable population-level inference. Statistical maps were thresholded at p < 0.05 and corrected for multiple comparisons using the FDR method in the SPM toolbox.

Regions of interest

Bilateral IFG and TPJ were determined from the above GLM analyses to target-evoked BOLD activity from the target-evoked fMRI data obtained in Experiment 3. Specifically, whole-brain GLM contrasts comparing unexpected versus expected targets were computed, and the resulting t maps were subjected to p < 0.05 (FDR-corrected) threshold. Voxels showing significant effects in this contrast and located near previously reported IFG (Weiss et al., 2018) and TPJ (Downar et al., 2000; Geng and Vossel, 2013) coordinates were selected as ROIs. Furthermore, the voxels were defined by localizing them to parceled anatomical areas as defined in the AAL atlas (Rolls et al., 2020).

To examine differences in VAN laterality and the effects of expectation on target-evoked BOLD activity, we extracted and averaged activation within the previously selected data-driven ROIs in the IFG and TPJ that comprise the left and right VAN. These were subjected to a repeated-measures ANOVA with the within-subject factors hemispheric laterality (left vs right VAN) and expectation (expected vs unexpected targets), conducted separately for auditory and visual targets. Parallelly, to avoid circularity in the laterality analysis, ROIs were also defined independently of the task-based fMRI contrasts. We adopted the 17-network Schaefer parcellation (Schaefer et al., 2018) variant from Kong et al. (2021), in which Schaefer ROIs were matched to individual-specific 17-network parcellations from Human Connectome Project (HCP) subjects. Parcels assigned to the salience/VAN were selected, encompassing IFG and TPJ in each hemisphere, and were used for all subsequent left–right laterality analyses. The results for these analyses can be found in the Supplementary Materials.

For functional connectivity analysis, we used a β series method to estimate the BOLD activity for each trial and voxel (Rissman et al., 2004). Each event (cues and targets) with a correct response was assigned an individual regressor. One extra regressor was included to model all the rest of the events with incorrect responses. Following this, the regressors were modeled using the GLM approach using custom MATLAB scripts within the SPM toolbox. FDR correction was applied where appropriate.

Functional connectivity

At the subject level, we calculated functional connectivity by averaging β values across voxels within each region of interest (ROI) and performing a Pearson’s cross-correlation analysis across trials. We averaged trials based on modality (auditory and visual) and expectation (expected vs unexpected). At the group level, we derived functional connectivity by averaging each subject’s correlation coefficients and then applied Fisher’s r-to-z transformation to approximate a Gaussian distribution, which allowed us to compare connectivity across cue types. Given the unequal number of trials across conditions, we controlled for trial count by identifying the minimum number of trials across all condition bins. For each condition, based on this minimum count, we then randomly sampled an equal number of trials using a uniform probability distribution. This subsampling procedure ensured balanced comparisons of functional connectivity across conditions while minimizing bias introduced by trial count variability.

Results

Behavioral performance

In our study, participants performed a trial-by-trial auditory-visual cross-modal cuing task, where cues directed them to anticipate the modality of the upcoming target. A 2 × 2 repeated-measures ANOVA was used to analyze task performance and reaction times (RTs), with expectation (expected vs unexpected) and target modality (auditory vs visual) as factors (Fig. 2; Tables 1, 2). Across all three experiments, no significant main effect of expectation was found on task performance, indicating that the practice phase successfully balanced task difficulty across modalities. However, in each experiment, a significant main effect of expectation on RTs emerged, with participants consistently responding faster when the target stimulus was expected compared with when it was unexpected. This effect was observed in Experiment 1 (F_(1,12) = 46.014, p < 0.001), Experiment 2 (F_(1,29) = 35.688, p < 0.001), and Experiment 3 (F_(1,19) = 22.8, p = 0.0001), confirming that the probabilistic cues effectively shaped participants' expectations of the target modality. In terms of target modality, differences in RTs were significant in Experiments 2 and 3, where participants responded more quickly to visual stimuli than to auditory stimuli (Expt. 2: F_(1,29) = 6.094, p < 0.05; Expt. 3: F_(1,19) = 20.269, p = 0.0002). Interestingly, in Experiment 1, we found a significant interaction between expectation and target modality on RTs (F_(1,12) = 13.47, p < 0.01), suggesting that the influence of expectation may have differed depending on whether the target was auditory or visual in this experiment. We did not observe any other significant interactions in Experiments 2 and 3. Although faster responses were generally observed for visual targets, task difficulty was carefully controlled across all conditions, ensuring that participants exerted equal effort when discriminating between auditory tones and visual gratings. As a result, the focus of our EEG analyses was on within-modality differences, rather than cross-modality comparisons.

Figure 2.

Behavioral performance. A, C, E, Participants' accuracy in discriminating target features in Experiments 1, 2, and 3, respectively. There were no significant differences in target discrimination accuracy between auditory and visual modalities across any of the experiments. B, D, F, The reaction times (RTs) to target stimuli in Experiments 1, 2, and 3, respectively. In Panel F, a significant main effect revealed that RTs to visual targets were faster than to auditory targets. (* denotes statistically significant main effects as indicated by ANOVA).

Table 1.

Reaction time (ms) for different targets across three experiments

	N	Auditory Targets		Visual Targets
		Expected	Unexpected	Expected	Unexpected
Experiment 1	13	604 (31)	708 (34)	549 (22)	608 (21)
Experiment 2	30	418 (29)	513 (29)	391 (24)	514 (29)
Experiment 3	20	709 (68)	858 (82)	482 (35)	665 (71)

*values are mean (SEM). N = number of participants.

Table 2.

Performances for different targets across three experiments

	N	Auditory targets		Visual targets
		Expected	Unexpected	Expected	Unexpected
Experiment 1	13	0.98 (0.01)	0.96 (0.03)	0.99 (0.00)	0.98 (0.06)
Experiment 2	30	0.92 (0.01)	0.9 (0.01)	0.9 (0.01)	0.9 (0.01)
Experiment 3	10	0.75 (0.03)	0.76 (0.03)	0.76 (0.02)	0.72 (0.04)

*values are mean (SEM). N = number of participants.

EEG results

Early ERPs to sensory expectations

Auditory targets

In Experiment 1 (Fig. 3A), a 2 × 2 repeated-measures ANOVA of ERP amplitude between 80 and 120 ms with expectation (expected vs unexpected) and scalp regions (frontal, central, parietal) revealed a significant main effect of expectation (F_(1,12) = 9.80, p < 0.01). Unexpected auditory targets generated more negative responses compared with expected targets. The main effect of scalp region was also significant (F_(1,12) = 50.98, p < 0.001), showing that the magnitude of this negative deflection was largest over frontocentral scalp regions. The interaction between expectation and scalp region was not significant (F_(1,12) = 1.03, p = 0.33), which suggests that the effects of prediction updating were consistent across scalp regions.

Figure 3.

Grand-averaged ERP waveforms and topographical maps. A, B, Auditory target responses in Experiments 1 and 2. C, D, Visual target responses in Experiments 1 and 2. Expected (black) and unexpected (red) condition waveforms are overlaid. Auditory targets show greater negative deflections for unexpected stimuli, particularly over frontocentral regions. Visual targets also show greater negative deflections for unexpected stimuli, especially over central and parietal regions. Topographical maps illustrate the scalp distribution of these effects, showing stronger responses for unexpected conditions across both modalities. *Star markers in the topographical maps indicate electrodes used for generating the grand averaged ERP waveforms. Note that the target stimuli were quite different in Experiments 1 and 2, and thus the sensory-evoked waveforms have different amplitudes and slightly different latency profiles, while the negative-going deflections in the waveforms for unexpected targets is quite similar across experiments.

In Experiment 2 (Fig. 3B), the ANOVA showed similar significant main effect of expectation (F_(1,29) = 6.83, p < 0.05), with larger negative deflections for unexpected stimuli, suggesting rapid updates to the participants’ predictive models in response to violation of sensory expectations. The main effect of scalp region was also significant (F_(1,29) = 22.76, p < 0.001), indicating differences in ERP amplitude, with the greatest deflections over frontocentral areas. Importantly, there was a significant interaction between expectation and scalp region (F_(1,29) = 3.50, p < 0.05). Post hoc Tukey's HSD tests showed that ERP amplitudes for unexpected stimuli were more negative in the frontal (p < 0.01), central (p < 0.05), and parietal (p < 0.05) regions compared with expected stimuli.

Visual targets

For visual targets (Fig. 3), a similar effect of switch ERP emerged during the early latency window. Unexpected visual stimuli elicited stronger negative ERP amplitudes, particularly over parietal and occipital regions divided bilaterally across the scalp, reflecting a rapid update of sensory predictions.

In Experiment 1, a 2 × 2 repeated-measures ANOVA on ERP amplitude with expectation and scalp region revealed a significant main effect of expectation (F_(1,12) = 6.8, p < 0.05), indicating that unexpected visual targets elicited stronger negative ERP responses compared with expected targets (Fig. 3C). However, there was no significant main effect of scalp region (F_(1,12) = 0.13, p > 0.05), suggesting that the effects of prediction updating were not topologically specific around the scalp. Finally, the interaction between expectation and scalp region was significant (F_(1,12) = 4.11, p < 0.05). Follow-up paired t tests revealed stronger negativity for unexpected targets at frontal (unexpected: M = −1.70, SD = 0.10; expected: M = −0.55, SD = 0.30) and central regions (unexpected: M = −1.39, SD = 0.42; expected: M = −0.34, SD = 0.80).

Similarly, in Experiment 2, unexpected visual targets elicited significantly stronger negative deflections. The ANOVA showed a significant main effect of expectation (F_(1,29) = 4.40, p < 0.05), reflecting the brain’s need to update visual predictions in response to unexpected targets (Fig. 3D). While there was no significant main effect of scalp region (F_(1,29) = 1.65, p = 0.21), there was a significant interaction between expectation and scalp region (F_(1,29) = 3.97, p < 0.05). Follow-up Tukey’s HSD tests indicated that in the central region, unexpected targets (M = −0.42, SD = 0.11) produced significantly more negative ERP amplitudes than expected targets (M = 0.24, SD = 0.19, p < 0.05).

Despite slight differences in the experimental paradigms, such as varying cue validity (75% in Experiment 1 and 80% in Experiment 2), type of stimuli, experimental conditions, and different participant samples, the switch ERP finding was replicated across both experiments. In both cases, we observed greater negative ERP amplitudes for unexpected stimuli, suggesting that this effect is robust across different task designs and participant groups. This pattern of enhanced negativity in response to unexpected targets across both auditory and visual modalities provides evidence for a mechanism in which the brain must rapidly update its sensory processing when prior expectations are violated.

To summarize, across two experiment using auditory and visual cuing and target presentation, we observed a short-latency negativity elicited by unexpected modality targets. In response to auditory targets, we observed an enhanced frontocentral negativity peaking between 70 and 130 ms across both experiments. This time window corresponds to the brain’s early response to sensory input, typically associated with the auditory N1 component (Vogel and Luck, 2000; Näätänen and Picton, 2007). Similarly, we observed a centro-parietal negativity to unexpected visual targets during 90–150 ms. Traditionally, attention-driven cueing effects are characterized by larger ERPs for valid cues due to the enhanced sensory gain (Hillyard et al., 1998; Thut et al., 2006). However, in the present study, we observed what appeared to be an enhanced negativity for unexpected (invalidly cued) targets.

Such an effect could be considered an enhanced sensory ERP to the unexpected targets, which would align with models of perceptual expectations, where early ERP components reflect deviation from prior expectations rather than a simple attentional enhancement (Friston and Kiebel, 2009; Garrido et al., 2009; Kok et al., 2012). However, we interpret them to the result of an overlapping negativity generated in response to the requirement of switching modalities which we term as the “switch-ERP” effect. The idea is that this effect results when expectations mismatch incoming sensory input (i.e., an unexpected modality is presented), and the brain must rapidly switch to process the relevant modality, triggering a negative polarity ERP. Thus, rather than reflecting a typical cuing effect (larger ERP for valid targets or expectation induced suppression of invalid targets), the switch-ERP negativity may index additional neural resources required to shift attention to an unexpected sensory modality, aligning with prior findings on task-switching and cross-modal expectation violations (Bekker et al., 2005; Mazaheri et al., 2010). This argument is supported by the fMRI results presented below.

Source localization of the enhanced negativity in early latency ERPs

To investigate the neural sources underlying the observed switch-ERP effect (unexpected > expected) in Experiment 2, we performed source localization on the difference wave (unexpected − expected), averaged between the predetermined time windows using sLORETA. As a result, we observed activation in different brain areas which are recruited when processing unexpected sensory information and reorientation to the relevant modality.

Auditory targets

We found key areas in the frontal and temporal regions of the brain that were significantly activated (Fig. 4A). Voxels in the superior (BA 8/9) and middle frontal gyrus (BA 10/11) exhibited bilateral activity, with more activation in the right hemisphere. The IFG (BA 46/47) also showed significant activation, with peak activation in the left but greater number of voxels in the right hemisphere. Moving to the posterior regions, the precuneus (BA 7) was highly active bilaterally, with the right hemisphere showing extensive activation, suggesting a role in higher-order cognitive processes related to auditory stimuli. The superior parietal lobule (BA 7) also revealed strong bilateral activation along with the superior temporal gyrus (STG; BA 22/38), again more dominant in the right hemisphere. Temporal regions, particularly the middle temporal gyrus (MTG; BA 21/39) and inferior temporal gyrus (ITG; BA 20), showed moderate activation bilaterally. For all activated regions and summary, see Table 3.

Figure 4.

Source localization. Results of sLORETA modeling for (A) auditory and (B) visual target responses (unexpected > expected). The t maps show cortical regions that were significantly activated during the processing of unexpected targets compared with expected targets. Source activations were processed using Statistical Non-Parametric Mapping (SnPM) and coregistered to the probabilistic MNI-152 template (Mazziotta et al., 2001). Highlighted areas in yellow and red show regions of significant activation (p < 0.01).

Table 3.

sLORETA analysis for auditory targets in the 70–120 ms time window (unexpected − expected)

Brain region	BA	L/R	MNI coordinates			T value		No. of activated voxels
			X	Y	Z	Mean	Max.
Superior frontal gyrus	8/9	L	8	−10	50	3.86	4.80	49
		R	8	10	50	3.38	4.17	76
Middle frontal gyrus	10/11	L	11	−40	35	3.74	5.68	66
		R	45	50	15	4.02	5.21	78
Inferior frontal gyrus	46/47	L	47	−55	35	3.99	6.20	106
		R	45	−60	55	3.69	4.31	116
Precuneus	7	L	−25	−75	50	4.65	5.88	113
		R	10	−80	45	4.43	5.62	119
Cuneus	17/18	L	19	−5	−90	4.11	5.22	68
		R	5	−90	35	3.94	5.41	78
Superior temporal gyrus	22/38	L	−55	10	−5	3.82	5.26	112
		R	65	−20	0	3.92	4.51	136
Middle temporal gyrus	21/39	L	−65	−5	−15	3.72	4.58	109
		R	65	−25	−15	4.17	4.65	106
Inferior temporal gyrus	20	L	65	−30	−20	3.88	4.73	54
		R	65	−30	−20	3.88	4.73	49
Superior parietal lobule	7	L	−20	−60	65	5.43	6.33	65
		R	20	−60	65	4.84	5.98	63
Inferior parietal lobule	40	L	40	−40	−65	5.29	3.74	54
		R	45	−60	55	4.54	3.62	74
Middle occipital gyrus	18/19	L	−20	−100	5	3.64	4.80	118
		R	20	−100	5	3.62	4.66	55

MNI coordinates correspond to peak activity in a brain region. BA, Brodmann area; L, left; R, right; MNI, Montreal Neurological Institute; mean, mean activation of all voxels; Max., peak activation of voxel in that region.

Visual targets

The sLORETA analysis of visual targets indicated extensive activation across different parts of the brain, overlapping with regions present in the visual processing networks. As shown in Figure 4B, brain regions in the right hemisphere were dominant. The frontal areas, including superior (BA 8/9), middle (BA 10/11), and IFG (BA 45/47), showed strong bilateral activations, indicating their role in visual reorientation and higher-order visual processing.

In posterior regions, the precuneus (BA 7) and cuneus (BA 17/18) demonstrated bilateral engagement, with stronger activation on the right side. Furthermore, the STG (BA 22/38) was involved bilaterally, but with stronger right hemisphere activation. Finally, we also observed significantly activated voxels in the superior (BA 7) and inferior parietal lobules (BA 39/40), particularly in the right hemisphere (see Table 4 for a list of activated regions and statistics).

Table 4.

sLORETA analysis for visual targets (unexpected − expected) during the 90−130 ms time window

Brain region	BA	L/R	MNI coordinates			t value		No. of activated voxels
			X	Y	Z	Mean	Max.
Superior frontal gyrus	8/9	L	−35	55	20	4.61	6.34	181
		R	15	65	−15	4.82	6.71	200
Middle frontal gyrus	10/11	L	−40	50	15	4.01	6.30	204
		R	50	40	−5	4.70	6.97	170
Inferior frontal gyrus	45/47	L	−45	40	15	3.61	5.52	154
		R	55	35	0	4.69	7.20	156
Precuneus	7	L	−5	−50	60	5.43	7.32	163
		R	5	−50	60	5.51	7.57	135
Cuneus	17/18	L	−30	−90	30	4.06	6.15	129
		R	15	−95	25	4.02	5.51	103
Superior temporal gyrus	22/38	L	−60	0	5	4.01	5.08	168
		R	50	20	−20	4.59	6.89	177
Middle temporal gyrus	21/39	L	−60	−65	5	4.67	6.05	167
		R	55	10	−25	4.35	6.28	176
Inferior temporal gyrus	20	L	−60	−65	−10	4.74	5.70	78
		R	50	−75	−5	4.06	5.38	73
Lingual gyrus	18/19	L	−25	−95	−10	4.20	5.63	89
		R	20	−100	−10	4.13	5.37	80
Superior parietal lobule	7	L	−60	−65	−10	4.74	5.70	78
		R	50	−75	−5	4.06	5.38	73
Inferior parietal lobule	39/40	L	−35	−55	60	4.32	5.96	117
		R	40	−70	45	4.11	6.00	116
Fusiform gyrus	20/37	L	−60	−5	−30	3.93	5.34	115
		R	20	−95	−20	3.92	5.08	110

MNI coordinates correspond to peak activity in a brain region. BA, Brodmann area; L, left; R, right; MNI, Montreal Neurological Institute; mean, mean activation of all voxels; Max., peak activation of voxel in that region.

fMRI results

To further investigate the enhanced ERP negativity observed when sensory expectations were violated during the early latencies, we conducted fMRI using the same stimuli and an adapted-to-fMRI experimental design (Das et al., 2023). Our primary goal was to determine whether unexpected targets evoke distinct neural activity compared with expected targets. Secondly, we aimed to identify the cortical areas responsible for processing unexpected information and relate them to the observed ERP differences. To this end, we directly compared BOLD activity between unexpected and expected targets using a GLM analysis.

fMRI activations (unexpected > expected targets)

Auditory targets

The auditory fMRI analysis revealed several brain regions that showed greater activation in response to unexpected auditory targets compared with expected ones. Significant differences were observed across multiple regions, including frontal, temporal, and parietal areas (Table 5; Fig. 5A,B).

Table 5.

MNI coordinates and corresponding t values for brain regions that showed greater activation in response to unexpected compared with expected auditory targets

Brain region	BA	L/R	MNI coordinates			t value
			X	Y	Z
Superior frontal gyrus	11	L	−12	50	−22	2.49
		R	42	23	47	2.49
Inferior frontal gyrus	44/47	L	−33	20	−7	3.62
		R	45	11	26	3.67
Superior temporal gyrus	22	R	57	−13	−10	3.41
Middle temporal gyrus	21	L	69	−10	−13	2.46
		R	−66	−10	−7	3.18
Inferior temporal gyrus	20	L	−33	−31	−19	2.71
		R	57	−46	−13	5.60
Lingual gyrus	18	R	12	−94	−16	2.26
Precuneus	5	R	3	−58	68	2.65
Angular gyrus	39	L	−42	−67	32	4.24

BA, Brodmann areas; L/R, left/right.

Figure 5.

Activation of the ventral attention network (VAN) in response to unexpected versus expected sensory stimuli across auditory and visual modalities. A, Axial and sagittal brain slices showing significantly greater activation in the right inferior frontal gyrus (rIFG) and right temporoparietal junction (rTPJ) for unexpected stimuli compared with expected stimuli, across auditory (red) and visual (green) targets. B, Left and right VAN activity for unexpected and expected targets. Increased activation in the rIFG and rTPJ for unexpected stimuli in both auditory (top) and visual (bottom) modalities. This suggests that the VAN is engaged in response to cross-modal prediction violations. C, Cue-related activations in left TPJ and right TPJ for auditory (top row) and visual (bottom row) cues. D, Cue-related activity in the left and right VAN regions. In both hemispheres, the IFG was deactivated during the cue period.

In the frontal areas, bilateral superior, middle, and inferior frontal gyri exhibited increased activation, suggesting the involvement of executive processes when the auditory target was not expected. Of particular interest was the increased activation of the IFG, known to play a key role in inhibitory control and the reallocation of attention to novel or unexpected events (Aron et al., 2003). Increased activity in right IFG is typically associated with suppression of expected responses so that the brain can reorient toward unexpected auditory stimuli (Hampshire et al., 2010). Additionally, in the temporal regions, STG on the left hemisphere, traditionally associated with primary auditory processing, displayed increased activation, indicating its role in processing the unexpected auditory information (Yi et al., 2019). Additionally, the right MTG showed greater activation, reflecting its involvement in processing complex auditory stimuli, such as semantic content or novel auditory features (Weiss et al., 2018; Ren et al., 2020). The TPJ, including regions surrounding the MTG and supramarginal gyrus, also demonstrated significant activation. The right TPJ is critical for bottom-up attention processes, particularly for detecting and responding to unexpected stimuli (Corbetta and Shulman, 2002; Solís-Vivanco et al., 2021). Other regions, such as the lingual gyrus and lateral occipital gyrus, also showed increased activation in response to unexpected auditory targets, potentially indicating cross-modal processing triggered by unexpected events (Downar et al., 2000; Sidlauskaite et al., 2014).

Visual targets

The fMRI analysis for visual targets similarly revealed stronger activation in several brain regions when participants processed unexpected visual stimuli compared with expected ones (for all regions, see Table 6). Similar to auditory targets, IFG and TPJ were more activated for unexpected visual targets compared with expected ones (Fig. 5A,B).

Table 6.

MNI coordinates and corresponding t values for brain regions that showed greater activation in response to unexpected compared with expected visual targets

Brain region	BA	L/R	MNI coordinates			t value
			X	Y	Z
Middle frontal gyrus	11	L	−27	11	56	4.88
		R	30	41	−10	3.45
Inferior frontal gyrus	45/47	L	−33	29	5	3.46
		R	30	23	−10	2.75
Middle temporal gyrus	21	L	−60	−22	−13	3.54
		R	57	−58	20	7.65
Inferior temporal gyrus	20/37	L	−54	−10	−31	2.74
		R	51	−10	−31	2.74
Lingual gyrus	18	R	21	−70	−13	1.93
Intraparietal sulcus	7	L	−36	−55	53	2.17
		R	33	−70	50	4.77
Intraparietal sulcus	40	L	−51	−46	50	2.51
Supramarginal gyrus	22	R	66	−37	5	2.02
Fusiform gyrus	20/37	L	−42	−40	−22	3.19
		R	39	−40	−22	3.19

BA, Brodmann areas; L/R, left/right.

Additionally, in the frontal cortex, we observed increased activation in the left superior frontal gyrus and the right middle frontal gyrus. Furthermore, in the temporal areas, bilateral MTG and the right ITG showed greater activation to unexpected visual targets. Importantly, significant activation was also found in the right TPJ, including regions such as the MTG, inferior parietal lobule, and lateral occipital gyrus.

Target-evoked activity in left versus right VAN

For both auditory and visual targets, a right laterality in VAN activity emerged (Fig. 5B). To directly test for effects of laterality, we subjected target-evoked BOLD activity in the VAN ROIs to a repeated-measures ANOVA with factors of VAN laterality (left and right) and expectation (unexpected and expected). The ROIs used in this analysis were defined from group-level, statistically thresholded t maps of the unexpected > expected GLM contrast (see Materials and Methods). To address potential concerns regarding circularity, we additionally performed a complementary analysis using independently defined, atlas-based ROIs derived from the Schaefer parcellations (Kong et al., 2021), which is reported in the Supplementary Materials (Fig. S2).

For auditory targets, the analysis revealed significant main effects of expectation (F_(1,20) = 290.91, p < 0.0001), and VAN laterality (F_(1,20) = 357.51, p < 0.0001). Importantly, there was also a significant interaction between expectation and VAN laterality (F_(1,20) = 948.01, p < 0.00001), indicating that the effect of expectation varied by hemispheric VAN laterality. Post hoc pairwise comparisons using Tukey’s HSD indicated that neural responses in the left VAN did not significantly differ between the expected and unexpected conditions (p = 0.287, 95% CI [−0.11, 0.34]). In the right VAN, unexpected targets elicited significantly greater neural responses than the expected condition (p < 0.00001, 95% CI [2.66, 3.04]).

For visual targets, the analysis revealed similar significant main effects of expectation (F_(1,19) = 394.87, p < 0.001) and VAN laterality (F_(1,19) = 83.67, p < 0.001). Additionally, we also found a significant interaction effect between laterality and expectation (F_(1,19) = 71.62, p < 0.001), indicating that the effect of expectation differed across hemispheres. Follow-up pairwise comparisons using Tukey's HSD showed that the deactivation elicited in the left VAN was significantly different between unexpected (M = −0.28) and expected (M = −1.03; p < 0.001, 95% CI [0.56, 1.01]) conditions. On the other hand, activation in the right VAN was more pronounced and significantly different between unexpected (M = 1.75) and expected (M = −0.09; p < 0.001, 95% CI [1.62, 1.87]) targets.

Finally, we also examined cue-related activity within the same VAN voxels that were activated during target processing (Fig. 5C,D). Interestingly, in contrast to the right-lateralized activations observed for targets, both left and right TPJ were activated in response to cues. However, we observed deactivation in bilateral IFG during this cue period. This is expected during the cue period, as the brain is actively forming predictions and preparing for the anticipated modality, which reduces the demand for inhibitory control typically associated with IFG activation (Shulman et al., 2003; Braga et al., 2013; Vossel et al., 2014).

Overlap between source localization and fMRI activations

The fMRI results from Experiment 3 revealed key activations within frontoparietal regions, indicating the engagement of VAN in response to unexpected stimuli, across both auditory and visual modalities. While these findings confirm VAN’s involvement in processing cross-modal violations, the poor temporal resolution of fMRI (Das et al., 2023) limits our ability to delineate the precise timing and dynamics of VAN. To solve this, we incorporated the source localization areas from Experiment 2, overlaying the thresholded (p < 0.05) sLORETA maps onto the fMRI-derived difference maps from Experiment 3. By calculating sLORETA maps of averaged difference of ERPs in successive 40 ms time windows around the peak of the switch ERP effect, we aimed to precisely identify the temporal evolution of neural responses within the VAN.

For auditory targets (Fig. 6A–C), the overlap between ERP source localization and fMRI activations began to emerge in the right IFG during the 40–80 ms window. Following that, during the 80–120 ms window, overlap occurred for both right IFG and right TPJ, coinciding with the observed auditory switch ERP effect. Following that, during the 120–160 ms window, the overlap reduced, remaining mainly in the right IFG. In case of visual targets (Fig. 6D–F), we started the 40 ms moving window from 48 ms, so that we could capture the preselected time window (90–150 ms). Here, the overlap between ERP and fMRI activations appeared earlier, with the right IFG activation beginning during the 48–88 ms window, even before the onset of the visual switch ERP effect. During the peak visual switch ERP window (88–128 ms), both right IFG and TPJ show prominent activation, paralleling the pattern seen in auditory processing. After this peak window, the overlap was confined to the right IFG only. Interestingly, in both auditory and visual modalities, the right TPJ overlap was prominent only during the peak switch ERP time window. This progression suggests that while both TPJ and IFG are involved early on, TPJ’s engagement is more transient and is related to the enhanced negativity of the switch ERP, while IFG continues to support the ongoing inhibitory control required to manage the task switch.

Figure 6.

Time-resolved fMRI and source localization activation overlap. Top row, Auditory targets. A–C, Brain activation maps for auditory targets (unexpected > expected) at different time intervals following target onset. A, At 40 ms poststimulus, initial activations are observed primarily in the superior temporal and frontal regions, particularly in the right inferior frontal gyrus (IFG). B, At 80 ms, overlap spreads, with significant engagement in the inferior frontal gyrus (IFG) and temporoparietal junction (TPJ). C, At 120 ms, the overlap is limited to right IFG. Red shading represents areas of fMRI activation, blue shading represents source localization using sLORETA, and purple areas indicate overlap between these maps. Bottom row, Visual targets. D–F, Brain activations for visual targets (unexpected > expected). D, At 48 ms, initial activations overlap in the right IFG. E, By 88 ms, the right TPJ and IFG show increased overlaps. F, Between 128 and 168 ms, the overlap is limited to right IFG only. Green shading represents areas of fMRI activation for visual targets, blue for source localization, and cyan areas indicate overlap between sLORETA and fMRI maps.

Connectivity between VAN and sensory areas

In bottom-up sensory processing, the IFG is essential for inhibitory control and reallocation of attention, especially when encountering novel or unexpected stimuli. It has been widely recognized for its role in inhibiting prepotent responses when unexpected or novel stimuli are encountered (Aron et al., 2003; Hampshire et al., 2010). The right TPJ is involved in attentional reorientation, particularly in detecting and responding to unexpected, novel, or salient stimuli (Geng and Mangun, 2011; Geng and Vossel, 2013). Together, the IFG and TPJ coordinate within VAN to support the process of disengagement from ongoing, goal-directed tasks (driven by the dorsal attention network) and reorient toward unexpected but potentially important events.

We conducted a functional connectivity analysis at the single-trial level using beta (β) values (Rissman et al., 2004). Our goal was to determine whether target-related functional connectivity between regions in the VAN–the IFG and the TPJ varied based on whether the target was expected or unexpected. The right IFG and TPJ showed heightened activity for unexpected auditory and visual stimuli, suggesting a role of the VAN in prediction violations across modalities. Research suggests that the VAN supports directing attention to a task relevant stimulus and interacts with sensory areas (Sabine et al., 1999; Hopfinger et al., 2000; Corbetta and Shulman, 2002; Egner et al., 2010; Li et al., 2012; Sreenivasan et al., 2021). Even though univariate analysis, including our study, shows evidence for shifts in frontoparietal activity in response to unexpected stimuli, in support of this theory, the evidence remains indirect. What is the relation of this activity within the VAN regions and regions outside VAN? The beta-series correlation method is a functional connectivity analysis technique that leverages trial-by-trial variability to assess covariance in activity across distant brain regions, offering a more direct approach to study network interactions (Rissman et al., 2004; Sreenivasan et al., 2021).

To understand how IFG interacts with TPJ in cross-modal predictive processing, we analyzed functional connectivity between these regions under different conditions. During unexpected auditory targets, functional connectivity between the right IFG and TPJ was significantly stronger (p = 0.004, d = 0.36) compared with expected targets (d = 0.19), as shown in Figure 7. A similar pattern emerged for visual targets, with higher functional connectivity (p = 0.009, d = 0.34) for unexpected versus expected targets (d = 0.14), as shown in Figure 8. No such connectivity differences were observed between the left IFG and TPJ, for auditory (p = 0.7) or visual targets (p = 0.30), indicating sensitivity in the right VAN to unexpected sensory events. Since we also observed greater activation to unexpected targets in sensory areas (Tables 5, 6), which are implicated to be lower order areas in the hierarchical system where prediction errors are generated (Kok et al., 2014), we tested the functional connectivity between them and areas in the VAN. In case of auditory modality, we found greater functional connectivity between left STG and right TPJ for unexpected (d = 0.25) compared with expected (d = 0.09) auditory targets (p = 0.03). We did not observe the same effect in the right STG or between right IFG and STG. Along similar lines, for visual modality, we found greater connectivity between left fusiform gyrus (FG) and right TPJ for unexpected (d = 0.34, p = 0.02) compared with expected targets (d = −0.19). We did not find any connectivity between right FG and regions in VAN. This indicates a likely involvement of the left FG (only) while processing visual gratings.

Figure 7.

Functional connectivity analysis of auditory target processing between regions in VAN, comparing expected and unexpected auditory targets. A, Connectivity between the left IFG and left TPJ shows no significant difference between unexpected and expected auditory targets. B, Connectivity between the right IFG and right TPJ shows significantly higher connectivity for unexpected auditory targets compared with expected ones (*p = 0.01). C, Connectivity between the right TPJ and left STG is also significantly higher for unexpected compared with expected auditory targets (*p = 0.03). D, Functional connectivity is higher in the right VAN during unexpected auditory targets. Further, right TPJ is functionally connected to the left STG. Solid orange lines indicate significant connectivity for unexpected auditory targets, while dashed lines represent nonsignificant connectivity. The color coding corresponds to expected and unexpected conditions across the regions: IFG, inferior frontal gyrus; TPJ, temporoparietal junction; STG, superior temporal gyrus; VAN, ventral attention network.

Figure 8.

Functional connectivity analysis of visual stimuli processing within regions of the ventral attention network (VAN), comparing expected and unexpected visual targets. A, Connectivity between the left IFG and left TPJ shows no significant difference between expected and unexpected visual targets. B, Connectivity between the right IFG and right TPJ is significantly higher for unexpected visual targets compared with expected visual targets (*p = 0.007). C, Connectivity between the right TPJ and left FG is also significantly greater for unexpected visual targets compared with expected ones (*p = 0.0). D, The schematic illustrates that functional connectivity in the right VAN is enhanced during unexpected visual targets, with additional connectivity between the right TPJ and left FG. Solid orange lines indicate significant connectivity for unexpected visual targets, while dashed lines represent nonsignificant connectivity. IFG, inferior frontal gyrus; TPJ, temporoparietal junction; FG, fusiform gyrus; VAN, ventral attention network.

Negative VAN–behavior coupling predicts perceptual advantage for unexpected targets

To further clarify the functional significance of right VAN activity in predictive processing and perception, we conducted a linear mixed-effects model analysis with reaction times (RT) and expectation as fixed effects and a random intercept for each subject. For auditory targets, we observed significant main effects of expectation (β = 0.33, p = 0.02) and RT (β = 0.002, p = 0.03) on right VAN activity. Importantly, the interaction between RT and expectation was also significant (β = 0.002, p = 0.009). For visual targets, we again observed significant main effects of RT (β = 0.05, p = 0.01) and expectation (β = 0.38, p = 0.02) on right VAN activity, but no significant interaction between the two factors. Together, these results indicate that both expectation and trial-by-trial behavioral performance modulate right VAN engagement.

To complement these findings, we next performed a correlation analysis between single-trial RTs and right VAN activation. As shown in Figure 9, both auditory and visual targets showed a negative relationship between right VAN activity and RTs for unexpected targets. For auditory targets (Fig. 9A), Z-transformed correlation coefficients were significantly more negative for unexpected compared with expected targets (p = 0.01). A similar pattern was observed for visual targets (Fig. 9B), with more negative correlation coefficients for unexpected compared with expected trials (p = 0.008). In both modalities, unexpected targets elicited stronger VAN–behavior coupling. These results indicated greater right VAN activity to be associated with successful attentional switching, enabling participants to rapidly disengage from ongoing task sets and reorient toward behaviorally relevant, unexpected targets. We did not find observe this effect between the left VAN activity and RTs.

Figure 9.

Correlation between reaction times and right VAN activity at the single-trial level. For both auditory (A) and visual (B) targets, right VAN activity negatively correlates with reaction times suggesting a perceptual advantage (in the form of faster RTs) related to successful switching (right VAN activity). Error bars represent SEM.

Discussion

We examined how the VAN supports reorienting attention across sensory modalities when expectations are violated. Using an auditory-visual probabilistic cueing paradigm, participants formed expectations about the upcoming task-relevant modality. Across three experiments combining EEG, fMRI, and connectivity analyses, we found convergent evidence that violations of cross-modal expectations engage the right-lateralized VAN.

Across experiments, unexpected stimuli elicited stronger negative-going ERPs for both auditory and visual targets, consistent across procedural variations. Source localization and fMRI confirmed right-lateralized activations in the TPJ and IFG, accompanied by increased VAN-sensory connectivity. These results suggest that VAN-mediated control processes enable attention to shift from attended to unattended modalities, integrating predictive and attentional mechanisms during expectation violations.

Further, by directly testing for hemispheric differences in target-evoked activity, we found support for a right-hemisphere biased VAN engagement during expectancy violations. We found significant interactions between expectation and laterality, with unexpected stimuli eliciting reliably stronger responses in the right VAN compared with the left. For both auditory and visual targets, unexpected stimuli selectively increased activation in the right IFG and TPJ, while left-hemisphere responses did not differ between expected and unexpected conditions. This right-hemisphere bias in TPJ and IFG activation aligns with the VAN’s known lateralized functions in attentional reorientation (Corbetta and Shulman, 2002; Vossel et al., 2012) and perhaps predictive processing (Ficco et al., 2021).

Early-latency ERP responses to cross-modal expectation violations

Across Experiments 1 and 2, unexpected targets evoked an early negative-polarity ERP (auditory, 70–120 ms; visual, 90–150 ms) larger than for expected targets. These effects resemble mismatch negativity (MMN) responses typically associated with prediction errors (Näätänen et al., 2007; Fitzgerald and Todd, 2020; vMMN; Stefanics et al., 2015) though our cross-modal paradigm implicates broader mechanisms. Rather than a pure prediction-error or expectation-suppression effect (Summerfield and Egner, 2016; Tang et al., 2018), the results indicate early recruitment of VAN regions involved in reorienting attention across modalities.

sLORETA and fMRI identified the IFG and TPJ as primary sources of the early ERP component. Similar early negativities observed in task-switching paradigms (Jamadar et al., 2010; Czernochowski, 2011) suggest activation of frontoparietal control systems that suppress ongoing responses and redirect attention toward relevant sensory input (Li et al., 2012; Zhuo et al., 2021). Consistently, Bekker et al. (2005) reported enhanced early frontocentral negativities during inhibitory control tasks, reinforcing the interpretation that the VAN, driven by IFG and TPJ, facilitates rapid disengagement from the expected modality and reallocation of processing to the unexpected, task-relevant stimulus, supporting efficient cross-modal reorienting under expectancy violations.

In our paradigm, the brain faced a unique form of mismatch processing involving cross-modal violations rather than unmet predictions within a single modality. When expectations in one sensory channel were violated, the predicted input failed to occur while an unexpected stimulus appeared in another modality (Foxe et al., 2005; Sidlauskaite et al., 2014). Consequently, the observed switch ERP reflects not merely prediction error or expectation suppression but a broader mechanism integrating perceptual violation, task-switching, and attentional reorienting. Engagement of the right IFG and TPJ within the VAN indicates that the brain actively resolves sensory conflict and reallocates attention when predicted modalities are violated. Although predictive coding models (Friston and Kiebel, 2009; Arnal et al., 2011; Tang et al., 2018) frame such responses as model updates following prediction errors, our ERP, fMRI, and connectivity evidence suggest that attentional reorienting predominates in driving enhanced neural responses to unexpected stimuli.

Importantly, this conclusion stems from a task demanding disengagement from one sensory modality and engagement with another, placing unusually high demands on the VAN (Spence and Driver, 1997; Downar et al., 2001). Nevertheless, our findings suggest that VAN-mediated reorienting could also contribute to expectation-suppression effects in unisensory contexts, particularly when violations are salient and behaviorally relevant (Summerfield and Egner, 2009; Kok et al., 2012; Alink and Blank, 2021). In such instances, VAN engagement may facilitate prediction-error mechanisms, with their relative influence varying according to task demands, sensory modality, and the degree of attentional capture.

Neural circuitry of the VAN

Combined fMRI and source analyses revealed temporally distinct but sequential engagement of VAN regions. Early IFG activity preceded the main switch-related ERP component, suggesting an inhibitory or “stop-signal” role (Chambers et al., 2009; Bollinger et al., 2010; Jahanshahi et al., 2015; Tomiyama et al., 2022), possibly suppressing prior expectations to enable adaptive responses (Eng et al., 2015). Subsequently, TPJ activation emerged during the peak ERP window and then declined, while IFG activation persisted. This sequence indicates that IFG initiates suppression of the irrelevant modality, whereas TPJ mediates reorienting toward the relevant input. These dynamics parallel findings from ERP–fMRI task-switching studies (Jamadar et al., 2010), which report early prefrontal activation followed by later parietal engagement. This pattern supports a flow of information within the VAN where IFG acts as a control-gate that triggers TPJ-driven attentional reorientation (Knight and Scabini, 1998; Vossel et al., 2014). Thus, VAN engagement reflects a temporally ordered coordination of inhibition and reorientation processes necessary for flexible cross-modal adaptation.

Functional connectivity analyses showed increased coupling between IFG and TPJ during unexpected trials for both auditory and visual targets (Figs. 7, 8). Prior research indicates that expectation violations modulate prefrontal–parietal communication to update top-down control (Bressler et al., 2008; Corbetta et al., 2008; Chambon et al., 2017; Masina et al., 2022). We suggest that this IFG–TPJ pathway enables suppression of defied predictions while enhancing attention to behaviorally relevant sensory input. Beyond this shared VAN pathway, modality-specific differences emerged. For auditory targets, VAN connectivity extended to the left STG, whereas visual targets showed links to the left FG. The STG’s involvement aligns with its established role in auditory prediction error processing (Kim, 2014; Liu et al., 2023), while FG engagement reflects visual error detection (D'Astolfo and Rief, 2017). Together, these findings indicate that VAN interacts dynamically with modality-specific regions, forming a feedback loop where top-down control from IFG–TPJ modulates sensory processing while bottom-up signals refine expectations (Rahnev et al., 2011). Increased VAN–sensory connectivity for unexpected stimuli suggests that VAN not only mediates attentional reorienting but also recalibrates sensory processing to enhance perception of relevant violations.

Trial-by-trial analyses revealed that stronger right VAN activation correlated with faster reaction times across both modalities (Fig. 9). This negative correlation indicates that effective VAN engagement facilitates perceptual efficiency under expectancy violation. Such coupling between VAN activity and behavior extends prior evidence for its role in attentional reorienting (Vossel et al., 2014), showing that trial-level variations in VAN recruitment predict behavioral efficiency.

Rapid TPJ and IFG responses to violated expectations

We observed robust right TPJ activation during expectancy violations across modalities, consistent with its role in attentional reorienting. Although TPJ activity is often associated with the P300 component (250–400 ms; Menon et al., 1997; Soltani and Knight, 2000; Bledowski et al., 2004), our findings revealed earlier activation ∼100 ms, suggesting TPJ involvement in rapid detection rather than later contextual updating. This early activation likely serves as an attentional “neural-warning,” flagging unexpected inputs and initiating reorientation, analogous to MMNs (Alain and Woods, 1997; Ritter et al., 1999).

Supporting this interpretation, MEG studies have shown gamma-band TPJ responses within 100 ms to unexpected visual changes (Beauchamp et al., 2012). Similarly, early neural markers of attentional reallocation predict later P300 magnitude (Naeije et al., 2016). We observed comparable correlations between the magnitude of early switch ERPs and later P300 amplitudes (Fig. S1), indicating that initial prediction-error detection shapes subsequent contextual updating.

The right IFG showed a parallel pattern of engagement. The IFG has long been identified as a core node in both predictive coding and cognitive control frameworks. On the one hand, it has been described as a stable component of predictive networks, particularly in facilitating top-down modulation of sensory processing and integrating prior expectations with incoming sensory evidence (Arnal and Giraud, 2012; Wang et al., 2016). On the other hand, the IFG is also strongly associated with inhibitory control and the suppression of prepotent or contextually inappropriate responses (Aron et al., 2003; Hampshire et al., 2010). In our paradigm, IFG activity emerged prior to the main switch-related time window, suggesting a preparatory role in rapidly interrupting ongoing task sets and enabling the reallocation of attention toward the unexpected, task-relevant modality. This dual characterization, as both a predictive hub and an inhibitory control region, positions the IFG as a key interface between prediction error signaling and attentional reorienting. By rapidly engaging, the IFG may facilitate conflict resolution when predicted and actual sensory events diverge, thereby enabling the TPJ and other VAN regions to execute a swift reorientation response.

Conclusion

Our findings underscore the right-lateralized VAN’s critical role in coordinating attention and prediction during cross-modal expectation violations. By suppressing outdated expectations, reallocating attention, and enhancing perceptual processing, the VAN supports adaptive control across modalities. Future research using high-temporal resolution and naturalistic paradigms should further elucidate how VAN-sensory interactions underpin flexible predictive processing in complex multisensory environments.