Mapping the spatiotemporal dynamics of processing task‐relevant and task‐irrelevant sound feature changes using concurrent EEG‐fMRI

Abstract

The cortical processing of changes in auditory input involves auditory sensory regions as well as different frontoparietal brain networks. The spatiotemporal dynamics of the activation spread across these networks has, however, not been investigated in detail so far. We here approached this issue using concurrent functional magnetic resonance imaging (fMRI) and electroencephalography (EEG), providing us with simultaneous information on both the spatial and temporal patterns of change‐related activity. We applied an auditory stimulus categorization task with switching categorization rules, allowing to analyze change‐related responses as a function of the changing sound feature (pitch or duration) and the task relevance of the change. Our data show the successive progression of change‐related activity from regions involved in early change detection to the ventral and dorsal attention networks, and finally the central executive network. While early change detection was found to recruit feature‐specific networks involving auditory sensory but also frontal and parietal brain regions, the later spread of activity across the frontoparietal attention and executive networks was largely independent of the changing sound feature, suggesting the existence of a general feature‐independent processing pathway of change‐related information. Task relevance did not modulate early auditory sensory processing, but was mainly found to affect processing in frontal brain regions. Hum Brain Mapp 37:3400–3416, 2016. © 2016 Wiley Periodicals, Inc.

INTRODUCTION

Auditory sensory input delivers a constant stream of information on the state of the surrounding environment. Rapidly detecting and evaluating changes in the auditory input and, if necessary, initiating adequate actions to cope with these changes is essential for adaptive cognition and behavior. Functional magnetic resonance imaging (fMRI) experiments provide compelling evidence that the neural processing of changes involves subcortical and cortical auditory sensory brain regions, but also a widespread network of frontal and parietal brain areas [Downar et al., 2000; Kiehl et al., 2001; Salmi et al., 2009]. The overall cortical activation pattern thereby largely overlaps with different frontoparietal brain networks involved in attention control and action selection. The right‐lateralized ventral attention network, consisting of the temporoparietal junction, the anterior insula, and parts of inferior and middle frontal gyrus, has been related to stimulus‐driven reorienting of attention, whereas the dorsal attention network, including the frontal eye fields, the intraparietal sulcus and parts of the superior parietal lobe, is thought to be involved in voluntary control of attention [Corbetta et al., 2008; Fox et al., 2006; Vossel et al., 2014]. Although these networks have been mainly researched in the context of visual processing, there is a growing body of evidence suggesting that similar brain regions are also engaged during the allocation and maintenance of auditory attention [Ahveninen et al., 2013; Green et al., 2011; Kong et al., 2014; Lee et al., 2014]. Both networks are regularly found to be jointly activated in response to changes in auditory input [e.g., Alho et al., 2014]. Activation differences between task‐relevant and irrelevant changes in auditory input have been mainly observed within the so‐called central executive network, which involves parts of the medial frontal and cingulate cortex, the inferior and middle frontal gyrus, and lateral parietal brain regions and is thought to be essentially involved in action selection and performance monitoring [Downar et al., 2001; Menon, 2011; Vincent et al., 2008].

While functional neuroimaging can reliably map the spatial pattern of brain regions involved in auditory change processing, it cannot, due to the sluggishness of the blood oxygenation level‐dependent (BOLD) response, provide useful information about the temporal order in which different brain regions or functional networks are engaged during this process. One attempt to overcome this limitation is to combine fMRI measurements with concurrently acquired electroencephalographic (EEG) data, thus gaining simultaneous information on both the spatial and temporal patterns of activation [Debener et al., 2006; Jorge et al., 2014; Ritter and Villringer, 2006]. Previous research successfully applied this technique to identify the cortical generator sites of specific EEG components related to auditory deviance processing [Benar et al., 2007; Goldman et al., 2009]. So far, no EEG‐fMRI study however mapped the entire spectrotemporal dynamics of auditory deviance processing from early sensory detection to attention reorienting and action selection.

We here aimed to close this gap using an fMRI‐informed EEG source reconstruction approach and an auditory stimulus categorization task in which auditory tone complexes had to be sorted either according to their pitch or their duration. The applied stimulus categorization rule was switched in random intervals, so that the spatiotemporal dynamics of change‐related activity could be studied depending on both the changing sound feature and the task relevance. For brain regions showing significant change‐ or task relevance‐related BOLD responses EEG source time courses were computed and activation peak latencies were assessed, allowing us to map the temporal dynamics of neural activity within these areas.

Based on previous electrophysiological and concurrent EEG‐fMRI evidence, we hypothesized to observe first major cortical change‐related activity about 100 ms after deviance onset in auditory sensory and frontal brain regions and to subsequently spread into parietal lobe for attention reorienting [Menon and Uddin, 2010; Näätänen et al., 2011]. Task relevance‐related differences in processing were mainly reported to occur in frontal and parietal lobe, but have in some studies also been observed at the level of auditory cortex [Kiehl et al., 2001; Kim, 2014].

MATERIAL AND METHODS

Subjects

Twenty‐four volunteers participated in the experiment (14 females, mean age: 22.7 years, age range: 18–26 years). All participants were right‐handed, had normal hearing (hearing loss less than 20 dB HL between 125 Hz and 8 kHz), and no history of neurological or psychiatric disorders. Two subjects showed severe head movements (total displacement > 3 mm, maximum scan‐to‐scan movement > 1 mm) during the fMRI data acquisition and were removed from the dataset. Similarly, one EEG dataset had to be discarded due to a high number of artifacts leading to a rejection of over 70% of all trials. Therefore, the statistical analysis of the data was based on a subsample of 21 participants only (13 females).

All experimental procedures were approved by the ethics committee of the University of Oldenburg and written informed consent was obtained from all participants. The study was conducted in accordance with the Declaration of Helsinki [World Medical Association, 2013].

Stimuli and Task

During the experiment, participants listened to sequences of harmonic tone complexes, which consisted of a fundamental tone (F0 = 500/630 Hz) and the first two overtones and had a duration of either 100 or 300 ms, including 15 ms cosine‐squared onset and offset ramps. The stimulus onset asynchrony (SOA) between two subsequent sounds was uniformly jittered in steps of 16.7 ms between 1,400 and 1,600 ms, resulting in a mean SOA of 1,500 ms. This jitter was implemented to (i) reduce expectancy effects and (ii) to minimize the temporal correlation between sound presentation and MRI volume acquisition.

The listener's task was to categorize each stimulus via button press either according to its pitch or its duration while ignoring changes in the other sound feature (Fig. 1). Button presses had to be made with the index finger (low pitch/short duration) or middle finger (high pitch/long duration) of the right hand. The current categorization rule was denoted by a visual cue, which was presented centrally on a screen throughout the task. Switches in the categorization rule were indicated by a change of this visual cue occurring 400–200 ms prior to the onset of the next sound (randomly jittered in steps of 16.7 ms).

Figure 1.

Experimental task: Participants were instructed to categorize harmonic tone complexes via button press either according to their pitch or their duration while ignoring changes in the other sound feature. The task was arranged in blocks of eight stimuli, in which the same stimulus categorization rule was maintained. The current categorization rule was denoted by a visual cue presented permanently on a screen. Switches in the categorization rule were indicated by a change of this visual cue occurring 400–200 ms prior to the onset of the next sound. Dotted lines mark borders between consecutive blocks.

The task was arranged in blocks of eight stimuli in which the same categorization rule was maintained. In total, the experiment consisted of 120 task blocks (60 for each categorization scheme) and 40 baseline blocks, in which no auditory stimuli or visual cues were presented. Block duration was 12 seconds each. Block order was pseudo‐randomized so that the experiment contained 40 categorization rule switches (20 in each direction). No delay was inserted in‐between adjacent blocks, so that switches occurred in‐between two successively presented stimuli. Note that succeeding blocks could also maintain the same categorization rule, leading to continuous sequences of 16 or 24 pitch or duration categorization trials. Pauses of 30 seconds duration were included after 30 blocks each. Throughout the experiment, a fixation cross was presented centrally on the screen to stabilize eye gaze.

Overall, the experiment had a duration of about 35 minutes and consisted of 480 trials in each categorization task (i.e., 960 trials in total). Three hundred eight of these were repetition trials, in which the categorization scheme, the sensory stimulation, and the required button response were identical to the preceding trial. In addition, in each categorization condition, we presented 50 task‐relevant sound feature change trials, in which the task‐relevant sound feature (i.e., pitch or duration) and the required button response changed, as well as 50 task‐irrelevant sound feature change trials, in which the task‐irrelevant sound feature changed. Concurrent changes of both the relevant and irrelevant sound feature were not permitted. Forty trials contained a categorization rule switch from one to the other categorization rule, half of those in each condition. Further, 60 trials contained task‐irrelevant visual distractors, which were presented simultaneously with the onset of the auditory stimulation and were implemented as a 300 ms color change of the visual categorization cue from black to red. These trials served to control for neural activity related to the processing of the changing visual input when analyzing fMRI responses related to categorization rule switching. Note that rule switching was beyond the scope of this manuscript, so trials were modeled but not further analyzed. The remaining 44 trials were those following baseline blocks or pauses and were labelled as return‐to‐task trials, which were again modelled but not analyzed.

To minimize learning effects during the experiment, participants were familiarized with the categorization task in a separate training session, which took place in a sound‐attenuated room at the University of Oldenburg. The training of the experimental task lasted for about 17 minutes (480 trials in total) and contained 30 relevant change trials, 30 irrelevant change trials, 30 visual distractor trials, and 10 rule switch trials in each categorization scheme.

Data Acquisition

Functional MRI data acquisition was performed on a 3 T Siemens MAGNETOM Verio MRI scanner (Siemens AG, Erlangen, Germany) with a twelve‐channel head array. Key‐presses were recorded using an MR‐compatible response keypad (LUMItouch, Photon Control Inc., Burnaby, BC, CDN). Acoustic stimuli were delivered via MR‐compatible headphones (MR confon OPTIME 1; MR confon GmbH, Magdeburg, Germany). Participants were equipped with ear‐plugs during the experiment to minimize effects of scanner background noise on task performance. To ensure that participants could hear all sounds despite ear protection, the sound level and the balance were individually adjusted under scanner noise before starting the experiment.

We obtained 1430 T2*‐weighted gradient echo planar imaging (EPI) volumes with BOLD‐contrast for each subject (time of repetition (TR) = 1,500 ms, time of echo (TE) = 30 ms, flip angle α = 70°, Field of View (FoV) = 200 × 200 mm², voxel‐size = 3.1 × 3.1 × 3.1 mm³). Volumes consisted of 23 transverse slices with a gap of 0.9 mm in‐between and were recorded in an ascending order. Subsequently, a high‐resolution structural volume was acquired for each participant using a T1‐weighted magnetization‐prepared rapid acquisition gradient echo (MPRAGE) sequence (TR = 1,900 ms, TE = 2.52 ms, α = 9°, FoV = 256 × 256 mm², voxel‐size = 1 × 1 × 1 mm³).

EEG was recorded during fMRI data acquisition from 64 electrodes using an MRI‐compatible amplifier system (BrainAmp MR Plus, Brainproducts, Gilching, Germany). The scalp electrodes were placed in a customized, equidistant layout with electrodes AFz serving as ground and Cz as online recording reference. Eye movements were monitored by an EOG electrode placed below the left eye and the electrocardiogram was recorded by an electrode placed on the left lower back. The hardware clock of the EEG system and the MRI scanner's master clock were synchronized. The data were recorded with a sampling rate of 5,000 Hz and analog filtered between 0.016 and 250 Hz. Electrode impedances were maintained below 20 kΩ prior to data acquisition.

fMRI Data Analysis

MRI data were processed and analyzed using SPM12 (FIL, Wellcome Trust Centre for Neuroimaging, UCL, London). To correct for head motion the functional time series was spatially realigned to the first image of the time series. The structural T1‐weighted volume was registered to a mean functional image and segmented in order to obtain spatial normalization parameters. Using these parameters functional and structural images were normalized to the Montreal Neurological Institute (MNI) template brain. Finally, normalized functional volumes were smoothed with a three‐dimensional Gaussian kernel of 8mm full‐width‐half‐maximum.

A generalized linear model and a random effects analysis were used for the statistical analysis of the fMRI data. For both categorization tasks, the single‐subject model contained regressors modeling hemodynamic responses to repetition trials, task‐relevant sound feature changes, and task‐irrelevant sound feature changes. Trials following categorization rule switches, visual distractors, return‐to‐task trials, and trials containing response errors (i.e., no button press or an incorrect stimulus assignment) were modeled by additional regressors of no interest. To account for the imbalance in trial numbers between repetition and sound feature change trials, repetition trials were split into two regressors, containing 50 and ≤258 randomly selected trials. All later statistical comparisons between sound feature change and repetition trials were then based on the numerically balanced regressors with 50 trials. Signal changes related to head movement were accounted for by including the six movement parameters as computed in the SPM12 realign procedure. In total, the single‐subject model, thus, consisted of 22 regressors and a constant term. The time series in each voxel were high‐pass filtered to 1/128 Hz and modeled for temporal autocorrelation across scans with an AR(1) process.

The statistical data analyses concentrated on the processing of task‐relevant and irrelevant sound feature changes. To identify brain regions which are responsive to sound feature changes per se, we first compared all trials containing a task‐relevant or irrelevant sound feature changes against repetition trials, containing no change in auditory input, using a t‐test on the group level. In a second step, we aimed to reveal differences in the processing of sound feature changes related to (i) the changing sound feature (i.e., pitch vs. duration) and (ii) the task relevance of the change (i.e., task‐relevant vs. task‐irrelevant). To this end, we computed the following differential contrasts for each subject: task‐relevant pitch change ‐ repetition, task‐irrelevant pitch change ‐ repetition, task‐relevant duration change ‐ repetition, and task-irrelevant duration change ‐ repetition. Please note that all feature change regressors were contrasted to repetition trials occurring under the same categorization rule to minimize the effects of general categorization rule‐dependent differences between conditions. This means that relevant pitch changes were contrasted to the repetition regressor modeling responses in pitch categorization blocks, whereas irrelevant pitch changes, occurring during duration categorization blocks, were contrasted to repetition trials occurring in those blocks. The four contrasts were then entered into a flexible‐factorial ANOVA model in SPM, containing the factors task relevance (task‐relevant/task‐irrelevant), changing sound feature (pitch/duration), and subject (21 levels) to account for between‐subject variability. Main effects of task relevance and the changing sound feature as well as the interaction between both factors were analyzed using F‐tests. To assess the direction of significant effects mean beta estimates were extracted from peak voxels.

Results of all fMRI analyses are presented at P < 0.05, corrected for multiple comparisons using a family‐wise error (FWE) correction on the cluster‐level. The voxel‐level threshold for cluster identification was thereby set to P < 0.001 (uncorrected). All differential contrasts were analyzed both on the whole‐brain level and restricted to a superior temporal lobe mask to specifically assess differential activations in brain regions involved in the sensory processing of the auditory stimulation. This mask encompassed the temporal transverse and the superior temporal gyrus as defined in the AAL template provided by the WFU PickAtlas extension for SPM [Maldjian et al., 2003].

EEG Data Analysis

EEGLAB [Delorme and Makeig, 2004] was used for preprocessing of the EEG data. As a first step, MRI artifact correction was performed using functions provided by the FMRIB plug‐in for EEGLAB [for a description of the methods see Niazy et al., 2005]. Scanner gradient artifacts were removed using a slice template obtained from averaging over 30 consecutive fMRI volumes (fmrib_fastr function). Cardio‐ballistic artifacts were detected and cleaned from the EEG using the functions fmrib_qrsdetect and fmrib_pas. EEG data was then down‐sampled to 500 Hz, re‐referenced to a common average reference, offline filtered from 0.1 to 20 Hz, and epoched from −100 to 600 ms relative to the onset of each auditory tone complex. Artifacts related to eye blinks and lateral eye movements were pruned from the data using independent components analysis (ICA). For this procedure, a copy of the re‐referenced EEG data was offline filtered from 1 to 40 Hz and epoched into continuous 2 s intervals. We conducted a principal component analysis (PCA) to reduce data dimensionality and computed 45 independent components (ICs) using the extended infomax algorithm implemented in EEGLAB. The demixing matrix obtained from this procedure was then applied to the original dataset used for the ERP analysis and ICs reflecting eye blinks and lateral eye movements were removed. Note that this two‐step procedure with different filter settings for ICA training and actual data processing follows the recommendation by Debener et al. [2010] to deal with the adverse influences of slow amplitude drifts (<1 Hz) and very substantial low‐pass filtering on ICA data decomposition. Afterward, all epochs containing absolute signal amplitudes of more than 150 µV were rejected and a baseline correction was performed using the pre‐stimulus interval from −100 to 0 ms. From the remaining trials, we computed event‐related potentials for task‐relevant (mean (± SD) number of trials after artifact rejection: 46 ± 5) and task‐irrelevant (46 ± 5) pitch changes, task‐relevant (45 ± 6), and task‐irrelevant (45 ± 5) duration changes, all pitch (91 ± 10) and all duration changes (91 ± 10), as well as for numerically matched and randomly selected subsets of repetition trials. Like in the fMRI analysis, repetition trials were matched according to the applied stimulus categorization rule.

Subsequently, sources underlying the observed event‐related potentials were modeled using the software package Brainstorm [Tadel et al., 2011]. To avoid a potential localization bias of EEG activity not reflected in BOLD signal changes toward regions exhibiting fMRI activations, the inverse modeling was conducted on the whole cortical surface without prior fMRI weighting [Huster et al., 2012]. Please note that, due to a lack of individual electrode position information, we refrained from computing source activity on the subjects' individual anatomy. Instead, the electrode layout was adjusted to the Colin27 brain template and all analyses were conducted in the normalized MNI space. First, an OPENMEEG boundary element method (BEM) head model was computed for the Colin27 brain template. Source reconstruction was then performed for a set of 15002 vertices spanning the cortical surface of the head model using a dSPM approach with depth weighting (order: 0.5, maximal amount: 10). The dipole orientation was constrained to be normal to the cortical surface. Noise covariance levels for the source reconstruction process were estimated from individual single‐trial pre‐stimulus baselines. For further analysis, we computed absolute values from the resulting source activation maps to facilitate the interpretation of activation strengths within cortical ROIs.

The statistical analysis of the source activation maps aimed to reveal the temporal progression of change‐induced activity across the spatial pattern of brain regions identified using the concurrently acquired fMRI data. As a first analysis step, we defined a set of anatomical regions‐of‐interest (ROIs) covering the main anatomical structures which showed significant change‐related activity in our fMRI data analysis. Using the Destrieux atlas labels of the cortical surface as provided by Brainstorm, we identified the following ROIs matching our activations: left and right auditory cortex (Atlas labels: G_temp_sup‐G_T_tranv + S_temporal_trans + G_temp_sup‐Plan_tempo), the anterior part of the left and right insula (G_insular_short), the left and right temporoparietal junction (G_pariet_inf‐Supramar), the left and right superior parietal lobe and intraparietal sulcus (G_parietal_sup + S_intrapariet_and_P_trans), the posterior medial frontal cortex and adjacent parts of the anterior cingulate cortex (G_and_S_cingul‐Mid‐Ant), left motor cortex (G_precentral + S_central + G_postcentral), as well as left and right middle (G_front_middle) and inferior frontal gyri (G_front_inf‐Triangul). Please note though that the middle frontal gyrus ROI of the Destrieux atlas provided by Brainstorm seems to encompass also parts of the superior frontal gyrus. We therefore manually modified this ROI to cover middle frontal gyrus only. Supporting Information Figure S1 depicts the anatomical location of all ROIs on the cortical surface. For each ROI and all event‐related potentials of interest (i.e., all pitch changes, all duration changes, relevant and irrelevant pitch and duration changes only, and matched subsets of repetitions trials) single‐subject activation time series were calculated by averaging the activation time series across all vertices within a ROI.

Difference waves between the change and the corresponding repetition conditions were computed from these data to assess the time course of change‐related activity. For bilateral ROIs we tested for lateralization effects in change‐related activity between the left and right hemisphere using point‐wise two‐tailed paired t‐tests. In case this analysis indicated no significant inter‐hemispheric differences in change‐related activity within a ROI, time courses were averaged over both hemispheres for further analysis to reduce data complexity. For each of the remaining ROI time series we then performed point‐wise two‐tailed t‐tests to identify time intervals of significant change‐related activity. All t‐tests were performed within the time window from 0 to 600 ms after stimulus onset. Differences were considered as statistically significant when a threshold of P < 0.05 was passed within continuous intervals of at least 30 ms duration (i.e., for at least 15 consecutive time points). This threshold was chosen to ensure that all sustained differences were reliably identified whereas more transient effects, lasting a few time points only, were ignored for further analysis.

For each time interval showing significant change‐related activity, activation peaks were assessed on the group level and the single‐subject activation peak latencies were identified within intervals of ±50 ms encompassing the group maxima. Note that multiple activation peaks within each ROI and each activation interval could enter this analysis. In case that peaks within a ROI differed by less than 50 ms in latency, only the main peak was analyzed though. Peak latencies were then compared pair‐wise (both within and across ROIs) using two‐tailed paired t‐tests. Peak latencies were reported to differ significantly when passing a statistical threshold of P < 0.05, corrected for multiple comparisons using the Benjamini–Hochberg procedure for FDR correction.

Behavioral Data Analysis

The analysis of the behavioral data focused on comparing response times and error rates (i.e., the proportion of incorrect or omitted responses) following task‐relevant and irrelevant pitch and duration changes. For each participant, we computed median response times and error rates in the two repetition (i.e., repetition in the pitch or duration categorization block) and the four feature change conditions. As a first analysis step, these measures were compared in‐between both repetition conditions using paired t‐tests. This test served as control to identify general behavioral differences related to the applied categorization rule, independent of changes in sensory stimulation.

Subsequently, response times and error rates were compared in‐between the four feature change conditions using repeated‐measures ANOVAs including the factors task relevance (task‐relevant/task‐irrelevant) and changing sound feature (pitch/duration). Since the comparison of both repetition conditions revealed significantly faster response times in the pitch categorization task, the analysis of response speed was not based on the absolute response times but rather on response time differences between feature change and corresponding repetition trials. Main effects of the ANOVAs were reported to be statistically significant when passing a statistical threshold of P < 0.05. Paired t‐tests at P < 0.05 (FDR corrected) were then used to investigate differences between conditions in more detail.

RESULTS

Behavioral Performance

The categorization task performance was quantified in terms of error rates and response times. As a first step, we analyzed responses in repetition trials to assess general differences between both categorization tasks using paired t‐tests. No differences in error rates were observed (P > 0.1; pitch: 0.8% ± 0.8%, duration: 1.2% ± 1.1%). The comparison of response times in both tasks showed however that participants responded significantly faster to repetition trials in the pitch than the duration categorization task (P < 0.001; pitch: 422 ± 66 ms, duration: 439 ± 62 ms).

Differences in response time and task accuracy in trials containing an auditory feature change were then investigated using two‐factorial repeated measures ANOVAs including the factors task relevance (task‐relevant/task‐irrelevant) and changing sound feature (pitch/duration). To account for the generally faster response times during pitch categorization, we did not analyze absolute response times following auditory feature changes but rather the response time differences between these conditions and the corresponding repetition conditions. Table 1 states mean response time differences and error rates obtained for task‐relevant pitch changes, task‐irrelevant pitch changes, task‐relevant duration changes, and task‐irrelevant duration changes. Performing an ANOVA on the response time differences revealed a significant main effect of task relevance (F(1,20) = 7.4; P < 0.05; η² = 0.27) as well as significant task relevance‐by‐changing sound feature interaction (F(1,20) = 9.7; P < 0.01; η² = 0.33). No main effect of the changing sound feature was observed (F(1,20) = 2.9; P = 0.1; η² = 0.13). Post‐hoc paired t‐tests showed that these effects were related to significantly increased response times following task‐irrelevant pitch changes as compared with all other conditions (all P < 0.05, FDR corrected).

Table 1.

Behavioural performance in trials containing task‐relevant or irrelevant sound feature changes

	Response time difference	Error rate
Pitch categorization task
Task‐relevant pitch change	139 ± 44 ms	5.4% ± 6.3%
Task‐irrelevant duration change	138 ± 72 ms	3.1% ± 2.3%
Duration categorization task
Task‐relevant duration change	152 ± 48 ms	21.6% ± 13.9%
Task‐irrelevant pitch change	188 ± 42 ms	8.7% ± 6.0%

The table states response time differences between trials containing a sound feature change as compared with repetition trials as well as the percentage of response errors in these trials.

For error rates, the ANOVA showed significant main effects of both task relevance (F(1,20) = 15.4; P < 0.001; η² = 0.44) and the changing sound feature (F(1,20) = 31.2; P < 0.001; η² = 0.61) as well as a significant interaction between both factors (F(1,20) = 56.5; P < 0.001; η² = 0.74). Paired t‐tests subsequently conducted between conditions revealed that error rates were higher in trials containing a relevant duration change than in the other three conditions (all P < 0.001, FDR corrected). Also, task‐irrelevant pitch changes, which occurred during the duration categorization task, were associated with higher error rates than similarly irrelevant duration changes during the pitch categorization task (P < 0.001, FDR corrected). In other words, increased error rates were observed in response to both task‐relevant and task‐irrelevant sound feature changes during the duration categorization task.

fMRI Correlates of Deviance Processing

The fMRI data analyses focused on the neural processing of task‐relevant and irrelevant sound feature changes. First, we contrasted all trials containing a sound feature change, independent of its task relevance and the changing feature, to a matched subset of trials containing a stimulus repetition. This contrast aimed to identify the overall pattern of change‐induced brain activity. As depicted in Figure 2A, increased BOLD responses following sound feature changes were observed in secondary, but not primary, auditory sensory areas in the planum temporale and the superior temporal gyrus, as well as in the anterior portion of the insula, the inferior and middle frontal gyrus, the superior parietal lobe and the intraparietal sulcus, the temporoparietal junction, the thalamus, inferior and superior colliculi, the posterior medial frontal cortex and the anterior cingulate cortex, in primary and secondary motor areas, and parts of primary visual cortex (all activations at P < 0.05 on the cluster‐level, corrected for multiple comparisons using the FWE approach).

Figure 2.

Brain regions involved in processing sound feature changes: (A) The overall spatial pattern of change‐induced activity following task‐relevant and irrelevant sound feature changes as compared with matched subsets of repetition trials, containing no feature change. The figure depicts T‐values surpassing the significance threshold of P < 0.05 (FWE corrected on the cluster‐level). Activated brain regions include the anterior cingulate cortex (ACC), the auditory cortex (AC), the inferior colliculus (IC), the insula (INS), the inferior frontal gyrus (IFG), the intraparietal sulcus (IPS), the middle frontal gyrus (MFG), the motor cortex (MOT), the posterior medial frontal cortex (pMFC), the superior colliculus (SC), the superior parietal lobe (SPL), the thalamus (THAL), and parts of the visual cortex (VIS). (B) A two‐factorial ANOVA revealed a significant main effect of the changing sound feature on change‐related activity in the right superior temporal gyrus (STG). Activity in this region was increased following both relevant and irrelevant pitch changes as compared with duration deviations. (C) A main effect of task relevance became evident in the superior parietal lobe, the insula, inferior and middle frontal gyri, and in the left motor cortex. While the motor cortex showed increased BOLD responses following task‐relevant changes, all other regions were more activated in response to irrelevant sound feature changes. Figures B and C depict masked F‐scores at a significance threshold of P < 0.05 (FWE corrected on the cluster‐level). [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Differences in change‐induced activity related to the changing sound feature (pitch/duration) and the task relevance of the change (task‐relevant/task‐irrelevant) were then investigated using a flexible‐factorial ANOVA model in SPM. This analysis demonstrated a significant main effect of the changing sound feature in the right posterior superior temporal gyrus and the adjacent superior temporal sulcus (Fig. 2B). BOLD responses in this region were higher following pitch as compared with duration changes, independent of their task relevance. Further, as shown in Figure 2C, a significant main effect of task relevance was observed in the superior parietal lobe and the intraparietal sulcus, bilateral inferior frontal gyri, the left anterior insula and middle frontal gyrus, the posterior medial frontal cortex, and left motor cortex (all activations at P < 0.05 on the cluster‐level, FWE corrected). In contrast, no relevance‐related differences were found in temporal lobe regions involved in the sensory processing of auditory information. Remarkably, in all areas but left motor cortex BOLD responses were higher for irrelevant as compared with task‐relevant changes. No significant interaction between task relevance and the changing sound feature was observed.

EEG Dynamics of Processing Sound Feature Changes

The fMRI data analysis showed increased BOLD responses following auditory feature changes in a widespread set of auditory sensory, thalamic, frontal, and parietal brain regions, which was largely independent of the changing sound feature (cf., Fig. 2B). The temporal progression of the activation spread across this spatial activation pattern was subsequently studied using the concurrently acquired EEG data. To this end, we extracted individual EEG source activation time courses from ROIs localized to auditory cortex, superior parietal lobe and intraparietal sulcus, the temporoparietal junction, the left motor cortex, the anterior insula, the medial frontal and the anterior cingulate cortex, the inferior frontal gyrus, and the middle frontal gyrus. We did not include thalamic and brainstem ROIs in our analysis, since activity originating from these regions cannot, due to their deep location, be captured by the applied EEG source analysis. We further refrained from analyzing visual cortex activity in more detail. In our view, visual activations are likely to reflect an orientation toward visually presented categorization cue following a sound feature change and are not directly related to the processing of the auditory information. Please also note, that unlike in our fMRI analysis, we could not directly compare ERPs for pitch and duration changes due to the differing onset latencies of both feature changes. Instead, two separate analyses were conducted.

Temporal progression of change‐induced activity

As a first analysis, we investigated the overall dynamics of the activation spread within our ROIs following a pitch or duration change, independent of its task relevance for the listener. For this, all trials containing a pitch and duration change, respectively, were pooled and compared with a numerically matched subset of repetition trials containing no change in the auditory stimulation. Figure 3 depicts the time courses of change‐induced activity (i.e., the difference wave between change and repetition trials) for pitch (top row) and duration changes (bottom), time‐locked to sound onset, within the left and right‐hemispheric ROIs as well as the mean ROI time course averaged over both hemispheres. For pitch changes, we observed statistically significant ROI laterality effects in the insula (from 340 to 380 ms after stimulus onset), the temporoparietal junction (214–262 ms), and the superior parietal lobe/intraparietal sulcus ROI (240–268 ms), whereas no interhemispheric differences were found in the auditory cortex and the middle and inferior frontal gyri (point‐wise paired t‐tests, P < 0.05 in contiguous intervals of at least 30 ms duration; significant intervals are depicted in Fig. 3). For duration changes, interhemispheric differences were observed in the auditory cortex (334–402 ms; 552–590 ms) and the temporoparietal junction (320–388 ms), but in none of the other ROIs. The further analysis of all ROIs showing no laterality effect was based on the mean time course over both hemispheres.

Figure 3.

The figure shows the EEG source time courses of change‐related activity following pitch (A) and duration (B) changes in a set of fMRI‐based regions‐of‐interest (ROIs), time‐locked to stimulus onset. ROIs include the auditory cortex (AC), the anterior insula (INS), the temporoparietal junction (TPJ), the superior parietal lobe and the intraparietal sulcus (SPL/IPS), the posterior medial frontal cortex (pMFC), the middle (MFG) and inferior frontal gyri (IFG), and the left motor cortex (MOTOR). Graphs depict left and right hemispheric ROI time courses (if applicable) as well as the mean time course, averaged over both hemispheres. In case of significant interhemispheric differences, left and right time courses were analyzed separately, whereas otherwise all further analyses were based on the averaged ROI time course. The time courses used for further analyses are marked in bold. The lower panel in each graph denotes time intervals in which the left, the right, or the mean ROI time course differs significantly from zero (i.e., phases of significant change‐related activity) as well as intervals showing significant interhemispheric differences. Vertical bars depict difference wave peaks which were included in the analysis of activation peak latencies. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Time intervals showing change‐induced activity were identified using point‐wise t‐tests performed from 0 to 600 ms after stimulus onset. Significant change‐induced activity was observed for all ROIs and following both pitch and duration changes (at P < 0.05 in contiguous intervals of at least 30 ms duration; significant intervals are depicted in Fig. 3). Activity increases related to a pitch change successively emerged in the auditory cortex (first significant difference at 116 ms), the left (122 ms) and right insula (130 ms), the left (138 ms) and right (134 ms) temporoparietal junction, the inferior (138 ms) and middle frontal gyrus (140 ms), and left motor cortex (138 ms), whereas change‐induced activity in the superior parietal lobe/intraparietal sulcus ROI (left: 196 ms; right: 214 ms) and the posterior medial frontal cortex (196 ms) was observed later. Following duration changes, first change‐induced activity emerged from about 250 ms on in the right (254 ms) and left (262 ms) temporoparietal junction and the right (254 ms) and left (266 ms) auditory cortex, followed by effects in the posterior medial frontal cortex (268 ms), the superior parietal lobe/intraparietal sulcus (268 ms), the left motor cortex (270 ms), the insula (284 ms), and later in the middle (478 ms) and the inferior (510 ms) frontal gyri.

For each time interval showing significant change‐induced responses we then identified group activation peaks (see Fig. 3; group peaks are demarked by vertical bars) and computed single‐subject peak latencies from intervals of ±50 ms around these group peaks. Peak latencies were compared statistically using paired t‐tests (at P < 0.05, FDR corrected for multiple comparisons). Note that in cases in which both hemispheres showed a similar time course, only peaks form the dominant hemisphere (i.e., the ROI showing a higher peak amplitude) were analyzed. Figure 4 visualizes the resulting progression of change‐induced activation across ROIs following changes in sound pitch (top) and duration (bottom). Please note that in this figure response latencies are not plotted relative to sound onset but rather to deviance onset (pitch: 0 ms after sound onset; duration: 100 ms) to ease the comparison across features. Boxes encompass peaks that do not differ significantly in their latency and can thus be considered to occur quasi simultaneously (at P < 0.05, FDR corrected for multiple comparisons).

Figure 4.

The figure depicts the temporal progression of change‐induced activation peaks across ROIs. Please note that latencies are given relative to deviance onset (i.e., pitch: 0 ms after stimulus onset; duration: 100 ms). Boxes encompass response peaks that do not differ significantly in latency and, thus, occur quasi simultaneously.

According to this analysis, the spread of change‐induced activity following pitch deviations can be divided into five successive phases. The first phase of change‐induced activity peaks emerged from about 150 ms after deviance onset on and includes the auditory cortex, the left insula, the inferior frontal gyrus, the temporoparietal junction, the middle frontal gyrus, and the left motor cortex. Please note that the peak latency of the temporoparietal junction response does not differ significantly from any of the other regions. The second phase, starting about 210 ms after change onset, encompasses successive activation peaks of the right insula, the posterior medial frontal cortex, the right temporoparietal junction, the inferior gyrus, the auditory cortex, the superior parietal lobe and the intraparietal sulcus, and left motor cortex. The third activation phase includes the middle and inferior frontal gyri as well as the left insula and the posterior medial frontal cortex. A subsequent series of peaks then encompasses again the auditory cortex, the medial frontal cortex, and left motor cortex, and later the superior parietal lobe and the intraparietal sulcus.

Performing a similar analysis, the cortical processing of duration changes can be roughly divided into four activation phases. Initially, change‐induced activity successively peaks in the superior parietal lobe and the intraparietal sulcus, the left motor cortex, the left temporoparietal junction, the posterior medial frontal cortex, the left auditory cortex, the insula, the right auditory cortex, and the right temporoparietal junction. From about 290 ms after stimulus onset on, changes are then processed further in the superior parietal lobe/intraparietal sulcus ROI and in the left motor cortex. The third activation phase, starting around 380 ms after stimulus onset, then shows widespread co‐activations of frontal and parietal brain regions as well as motor cortex responses. From there, activation spreads back to the left auditory cortex and the left temporoparietal junction. The overall dynamic of this activation spread, progressing from an early stage of deviance detection to parietal lobe and then frontal cortex, is in line with the temporal dynamics observed for the processing of pitch changes. It should be noted, though, that, unlike for pitch changes, we observed additional early change‐related activity in the superior parietal lobe/intraparietal sulcus and the posterior medial frontal cortex, but no early effects in the inferior and middle frontal gyri. This suggests that the initial detection of pitch and duration changes may partially rely on differing frontoparietal brain regions.

In the analyses described above, we focused on the temporal dynamics of activity within predefined ROIs as identified in our fMRI analysis. It may be questioned whether this approach adequately captures the spread of EEG activity over the cortex. To qualitatively control for additional EEG activation sites we additionally visualized the time courses of change‐induced activity following pitch and duration changes on the whole‐brain level (Supporting Information Fig. S2). Matching the results of our fMRI analysis, major activation foci were observed in bilateral auditory cortex and adjacent parts of the medial temporal lobe, the insula, the temporoparietal junction, the superior parietal lobe, medial frontal and anterior cingulate cortex, motor areas, primary visual cortex, as well as in the middle and inferior frontal gyri. As in the fMRI, activations sites were largely comparable for pitch and duration changes. This suggests that the applied ROI‐based analysis adequately describes the dynamics of EEG activity.

Effects of task relevance

In the first EEG data analysis, the temporal progression of change‐related activity was described irrespectively of the task relevance of the changing sound feature. Subsequently, we aimed to identify differences in the processing of task‐relevant and task‐irrelevant sound feature changes. To this end, we computed change‐induced responses from the difference waves between task‐relevant or task‐irrelevant feature changes and matched subsets of repetition trials (i.e., sets of repetition trials occurring under the same categorization rule). Building up on our fMRI data, relevance‐related differences in activation amplitudes were then investigated in ROIs covering the superior parietal lobe and intraparietal sulci, inferior frontal gyri, middle frontal gyri, the anterior portion of the insula, as well as the posterior medial frontal cortex. Similar to the previous analysis of change‐induced activity, a first test aimed to reveal interhemispheric differences in the processing of task‐relevant as compared with task‐irrelevant changes. For relevant and irrelevant pitch changes, only the superior parietal lobe/intraparietal sulcus ROI showed significant differences (252–280 ms and 476–508 ms after stimulus onset; point‐wise paired t‐tests, P < 0.05 in contiguous intervals of at least 30 ms duration). Interhemispheric differences following duration changes were only observed in the insula (492–546 ms). For all other ROIs, we therefore averaged time courses over both hemispheres for the further analysis.

Figure 5A depicts ROI time courses of change‐induced activity following task‐relevant and task‐irrelevant pitch changes. First relevance‐related amplitude differences emerged in the insula in the interval from 166 to 194 ms after stimulus onset. This effect was followed by later differences in the inferior frontal gyrus (398–426 ms) and the left motor cortex (404–434 ms). No sustained relevance‐related differences were observed in the superior parietal lobe/intraparietal sulcus, the middle frontal gyrus and the posterior frontal cortex ROIs (point‐wise paired t‐tests, P < 0.05 in contiguous intervals of at least 30 ms duration).

Figure 5.

ROI time courses of change‐related activity for task‐relevant and irrelevant pitch (A) and duration (B) changes, time‐locked to stimulus onset. In case of significant inter‐hemispheric differences between relevant and irrelevant changes, graphs depict left and right hemispheric time courses; otherwise only the mean time course over both hemispheres is shown. The lower panel in each graph marks time intervals showing significant relevance‐related differences in the left, the right, or the mean ROI time course as well as intervals showing significant interhemispheric differences. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

A similar analysis comparing change‐related processing following task‐relevant and irrelevant duration changes revealed relevance‐related differences in all structures of interest. As shown in Figure 5B, the left insula ROI showed significant increased activity levels following task‐relevant duration changes from 300 to 338 ms after stimulus onset and, later, from 464 to 568 ms. No effect of task relevance was found in the right insula ROI though. At later processing phases, significantly increased activity following relevant duration changes was further observed in the middle frontal gyrus (418–580 ms), the superior parietal lobe/intraparietal sulcus (438–564 ms), the posterior medial frontal cortex (444–582 ms), the inferior frontal gyrus (454–598 ms), and the left motor cortex (422–584 ms).

DISCUSSION

We here studied the spatiotemporal dynamics of processing task‐relevant and task‐irrelevant pitch and duration changes in an auditory stimulus categorization task with changing categorization rules. Our data demonstrate that the processing of sound feature changes recruits a widespread network of brain regions, including auditory sensory areas, the insula, the thalamus and the superior colliculi, the temporoparietal junction region, superior parietal lobe and intraparietal sulcus, the posterior medial frontal cortex, left motor cortex, and inferior and middle frontal gyri. While this overall spatial activation pattern was largely consistent for both features, feature‐specific differences became evident in the temporal dimension. In particular during the early cortical stages of change processing, we observed differing frontal and parietal contributions following changes in sound pitch and duration. Later stages of change processing responses then, however, showed a rather consistent spread of activity into parietal lobe and further into frontal cortex, providing evidence that subsequent deviance processing in frontoparietal brain regions is largely independent of the changing sound feature. Differences in activation levels related to the task relevance of a sound feature change did not occur in auditory sensory regions, but were first found to emerge at the level of the anterior insula, followed by effects in frontal and parietal regions, and the left motor cortex.

Feature‐Related Differences in Early Change Detection

The overall fMRI activation pattern associated with changing pitch or sound duration was largely identical. Only a portion of the right posterior superior temporal gyrus and the superior temporal sulcus showed consistently higher BOLD responses following both task‐relevant and irrelevant pitch changes as compared to changes in sound duration. This rightward lateralization is in good agreement with previous data reporting a right‐hemispheric dominance for processing pitch information and pitch changes, suggesting that the observed activation may be related to the sensory processing of the changed pitch information in auditory cortex [Patterson et al., 2002; Puschmann et al., 2013; Zatorre and Belin, 2001]. Most studies however reported pitch‐induced sensory activity to be localized to more anterior parts of the auditory cortex, in particular to the lateral aspect of Heschl's gyrus and the anterior portion of the planum temporale [Hall and Plack, 2009; Penagos et al., 2004; Puschmann et al., 2010]. In contrast, no brain region was generally more activated following duration as compared with pitch changes in our study.

Despite this largely identical overall activation pattern the concurrently acquired EEG data indicate feature‐dependent differences in the activation sequence, in particular concerning early deviance processing. While, for pitch changes, first change‐induced response peaks emerged about 150 ms after deviance onset in the left insula and the auditory cortex, followed by effects in the inferior frontal gyri, the temporoparietal junction, the middle frontal gyri, and the left motor cortex, change‐induced activity following duration changes became evident at about 190 ms after deviance onset only and additionally involved the superior parietal lobe/intraparietal sulcus and the posterior medial frontal cortex, but not the inferior and middle frontal gyri. Response latencies are in good agreement with the time window of the mismatch negativity component, which represents an early marker of auditory deviance detection [May and Tiitinen, 2010; Näätänen et al., 2011]. Also, the delayed responses for duration as compared with pitch deviants correspond to previous observations [Leung et al., 2012]. The mismatch negativity potential was localized to multiple generator sites, in particular in auditory and frontal cortex [e.g., Doeller et al., 2003]. The exact source distribution, however, seems to depend on both the nature of the stimulus deviation and the applied experimental paradigm [Chakalov et al., 2014; MacLean et al., 2015; Rissling et al., 2014]. In some agreement with our data, a comparison of mismatch negativity generator sites for pitch and duration deviants using functional MRI revealed not only differing activation sites in auditory and frontal cortex, but also increased activation levels in superior parietal lobe and the cingulate cortex in response to duration changes [Molholm et al., 2005]. Also, recent EEG experiments showed contributions of parietal and medial frontal cortex generators to mismatch negativity responses following duration changes [Rissling et al., 2014; Takahashi et al., 2013]. Based on this, it may by suggested that differences in the early processing of pitch and duration deviants may reflect differing regional contributions to mismatch negativity generation. In particular, our data suggest that the early detection of pitch deviants may rely on an auditory‐frontal cortex network, whereas the detection of duration changes may rather involve auditory sensory, medial frontal, and parietal brain areas.

Activation Spread of Change‐Induced Activity Following Initial Deviance Detection

The mismatch negativity component is commonly thought to represent a pre‐attentive and automatic marker of cortical deviance processing, but to be involved in calling attention to the changing auditory input and, thus, initiating further attentive deviance processing [Näätänen et al., 2011]. Consistent with this view, early change‐induced responses to pitch changes were followed by right‐lateralized activations of the insula (about 200 ms after deviance onset) and the temporoparietal junction (230 ms) as well as by rather bilateral responses in the medial frontal cortex (230 ms) and the inferior frontal gyri (250 ms). The right temporoparietal junction and the right anterior insula represent core structures of the ventral frontoparietal attention network, which is commonly thought to be essentially involved in target detection and stimulus‐driven reorienting of attention to salient sensory events [Corbetta and Shulman, 2002; Fox et al., 2006; Kim, 2014; Linden et al., 1999; Sadaghiani et al., 2009; Salmi et al., 2009]. Noteworthy, we also observed a pattern of insula and right temporoparietal junction activations with similar response latencies following duration changes, suggesting that the recruitment of the ventral attention network after early change detection occurs in a feature‐independent manner. Subsequently, our data show, also consistent across features, a spread of change‐induced activity to the superior parietal lobe and the intraparietal sulcus from about 270 to 290 ms after stimulus onset on. This region is part of the dorsal frontoparietal attention network and is assumed to be involved in voluntarily directing auditory attention to sound sources [Hill and Miller, 2010; Smith et al., 2010]. Our findings of change‐induced responses in parts of both the ventral and dorsal attention networks is in line with previous data showing joint activations of networks during stimulus‐driven reorienting of attention [Alho et al., 2014; Thiel et al., 2004]. So far, there is mixed evidence whether signal processing in the ventral attention network precedes activity in dorsal network and thus initiates the reorienting process [for a review on this issue see Corbetta et al., 2008]. Given the significant temporal delay between activations in the right insula and the right temporoparietal junction as compared with the superior parietal lobe/intraparietal sulcus, our data may argue in favor of such a role of the ventral attention network. However, we here did not investigate responses in other regions of the dorsal attention system, in particular the frontal eye fields. Therefore, our data cannot provide any conclusive evidence for a clear temporal order of ventral and dorsal attention network activations during auditory deviance processing.

Following up on parietal activations related to reorienting attention to the sound feature change we observed a rapid series of neural activity in areas including the anterior portion of the insula, the posterior medial frontal cortex, the inferior and middle frontal gyrus, as well as superior parietal regions. Large parts of the prefrontal cortex as well as the superior parietal lobe were previously demonstrated to be essentially involved in action selection, response inhibition, and performance monitoring [Ridderinkhof et al., 2004; Tanji and Hoshi, 2008]. The dorsolateral prefrontal and the posterior parietal cortex show strong intrinsic resting state coupling and are regarded to form a central executive network [Menon, 2011]. The anterior portion of the insula is also commonly observed to be activated in the context of performance monitoring and the processing of response errors [Ullsperger et al., 2010]. It has been argued, though, that the insula is not part of the central executive network itself, but rather plays an essential role for switching activation between the default mode and the central executive network when processing salient sensory events [Menon and Uddin, 2010; Sridharan et al., 2008]. In line with this, insula responses tended to precede other frontal activations in our experiment. Please note, though, that although the general activation pattern was again largely consistent across both features, frontal cortex activation qualitatively occurred later following duration than pitch changes. Given the argued role of these brain regions in action selection it may be speculated that latency differences translate into differing response times between conditions. No such effect was however found in the behavioral data.

The auditory cortex was not only activated during initial stimulus processing and early deviance detection, but also showed later re‐activations. It may be argued that these effects represent offset responses to the 100 and 300 ms tone complexes. However, such offset responses should similarly occur for both change and repetition trials, and should therefore not be observed in the difference wave of change‐induced activity. Also, late auditory cortex activations were not stably locked to stimulus offset times, but occurred 160 and 78 ms after stimulus offset in pitch change trials and 257 ms (or 457 ms) after stimulus offset in duration change trials. Therefore, it seems unlikely that the observed responses are driven by stimulus offsets. Instead, we suggest that auditory cortex re‐activations may be related to top‐down feedback from hierarchically higher processing stages. In our study, late auditory cortex peaks occurred simultaneously with responses in brain areas involved in attention reorienting and subsequent stimulus evaluation and response selection. In agreement with this view, studies in both the auditory and visual domain provided evidence for a coupling of the intraparietal sulcus and sensory areas during spatial orienting of attention [Rossi et al., 2014; Vossel et al., 2012; Weisz et al., 2014] and reported re‐activations of auditory cortex following perceptual choices and feedback on stimulus relevance [Weis et al., 2013a, 2013b].

Similar to auditory sensory regions, our EEG data show multiple activation phases for the left motor cortex, with the first response peak occurring already during initial cortical deviance processing. In our view, this early effect may reflect preparatory activity to initiate rapid behavioral responses. Subsequent phases of motor activity were observed about 300 ms after deviance onset in conjunction with superior parietal lobe/intraparietal sulcus responses as well as about 400 ms after deviance onset, following up on the sequence of central executive network activations. Given the role of these structures in voluntary orienting of attention and executive control, later motor responses may thus be related to voluntary response selection.

Our fMRI data show that cortical change‐induced activity was accompanied by substantial subcortical activations, covering large portions of the thalamus as well as parts of the superior and inferior colliculi. These findings agree with previous studies reporting deviance‐related processing at subcortical level, in particular in the inferior colliculus and the medial geniculate nucleus of the thalamus [Antunes and Malmierca, 2014; Cacciaglia et al., 2015; Duque et al., 2015; Mitchell et al., 2015; Slabu et al., 2012]. In animals, subcortical effects have been shown to occur earlier than on the cortical level, but to be modulated by corticofugal auditory cortex projections [Antunes and Malmierca, 2014; Antunes et al., 2010; Malmierca et al., 2009]. Please note though that change‐induced thalamic activity in our current study was not observed in the medial geniculate nuclei but rather in the ventral and reticular nuclei of the thalamus. Our finding agrees with animal electrophysiology by Yu et al. [2009], showing stronger responses to sound deviants in the thalamic reticular nucleus than in the medial geniculate body, which is a target region of GABAergic input from this structure. Given that the thalamic reticular nucleus however receives strong corticofugal prefrontal inputs, it is also suggested to play a major role in subcortical information filtering and stimulus selection [Wimmer et al., 2015; Zikopoulos and Barbas, 2006]. Since the EEG source reconstruction approach used in our current study cannot extract source time courses from deep subcortical regions, it therefore remains open whether the observed thalamic fMRI activations are related to an early deviance detection process or rather to later stimulus selection.

Effects of Task Relevance

Relevance‐related differences in the spatial activation pattern observed for task‐relevant and task‐irrelevant sound feature changes emerged in the insula, the motor cortex, as well as in several brain regions of the central executive network, including the inferior and middle frontal gyri, the posterior medial frontal cortex, and the superior parietal lobe and the intraparietal sulcus. This overall pattern of brain regions agrees with previous neuroimaging work comparing relevant and irrelevant changes in sensory input. Please note, though, that while previous experiments also found increased responses to task‐irrelevant stimuli within regions of the central executive network, presumably reflecting response inhibition and performance monitoring, the insula was commonly reported to be more activated when processing relevant target stimuli [Downar et al., 2001; Kiehl et al., 2001; Kim, 2014]. Other fMRI experiments which used physically identical sounds with differing relevance, as we did here, however also failed to observe increased insula activity to relevant response targets, suggesting that this effect is modulated by stimulus characteristics [Alho et al., 2014; Salmi et al., 2009]. Furthermore, in contrast to some prior studies, which found increased auditory cortex activity following relevant as compared with irrelevant changes in auditory input, we here did not observe any differences in auditory sensory regions [Kiehl et al., 2001; Ross et al., 2010; Salmi et al., 2009]. In these studies, though, irrelevant changes either differed physically form relevant changes [Kiehl et al., 2001] or were presented within a spatially separated and unattended stream [Ross et al., 2010; Salmi et al., 2009], which may weaken the stimulus representation in auditory cortex [e.g., Bidet‐Caulet et al., 2007; Ding and Simon, 2012].

Our EEG data suggest that first differences between task‐relevant and irrelevant sound feature changes start to emerge about 150–200 ms after deviance onset in the anterior insula. This effect was consistently observed for both pitch and duration changes, thus indicating a general and feature‐independent role of the insula for an early evaluation of stimulus relevance. It was followed by a second phase of differential activity, encompassing parts of the central executive network and the motor cortex, starting about 300 ms after deviance onset. Noteworthy, while all ROIs showed significant relevance‐related modulations following duration changes, differences between task‐relevant and irrelevant pitch changes were only found in the middle frontal gyrus and the left motor cortex. Also, relevance‐related effects were generally less pronounced for pitch changes. It may be speculated that this lack of late relevance‐related differences in large parts of central executive network following pitch changes reflects the differing behavior observed in both categorization tasks. Irrelevant pitch changes were associated with both a significant response time slowing and a high proportion of response errors, whereas irrelevant duration changes hardly influenced stimulus categorization accuracy. Based on this, it seems plausible that relevant and irrelevant pitch changes were represented similarly in the brain, up until high processing stages involved in action selection, where motor responses to irrelevant pitch changes were inhibited. This is also in line with the role of the inferior frontal gyrus in error monitoring and response inhibition [Aron et al., 2004]. In contrast, change‐induced activity following irrelevant duration changes was attenuated quickly and late responses were largely identical to repetition trials, thus hardly affecting response accuracy.

The EEG results seem to be in some disagreement with our fMRI data, which show no significant interaction between the changing sound feature and task relevance. A likely explanation for this apparent mismatch may be that the EEG data analysis was restricted to event‐related potentials, time‐locked to the stimulus onset. Using this approach, activations exhibiting varying onset times across trials in response to the stimuli (so‐called induced activity) are not captured adequately. In contrast, the BOLD signal contains information about these responses as well. In addition to the differing activation sites, the direction of relevance‐related effects seem to vary between EEG and fMRI in our dataset. While fMRI responses were found to be increased for irrelevant as compared with relevant changes in all brain regions but in the left motor cortex, the EEG data generally shows higher amplitudes in response to task‐relevant changes. Such findings are, however, not uncommon in concurrent EEG‐fMRI and have, for example, also been observed in several previous auditory oddball experiments [Benar et al., 2007; Goldman et al., 2009]. Investigating the relationship between BOLD amplitude and oscillatory EEG power in a visual attention task, Scheeringa et al. [2011] demonstrated differential relationships between BOLD signals and high and low frequency EEG power. In their study, alpha and beta band power was inversely related to visual cortex BOLD signals, whereas gamma power showed the opposite relationship. In addition, the directionality of associations between EEG power in certain frequency bands and BOLD signal seems to be regionally specific [Michels et al., 2010]. Based on these findings, differences between relevance‐related effects in BOLD and EEG in our dataset should not be misinterpreted as showing opposing effects.

CONCLUSION

Using concurrent fMRI and EEG we here revealed the spatiotemporal dynamics of processing task‐relevant and irrelevant sound feature changes. Our data show the successive progression of change‐induced neural activity throughout different brain networks involved in auditory deviance detection, reorienting of attention to the changing sound, and stimulus evaluation and action selection. While early change detection was found to recruit feature‐specific networks involving auditory sensory but also frontal and parietal brain regions, the later spread of activity across the ventral and dorsal frontoparietal attention networks and the central executive network was largely independent of the changing sound feature, suggesting the existence of a general processing pathway of change‐related information. Task relevance did not modulate early auditory sensory processing, but was mainly found to affect late stimulus processing within frontal brain regions. We however also identified an early marker of task relevance at the level of the insula, where first relevance‐related activation differences were already observed from about 150–190 ms after deviance on.

Supporting information

Supporting Information Figure 1

Supporting Information Figure 2