Hierarchy of prediction errors for auditory events in human temporal and frontal cortex

Significance

To survive, organisms must constantly form predictions of the future based on past regularities. When predictions are violated, action may be needed. Different scales of environmental regularity need to encompass both subsecond repetitions and complex structures spanning longer timescales. How different parts of the brain monitor these temporal regularities and produce prediction error signals is unclear. Utilizing subdural electrocorticographic electrodes with an auditory paradigm involving local and global regularities, we show that frontal cortex is sensitive to the big picture, responding with high γ-band activity exclusively to globally unpredictable changes, whereas the temporal cortex equally responds to any change in the immediate history. These results reveal a hierarchy of predictive coding recorded directly from the human brain.

Abstract

Predictive coding theories posit that neural networks learn statistical regularities in the environment for comparison with actual outcomes, signaling a prediction error (PE) when sensory deviation occurs. PE studies in audition have capitalized on low-frequency event-related potentials (LF-ERPs), such as the mismatch negativity. However, local cortical activity is well-indexed by higher-frequency bands [high-γ band (Hγ): 80–150 Hz]. We compared patterns of human Hγ and LF-ERPs in deviance detection using electrocorticographic recordings from subdural electrodes over frontal and temporal cortices. Patients listened to trains of task-irrelevant tones in two conditions differing in the predictability of a deviation from repetitive background stimuli (fully predictable vs. unpredictable deviants). We found deviance-related responses in both frequency bands over lateral temporal and inferior frontal cortex, with an earlier latency for Hγ than for LF-ERPs. Critically, frontal Hγ activity but not LF-ERPs discriminated between fully predictable and unpredictable changes, with frontal cortex sensitive to unpredictable events. The results highlight the role of frontal cortex and Hγ activity in deviance detection and PE generation.

The ability to detect unexpected environmental events results from a comparison of the actual state of our sensory world with predictions based on immediate and long-term contextual knowledge. Predictive coding theory, first articulated within the visual domain, postulates that distributed neural networks learn statistical regularities of the natural world, generating a prediction error (PE) signal as deviations from these predictions occur (1). Because of the difficulty of recording high-frequency activity in scalp EEG recordings, studies on PE in audition have focused on low-frequency event-related potentials (LF-ERPs). The mismatch negativity (MMN) is considered the classic PE signal elicited during passive listening to deviant sounds interrupting the context provided by a sequence of repeated standard stimuli (2). Converging evidence suggests that the MMN has interacting generators in the secondary auditory cortex on the superior temporal plane and superior temporal gyrus (STG) as well as in the prefrontal cortex (3, 4), but the distinct contribution of each part of this network, especially the prefrontal part, is not clear. Evidence from neuropsychological event-related potentials and neuroimaging studies supports a key role of the prefrontal cortex in contextual processing (5, 6), suggesting a crucial role of this brain region in predictive coding.

Importantly, low-frequency scalp-recorded responses, like the MMN, do not reveal the full spectrum of the neuronal response to prediction violation. Whereas recording high frequencies with scalp EEG has major methodological issues related to low signal to noise (7, 8), numerous studies using electrocorticography (ECoG; recorded on the cortical surface) have shown high γ-band (Hγ) response to be a localized index for functionally selective activity (9, 10). It is not clear whether cortical neuronal activity responsible for deviance detection is best indexed by low- or higher-frequency bands. This differentiation is critical, because the Hγ has distinct response properties compared with LF-ERPs (11). Using intracranial recordings, involvement of low frequency-evoked activity and Hγ-induced activity in auditory PE signals were found in temporal regions (12, 13), where Hγ amplitude was shown to increase earlier than lower-frequency bands (12). In inferior frontal regions, previous ECoG studies (12, 14) did not find evidence for Hγ frontal activity in response to local deviations [as opposed to global ones; discussed in the work by El-Karoui et al. (14)], although low-frequency effects were reported in some (15, 16) but not all (17) studies.

Using the high temporal and spectral resolution of direct cortical recordings from subdural ECoG electrodes, we compared frontal and temporal cortical patterns of LF-ERPs and Hγs in five patients listening to trains of task-irrelevant auditory stimuli in two conditions. The conditions differed in the predictability of deviation from repetitive background stimuli (fully predictable: four standards always followed by a deviant vs. unpredictable: deviants randomly embedded in trains of standard stimuli). Subjects were instructed to ignore the sounds and watch a visual slideshow. We focused on the amplitude and latency variation of both LF-ERPs and Hγs as metrics of the PE (mismatch signal). Based on previous findings, we hypothesized that Hγ activity signals the mismatch earlier than LF-ERPs and that the temporal (auditory) cortices would be sensitive to local probabilities and not affected by the predictable vs. unpredictable manipulation. In contrast, the frontal cortex, assumed to be sensitive to higher-order regularities, would be differently affected by periodic vs. nonperiodic deviations. This differential frontal response could be either a stronger response to predictable than unpredictable deviants, signaling a mechanism of suppression of orienting response toward the expected repeating deviant, or a stronger response to unpredictable deviants compared with predictable deviants, signaling the PE.

Results

Participants (n = 5) (Methods) listened to sound trains of high-probability standards (P = 0.8; F₀ = 500 Hz) mixed with low-probability deviants (P = 0.2; F₁ = 550 Hz) in blocks of 400 sounds, with a stimulus onset asynchrony (SOA) of 600 ms. The order of the sounds was either pseudorandom, with a minimum of three standard tones before a deviant, or regular, such that exactly every fifth sound was a deviant (Fig. 1A). Thus, in the regular condition, deviants were fully predictable, whereas in the irregular condition, exact prediction was not possible. In both conditions, the participants were instructed to ignore the sounds and watch a slideshow of a variety of visual images changing at an unpredictable slow pace (∼3 s per picture; unsynchronized with the auditory stimuli). The pictures were displayed on a liquid crystal display (LCD) monitor positioned over the patient’s bed. Channel time series were used for the following analysis steps that are explained in more detail in Methods. We first selected channels showing stimulus-responsive activity modulation in the Hγ and the LF-ERP band (Methods, I: Stimulus-responsive activity modulation and Fig. 1). In each of these channels, we calculated a time point by time point ANOVA on stimulus type (standard or deviant) and determined the F_{stimulus type} value time series (Methods, II: Mismatch signal). We used a principal component analysis (PCA) and found the course of F_{stimulus type} across time accounting for the highest variance within the set of stimulus-responsive channels (Methods, III: PCA). We selected channels loading highly on the principal component (Methods, IV: Data reduction). We then compared time points of onset and peak (maximal) F values across channels (Methods, V: Comparison of mismatch signal timing) and verified the results on a group level (within subject) [Methods, VI: Group (within-subject) analysis]. We verified that differences in onset and peak latency between frequency bands are independent of anatomical locations [region of interest (ROI)] (Methods, VII: ROI-specific analysis). Finally, we tested in which anatomical location a predictability effect is represented (Methods, VIII: Predictability effect). In Methods, we provide detailed descriptions of each of these steps. The steps were taken separately for the LF-ERPs and Hγ signals. We studied 287 channels across all subjects. Stimulus-related activity was found for the Hγ (n = 40 channels; 13%) and the LF-ERP (n = 116 channels; 40%) (Table 1 shows numbers of stimulus-responsive channels per subject) bands across multiple frontal and temporal recording sites (Fig. 1B). The F_{stimulus type} values in highly loading LF-ERP channels (n = 14) passed the empirical threshold around 143 ms (SD across channels = 44.9 ms) and peaked around 221 ms on average (SD = 8.8 ms) (Fig. 1C); the small SD is because of the fact that these are the channels loading highly on a single temporal PCA component. That is, these channels should necessarily have high resemblance in their temporal structure. F values in stimulus-responsive Hγ channels loading highly on the first Hγ principal component (n = 7) passed the empirical threshold around 72 ms (SD = 34 ms) and peaked around 141 ms on average (SD = 9 ms). The temporal differences between LF-ERP and Hγ for onset latency (t₂₀ = 3.8; P = 0.0011) as well as for peak F values (t₂₀ = 13.92; P < 0.0001) (Fig. 1C) across channels were significant. These differences were replicated in a within-subject group analysis (Hγ: 100.8 ms; SD = 70.4 ms; LF-ERP: 286.1 ms; SD = 140.2 ms; Wilcoxon rank sum test for onset P = 0.024; peak latency difference: Hγ: 139.3 ms; SD = 90.3 ms; LF-ERP: 343.5 ms; SD = 168.1 ms; Wilcoxon rank sum test for peak P = 0.039) (Fig. 1D).

Fig. 1.

Temporal profile of mismatch signal of Hγ and LF-ERP bands. (A) Participants listened to stimuli consisting of 180-ms-long (10 ms rise and fall time) sounds. High-probability standards (500 Hz) mixed with low-probability deviants [550 Hz; stimulus onset asynchrony (SOA) = 600 ms] were presented either unpredictably (pseudorandom sequence: minimum three consecutive standards) or fully predictable (regular: exactly every fifth sound was a deviant). (B) Stimulus-responsive regions in the Hγ and low-frequency (LF) band for all subjects. C, Left shows the principal components (PCs) of F-value time series of all task active channels for (Upper) the Hγ and (Lower) the LF bands. C, Right shows channels of F_{stimulus type} time series of highly loading channels. Differences in color were chosen to better distinguish the F time course of different channels. The time course of the PC reveals a statistically significant difference (asterisk) in latency with an earlier maximum of the peak F values (black dots) in the Hγ band than in the LF-ERP band. (D, Upper) Averaged subject-specific F-value time course for the Hγ (red) and event-related potential (blue). (D, Lower) Onset and peak latency of PCs. Onset and peak latencies differ significantly across subjects between Hγs and LF-ERPs. Error bars indicate the SE across subjects. E, Upper shows F_{stimulus type} time series separately averaged across highly loading frontal Hγ (magenta), temporal Hγ (blue), temporal LF (black), and frontal LF channels (red). (E, Lower) Mean onset and peak latencies of F values for each frequency band and anatomical ROI (Table 2 shows mean onset and peak latency). Error bars show SE across channels.

Table 1.

Number of stimulus-responsive LF-ERPs and Hγ channels per subject

Patient	Stimulus-responsive LF-ERP channels (temporal/frontal/parietal)	Stimulus-responsive Hγ channels (temporal/frontal/parietal)	Total no. of electrodes*
I	31 (14/12/5)	18 (6/7/5)	60
II	25 (12/1/12)	6 (4/1/1)	59
III	19 (10/5/4)	9 (4/3/2)	52
IV	10 (4/3/3)	2 (1/1/0)	56
V	31 (17/2/12)	5 (2/1/2)	60
∑	116	40	287

^*Excluding electrodes rejected for epileptic activity or excessive artifacts.

ROI Analysis.

We tested for the differential effect of the mismatch signal over frequency bands, comparing electrodes placed over the lateral temporal lobe and electrodes placed over the lateral frontal cortex. Fig. 2 shows the F_{stimulus type} variation across time averaged across highly loading channels separately for the frontal and temporal Hγ and LF-ERP bands. We found a significant effect of frequency band (F_onset = 6.53; P = 0.02 and F_peak = 27.5; P < 0.0001; df = 1, 28) (Fig. 2 and Table 2 show average onset and peak latencies of Hγ and LF-ERP band differences between ROIs) but no main effect of ROI or effect of interaction (P > 0.1). Hence, Hγ activity shows an earlier discrimination between deviants and standards than LF-ERP in both lateral temporal and frontal cortex.

Fig. 2.

Predictability effect. Frontal and temporal Hγ and LF-ERP activities differ with respect to effect of predictability. Colored lines show the evoked response to stimuli separately for frontal and temporal Hγs and LF-ERPs. Zero marks the auditory stimulus presentation. Blue lines show response to fully predictable standards, green lines show response to unpredictable standards, red lines show response to fully predictable deviants, and cyan lines show response to unpredictable deviants. Shaded areas denote the SE across highly loading channels. Corresponding F-value time series for the ANOVA across channels are set above, depicting the strength of statistical significance of the main effect for stimulus type (first row), the main effect for predictability (second row), and the effect of interaction (third row) in gray scale. Darker shades denote higher F values; time windows with F values smaller than the corresponding statistical significance threshold are shown in white.

Table 2.

Onset and peak F values

Amplitude	Hγ frontal	Hγ temporal	LF-ERP frontal	LF-ERP temporal
Onset	92	85	202	141
Onset SD	31	65	102	68
Peak	132	143	306	229
Peak SD	15	85	20	59

Summary of mean latency and SD of peak F-value latency across channels per ROI and frequency band in milliseconds.

Predictability Effects.

Fig. 2 shows the amplitude variation (averaged across highly loading channels) in response to standard and deviant trials for the LF-ERP and the Hγ band in the frontal and temporal ROIs together with the time point by time point F statistic for the main effects of stimulus type, predictability, and their interaction across highly loading channels. Because the channels were selected to show a stimulus type effect in the first stages of the analysis, significance values of the main effect of stimulus type in this analysis may be inflated. However, the focus here is on the critical effects of predictability on neural responses. Table 3 summarizes maximal F values and corresponding P values for each ROI and frequency band. The threshold F value derived from the empirical distribution center is around 4.4 for all tests. LF-ERPs differentiate between standards and deviants starting around 200 ms but do not show amplitude variation as a function of predictability (Fig. 2) or an interaction between stimulus type and predictability. In contrast, the frontal Hγ channels show an effect of interaction between stimulus types (standards vs. deviants) and predictability (Fig. 2) driven by the stronger response to deviants than standards when the deviants were unpredictable than when they were predictable. The corresponding within-subject analysis also revealed a significant interaction in frontal but not the temporal sites for Hγ activity (Fig. S1) as indicated by a strong Hγ mismatch response (MMR) to unpredictable deviants but nearly no MMR for predictable deviations in frontal electrodes. Furthermore, only frontal Hγ showed sustained activity (Methods and Fig. S2). These results indicate that frontal Hγ discriminates between predicted and unpredicted deviants, with a selective response to unpredicted deviations.

Table 3.

F values per ROI and frequency band

Effect type	Temporal	Frontal
Hγ
MEst	F(1,32) = 13.9; P < 0.00001*	F(1,20) = 5.07; P = 0.004*
MEbt	n.s.	n.s.
IEst-x-bt	n.s.	F(1,20) = 8.2; P < 0.00001*
LF-ERP
MEst	F(1,24) = 7.56; P = 0.011*	F(1,8) = 27.86; P < 0.00001*
MEbt	n.s.	n.s.
IEst-x-bt	n.s.	n.s.

Summary of maximal F values per ROI and frequency band and corresponding P values. Only significant F values are reported. Differences in df are because of the different numbers of significant electrodes selected per ROI and frequency band. IE_st-x-bt, effect of interaction between stimulus type and block type; ME_bt, main effect of block type; ME_st, main effect of stimulus type; n.s., not significant.

^*Statistically significant.

Fig. S1.

Frontal and temporal Hγ differ with respect to effect of predictability on a group level. (Middle) Colored lines show the evoked response to stimuli separately for frontal and temporal Hγ. Zero marks the auditory stimulus presentation. Blue lines show response to fully predictable standards, green lines show response to unpredictable standards, red lines show response to fully predictable deviants, and cyan lines show response to unpredictable deviants. Shaded areas denote the SE across subjects. (Top) Corresponding F-value time series for the ANOVA across subjects were set, depicting the strength of statistical significance of the main effect for stimulus type (first row), the main effect for predictability (second row), and the effect of interaction (third row) in gray scale. F values smaller than the F value corresponding with the statistical significance threshold are shown in white. Bottom shows evoked Hγ responses of a representative subject (S03). Colored lines show the average across channel.

Fig. S2.

Comparison of amplitude modulation time over baseline. (Left) Depiction of standardized amplitude modulation to unpredictable deviants of the Hγ (HG) band in the frontal (green) and temporal (blue) ROIs. We standardized the amplitude to (i) correct for differences in modulation and (ii) estimate the CI of a joint baseline distribution (red lines). (Right) Frontal Hγ amplitude was longer activated over baseline (>97.5%) than temporal Hγ amplitude.

Discussion

We examined the role of frontal and temporal cortices in generation of a PE signal for auditory deviants operationalized as the difference between the response to deviant and standard stimuli. Deviations from auditory background stimuli modulated the response to the sounds in both the lower frequencies event-related potentials, typically associated with the scalp-recorded MMN, and the power of the Hγ band recorded directly from the cortex. The PE signal emerged earlier in the Hγ amplitude than in the LF-ERPs and was evident at both temporal and frontal channel locations. However, only the frontal cortex Hγ differentiated between fully predictable and unpredictable deviations, emphasizing the key role of frontal cortex in PE.

The effect that we found with ECoG started at ∼140 ms and peaked at 230 ms. Previous scalp MMN studies reported response differences between standard and deviant stimuli onsetting and peaking between 100 and 250 ms (reviewed in ref. 18). The LF-ERPs effects observed in our study are at the longer latency range of these scalp findings, which may reflect the difference between scalp and epicortical recordings. For example, most studies of MMN have not dissociated N1 refractoriness effects from the memory-based MMN (19, 20). When measures are taken to isolate the MMN from N1 refractoriness effects, the MMR has a longer latency than when the traditional MMN (deviant–standard) derivation is used (20–23). We used the traditional contrast of deviant–standard. However, our ECoG electrodes, located on the lateral surface of the brain, are less sensitive to refractoriness-sensitive N1 or earlier (24) sources located on the supratemporal plane within the sylvian fissure (25, 26) than at frontal/central scalp electrodes, where the scalp MMN is typically measured. Hence, whereas ECoG allows high accuracy in spatial localization of effects, scalp recordings may provide a more global picture of the evolution of deviance related activity in time at the cost of spatial uncertainty. In addition, we cannot rule out the possibility that the special conditions of ECoG intensive care unit recordings might have slowed neural responses relative to laboratory conditions typical of EEG.

Studies using the mismatch paradigm with scalp EEG or magnetoencephalography (MEG) support the presence of separate temporal and frontal generators of the MMR (3, 27, 28). However, because of the ill-posed inverse problem, EEG and MEG are not well-suited for localizing brain sources with certainty or spatially resolving adjacent sources. Moreover, recording high-frequency activity from the scalp is limited by low signal to noise ratio in this band (7, 8). Functional MRI data using similar paradigms support the presence of both temporal and prefrontal activity, but whether they have distinct response profiles could not be clearly discerned, partly because of the lack of temporal resolution of functional MRI (3, 29).

The use of intracranial recording directly from the surface or depth of the cortex allows simultaneously high spatial and temporal resolution of local neural activity in the human cortex. Previous intracranial findings using a mismatch paradigm have converged on showing responses to deviants over the STG (30, 31). In contrast, although some studies reported MMRs over inferior frontal cortex, others did not find such evidence (3), perhaps because of the sparse and variable spatial sampling of ECoG, and in some cases, the reported frontal responses could have been caused by volume conduction from temporal sources (15). Most of these previous studies have only examined low-frequency event-related responses. However, Edwards et al. (12) have also shown that broadband temporal Hγ responses to deviants are stronger than for standards. Eliades et al. (31) suggested that this effect is a result of adaptation to the repeating standard. Our results, showing clear deviant-related responses over both the temporal and frontal cortex for both frequency bands, support a frontal cortex contribution to the MMR. Moreover, the dissociation between the pattern of response to the predictable and unpredictable stimuli across the two regions provides evidence for distinct processing in these two regions.

Indeed, the major finding of our study is that predictability affected the PE response of the frontal cortex as measured by the Hγ activity, whereas no such modulation was found in temporal electrodes. Although the response measured over temporal cortex revealed a mismatch signal, regardless of the global structure of the sequence (i.e., its predictive value), the frontal PE was seen almost exclusively in response to unpredictable deviants.

The lack of predictability effects at the level of the auditory cortex is in line with previous scalp EEG studies using a similar task design. Volosin and Horváth (32) found the P3 response to be sensitive to periodicity, whereas early components, such as the N1 and the MMN, which are generated by sources in the auditory cortex, are unaffected, especially when participants were instructed to ignore the auditory stimulation (33, 34). Effects of predictability at the MMN time window also depend on the interstimulus interval (ISI). Sussman et al. (35) found that the scalp MMN was suppressed when the sequence in the fully predictable condition had very short ISIs (100 ms), such that repeating sequences of stimuli could be integrated and perceived as united auditory objects. In contrast, longer ISIs, in the range used here, yielded similar MMN amplitudes in response to both fully predictable and unpredictable deviants. One explanation for the dissociation between predictable and unpredictable deviants that we found at the frontal sites could be that the frontal cortex integrates over longer timescales than the auditory cortex. Under this premise, although for the auditory cortex, with the ISIs used, the unit of processing would be the individual tone and the deviant tone would be an oddball in both predictable and unpredictable sequences, for the frontal cortex, the predictable sequence would be seen as repeating identical units, each composed of four low tones and one high tone.

A recent intracranial study using depth electrodes also examined local and global deviations. El-Karoui et al. (14) used a design in which local and global deviations were embedded in the same sequence. Trials were composed of a rapid sequence (SOA = 150 ms) of either five identical tones (SSSSS) or four identical tones followed by one deviant tone (SSSSD). In some blocks, SSSSS trials were frequent (80%), and SSSSD trials were rare (20%), whereas in the other blocks, this probability was reversed. The final D tone compared with a final S tone represents local deviation in either block. It was also a global deviant when the SSSSD trials were rare. The final S tone is always a local standard, but it represents a global deviant when SSSSS trials are rare and a global standard when SSSSS trials are frequent, arguably allowing for a pure measurement of a global effect. For local deviations, both LF-ERP responses and broadband Hγ responses were restricted to mainly superior temporal lobe contacts, with the exception of one frontal electrode that showed Hγ response. In contrast, global deviations elicited more widespread and protracted responses, including low-frequency effects and Hγ augmentation in temporal and left frontal contacts and a frontotemporoparietal depression in the β-band.

In our study, all deviants may be considered local deviants, but only in the nonpredictive block are they also global deviants. Consistent with the work by El-Karoui et al. (14), in our study, both local and global deviations elicited a response in both the LF-ERP and Hγ bands in lateral temporal contacts. Furthermore, in both studies, global deviations included a significant frontal cortex response. However, unlike in the work by El-Karoui et al. (14), we observed an LF-ERP frequency response to local deviation (predictable or not) in frontal electrodes, as did others (15, 16, 36). There are two limitations that preclude direct comparisons between our study and the work by El-Karoui et al. (14). First, our study compared responses to predictable and unpredictable tones that were both irrelevant to the task and in an unattended modality. This scenario probed the automatic response to the environment to enable orienting attention to critical or unexpected events. The global deviation in the study by El-Karoui et al. (14) was the target, and subjects had to count and memorize the total number of global deviations. Consequently, their global effect is a target-related response and not an automatic response. Second, although the design of the study by El-Karoui et al. (14) was a 2 × 2 design, they only examined the main effects of local and global deviation and did not examine the interaction that would be important if local deviants elicit stronger responses in blocks when they are also global deviants.

The findings of earlier responses in the γ-band than in the lower frequencies are consistent with the findings in the work by Crone et al. (11), which found that the functional response properties of Hγ activity are distinct from lower-frequency modulations in timing and spatial location. Specifically, lower frequencies, such as α-activity, reach peak amplitude later than Hγ. The fact that Hγ activity signals the PE earlier than LF-ERPs in our data could be because of enhanced signal to noise ratio in the higher band. However, the distinct effect of predictability, found only in the Hγ signals, suggests that the two frequency bands reflect different neuronal mechanisms.

Limitations and Future Research.

We did not find sensitivity to periodic structure in auditory cortex contacts, such as it was shown for the rat or macaque auditory cortex (37, 38). Yaron et al. (37) found neurons responding differently to fully predictable and unpredictable standards and deviants. However, the largest fraction of neurons in the rat auditory cortex responded similarly in the fully unpredictable condition. The registered cortical signal in ECoG recordings is determined by population activity. Hence, the activity patterns of a minority of neurons may be missed by ECoG macroelectrodes. Nevertheless, even if a periodicity effect in the human temporal cortex is present at a finer neuronal level, this effect is more prominent in the human frontal cortex.

The predictability in our study results from periodicity–every fifth sound was a deviant. However, context effects on the MMN or adaptation can be experimentally studied by manipulating different sources of expectation, which may depend on dissociable neural mechanisms. For example, Todd et al. (39, 40) found that, when the identities of the standard and deviants were switched during the experiment, the MMN for the initial order was larger than for the subsequent order, especially when longer sequences were played before the switch. This “primacy effect” suggests a long-term memory of the initial standard. Other studies found that, for fast, isochronous sequences (SOA = 150 ms), two MMNs occur for a pair of deviants only when some deviants also occur alone (41, 42), suggesting that, when the second deviant is highly expected, MMN is attenuated. This expectancy suppression seems to be dissociable from pure repetition suppression (43). Another source of expectancy may depend on the variance of the standards (44).

Our critical conclusion is that frontal and temporal cortices have different functions in signaling the deviation or PE. Frontal Hγ selectively signals unpredictable deviants with sustained Hγ activation, whereas temporal Hγ shows responses to both unpredictable and fully predictable deviants. This result highlights a selective role of frontal structures in computing a PE. A feature-based adaptation mechanism, as seen in the auditory cortex (13, 31, 45), is expected to produce a response independent of the degree of periodicity of occurrence of deviants. Because on average, the probability of stimuli in the predictable and unpredictable sequences was identical, a purely adaptation-based model (31) would not predict the differential activity that we found in frontal electrodes. Thus, the frontal cortex Hγ provides evidence of a selective PE to unpredictable events. The selective Hγ amplitude modulation to unpredictable deviants might also reflect a switch of attention (4). However, both functional explanations (selective response to unpredictable events or a switch of attention) indicate that frontal Hγ activity reflects a mechanism that tracks the expected input and generates a response when predictions are violated.

In sum, our findings support the notion that Hγ activity in the frontal cortex signals detection of unpredictable deviations from the auditory background.

Methods

Patients.

Five epilepsy patients (mean age = 33 y old; SD = 9.23) undergoing presurgical monitoring with subdural electrodes participated in the experiment after providing their written informed consent. Experimental and clinical recordings were taken in parallel. Recordings took place at the University of California, San Francisco and approved by the local ethics committees (Committee for the Protection of Human Subjects at the University of California, Berkeley).

Stimuli.

Participants listened to stimuli consisting of 180-ms-long (10 ms rise and fall time) harmonic sounds with a fundamental frequency (F₀) of 500 or 550 Hz and the three first harmonics with descending amplitudes (−6, −9, and −12 dB relative to the fundamental). The stimuli were generated using Cool Edit 2000 software (Syntrillium). The stimuli were presented from loudspeakers positioned at the foot of the subject’s bed at a comfortable loudness. High-probability standards (P = 0.8; F₀ = 500 Hz) were mixed with low-probability deviants (P = 0.2; F₁ = 550 Hz) in blocks of 400 sounds, with an SOA of 600 ms (Fig. 1A). The order of the sounds was either pseudorandom, with a minimum of three standard tones before a deviant, or regular, such that exactly every fifth sound was a deviant. Thus, in the regular condition, deviants were fully predictable, whereas in the irregular condition, exact prediction was not possible. In both conditions, the participants were instructed to ignore the sounds and watch a slideshow of a variety of visual images changing at an unpredictable slow pace (∼3 s per picture; unsynchronized with the auditory stimuli). The pictures were displayed on an LCD monitor positioned over the patient’s bed.

Data Processing.

Details of data recording and data preprocessing are in SI Methods. The resulting time series were used to characterize brain dynamics over the time course of auditory mismatch detection in terms of the LF-ERPs and Hγ activity. For each trial (−1–2 s around stimulus onset—sufficiently long to prevent any edge effects during filtering), we band pass filtered each electrode’s time series at two frequency bands: a low-frequency band (LF-ERP: 1–20 Hz; the “LF-ERP range” traditionally used for scalp MMN studies) and a high-frequency band (Hγ range: 80–150 Hz) (selection of frequency bands is discussed below and in Fig. S3 A and B). We obtained the Hγ analytic amplitude $A_{HG}\left(t\right)$ by Hilbert transforming the Hγ filtered time series. We smoothed both the LF-ERP and the Hγ band time series, such that amplitude value at each time point n is the mean of 10 ms around each time point n. We then baseline corrected the trial activity by subtracting the mean activity from the 100 ms preceding the stimulus onset in each trial of each channel.

Fig. S3.

(A) Depiction of differences of mismatch signal as a function of frequency. Each line shows the average of highly loading channels on the frequency-specific principal component accounting for the highest variance across time. Note that the profile of the response (maximal F) displayed in B shows that the PE is concentrated at low frequency (LF; ∼1–20 Hz) and Hγ (∼80–150 Hz), whereas the response between 20 and 80 Hz is lower. (C) Depiction of grand averages of Hγ and LF amplitude modulation. In each frequency band, the amplitude modulation to the onset of auditory stimulation was averaged across trials and then, across all channels pooled across all subjects. In each frequency band, we compared the first 250 ms with 100 ms baseline preceding the onset of standard and deviant tones. We obtained the Hγ analytic amplitude $A_{HG}\left(t\right)$ by Hilbert transforming the Hγ filtered time series. Note that, in contrast to the analytic amplitude of the Hγ band, which is always equal to or larger than zero, the LF band is a time series of activity and has both positive and negative going deflections (e.g., like P1-N1). Hence, channels with an LF amplitude modulation in interval I or III greater than the 97.5th percentile (significantly elevated amplitude over baseline) and channels with an LF amplitude modulation in interval II smaller than the 2.5th percentile were classified as showing a significant LF-ERP modulation after the standard or deviant stimuli or both.

I: Stimulus-responsive activity modulation.

We first identified stimulus-responsive channels showing a significant (compared with an empirical distribution; see below) amplitude modulation in either the Hγ or LF-ERP band or both after the onset of standard stimuli, deviant stimuli, or both. Standard and deviant trials were averaged separately. For each type of stimulus, we first calculated the average baseline activity ${\overline{B}}_{\text{H}\gamma }$ and ${\overline{B}}_{LF}$ across 100 ms preceding the stimulus onset. For the Hγ activity, we subtracted ${\overline{B}}_{\text{H}\gamma }$ from the activity modulation ${\overline{A}}_{\text{H}\gamma }$ averaged across 250 ms after the stimulus onset (Fig. S3C). For the LF-ERP band, we subtracted ${\overline{B}}_{LF}$ from the activity modulation ${\overline{A}}_{LF}$ in three different intervals centered on the main peaks of the mean response (I, 0–60 ms; II, 60–120 ms; and III, 120–250 ms in Fig. S3C). This early time window allowed us to select fast-responding channels in both frequency ranges (confirmation that the selected length of stimulus response intervals had no effect on the selection stimulus-responsive channels is in SI Methods and Fig. S4). The difference between $\overline{B}$ and $\overline{A}$ was compared against an empirical distribution (SI Methods).

Fig. S4.

Depiction of F-value time series averaged across stimulus-responsive channels as a function of length of activation period. Amplitude modulation of Hγ and low-frequency (LF) activity of 300–500 ms after stimulus onset was compared with baseline activity (−100–0 ms). Selected principal components estimated within the set of stimulus-responsive channels in each interval are similar.

II: Mismatch signal.

Within each stimulus-responsive channel and separately for Hγs and LF-ERPs, we carried out a one-way ANOVA with factor stimulus type (standard vs. deviant) at each time point, with single trials as random variable. This analysis yielded a time series of F values for each channel representing the main effect of stimulus type between −100 and 600 ms. In both conditions, we left out the first two trains, because the periodicity can only be detected after a repeated completion of whole trains of stimuli. The F value of the main effect “stimulus type” parameterizes the mismatch signal with high F values, indicating a large difference in amplitude between the standard and deviant stimuli. To set a threshold for significant difference, an empirical distribution of the main effect was constructed by randomly reassigning the labels (standard or deviant) to the single trials in 1,000 permutations.

III: PCA.

Using a PCA, we identified consistent temporal patterns of Hγ/LF-ERP activity among the entire set of stimulus-responsive channels pooled across patients. The PCA was used to find the course of the mismatch signal (F_{stimulus type}) across time, accounting for the highest variance within the set of stimulus-responsive channels.

IV: Data reduction.

Channel time series strongly resembling the mismatch signal determined in III (highly loading on the first principal component of the PCA) are those that exhibit large differences between standard and deviant stimuli in terms of amplitude. The degree of resemblance with the mismatch signal is given as Pearson’s r. We chose Pearson’s r exceeding the 75th percentile of all positive r values as the cutoff criterion for highly loading channels. We set this level as a tradeoff, because setting the criterion too high would exclude too many channels and reduce generalization across the cortex, whereas setting it too low would include channels with minor effects.

V: Comparison of mismatch signal timing.

In each highly loading channel (exceeding the cutoff criterion), we tested which frequency band (LF-ERP vs. Hγ) showed a mismatch signal first by determining the onset and peak latency of significant F values. The onset latency of the mismatch signal was determined as the first point in which the F value exceeded the confidence interval of the empirical distribution. Both latency of maximal F values in the Hγ and LF-ERP band and the onset of significant F values were used to quantify temporal differences of the PE between the frequency bands. Onsets and peak latencies were compared in a two-sample (because of different numbers of channels per frequency band) t test comparing the two frequency bands.

VI: Group (within-subject) analysis.

To verify that the results presented are valid at a group level and not driven by single subjects, a PCA was used in each subject to find the course of the mismatch signal across time, accounting for the highest variance for both the LF-ERP and the Hγ bands. The onset and peak latencies of the LF-ERP and Hγ principal components were compared in a within-subject analysis using a Wilcoxon rank sum test.

VII: ROI-specific analysis.

In the next step, we tested whether differences regarding the onset and maximum of deviance detection between frequencies depend on anatomical locations. An interaction between the frequency band and the ROI effects would indicate such dependence. The pool of all LF-ERP and Hγ stimulus-responsive channels were grouped according to a frontal and temporal ROI to analyze ROI-specific patterns of PE. Steps III and IV were then performed separately for each ROI. We determined the peak and onset latencies of the mismatch signals for each channel and frequency band as described in step V and conducted a two-way ANOVA across channels with the factors frequency band (Hγ vs. LF-ERP) and ROI (temporal vs. frontal).

VIII: Predictability effect.

Finally, to test the effect of the predictability of deviance, we used a time point by time point ANOVA to look for an interaction of the block type (predictable and unpredictable) with the effect of stimulus type across channels separately for each ROI and frequency band limited to the channels that loaded highly on the first principal component (from step IV). Because the channels were selected to show a stimulus type effect in the first stages of the analysis, significance values of the main effect of stimulus type in this analysis may be inflated (“double dipping” effect). However, the critical effects of predictability and the interaction are not. The significance values were assessed using bootstrap procedures as outlined above.

SI Methods

Data Recording.

The ECoG was recorded at the University of California, San Francisco using 64 platinum–iridium–electrode grids arranged in an 8 × 8 array with 10-mm center to center spacing (Ad-Tech Medical Instrument Corporation) (grid location is shown in Fig. 1). Exposed electrode diameter was 2.3 mm. The data were recorded continuously throughout the task at a sampling rate of 2,003 Hz.

Data Preprocessing.

We used Matlab 2013b (Mathworks) for all offline data processing. All filtering was done using zero-shift infinite impulse response (IIR) filters [Butterworth filter of order 4: filtfilt() function in Matlab]. We excluded channels exhibiting ictal activity or excessive noise from additional analysis. In the remaining “good” channels (Table 1), we then excluded time intervals containing artefactual signal distortions, such as signal steps or pulses, by visual inspection. Finally, we rereferenced the remaining electrode time series by subtracting the common average reference

$$x_{CAR}\left(t\right)=\frac{1}{n}{\displaystyle \sum }_{c=1}^n{x}_c\left(t\right)$$

calculated from each channel time series x_c.

Empirical Distribution of Stimulus-Responsive Activity Modulation.

The difference between $\overline{B}$ and $\overline{A}$ was compared against an empirical distribution derived from randomly shifted time series (1,000 permutations). In each iteration, time series of each channel and each trial were shifted (circular shift of the entire trial time series between −0.1 and 0.6 s) separately, and new (surrogate) trial averages ($\overline{B}$ and $\overline{A}$) were calculated from the shifted trials. Channels exceeding the 97.5th percentile of the channel-specific surrogate ${\overline{B}}_{\text{H}\gamma }-{\overline{A}}_{\text{H}\gamma }$ distribution were classified as showing a significant Hγ modulation after the standard or deviant stimuli or both. Note that, in contrast to the analytic amplitude of the Hγ band, which is always equal or larger than zero, the LF-ERP band is a time series of activity and comprises both positive and negative going deflections (e.g., like P1-N1). Hence, channels with an LF-ERP amplitude modulation in interval I or III greater than the 97.5th percentile (significantly elevated amplitude over baseline) and channels with an LF-ERP amplitude modulation in interval II smaller than the 2.5th percentile were classified as showing a significant LF-ERP modulation after the standard or deviant stimuli or both. Extending the critical windows up to 300, 400, 500, or 600 ms did not affect the selection of channels or the principal components.

Selection of Frequency Bands.

To test which frequency band was the most informative in signaling the PE, we filtered all time series of each channel in each trial in 39 bands between 2 and 180 Hz (log spaced), each with a bandwidth of 10% of the center frequency. We applied the same pipeline as in the main analysis up to stage V, except that all channels were included in this analysis (skipping stage I). On average, 8.15 channels were chosen in each frequency band (SD = 1.2) based on their high loading on the first principal component within each band. The profile of the response shows that the PE is concentrated in the low-frequency (∼1–20 Hz) and Hγ (∼80–150 Hz) range, whereas the response between 20 and 80 Hz is lower (Fig. S3). Thus, for the main analysis, a low (1–20 Hz) band and a high band (80–150 Hz) were tested (Methods).

Sustained Activity.

Fig. 2 suggests that, in contrast to temporal activity, frontal Hγ activity to unpredictable deviants plateaus, indicating sustained activity. We tested posthoc whether frontal Hγ activity to unpredictable deviants persists longer than temporal Hγ (Fig. S4). Hence, we compared the length (time in milliseconds) of Hγ activity over baseline between the frontal and temporal ROIs across channels. We standardized frontal and temporal activity between zero and one.

With

$${\mathit{Y}}_{roi}={\mathit{X}}_{roi}-\text{min}\left({\mathit{X}}_{roi}\right),$$

we set Y_roi—denoting all time series of ECoG channels in a given ROI—above zero. The next step,

$${\mathit{Z}}_{roi}={\mathit{Y}}_{roi}\times \frac{1}{\text{max}\left({\mathit{Y}}_{roi}\right)},$$

compresses all values of Y_roi, such that the resulting matrix Z_roi ranges between zero and one. This standardization preserves the typical time course but enables a joint baseline distribution (100 ms preceding stimulus presentation). We estimated the upper confidence interval (CI; CI_97.5%) from the baseline distribution across all channels in both ROIs. For each channel, we determined the duration of activity over baseline (greater than CI_97.5%) in milliseconds. The durations were compared in a t test between ROIs across channels. On average, Hγ activity measured in frontal electrodes persists longer over baseline (M = 337 ms; SD = 176 ms) than temporal Hγ (M =148 ms; SD = 102 ms; t₁₅ = 2.85; P = 0.012).