A Switch and Wave of Neuronal Activity in the Cerebral Cortex During the First Second of Conscious Perception

Abstract

Conscious perception occurs within less than 1 s. To study events on this time scale we used direct electrical recordings from the human cerebral cortex during a conscious visual perception task. Faces were presented at individually titrated visual threshold for 9 subjects while measuring broadband 40–115 Hz gamma power in a total of 1621 intracranial electrodes widely distributed in both hemispheres. Surface maps and k-means clustering analysis showed initial activation of visual cortex for both perceived and non-perceived stimuli. However, only stimuli reported as perceived then elicited a forward-sweeping wave of activity throughout the cerebral cortex accompanied by large-scale network switching. Specifically, a monophasic wave of broadband gamma activation moves through bilateral association cortex at a rate of approximately 150 mm/s and eventually reenters visual cortex for perceived but not for non-perceived stimuli. Meanwhile, the default mode network and the initial visual cortex and higher association cortex networks are switched off for the duration of conscious stimulus processing. Based on these findings, we propose a new “switch-and-wave” model for the processing of consciously perceived stimuli. These findings are important for understanding normal conscious perception and may also shed light on its vulnerability to disruption by brain disorders.

Introduction

What are the fundamental changes in brain activity that accompany conscious experience? Recent work using measurements at different spatiotemporal scales has demonstrated diverse brain changes observed specifically for consciously perceived in contrast to non-perceived stimuli. For example, event-related potential, intracranial EEG, and magnetoencephalography studies variably suggest that conscious perception is associated with specific transient potentials, broadband gamma activity increases, or sustained cortical potentials (Del Cul et al. 2007; Wyart and Tallon-Baudry 2008; Gaillard et al. 2009; Li et al. 2014; Pitts et al. 2014) while functional neuroimaging suggests a role for widespread activation of the frontoparietal association cortex (Rees et al. 2002; Hesselmann 2013). However, a unified understanding of how these changes may contribute to conscious perception has not yet been achieved. Our goal was to study brain activity in widespread cortical networks as directly as possible and at high temporal resolution to move closer to such a unified understanding. We hypothesize that changes in brain activity during conscious perception may be summarized through two fundamental processes (Fig. 1). When signals entering the brain reach conscious perception we propose that this triggers a “switch” and “wave” in long-range neuronal activity. For example, in this model when signals enter the primary visual cortex and combine with top-down signals from higher cortices to reach conscious detection (Fig. 1a), this initiates a wave of processing through a sequential hierarchy of cortical networks including frontoparietal association cortex and medial temporal memory systems (Fig. 1b). In addition to triggering this wave, conscious detection meanwhile initiates several switches in long-range network activity. For example, activity in primary visual cortex and higher cortex may be turned off to prevent additional external inputs from mixing with the consciously perceived stimulus, and similarly the default mode network may be inhibited to temporarily prevent internal stimuli from entering the processing stream. At later times when the fully mature wave of sequential processing has completed, an additional switch may occur allowing information to reenter primary visual and higher cortices producing sustained, persistent activity for working memory and subsequent report (Fig. 1c). This unified model could explain a number of divergent findings in prior studies and predicts that cortical activity patterns during conscious perception and processing should fall into either monophasic propagating increases contributing to the wave, or polyphasic stationary increases/decreases contributing to long-range network switching.

Figure 1.

Switch and wave of neuronal activity in the first second of conscious perception. (a) Shortly after stimulus onset (0–200 ms), information enters primary visual cortex causing activation (red). Stimuli that are detected and later reported as Perceived also activate higher order association cortices. This triggers a polysynaptic “Switch” that inhibits the default mode network, inhibits the primary visual and higher order association cortex, and initiates a processing “Wave.” (b) During the processing phase (200–600 ms), a wave of activation propagates through hierarchical unimodal visual association cortices and higher-order heteromodal association cortices including the medial temporal lobe for memory encoding. The default mode network, primary visual and higher association cortex remain deactivated (blue). (c) During the late periods (~600+ ms) following processing, the visual cortex and higher association cortices reactivate as subjects prepare to report their perception. Deactivation continues in default mode network.

To investigate this hypothesis we sought techniques to obtain a comprehensive view of neuronal activity throughout the cerebral cortex at high temporal resolution. Recent work has shown that direct recordings from the cerebral cortex with electroencephalography or local field potential electrodes and analysis of overall broadband power changes especially in the gamma frequency range (typically 30–150 Hz) can serve as an excellent surrogate for the local population neuronal firing (Bechtereva and Abdullaev 2000; Mukamel et al. 2005; Manning et al. 2009; Miller 2010; Miller et al. 2013). Thus electrocorticography recordings from human subjects with electrodes directly in contact with brain tissue provide a unique opportunity to obtain high fidelity and high temporal resolution recordings of neuronal activity in the cerebral cortex. Although each electrode samples from a relatively local region within a few millimeters of the contact (Menon et al. 1996), we were interested in mapping activity throughout the cortical surface. To achieve this we combined data across hundreds of electrode contacts using a data-driven analysis approach based on clustering algorithms. This approach enabled us to visualize both the wave and switch components of neuronal activity in the cerebral cortex accompanying conscious perception.

Choice of an appropriate behavioral paradigm to study conscious perception depends on a number of considerations. A widely used paradigm which we adopt here is to compare brain activity for external stimuli that subjects report as perceived in contrast to stimuli that are not perceived. To produce similar stimuli while varying perception, some experiments have varied the duration (Pins and Ffytche 2003; Koivisto et al. 2016) or intensity (Wyart and Tallon-Baudry 2008; Auksztulewicz and Blankenburg 2013) of stimuli on a trial-by-trial basis, or introduce a mask in which another stimulus follows or precedes some targets to obscure their perception (Del Cul et al. 2007). To avoid potential confounds of variations in the stimulus itself or contributions of the mask to responses we used a threshold perception task (Pins and Ffytche 2003; Ress and Heeger 2003; Wyart and Tallon-Baudry 2008) with an identical face visual stimulus held at threshold perceptual levels adjusted for each individual. Practically, this method may be less widely used because the response to a threshold stimulus is relatively small and demands more numerous trials to differentiate. However, it has the advantage of keeping all aspects of the stimulus constant so that only conscious perception differs between trials. Another important consideration is the ability to cleanly separate early activity related to conscious perception of the stimulus from later activity elicited by questions or responses at the time of report. Although some progress has been made in no-report paradigms (Frassle et al. 2014; Pitts et al. 2014), for now subsequent report is still the most reliable way to evaluate neuronal responses to consciously perceived vs. not-perceived events on a trial-by-trial basis. It is crucial, however to allow enough time to pass between the stimulus and report, and to use methods with sufficient temporal resolution so that activity related to conscious perception can be readily distinguished from activity related to the later queries and motor response. To achieve this separation, we used relatively high temporal resolution electrophysiology measurements, introduced a delay between the stimulus and report, and only included data before the elicitation of report in the analysis.

Previous studies have relied on various types of brain imaging or physiological measurements. Functional MRI is able to produce whole-brain images showing contrasts in fronto-parietal association areas between perceived and not perceived stimuli (Dehaene et al. 2001; Rees 2007; Hulme et al. 2009) pointing to their involvement in perception. This is not surprising as the fronto-parietal association cortices have repeatedly been implicated in top-down modulation of visual processing and attention through work with humans (Barcelo et al. 2000; Gazzaley et al. 2007; Zanto et al. 2011; Gazzaley and Nobre 2012) and animals (Moore and Armstrong 2003). However, given the temporal resolution of fMRI on the order of several seconds, fMRI is hardly able to characterize the temporal progression of conscious processing that may be complete within several hundred milliseconds unless they are widespread and sustained, but may instead mainly reflect longer time-scale processes. Scalp electroencephalography (EEG) and magnetoencephalography (MEG) have a much higher temporal resolution on the order of milliseconds. However, these measurements are prone to extracranial movement artifacts, especially saccades and other myogenic noise that can be confounded with gamma band power at the time of cognitive events (Yuval-Greenberg et al. 2008; Yuval-Greenberg and Deouell 2009; Muthukumaraswamy 2013). Scalp EEG and MEG also have limited access to brain structures not directly adjacent to the surface, such as midline cingulate and frontoparietal cortex, medial temporal or orbitofrontal cortex, which may be intimately involved in cognition (Menon and Uddin 2010; Lovstad et al. 2012). Despite these limitations, masking studies using scalp EEG have shown late event-related potential (ERP) differences in occipital, central, and fronto-polar contacts between perceived and not-perceived stimuli, whereas early activity apparent in the occipital contacts does not vary with perception (Del Cul et al. 2007). These results suggest that early visual cortex activation around 100 ms poststimulus may be similar regardless of percept, but later activity beginning around 300 ms is present throughout the brain only when perception is present (Dehaene et al. 1998). While scalp EEG ERPs indicate the presence of changes in activity and ERP amplitude may be related to strength of response, they do not enable the distinction between increases and decreases in underlying neuronal activity. In summary prior work has not examined the important sequential increases and decreases in cortical activity at sufficient temporal resolution or spatial coverage to view the detailed changes that occur with conscious perception.

We recruited subjects each implanted with 100–300 subdural and depth electrodes to perform our threshold perception task comparing identical stimuli that were later reported as perceived or not perceived. Our analysis focused on broadband gamma power changes, which are more directly related to the local population neuronal spiking activity (Mukamel et al. 2005; Manning et al. 2009; Miller et al. 2013) rather than lower frequency or ERP signals which are less well-localized and more susceptible to the location of the reference electrode and potential remote generators. An interesting previous study on conscious perception using electrocorticography showed long-range network changes, but relied on masking paradigms and included fewer electrodes (Gaillard et al. 2009), and therefore could not view the progression of changes in broadband gamma activity across the cortical surface as we did with more comprehensive coverage and different analysis methods. To our knowledge this is the first time that the full spatiotemporal sequence of changes in neuronal activity has been mapped over the surface of the cerebral cortex in the first second of conscious perception.

Materials and Methods

Subjects and Electroencephalography (EEG) Recording Electrodes

Twelve right-handed adult subjects undergoing craniotomy with intracranial electrodes were recruited from the Yale Comprehensive Epilepsy Program. Three subjects were excluded from analysis due to inadequate data collection or poor behavioral performance, and the remaining 9 (4 male) had between 114 and 286 implanted subdural (grid or strip) and depth electrode contacts per subject (Fig. 3) (see also Supplemental Table S1; Supplemental Methods).

Figure 3.

Distribution of electrodes in the 9 study participants. (a) Surface representation of all electrodes included in the study. Each patient’s electrodes are coded by a different color. Lateral, medial, ventral, and posterior views are shown for each hemisphere. Contacts included grid, strip, and depth electrodes. For depth electrodes only the two contacts closest to the medial gray matter surface were used. (b) Voltage changes were represented over the surface of the cortex within 15 mm of each electrode center (see Methods for details). Color maps show number of subjects contributing to signals for each location on the brain surface. Total number of subjects = 9. Total number of electrode contacts across all subjects = 1621.

Behavioral Testing

Behavioral testing began at least 2 days after electrode implantation and continued daily as tolerated. The behavioral task tested conscious visual perception of face stimuli appearing in 1 of 4 visual field quadrants (Fig. 2). Subjects first underwent a calibration run to attain a stimulus that they would perceive 50% of the time. The task contained two different backgrounds: one of static white noise video with no audio, and one of the “Coral Seas” episode of the BBC series Blue Planet, which included audio as well as video. Calibration was done first with the noise background and then with the movie background, and subsequent trials alternated between runs with the movie and noise backgrounds, counterbalanced across individual testing days for each patient. The stimulus consisted of a single black and white picture of a face taken from a neutral face bank created by Natalie Ebner of the Berlin Max Plank Institute, originally from the CAL/PAL Face Database. This face was used for all trials and appeared for 50 ms in 1 of 4 quadrants in random order (Fig. 2a). The center of each face was 16.2 cm (radial displacement) from the central fixation cross or 10.8 degrees at 85 cm distance and the dimensions of the face were 3.7 × 4.6 degrees along the short and long axis of the oval. At the end of each calibration run, a detection curve was calculated based on detection accuracy at each opacity fitted with a sigmoidal cumulative Weibull distribution function.

Figure 2.

Behavioral task elicits validated report of perception. (a) Behavioral task for a single trial. A 6–10 s Prestimulus Interval with randomly jittered duration is followed by a 50 ms (3 frames at 60 Hz screen refresh rate) Face Presentation in 1 of 4 quadrants. The face is previously titrated per subject to 50% detection threshold. After either a 1 s or 15 s Poststimulus Delay, the subject is prompted to report their perception and the location of the stimulus in 2 forced-choice questions. The next trial immediately follows with a total of 32 trials per run (average of 10 min per run). Background during the face presentation and delay periods consists of either white noise (shown here) or a movie about undersea life. (b) Responses to perception question. In trials in which a face was present, 52% were reported as seen. However, when no face was present (12.5% of trials were blanks), only 9.6% of those trials were reported as seen (false positives). There was no difference in percentage reported as seen for the 1 s versus 15 s delay trials. Wilcoxon paired rank sum test *P < 0.05; NS, not significant. N = 9 subjects. Error bars are SEM. (c) When faces were reported as seen, subjects located the face in the correct quadrant 89% of the time. In contrast, subjects correctly located only 28% of the faces when they were reported as not seen. Outlined in red are the validated “confirmed perceived” and “confirmed not perceived” trials analyzed in this study, heretofore referred to simply as “Perceived” and “Not Perceived.” N = 9 subjects, with average of 391 face-present trials per subject.

Once a patient completed calibration, we performed testing runs each containing 28–32 face trials. Each run started with 10 s of fixation on a white cross with a solid gray background. Each trial would then start with a randomly jittered prestimulus interval of 6–10 s, after which a 50 ms face stimulus was presented in 1 of 4 quadrants on the screen (see Fig. 2a). The target stimulus was followed by either a 1 s or 15 s delay period, varied randomly. Then two forced-choice questions were presented on a solid gray background, first “Did you see a stimulus? If so, press the blue (first) button, if not, press the yellow (second) button” (with order of buttons for affirmative and negative responses counterbalanced across subjects), and second “Where was it located?” with large numbers 1–4 appearing in each quadrant (Fig. 2a). The subject was instructed to answer the location question truthfully if they detected a stimulus and to press a random button if they did not. They were also informed that some of the trials would be blank (no face), but were not informed that the blank rate was 12.5%. At the end of the second question, the movie or noise background would resume with the next prestimulus interval to start the next trial. Total duration of each run varied from 8 to 15 min (mean 10 min). At the end of each run, the 50% opacity was re-calculated based on the subject’s performance at that opacity in that run according to updated Weibull distribution parameters. See Supplemental Methods for additional details regarding behavioral testing.

EEG Recording, Epoch Extraction and Artifact Rejection

The EEG of each testing session was recorded on NATUS/Neuroworks software at 1024 Hz with a high-pass filter of 0.1 Hz and low-pass filter of 400 Hz. Signals were recorded relative to a reference electrode chosen by the clinical team to minimize visible noise artifact on the EEG.

The EEG of each testing session was pruned from the clinical recordings in NATUS/Neuroworks. The recording channel containing the TTL synchronization pulses was pattern-matched to template pulses to identify times at which faces, questions, button presses, and trial starts appeared, as they all had TTL pulses of differing durations. An epoch of 2 s (2048 samples) was then extracted centered on each face stimulus onset. The epochs from each session were then aggregated into categories such as validated “Perceived” or “Not Perceived” trials (see Fig. 2c) before artifact rejection and subsequent analysis.

The intracranial EEG from all sessions was reviewed by the clinical team and we did not include any EEG data containing clinical or subclinical epileptic events in the analysis. Each individual face stimulus trial epoch underwent conservative artifact rejection through 4 steps applied separately to each electrode, including criteria for 60 Hz noise, large amplitude deflections, flat lines from potentially unplugged electrodes, and large deviations from the mean signal of each electrode (see Supplemental Methods for details).

Time–Frequency Analysis

Time–frequency analysis was done on each non-rejected trial using the spectrogram function in MATLAB to perform a short-time Fourier transform. We used a 256-sample (250 ms) time window shifted by increments of 32 samples (31.25 ms) and displayed results using the center time point of each window. Although sliding windows decrease the temporal resolution of the power change time-series, the resulting smoothing effects enhance statistical data analysis (Simonoff 1996). To obtain an overview of changes we averaged the time–frequency results across Perceived and Not Perceived trials from all electrodes within each subject and then across all 9 subjects to produce a grand average time–frequency plot (Fig. 4). We used the frequency band 40–115 Hz to investigate changes in broadband gamma power due to both a consensus in the literature regarding the range of gamma power changes associated with behavioral tasks (Mukamel et al. 2005; Sederberg et al. 2007; Merker 2013; Miller et al. 2013; Sedley and Cunningham 2013) and because our analysis of spectral changes across subjects confirmed high-frequency increases from baseline within this range (Fig. 4). The gamma power time-series were next averaged across all frequencies in the 40–115 Hz range, then averaged across trials for each patient, and finally converted to z-score values at each time point using the mean and standard deviation of the 28 baseline (prestimulus) values for each electrode. See also Supplemental Methods for details.

Figure 4.

Overall view of changes through time–frequency analysis across all electrodes, subjects and trials. Both low and high frequency changes occurred during Perceived trials that were not present in Not Perceived trials. (a) 40–140 Hz power in Perceived trials. (b) 40–140 Hz power in Not Perceived trials. (c) 0.1–40 Hz power in Perceived trials. (d) 0.1–40 Hz power in Not Perceived trials. Marked broadband gamma frequency (40–140 Hz) power increases are seen in Perceived but not in the Not Perceived trials (a,b). In addition, delta range (0.1–4 Hz) power increased from baseline in both Perceived (c) and Not Perceived (d) trials, although with higher amplitude increases in Perceived trials. Perceived trials also had alpha (8–12 Hz) and beta power (12–30 Hz) increases and decreases that were absent in Not Perceived trials (c, d). Data for Perceived and Not Perceived trials were first averaged across electrodes within subjects and then results were averaged across all 9 subjects.

Mapping Gamma Power Z-scores on the Standard MNI Brain Surface

To produce composite brain images including all patient electrodes on one brain, each patient’s pre-op MRI was transformed into MNI space using BioImage Suite (http://bioimagesuite.yale.edu/). The electrodes that had been co-registered to the pre-op MRI space were then adjusted using the same transformation for each patient to bring the electrodes into MNI space. The electrodes for each patient had previously been co-registered with the pre-op MRI using post-op CT scans to identify electrodes, then co-registering the post-op CT to a post-op MRI, and finally co-registering the post-op MRI to the pre-op MRI. See Supplemental Methods for details regarding patient electrode projection onto the standard MNI brain.

To map gamma power z-scores at each time point to the standard MNI brain surface, we first assigned z-score values in spherical regions surrounding each electrode within individual subjects, and then performed a weighted average of these values across subjects. Because previous work has shown that intracranial electrode recordings have spatial resolution of 14–20 mm (Menon et al. 1996) we assigned values within a 15 mm radius of each electrode contact. Vertices within 1 mm of the electrode center were assigned the full z-score value for that electrode contact. Vertices at distances of 1–15 mm received a z-score decreasing linearly from the full value at 1 mm to a value of 0 at 15 mm. If a vertex fell within 15 mm of multiple electrode contacts, it received a value equal to the sum of the distance-scaled z-score values from those electrode contacts. We used the sum or “aggregate z-score” rather than the average to avoid artificially reducing the z-score values of active electrode contacts by low values that sometimes occurred in nearby electrode contacts with poor signals from the brain surface. To combine data across subjects, each vertex with z-score values from any subject was assigned the average of all z-score values at that vertex. This value was then weighted by multiplying by the square root of the number of subjects contributing data at that vertex (Fig. 3). This weighting was intended to represent the improved signal/noise ratio (which scales by square root of sample size) expected for vertices with data from more patients. Additional details for analyses presented in supplemental figures including propagation velocity calculations can be found in the Supplemental Methods.

Statistical Analysis

Statistical analysis was performed in MATLAB. Behavioral and electrophysiology data were evaluated by two-tailed Wilcoxon rank sum test without assuming a parametric-distribution of unmatched values, using a threshold P < 0.05 with Bonferroni correction. For electrophysiology data, we used k-means clustering to identify vertices on the cortical surface with similar timecourses based on Pearson’s correlation and to organize these vertices into anatomical clusters. We then calculated mean timecourses for all vertices within each cluster. For a given cluster, gamma power z-scores timecourses were considered significant if the following criteria were met: mean silhouette value >0; Wilcoxon rank sum test on the absolute values of all poststimulus vs prestimulus time points with P < 0.05, Bonferroni-corrected for the number of clusters. To provide additional insight into which individual time points showed the most significant changes we also compared each poststimulus value to the 28 prestimulus baseline values using a z-test, indicating individual times with P < 0.05 (Bonferroni-corrected for the 29 poststimulus times) by an asterisk.

Results

Behavioral Paradigm

The behavioral testing sequence for a single trial is shown in Figure 2a. Subjects’ perception level was on average at 52% (±2% SEM) when a face was present and 9.7% (±3% SEM) when there was no face present in a blank trial (Fig. 2b). When a trial contained a face and it was detected, the accuracy of locating the face was 89% (±3% SEM). However, if the face was not detected, the location accuracy dropped to 28% (±2%), or near chance levels (Fig. 2c). These results demonstrate that our subjects were appropriately engaged when performing the task and not randomly pressing buttons. Trials in which a face was present, detected, and correctly located can be described as “confirmed perceived” faces whereas trials in which a present face was not perceived and incorrectly located “confirmed not perceived.” Hereafter in this article, we will refer to these as simply “Perceived” and “Not Perceived” with the understanding that the perception in all trials analyzed is validated by location accuracy or inaccuracy.

We were interested in studying conscious perception with sufficient delay between the experience and report to ensure that responses were not simply automatic. Therefore, we tested perception after a 15 s delay in half of the trials. On the other hand, we also wanted to ensure that “not perceived” trials at 15 s were not simply faces that had been perceived but then forgotten. Therefore, we also included testing after a delay of 1 s in half of the trials. There was no significant difference in target detection between the 1 s and 15 s delays at 54% versus 49% respectively (Fig. 2b). We also found no significant difference in location accuracy between the 1 s and 15 s delays for both the perceived (P = 0.57) and not perceived trials (P > 0.99; not pictured). Therefore, because we were most interested in events during the first 1 s before any questions were asked we combined data from both trial types in all analyses below. The background view consisted of white noise in half of the trials and a movie about marine life in half of the trials. The movie was intended to more closely resemble events in real life but we included the trials with simple white noise in case the movie interfered with detection of neurophysiological events related to visual perception. There was no significant difference in behavioral detection rates between the movie and noise backgrounds (50% and 54%, respectively) and neurophysiological events were also similar in response to stimuli presented with either background (data not shown). Therefore, to increase sample size we combined these two trial types as well for all subsequent analyses.

Temporal Progression of Signals With Conscious Perception

To investigate signals from the brain during conscious events, we collected data from a large number of intracranial EEG electrodes providing broad sampling across most of the cerebral cortex in both hemispheres (Fig. 3). We performed time–frequency analysis across a wide spectrum of frequencies (0.1–140 Hz) and across all electrodes to obtain an initial overview of changes during perceived and not perceived trials (Fig. 4). This analysis revealed a broad increase in high frequency (>40 Hz) power in perceived compared to not perceived trials (Fig. 4a, b) which was in the range of cortical activity reported in previous literature for task-related changes. Broadband gamma power has previously been found to be closely linked to the overall rate of neuronal spiking in the population of neurons near the recording electrodes (Mukamel et al. 2005; Miller 2010; Miller et al. 2013). We also observed larger increases in delta frequency (0.1–4 Hz) and combined increases and decreases in alpha (8–12 Hz) and beta (12–30 Hz) activity in the perceived compared to not perceived trials (Fig. 4c, d). These lower frequency changes, which likely include well-known event related potentials and other lower-frequency changes associated with perception and postperceptual processing (Pitts et al. 2014), were not investigated further in the present study. We instead focused our investigation on the 40–115 Hz range, referred to as “gamma power” for the remainder of this article, to capitalize on the relatively high spatiotemporal resolution of these signals and more direct link to local neuronal activity.

To map the temporal progression of gamma power across the cortical surface we calculated the z-scores for changes relative to baseline in spherical regions around each electrode and then combined data across patients with a z-score display threshold of 2 (see Methods for details). Both Perceived and Not Perceived trials showed initial increases in the primary visual cortex immediately after stimulus presentation (Fig. 5). In the Not Perceived trials these signals diminished after about 250 ms and were followed by scant changes in other regions (Fig. 5b). In marked contrast, the Perceived trials showed an intriguing progression of signals over the cortical surface with time (Fig. 5a). The occipital increases shifted forward within 250 ms to travel to the ventral and lateral occipital and temporal-parietal visual association cortices. This was followed soon after by a progressive forward-sweeping wave of activity through the lateral frontal cortex, as well as activity reaching the medial temporal lobe parahippocampal and entorhinal cortical regions. In another marked departure from the Not Perceived trials, in the Perceived trials the primary visual cortex in the occipital lobe showed “decreases” within about 200 ms of stimulus onset with a nadir at about 375 ms, before showing late reactivations after about 625 ms. Interestingly, these transient decreases followed by late increases were echoed in several areas of higher-order association cortex including the superior parietal lobe, inferior frontal gyrus and orbital frontal cortex (see also further analyses below). Finally, several areas that overlap with the default mode network, such as the precuneus, ventral medial frontal, posterior parietal, and lateral temporal cortices showed progressive and sustained decreases beginning about 375 ms after stimulus onset. The full temporal progression of these changes are visualized in Supplementary Video 1. Subtraction analysis of Perceived minus Not Perceived gamma power maps confirmed that there were very few differences between Perceived and Not Perceived trials in the primary visual cortex at early times (see Supplementary Video 2). In addition, analysis of changes contralateral versus ipsilateral to the side of stimulus onset revealed that changes for the first 200–300 ms were somewhat asymmetrical, with larger changes seen in early visual association cortex areas contralateral to stimulus onset, however, changes for the remaining times were highly symmetrical in both hemispheres (see Supplementary Video 3). Because many of the changes were symmetric we also repeated the analysis combining results across the two hemispheres and found a similar overall sequence of changes to the results with each hemisphere analyzed separately (see Supplementary Video 4).

Figure 5.

Dynamic pattern of cortical broadband gamma activity in the first second after consciously perceived visual stimuli. Gamma power z-score maps combined across subjects. At 125 ms poststimulus, Perceived and Not Perceived trials appear similar, with both showing visual cortex increases. However, by 250 ms, in Perceived trials the visual cortex signal begins to decrease along with parallel decreases in several areas of higher association cortex, while increases spread to visual association areas. At that time point (250 ms), increases continue to be sustained in the visual cortex for Not Perceived trials, and then all but disappear by 375 ms poststimulus. In Perceived trials, the wave of increases persists along the dorsal and ventral visual streams while the default mode network shows sustained decreases. Finally, by 625–750 ms, the visual cortex and the higher order association cortex that displayed decreases now begin to show a second late increase in gamma power. Display threshold for positive and negative z-score values = 2. Number of subjects = 9; number of Perceived trials = 1552; number of Not Perceived trials = 1105. Medial view of left hemisphere and lateral view of right hemisphere only are shown here but similar changes were seen with other views as well (see Supplementary Video 1). Medial view is rotated by 30 degrees along the anterior-posterior axis to enable the ventral surface to be seen while the lateral view is rotated by 15 degrees.

Anatomical Location of Signals in Conscious Perception

To further identify the anatomical regions showing similar temporal profiles shortly after consciously perceived stimuli, we applied k-means correlational clustering to the time series of gamma power z-score values for all points on the cortical surface in the group data. This approach yielded 4 regions which remained relatively constant anatomically as the total number of clusters varied from 3 to 10 for Perceived stimuli (see also below). These 4 regions of high within-cluster temporal correlation were relatively symmetric between the two hemispheres, and revealed functional brain networks with distinct time series (Fig. 6; see also Supplementary Fig. S1 and S2). The first time series to show change was Cluster 1 (in green), which spans the primary visual cortex, superior parietal lobe, inferior lateral frontal cortex, orbitofrontal cortex and several additional smaller regions. Its time-series is characterized by an early increase, followed by a sharp decrease, followed by a second increase around 600 ms poststimulus (Fig. 6b). Cluster 2 (in blue) localizes to the areas showing sustained deactivation, such as the precuneus, ventral medial frontal, posterior parietal, and lateral temporal cortices. Clusters 3 (red) and 4 (cyan) show similar monophasic intermediate to long-latency increases and manifest in the visual association cortex (ventral temporo-occipital, lateral parieto-occipital), medial temporal, and lateral frontal and parietal association cortices. All 4 clusters for the Perceived stimuli had robust positive mean silhouette values indicating greater within-cluster than inter-cluster integrity (Fig. 6b). In contrast, when k-means clustering was performed with data from the not Perceived trials, none of the clusters had positive mean silhouette values and no coherent symmetric networks emerged (see Supplementary Fig. S3). When the anatomical regions from the Perceived data were applied to the Not Perceived data, the only region with a weakly positive mean silhouette value was the green Cluster 1 which included the primary visual cortex (Fig. 6c). In the Not Perceived trials, this region showed a transient increase lasting 200–300 ms after the stimulus, with the notable absence of both the large decrease and subsequent late increase seen in the Perceived trials (Fig. 6b, c).

Figure 6.

Anatomical regions and time courses of cortical changes with conscious perception revealed by k-means clustering. (a) Four regions obtained by k-means clustering of cortical areas with similar within-cluster gamma power z-score time courses for Perceived stimuli. These clusters are roughly symmetric and resemble functional networks previously identified in the brain. (b) Mean time courses for each cluster with Perceived stimuli. Colors correspond to same regions as in a. Stimulus onset was at Time = 0. Clusters were considered significant if they had positive mean silhouette values (mean s) indicating higher within-cluster than between-cluster temporal correlations, and if the Wilcoxon rank sum test for the poststimulus vs. prestimulus timecourse had Bonferroni-corrected P < 0.05. All 4 clusters met these criteria for the Perceived trials (b). Asterisks further indicate individual time points with the most significant changes relative to prestimulus baseline by z-test with Bonferroni-corrected P < 0.05. In (a), cluster 1 is localized to the visual and higher association cortices and shows the first small peak of increases, followed by decreases, and late secondary increases. Cluster 2 includes the default mode network and shows the next small peak but continues with sustained decreases. Clusters 3 and 4 include the visual association cortex, medial temporal lobes, and fronto-parietal association areas showing intermediate and late increases. (c) Mean time courses during the Not Perceived stimuli for same regions shown in (a). Only Cluster 1 (green), which included the visual cortex, exhibited a positive mean silhouette value and a significant poststimulus P value; individual time points with the most significant changes relative to baseline are therefore indicated by asterisks for this cluster. Same subjects and data as in Figure 5.

Because the decreases in visual cortex occurred after a delay of several hundred milliseconds, and also were accompanied by decreases in several regions of higher order association cortex, we thought that it was unlikely that these decreases were simply caused by surround inhibition (Bair et al. 2003). However, to further investigate this possibility, we analyzed the signals for each quadrant separately and found that electrodes found in the 4 quadrants of the visual cortex (left vs. right, above vs. below the calcarine fissure) respond equally to stimuli presented in all 4 quadrants. (Supplementary Fig. S4). This analysis showed similar time-courses for both increases and decreases in gamma power in all 4 quadrants further suggesting that the decreases are not due to surround inhibition.

After Cluster 1, the next region showing transient increases was Cluster 2 which overlapped with the default mode network (Fig. 6). However, in contrast with Cluster 1 the subsequent decreases in Cluster 2 were sustained and there was no late second increase. Clusters 3 and 4 next showed sequential monophasic activations of fronto-parietal association cortices and of the medial temporal lobes (Fig. 6). These clusters and especially Cluster 4 (Fig. 6a, b, in cyan) showed some overlap with the so-called “task-positive network,” whereas Cluster 2 (Fig. 6a, b, in blue) contains areas traditionally identified with the default mode network (Fox et al. 2005). To compare our results with prior work we created ROIs using coordinates from a separate previously published data set and found that the time courses for task positive regions resembled the monophasic changes in Clusters 3 and 4, and the time courses for default mode regions resembled the transient early increases and sustained decreases in Cluster 2 (Supplementary Fig. S5).

The anatomical regions defined by k-means clustering were robust to changes in the total number of clusters chosen for analysis. We repeated the analysis varying the number of clusters from 2 to 10 and found that beginning with 4 clusters, the anatomical regions for the clusters remained relatively constant (Fig. 7). With successive increases in the number of clusters these 4 regions became more anatomically focused but remained in the same general locations (Fig. 7). To identify corresponding clusters, we performed Pearson’s correlations between the time-series for each cluster and each time-series from the subsequent higher number of clusters (see Methods). We found that all time-series present at 4 clusters in Figure 6 correlated with one time-series from a subsequent cluster level with a Pearson’s correlation coefficient >0.95 and P < 0.0001 until 10 clusters (Supplementary Fig. S6).

Figure 7.

Anatomical regions are conserved across consecutively greater cluster levels. The boundaries of each corresponding cluster in a series are overlaid, with darker colors indicating greater spatial overlap across cluster levels. (a) Clusters with high temporal correlation to Cluster 1 from Figure 6. (b) Clusters with high temporal correlation to Cluster 2. (c) Clusters with high temporal correlation to Cluster 3. (d) Clusters with high temporal correlation to Cluster 4. Criteria for temporal correlations were Pearson’s correlation coefficient >0.95 and P < 0.0001. Regions shown are from k-means clustering with number of clusters ranging from 4 through 10. Data and subjects were the same as Perceived trials in Figure 5. See also Supplementary Figure S6.

Interestingly, at higher numbers of clusters the two networks showing monophasic increases (Clusters 3 and 4 from Fig. 6) are further divided into clusters peaking at successive latencies (Fig. 8). The anatomical areas corresponding to these clusters map spatially with the temporal succession. K-means clustering further resolves the marching of the “wave” of conscious processing from the visual association areas to the higher-order association areas. Beginning from the occipital cortex, a wave of increased broadband gamma power is apparent throughout the visual association cortex both ventrally and laterally. This sequence can also be traced in both lateral frontal lobes (Fig. 8). K-means clustering with 10 clusters analytically demonstrates the wave of activation related to conscious perception sweeping through the visual association cortex and the higher fronto-parietal-temporal association cortices. The propagation velocity of this wave was fairly consistent in lateral frontal, lateral posterior temporal-occipital as well as ventral temporal-occipital cortex at 149 mm/s (Supplementary Fig. S7). A view of all 10 k-means clusters can be seen in Supplementary Figure S8.

Figure 8.

A “wave” of visual conscious processing. At 10 clusters of Perceived gamma power z-score data, activation can be followed across both the dorsal and ventral visual processing streams. (a) The primary visual cortex/higher association cortex cluster is labeled as cluster 1, and the 4 intermediately activating clusters are labeled cluster 2–5, with the progression then returning to cluster 1 as it reaches the higher association cortex (arrows). (b) Time courses demonstrate sequential peaks in activity in clusters 1–2–3–4–5 and then back in cluster 1. Colors correspond with anatomical regions shown in (a). Clusters 3 and 4 in Figure 6 showing monophasic intermediate and late increases correspond approximately to clusters 2–3 and clusters 4–5 here, respectively. Data and subjects are the same as Perceived trials in Figures 5 and 6. See also Supplementary Figures S7 and S8.

It is interesting to note that gamma power z-scores did not return to 0 at the end of the nearly 1-s poststimulus epoch in the Perceived trials (Fig. 6). With 4 clusters, only Cluster 3 returned to baseline (Fig. 6b). Although sample size was reduced for time points after 1 s, repeat analysis on only the 15-s delay trials allowed us to obtain a preliminary analysis of later time points (>1 s). We found that that the late increases in the visual/higher association cortex cluster (Cluster 1) as well as the increases in the second fronto-parietal-temporal cluster (Cluster 4) were persistently increased for at least several seconds after the stimulus, while the default mode cluster (Cluster 2) continued to show sustained decreases (Supplementary Fig. S9).

Discussion

We have found that a rapid series of neural events can be traced through the brain in association with consciously perceived stimuli. In this experiment, we recruited patients with intracranial electrodes to perform a threshold visual perception task that employed target location to confirm perception. Using broadband gamma as a proxy for population neuronal spiking activity, we mapped the course of concerted network activations and deactivations across the brain during processing of conscious perception as well as during the lack of conscious perception. Examination of activity through time showed a stark contrast between dynamic activations, deactivations, and sustained reactivations during Perceived trials, compared to the gradual dissolution of activity from the visual cortex in confirmed Not Perceived trials within a few hundred milliseconds of stimulus onset.

Our findings build upon previous models of conscious access that propose a nonlinear “ignition” of widespread brain activity following conscious visual perception that is not found in unseen trials (Fisch et al. 2009) by showing with greater spatiotemporal resolution the exact process of those changes. We find visual cortex activation in relation to stimulus presentation that occurs regardless of conscious states, but we find activation as well as deactivation across many brain regions only during perceived trials. Previous “ignition” theories of conscious processing (Dehaene and Changeux 2011) have rarely discussed the importance of concerted network switching and deactivations. Our findings support a model in which consciously perceived stimuli elicit a specific sequence of neural events in the brain which can be summarized follows (Fig. 1): (1) A monophasic wave of neuronal activity propagates through successive hierarchical levels of association and medial temporal cortex, thought to be involved in processing and memory encoding; (2) long-range switching occurs with polyphasic changes in several networks, including primary and higher cortices as well as the default mode network, thought to be involved in signal detection, selection, and later preparation for report. In more detail, in the first 200 ms, detection of stimuli is associated with increased activity in visual cortex and higher association cortex (Fig. 1a). In the next 200 to ~600 ms of neural processing, a sequential wave of activity travels through the lateral frontal, lateral temporal-occipital, and ventral temporal-occipital association cortex as well as into the medial temporal memory systems at a rate of approximately 150 mm/s (Fig. 1b). Simultaneously activity levels in the visual cortex, higher order association cortex, and default mode network all decrease below baseline (Fig. 1b). Finally, there is a late secondary increase in activity in visual cortex and higher order association cortex (Fig. 1c). The neuroanatomical sequence of these events is well-placed to carry out many of the important functions necessary for conscious perception and postperceptual processing including stimulus detection, blocking additional inputs, hierarchical processing, memory encoding, and working memory/planning for subsequent report.

We observed that the spatial and temporal propagation of activity in the brain during the 1000 ms after a stimulus that is consciously perceived clusters into 4 symmetric functional networks. Even with increasing numbers up to 10 clusters designated for k-means clustering, these 4 clusters maintain highly conserved spatiotemporal form, suggesting that they are functionally robust to variations in analysis.

Cluster 1 (Fig. 6) includes the primary visual cortex in the occipital lobe along with regions of higher order association cortex such as the orbital frontal, lateral frontal, anterior temporal, and superior parietal association cortex. The gamma power z-score time-series for this cluster shows an early increase before 200 ms, followed by a decrease with a trough at around 350 ms poststimulus, and reactivating again after 600 ms. The triphasic nature of broadband gamma activity in this cluster is novel and several possible explanations exist for this pattern. The increase in primary visual cortex activity following stimulus presentation is found in both Perceived and Not Perceived trials, but only in Perceived trials does a sharp deactivation occur. The shape of this time series resembles that of second target detection performance during the attentional blink (Sergent et al. 2005), in which a second stimulus presented about 300 ms after a first target is unlikely to be consciously perceived. Prior theories have focused on working memory capacity as the bottleneck, and indeed they have shown the lack of a P3 ERP during the attentional blink as evidence of absent memory consolidation (Vogel and Luck 2002; Dux and Marois 2009). Yet our data suggest that the visual cortex itself may also be depressed at the time of the attentional blink which could prevent additional stimuli from entering the processing stream. In support of this mechanism, a different type of behavioral paradigm measuring visual discrimination has demonstrated a trough amplitude of the steady-state visual evoked potentials (SSVEP) over the occipital cortex at around 300 ms poststimulus, very similar in timing to the decreases we observed (Hindi Attar et al. 2010; Li et al. 2013). The magnitude of this trough in SSVEP amplitude was modulated by attentional salience in both studies. These prior studies, combined with our direct demonstration of reduced broadband gamma activity in visual cortex following consciously perceived stimuli suggests a new possible mechanism for the attentional blink. One can postulate that the tagging of salient visual information occurs within those critical first 200 ms or so poststimulus, and successful tagging precludes processing of subsequent targets during attentional blink due to the observed transient depressed cortical activity. The activity decrease we observed in the visual cortex is dissimilar from center-surround inhibition, which occurs in a matter of 1–30 ms after center receptive field activation and is only sustained for neurons representing areas near the center receptive field and transient for farther areas (Bair et al. 2003). We also repeated our own analysis after grouping trials by quadrant of target presentation, and found no differences in the latency of activation and deactivation in the visual cortex regardless of quadrant, further suggesting that surround inhibition does not explain the observed decreases. Another interesting new phenomenon that we observed in Cluster 1 was the late and sustained reactivation of the visual cortex and the higher association cortex regions at times greater than 600 ms after the stimulus. We can speculate that these late changes may be an aspect of postperceptual processing in which higher-order working memory systems access visual cortex to reflect on the memory of the target in preparation for imminent report. Studies have shown that both early and late visual cortex are activated during successful retrieval (Thakral et al. 2013). The ability of higher-order association cortices to modulate primary visual cortex activity during working memory tasks has been well studied (Barcelo et al. 2000; Premereur et al. 2013), as has the storage of short-term visual memory in visual areas (Agam et al. 2009). Our data could be further examined using causality analysis to find the driver of the late activation as possible sequelae of top-down processing back to the visual cortex or as intrinsic reactivation from the visual cortex itself following a period of suppression.

Cluster 2 (Fig. 6) includes many areas traditionally identified as default-mode network regions including the ventral medial frontal, posterior parietal, lateral temporal cortices, and the precuneus (Fox et al. 2005). This cluster’s time-series shows an initial activation peaking at 250 ms, followed by a long, slow deactivation that is sustained up to the end of the measured epoch poststimulus. Even though our subjects are presumably already in a vigilant state while awaiting possible stimulus onset during the trial period, stimulus detection may initiate further default mode network deactivation in order to facilitate processing of external stimuli. The default mode network is primarily shown to be active during periods of internal reflection of memories or future planning (Smallwood et al. 2012) and is deactivated during the onset of task blocks observed through both fMRI (Raichle and Snyder 2007) and electrophysiology (Miller et al. 2009). Furthermore, default mode network activity is inversely related to the working memory demands of an external task (Singh and Fawcett 2008; Ossandón et al. 2011) and performance on the task (Weissman et al. 2006; Li et al. 2007). Further analysis of the 15-s delay trials shows the fate of default mode network following target detection is sustained deactivation extending to 2.5 s poststimulus.

Clusters 3 and 4 show monophasic activation peaks at different latencies involving unimodal and heteromodal association cortex as well as the medial temporal lobe. The association cortical areas and medial temporal lobe regions in these clusters are similar to regions found in a previous intracranial EEG study of episodic memory encoding (Burke et al. 2014) and also resemble the forward-sweeping wave of processing in a prior MEG study of novel stimuli (Marinkovic et al. 2003). Cluster 3 peaks around 400 ms after the stimulus and returns to baseline by the end of the 1 s epoch, whereas Cluster 4 peaks around 600 ms poststimulus and its activity is sustained up to 2500 ms. Areas within these clusters are possibly involved in maintaining working memory and online processing of the stimulus, and overlap with the “task-positive network” identified through previous fMRI studies (Fox et al. 2005; Dosenbach et al. 2006). We found that the splitting of these two clusters into several intermediately peaking clusters marks the sequential “wave” of activation across the cortex as the stimulus is processed in preparation for subsequent conscious report. The propagation velocity of this wave was approximately 150 mm/s, much slower than theoretically attainable propagation speeds of 3500 mm/s based on axonal conduction velocities alone (Girard et al. 2001). To our knowledge this is the first direct measurement of the propagation velocity of sequential cortical processing in humans, although the value of approximately 150 mm/s is consistent with the expected speed based on human behavioral studies, as well as on measurements of cortical signal propagation in non-human primates (González-Burgos et al. 2000; Sato et al. 2012; Muller et al. 2014).

The current behavioral paradigm limits interpretation of our results in several ways. For one, the categorization of trials into Perceived or Not Perceived requires overt report by the subject. Report of perception may involve postperceptual processing unnecessary for perception itself, but unresolvable from the necessary aspects using our current paradigm. Previous studies that attempt to dissociate processing unnecessary for conscious perception have not isolated conscious processing with absolute success. A future experiment using intrinsic covert measures of conscious perception could exclude postperceptual processing unnecessary for conscious perception. For example, previous studies have shown that pupil diameter increases with arousal during target detection, and could be used as a marker of covert report (Hakerem and Sutton 1966; Kang and Wheatley 2015). Preliminary measurements show selective pupil dilation with the same perceptual task used here (Ding et al. 2017). In addition, simultaneous high-resolution eye tracking did not show saccadic eye movements after target stimuli, allaying potential concerns that saccades could contribute to intracranial signals with the present paradigm (Ding et al., 2017 unpublished observations). Our behavioral results showed a general lack of blindsight with location accuracy at 28% when the face was not perceived only slightly above chance levels of 25%. It would be interesting to use a paradigm in which a larger proportion of trials show blindsight in order to determine whether some of the changes we observed occur even when stimuli are not consciously perceived but can still be accurately located in space. Similarly, because our current behavioral paradigm contained only a small proportion of blank trials (12.5%) we could not investigate the interesting question of whether or not false positive trials might have similar patterns of early or late activation to those trials with an actual target, as has been studied previously with other methods (Ress and Heeger 2003). Finally, our paradigm does not allow investigation of the intriguing phenomenon of differential decay of attended versus unattended features of stimuli, as previously reported in many masking and iconic memory studies (Sperling 1960; Greenwald et al. 1996; Rossetti 1998; Naccache et al. 2002). Because subjects in our study attended to both the presence and location of stimuli, they likely performed both the perception and location tasks shortly after target onset and then stored their answers in working memory until they were questioned afterwards regardless of the delay.

In summary, we mapped the temporal dynamics immediately following consciously perceived events using intracranial EEG recordings of broadband gamma activity in the human brain. We observed early visual cortical increases both for events that were subsequently reported and those that were not. However, only events that were consciously perceived elicited a logical sequence of dynamic concerted network increases and decreases involving a switch and wave of activity in the visual cortex, multilevel association cortex, medial temporal memory systems and the default mode network. This initial picture will hopefully form the basis for future more detailed investigations of the neuroanatomy and neurophysiology of conscious events occurring over time.

Funding

This work was supported by the National Institutes of Health (NIH R01NS055829, UL1TR000142, T32NS00722429, T32GM00720538); the Betsy and Jonathan Blattmachr Family; the Loughridge-Williams Foundation; and a Gustavus and Louise Pfeiffer Research Foundation research fellowship.