Dissociating the Neural Correlates of Consciousness and Task Relevance in Face Perception Using Simultaneous EEG-fMRI

Abstract

Current theories of visual consciousness disagree about whether it emerges during early stages of processing in sensory brain regions or later when a widespread frontoparietal network becomes involved. Moreover, disentangling conscious perception from task-related postperceptual processes (e.g., report) and integrating results across different neuroscientific methods remain ongoing challenges. The present study addressed these problems using simultaneous EEG-fMRI and a specific inattentional blindness paradigm with three physically identical phases in female and male human participants. In phase 1, participants performed a distractor task during which line drawings of faces and control stimuli were presented centrally. While some participants spontaneously noticed the faces in phase 1, others remained inattentionally blind. In phase 2, all participants were made aware of the task-irrelevant faces but continued the distractor task. In phase 3, the faces became task-relevant. Bayesian analysis of brain responses demonstrated that conscious face perception was most strongly associated with activation in fusiform gyrus (fMRI) as well as the N170 and visual awareness negativity (EEG). Smaller awareness effects were revealed in the occipital and prefrontal cortex (fMRI). Task-relevant face processing, on the other hand, led to strong, extensive activation of occipitotemporal, frontoparietal, and attentional networks (fMRI). In EEG, it enhanced early negativities and elicited a pronounced P3b component. Overall, we provide evidence that conscious visual perception is linked with early processing in stimulus-specific sensory brain areas but may additionally involve prefrontal cortex. In contrast, the strong activation of widespread brain networks and the P3b are more likely associated with task-related processes.

SIGNIFICANCE STATEMENT How does our brain generate visual consciousness—the subjective experience of what it is like to see, for example, a face? To date, it is hotly debated whether it emerges early in sensory brain regions or later when a widespread frontoparietal network is activated. Here, we use simultaneous fMRI and EEG for high spatial and temporal resolution and demonstrate that conscious face perception is predominantly linked to early and occipitotemporal processes, but also prefrontal activity. Task-related processes (e.g., decision-making), on the other hand, elicit brain-wide activations including late and strong frontoparietal activity. These findings challenge numerous previous studies and highlight the importance of investigating the neural correlates of consciousness in the absence of task relevance.

Introduction

Recent years have seen considerable progress in the search for the neural correlates of consciousness (NCCs)—“the minimum neural mechanisms jointly sufficient for any one specific conscious experience” (e.g., seeing a face; Koch et al., 2016). However, dominant theories of conscious perception strongly disagree about when and where these NCCs occur in the brain.

Some theories of visual consciousness, most notably the recurrent processing theory (RPT; Lamme, 2003, 2006, 2010), suggest that it emerges in occipitotemporal cortical areas at early stages of processing (Koch et al., 2016; Boly et al., 2017). Accordingly, these accounts propose an enhanced negative event-related potential (ERP) at ∼200 ms after stimulus onset, the “visual awareness negativity” (VAN), as an electrophysiological NCC candidate (Koivisto and Revonsuo, 2010; Railo et al., 2011; Förster et al., 2020).

Other theories, most prominently the global neuronal workspace theory (GNWT; Dehaene et al., 1998; Dehaene, 2014; Mashour et al., 2020), argue that consciousness occurs only later when frontoparietal regions become involved in widespread sharing of perceptual information across the brain (Sergent et al., 2005; van Vugt et al., 2018). Concerning ERPs, the GNWT posits a close link between a late positive potential after ∼300 ms, the “P3b” (Polich, 2007), and conscious perception (Sergent et al., 2005; Mashour et al., 2020). Similarly, some higher-order theories of consciousness (Lau and Rosenthal, 2011) argue against first-order theories such as RPT and advocate a key role for the prefrontal cortex (PFC) in consciousness (Brown et al., 2019).

Critically, most previous research on NCCs has systematically confounded consciousness with processes that precede or follow the experience, for example, introspection, self-monitoring, and reporting (Aru et al., 2012). Therefore, the necessity of suited paradigms without immediate trial-by-trial responses has been emphasized (Tsuchiya et al., 2015). In monkeys, conscious perception without report could be decoded from prefrontal (Panagiotaropoulos et al., 2012; Kapoor et al., 2020) and inferotemporal neurons (Hesse and Tsao, 2020), but evidence from humans is scarce.

In pioneering electroencephalography (EEG) experiments, Pitts et al. (2012, 2014, 2018) capitalized on the phenomenon of inattentional blindness (IB), that is, the suppression of conscious awareness of an unexpected item because of attentional engagement in another task (for review, see Hutchinson, 2019). Using various stimuli, they demonstrated that the VAN and early stimulus-specific responses (e.g., the N170 for faces; Rossion, 2014) covaried with conscious perception, while the P3b was only elicited by task-relevant stimuli (Pitts et al., 2012; Shafto and Pitts, 2015; Schelonka et al., 2017; Schlossmacher et al., 2020; see also Koivisto et al., 2016; Cohen et al., 2020; Schlossmacher et al., 2021; Sergent et al., 2021), which clearly contradict theories postulating the P3b as an NCC (Mashour et al., 2020).

Despite its superior spatial resolution, functional magnetic resonance imaging (fMRI) has rarely been used to isolate content-specific NCCs in the absence of task-related processes, as recent reviews show (Koch et al., 2016; Boly et al., 2017; for an auditory study, see Wiegand et al., 2018). One pioneering fMRI study on binocular rivalry demonstrated that previously reported frontal activity (Lumer and Rees, 1999) was likely related to introspection (Frässle et al., 2014; see also Brascamp et al., 2015). However, to the best of our knowledge, there is no study relating electrophysiological and hemodynamic candidate markers of conscious perception in humans using a no-report or delayed-report design. Combining the advantages of high temporal (EEG) and spatial (fMRI) resolution may provide unique insights directly relevant to current theoretical debates.

Thus, in the present study, we used simultaneous EEG-fMRI to test the conflicting predictions of the predominant theories of consciousness concerning the role of early sensory versus late frontoparietal activity in conscious face perception in both the absence and presence of task relevance. Thereby, we aimed to better understand (1) the exact spatial pattern of brain activation related to consciousness when controlling for task-related effects and (2) the relation between hemodynamic and electrophysiological signatures of conscious perception.

Materials and Methods

Participants

The sample consisted of 75 participants recruited from the local student community of the University of Münster via public advertisements. All had normal or corrected-to-normal visual acuity and no history of psychiatric or neurologic illness. The sample size was chosen to be at least twice as large as in the original EEG study (Shafto and Pitts, 2015) based on the expectation that the signal-to-noise ratio of ERPs in the MRI environment would be lower. Participants provided written informed consent before the experiment and received monetary compensation. All procedures were approved by the ethics committee of the Medical Faculty of the University of Münster and were conducted in accordance with the Declaration of Helsinki (sixth revision). For the final sample, only participants with sufficient behavioral and EEG/fMRI data were included (see Results), resulting in 52 participants (32 female) with a mean age of 24.1 years (SD = 4.1) and a range of 19–34 years. This permitted the comparison of EEG and fMRI analyses based on identical stimulation, environment, participants, and reported conscious experience.

Experimental design and statistical analysis

Stimuli and presentation.

Stimulus presentation (Fig. 1) was based on the study by Shafto and Pitts (2015), adapted to fMRI and implemented in Presentation 17.2 (Neurobehavioral Systems). The image was projected onto a semitransparent screen positioned at the head end of the scanner with a resolution of 1024 × 768 pixels at 75 Hz. Participants viewed the screen in a mirror attached to the head coil so that the stimulus size corresponded to 8.3° of visual angle horizontally and vertically.

Figure 1.

Stimulus presentation and tasks. In all three experimental phases, each trial began with random lines presented in the background, followed by a critical stimulus (a schematic face or random lines). ERPs and BOLD responses were time locked to these critical stimuli. In the foreground, green dots continuously rotated around three concentric rings. For the distractor dot task in phases 1 and 2, participants were asked to respond whenever one of the dots briefly illuminated. For the face task in phase 3, participants were instructed to respond whenever they detected a face with a missing feature (e.g., an eye). Adapted from Shafto and Pitts (2015).

The stimulus material is described in more detail in the study by Shafto and Pitts (2015). In short, the face stimulus consisted of 36 short line segments based on an average female face. Additional (pseudo)randomly oriented and positioned copies of these line segments were distributed around the central face. In the control stimuli (hereinafter referred to as “random”), all line segments were (pseudo)randomly oriented and positioned. The background was white.

As it was necessary for the fMRI adaptation of the paradigm, the total number of stimuli was reduced from 720 to 240 per phase (120 faces, 120 random), maintaining a 50:50 ratio. To retain statistical power, only 10% instead of 20% of the face stimuli were missing a feature (e.g., an eye), which defined such faces as targets for the task-relevant condition described below. All EEG and fMRI analyses were based on intact faces and random control stimuli only. ERPs and hemodynamic responses were time locked to the onset of these critical stimuli.

Each stimulus was presented for 293 ms. The interstimulus intervals (ISIs), during which additional random line configurations were presented, lasted between 0.79 and 26 s [truncated exponential distribution; mean (M) = 3.94, SD = 3.24], as optimized for estimation efficiency in rapid event-related fMRI. The same 240 stimuli and ISI configurations were repeated (in randomized order) in each of the three phases of the experiment.

The distractor stimuli superimposing the critical face stimuli are also described in the study by Shafto and Pitts (2015). In short, 12 small green dots rotated on three concentric green rings (4 on each) around a fixation cross. Occasionally, one of them was illuminated, serving as a target for the distractor “dot task.” The number of dot targets matched the number of degraded-face targets. At all times, participants were asked to keep their gaze at the fixation cross. All trials with dot or degraded-face targets were excluded from EEG and fMRI analyses.

Procedure.

Participants went through three experimental phases with identical stimuli but two different tasks (Table 1). Each phase lasted 12–15 min including a break after half of the trials (one run). In phases 1 and 2, participants performed the distractor dot task, pressing a response box button with their right index finger whenever one of the rotating green dots briefly illuminated. This distractor task required distributed attention across the display. Before phase 1, but already in the scanner with the EEG cap attached, participants familiarized themselves with the setting and the dot task in three practice blocks without any faces presented. Immediately after phase 1, subjects completed an unheralded awareness assessment regarding the faces. This awareness assessment allowed us to divide subjects into an inattentionally “blind” and a spontaneously “aware” group for later analyses. Additionally, this assessment served the function of alerting subjects about the presence of faces in the background to make all subjects aware of the faces in phase 2. Importantly, the initial question was open-ended, asking whether participants had noticed “any coherent patterns” in the background. If they described faces, they were asked when they first noticed them (immediately, first run, or second run) and how many faces they had seen. In phase 2, irrespective of their answers, participants were asked to keep performing the same dot task. Afterward, the awareness assessment was repeated. For the final phase 3, participants were instructed to perform the face task, that is, to ignore the rotating dots, attend to the faces, and respond whenever they detected a face with a missing feature (e.g., an eye). Last, subjects were informed about the purpose of the study.

Table 1.

Overview of the experimental procedure

Phase	Task	Awareness of faces	Task relevance of faces
1	Dot task	No/yes	No
Awareness assessment
2	Dot task	Yes	No
Awareness assessment
3	Face task	Yes	Yes

Behavioral data analysis.

Task performance was quantified by d′ (Macmillan and Creelman, 2005) and response times (RTs) for each subject and phase, and was analyzed in the statistics program R (R Core Team, 2021). Because of the continuous stimulus presentation, false alarms were evaluated relative to the number of nontarget time intervals of the same length as the 2 s response interval for hits (Bendixen and Andersen, 2013). To summarize single-trial (ST) RTs while protecting against the influence of skewness and outliers, the median was used (Wilcox and Rousselet, 2018; but see also Rousselet and Wilcox, 2020). To investigate whether neural differences in the critical awareness contrast could also be attributed to differences in attentional allocation, d′ and median RTs were compared between blind and aware subjects in phase 1, using Welch's t test to account for heteroscedasticity (Welch, 1947; Wilcox and Rousselet, 2018). With regard to the task relevance contrast, a paired t test compared all subjects between phases 2 and 3. Both contrasts were two-sided. In line with recent recommendations, we report both classical and Bayesian inference as well as effect sizes (Wetzels et al., 2011; Keysers et al., 2020). Classical tests used the conventional a priori threshold of p < 0.05 (for a critical discussion, see Lakens et al., 2018). Bayes factors (BFs) quantify how much more likely one hypothesis is compared with another, with BF₀₁ denoting the evidence for the null hypothesis and BF₁₀ the evidence for the alternative hypothesis. A point-null prior (i.e., standardized effect size δ = 0) and a default Jeffrey–Zellner–Siow prior were used, respectively (Morey and Rouder, 2011; Morey et al., 2018). BFs are interpreted based on the conventions by Jeffreys (1961). Effect sizes are reported as Cohen's d (Cohen, 1988).

fMRI data acquisition.

MRI data were collected using a 3 tesla Siemens Magnetom Prisma and a 20-channel Siemens Head Matrix Coil (Siemens Medical Systems). First, a high-resolution T1-weighted scan with 192 slices was recorded for anatomic localization with a repetition time (TR) of 2130 ms, an echo time (TE) of 2.28 ms, a flip angle (FA) of 8°, and a voxel size of 1.0 × 1.0 × 1.0 mm within a field of view (FOV) of 256 × 256 mm. Subsequently, six functional datasets per subject (three phases × two runs) were acquired with a T2*-weighted echoplanar sequence sensitive to blood oxygenation level-dependent (BOLD) contrast (TR = 2080 ms, TE = 30 ms, FA = 90°, FOV = 208 × 208 mm, voxel size = 2.3 × 2.3 × 3.0 mm). They consisted of 288 volumes with 36 axial slices. A shimming field was applied before functional imaging to minimize magnetic field inhomogeneity.

fMRI data preprocessing.

All preprocessing steps were conducted using SPM12 version 7771 (The Wellcome Centre for Human Neuroimaging, UCL Queen Square Institute of Neurology, London, UK; https://www.fil.ion.ucl.ac.uk/spm/software/spm12/) and the Data Processing & Analysis of Brain Imaging (DPABI) 4.3 toolbox (Yan et al., 2016) in MATLAB 9.3 (MathWorks). The first five data volumes were discarded because of spin saturation effects. The remaining volumes were slice time corrected and realigned using a six-parameter (rigid body) linear transformation. The anatomic and functional images were coregistered and then segmented into gray matter, white matter, and CSF. The data were then nonlinearly spatially normalized to Montreal Neurological Institute (MNI) standard space with DARTEL (Ashburner, 2007), resampled to 2 mm isotropic voxels, and spatially smoothed with an 8 mm full-width at half-maximum Gaussian kernel. Finally, a gray matter mask was generated using the automatic optimal-threshold function in the SPM Masking Toolbox (Ridgway et al., 2009).

fMRI data analysis.

For first-level analysis, a general linear model (GLM) was estimated for each subject. A high-pass filter with a cutoff of 128 s was applied to remove slow signal drifts. Autocorrelations were modeled using the SPM prewhitening method FAST (Corbin et al., 2018), as recommended by Olszowy et al. (2019). The GLM design matrix included the presentation of faces, random lines, degraded faces, and dot targets as predictors. Six head movement parameters were defined as predictors of no interest. These protocols yielded onsets for reference functions that were convolved with a 2-gamma hemodynamic response function to model the BOLD signal change for each predictor. Contrast images (face − random) of the β-estimates were created for each participant in each phase. To increase sensitivity in voxels where some subjects had missing values (e.g., because of bounding box limitations and variable head size), multiple imputation by chained equations (van Buuren and Groothuis-Oudshoorn, 2011) for fMRI data was used as recommended by Vaden et al. (2012).

All second-level analyses reported here rely on Bayesian inference as implemented in SPM12 (Friston et al., 2002a,b) because of the following reasons. First, classical statistical parametric maps (SPMs) are insensitive in large search volumes (Friston and Penny, 2003). Because of the strong divergence of current theories of consciousness, the present study is concerned a priori with very extensive regions of the brain, most importantly the occipital cortex (Lamme, 2003, 2006, 2010; Koch et al., 2016) and a widespread frontoparietal network (Dehaene et al., 2006; Dehaene and Changeux, 2011; Lau and Rosenthal, 2011; Mashour et al., 2020). Additionally, regions of interest comprise face-sensitive (Haxby et al., 2000; Wang et al., 2016) as well as dorsal and ventral attention networks (Corbetta and Shulman, 2002; Corbetta et al., 2008). This large search volume would induce a massive multiple-comparison problem using classical inference, which Bayesian inference avoids altogether: the probability that any particular voxel is activated, given the data, is the same regardless of whether that voxel or the entire brain is analyzed (Friston and Penny, 2003). Thus, Bayesian compared with classical inference ensures more comparable sensitivity for all NCC candidates independently of the size of the underlying structure. Furthermore, Bayesian inference allows for evidence for null effects and is therefore suited to argue against the role of a given brain region in conscious perception.

Instead of classical SPMs, the results of Bayesian inference in the SPM software are posterior probability maps (PPMs), which are images of the probability or confidence that the contrast exceeds a specified threshold (e.g., zero), given the data (Friston and Penny, 2003). These probabilities are represented as logarithmic BFs (logBFs), with values > 5 indicating a probability of > 99% and thus “very strong evidence” (Han and Park, 2018) that the effect is > 0. Classical SPMs were computed additionally.

To directly test the neural correlates of consciousness and task relevance in face processing, two a priori defined contrasts were calculated using the difference maps (face − random) from first-level analysis. First and most importantly, the awareness contrast compared neural responses between aware and blind subjects in phase 1 in the absence of task relevance (independent test). Activations exceeding a logBF of 5 (99% probability) were used to extract and plot β-estimates. Since, a priori, any reliable NCC candidate must be present in all aware conditions, only those cortical clusters are reported in which post hoc analyses confirmed an average difference β > 0 (logBF > 5) in aware subjects in phase 1 and in all subjects in phases 2 and 3. Second, the task relevance contrast compared differential responses (face − random) between phase 3 (aware, task-relevant) and phase 2 (aware, task-irrelevant) in all subjects (paired test). All planned tests were unidirectional. In case of very large clusters extending over different brain areas, β-values were extracted from gray matter voxels with logBF > 5 in 8 mm spheres around local peaks of interest. Effect sizes as Cohen's d (Cohen, 1988) were computed using the averaged β-values. Finally, the SPM function spm_bms_null was used to map evidence for null in the awareness and task relevance contrast. All brain regions were identified based on the human Brainnetome atlas (Fan et al., 2016).

EEG data acquisition.

Concurrent EEG data were recorded from a 32-channel MR-compatible EEG cap (BrainCap MR, Brain Products) whose scalp electrodes, including reference and ground, were distributed according to the extended international 10–20 system. An additional electrode at the subjects' back monitored their electrocardiac activity. Electrode gel (Abralyt HiCl, EasyCap) was applied to electrode–scalp junctions until impedances fell to < 10 kΩ. A BrainAmp MR amplifier (Brain Products) was fixed inside the scanner bore and connected to a computer in the console room via a fiber-optic cable. EEG was recorded with a sampling rate of 5000 Hz and an online bandpass filter of 0.016–250 Hz. A SyncBox (Brain Products) was used to synchronize the scanner gradient and EEG acquisition system clocks. The exact timing of the stimulus presentation was measured using a photodiode in the scanner room connected to a BrainAmp ExG MR amplifier (Brain Products).

EEG data preprocessing.

The preprocessing of the raw EEG data began with the removal of scanner gradient artifacts using a moving average subtraction method (Allen et al., 2000) and the correction of cardioballistic artifacts in BrainVision Analyzer 2.1 (Brain Products). Afterward, data were downsampled to 250 Hz and voltages were rereferenced to the common average. Exceedingly noisy channels were interpolated. Eye movements and other artifacts were corrected using a dipole source-modeling procedure (Berg and Scherg, 1994; Ille et al., 2002) in Brain Electrical Source Analysis (BESA) Research 6.0 (BESA). The continuous signal was segmented into epochs of 300 ms before to 800 ms after the onset of the critical background stimuli, whose exact timing was determined from the photodiode signal. Further processing was performed using the FieldTrip toolbox (Oostenveld et al., 2011) in MATLAB. Trials were baseline corrected based on the average of the prestimulus interval from −100 to 0 ms. Slow drifts were removed using a second-order Butterworth high-pass filter at 0.1 Hz, which was applied in forward direction to avoid phase distortions (Rousselet, 2012). Low-pass filtering used a fourth-order, two-pass Butterworth filter at 30 Hz for statistical analyses and at 15 Hz for visualization purposes (Shafto and Pitts, 2015). Trials containing muscle artifacts, electrode jumps, or amplitudes exceeding a threshold of 100 µV were rejected based on visual inspection.

EEG data analysis.

Electrodes and intervals of interest (IOIs) for ERP analyses were based on the literature and data orthogonal to the statistical comparisons (grand averages across all subjects and phases). The N170 was analyzed in electrodes P7 and P8 (Bentin et al., 1996; Rossion and Jacques, 2012; Navajas et al., 2013; Rossion, 2014) between 160 and 230 ms, accounting for the delayed peak latency for line drawings of faces compared with natural face photographs (Sagiv and Bentin, 2001; Shafto and Pitts, 2015). Please note that electrodes T5/T6 correspond to P7/P8, respectively, and PO7/PO8 are not available in the cap used. The VAN was assessed in electrodes O1, O2, P7, and P8 between 210 and 270 ms (Koivisto and Revonsuo, 2010; Railo et al., 2011; Förster et al., 2020), and the P3b in electrode Pz between 400 and 600 ms (Polich, 2007; Shafto and Pitts, 2015). ERPs were analyzed analogously to BOLD responses. Difference waves between responses to faces and random control stimuli were computed within each phase to control for effects of repeated stimulus exposure. Difference amplitudes were averaged within the IOIs in the selected electrodes and were analyzed using R (R Core Team, 2021).

The same a priori defined contrasts for effects of awareness and task relevance as in the fMRI analyses were calculated using both classical and Bayesian inference (Keysers et al., 2020; for details on the Bayesian tests, see Behavioral data analysis section). In short, the independent awareness contrast compared responses between blind and aware subjects in phase 1 (Welch, 1947; Wilcox and Rousselet, 2018). To probe the NCC eligibility of awareness-related ERPs further, post hoc analyses tested their difference amplitudes in aware conditions against zero. The task relevance contrast compared differential amplitudes between phases 3 and 2 in all subjects. All planned tests were unidirectional depending on the respective ERP (more negative for the N170 and VAN, and more positive for the P3b) and used α < 0.05 and the BF conventions of Jeffreys (1961). ERPs and topographies were plotted in MATLAB and FieldTrip (Oostenveld et al., 2011).

Following these main analyses, supplementary exploratory analyses were performed. The first aimed at detecting potential additional awareness effects outside of our a priori defined sensors and intervals of interest. To this goal, we used a nonparametric cluster-based permutation (CBP) test (Maris and Oostenveld, 2007; Groppe et al., 2011) in FieldTrip (Oostenveld et al., 2011) with both a sample-specific and overall significance threshold of α = 0.05 and 5000 permutations. In a first step, the awareness contrast was computed using a two-sided CBP test across all electrodes and time points (0–600 ms). Second, to enhance power and to ensure that no relevant sensors were missed for the ERPs of interest, one-sided (i.e., polarity-dependent) CBP tests were performed separately in each IOI (N170, VAN, P3b) in all electrodes posterior to the midline. The same analyses were repeated in all anterior electrodes to address potential frontal effects (e.g., for the N170; Babiloni et al., 2010).

The second exploratory analysis concerned the finding that a decrease in average N170 amplitude can result from a decreased ST timing consistency because of stimulus uncertainty (Navajas et al., 2013). In our case, blind subjects may have exhibited a greater ST N170 latency jitter (LJ) than aware subjects either across trials within subjects (LJ_WS) or between subjects (LJ_BS), potentially causing the average amplitudes to decrease. Thus, to obtain reliable ST latencies, we performed automatic wavelet denoising of ST N170 responses (Ahmadi and Quian Quiroga, 2013; Navajas et al., 2013). The mean and SD of the ST latencies were then calculated for each participant. The LJ_BS was compared between the two groups using Levene's test for equality of variances, and the LJ_WS by independent Bayesian and classical t tests.

Finally, we addressed potential differences in eye movements between blind and aware subjects in phase 1. Since eye tracking or electrooculography (EOG) was not available in the scanner, virtual EOG channels were calculated using the EEG data before ocular artifact correction. Specifically, vertical and horizontal eye movements were analyzed based on the mean and differential absolute activity in the frontal electrodes (FP1, FP2), respectively. The average activity in all IOIs (N170, VAN, P3b) was compared analogously to the ERPs, but using two-sided tests because of the absence of directional hypotheses.

EEG–fMRI correlations.

To investigate the relation between electrophysiological and imaging results, EEG and fMRI data were correlated across subjects using both classical and Bayesian inference in R (Morey et al., 2018; R Core Team, 2021). Specifically, correlations were calculated for difference amplitudes (face − random) averaged in the respective intervals and electrodes of interest and difference β-values (face − random) averaged in the respective voxels of interest. Awareness-related ERPs and fMRI clusters were correlated across all subjects in phase 1, having the largest variance in conscious perception. With regard to task relevance, the P3b effect (phase 3 > phase 2) was correlated with the respective BOLD effects. Since we expected linear relationships between electrophysiological and hemodynamic effects, Pearson's correlation coefficient was used. To account for the potential influence of outliers, robust skipped correlations were computed additionally (Rousselet and Pernet, 2012; Wilcox and Rousselet, 2018). Test directions again depended on the polarity of each ERP. To account for multiple classical tests (i.e., 8 for awareness and 25 for task relevance), α was set to 0.0063 and 0.002, respectively, according to the Bonferroni method. For BFs, a default medium prior scale was used (Morey et al., 2018).

Data availability

Data and materials for this study are available through the Open Science Framework at https://osf.io/qy8ba.

Results

Behavior

After phase 1 of the experiment, 32 of the 75 participants reported inattentional blindness to the faces, while 43 were spontaneously aware of them. Five subjects were excluded because of incomplete EEG and/or fMRI data. In the remaining sample, six were excluded from the blind group because they did not perceive any faces even when they were made aware of their presence (in phase 2). From the aware group, 12 subjects were excluded because they reported having seen < 10% of the faces presented. This resulted in final samples of 25 blind subjects and 27 aware subjects for the integrated EEG-fMRI analyses.

Considering task performance (d′ values and RTs; Fig. 2), nine participants (four blind, five aware) lacked data because of a technical defect of the response box. Crucially, Bayesian tests indicated that the awareness contrast between blind and aware subjects in phase 1 was not confounded by differences in task performance (d′: t_(36.22) = −0.16, p = 0.878, BF₀₁ = 3.30, d = −0.05; median RT: t_(40.38) = −0.67, p = 0.508, BF₀₁ = 2.78, d = −0.20). With regard to the task relevance contrast, performance was significantly lower in the face task in phase 3 compared with the dot task in phase 2 (d′: t₍₄₂₎ = −7.45, p < 0.001, BF₁₀ = 3.63 × 10⁶, d = −1.53; median RT: t₍₄₂₎ = 12.88, p < 0.001, BF₁₀ = 1.75 × 10¹³, d = 2.17).

Figure 2.

Behavioral results. Performance in the distractor dot task (phases 1 and 2) and the face task (phase 3) was quantified by d′ and median RTs. Error bars represent SEs.

fMRI

Awareness

The awareness contrast comparing differential (face > random) responses in aware versus blind subjects in phase 1 revealed strongly enhanced activation in the left and right fusiform gyrus (FG) with peaks at MNI coordinates −36, −54, −14 (logBF = 10.45, t₍₅₀₎ = 4.33, d = 1.00) and 36, −52, −18 (logBF = 8.65, t₍₅₀₎ = 3.75, d = 1.01) and cluster sizes (CSs) of 173 and 168 voxels, respectively (Fig. 3A). Additionally, smaller activations (CS = 1) were found in the left inferior lateral occipital cortex (LOcC; peak MNI coordinates: −36, −74, −4; logBF = 5.28; t₍₅₀₎ = 2.71; d = 0.75) and the right inferior frontal junction (IFJ; peak MNI coordinates: 34, 16, 30; logBF = 5.17; t₍₅₀₎ = 2.71; d = 0.75). Post hoc analyses confirmed average β-differences > 0 in aware conditions in all of these clusters (left FG: all logBF > 14.81; right FG: all logBF > 12.35; left LOcC: all logBF > 9.91; right IFJ: all logBF > 6.97). While no conclusive awareness effects were found in other regions, PPMs in favor of the null hypothesis did not show substantial evidence against them.

Figure 3.

Imaging results. PPMs are presented using a dual-coded approach (Allen et al., 2012; Zandbelt, 2017). The β-differences [face (F) – random (R)] are mapped to color hue, and the logBF statistic to transparency. Black contours outline gray matter activations with logBF > 5 (i.e., 99% probability of an effect). Numbers next to maps indicate z-coordinates (MNI). Scatter plots show β-differences (F − R) of single subjects split by the groups from phase 1 across all phases. Error bars denote SEs. A, Awareness contrast of BOLD responses (F – R) between blind and aware subjects in phase 1. Axial slices are centered at clusters meeting NCC criteria (i.e., the left and right FG, the left LOcC, and the right IFJ). B, Task relevance contrast of BOLD responses (F – R) between phase 3 (aware, task-relevant) and phase 2 (aware, task-irrelevant) in all subjects. Task relevance strongly activated extensive occipitotemporal, frontoparietal, and attentional networks. The scatter plot presents β-differences (F − R) averaged across all voxels with task-related activation for an overview. Additionally, pairwise differences (phase 3 – phase 2) are shown. Statistics for local effects are displayed in Table 2.

Task relevance

To identify the brain regions primarily recruited when face perception was task-relevant, all subjects' differential BOLD responses were compared between phases 2 (aware, task-irrelevant) and 3 (aware, task-relevant). Task relevance led to strong and extensive activations in widely distributed brain regions (Fig. 3B). Average difference β-values from gray-matter voxels within a radius of 8 mm around the most prominent peaks in both hemispheres are presented (Table 2). In occipitotemporal areas, the strongest task effects were in bilateral medioventral occipital cortex (mvOcC) and FG. In parietal cortex, they peaked in the bilateral superior parietal lobule (SPL), inferior parietal lobule (IPL), and postcentral gyrus (PoG). In the frontal cortex, peaks were determined in the bilateral precentral gyrus (PrG) and right superior frontal gyrus (SFG), as well as bilateral supplementary motor area (SMA), middle frontal gyrus (MFG), and right orbital gyrus (OrG). Finally, bilateral peaks were identified in the insular cortex (INS).

Table 2.

Results for the task relevance contrast (all subjects, phase 3 > 2) in BOLD responses (face > random)

Lobe	Area	Hem	Peak MNI coordinates			Peak statistics		CS	β (Ph 2)	β (Ph 3)	ES
			x	y	z	logBF	t (51)		M (SE)	M (SE)	d
Occ	mvOcC	L	−10	−66	4	19.09	6.36	214	0.79 (0.26)	2.59 (0.28)	0.76
		R	2	−68	6	16.54	6.06	243	1.03 (0.30)	3.09 (0.31)	0.79
	FG	L	−44	−60	−7	32.59	8.16	228	0.90 (0.16)	2.33 (0.22)	0.97
		R	48	−54	−14	27.98	7.50	240	1.17 (0.22)	2.67 (0.24)	0.85
Par	SPL	L	−30	−46	46	36.04	8.90	200	0.66 (0.18)	2.47 (0.22)	1.12
		R	32	−54	50	36.04	10.28	247	1.27 (0.21)	3.96 (0.27)	1.41
	PoG	L	−44	−34	50	30.07	8.01	215	0.20 (0.23)	2.17 (0.21)	1.10
	IPL	R	50	−34	52	36.04	10.86	196	0.47 (0.23)	2.94 (0.23)	1.41
	PoG	L	−42	−22	52	32.50	8.16	148	0.05 (0.22)	2.06 (0.21)	1.06
		L	−54	−18	22	22.20	6.62	168	0.07 (0.17)	1.38 (0.20)	0.81
		R	62	−18	34	23.36	6.86	204	0.44 (0.22)	1.77 (0.21)	0.79
Fro	PrG	L	−42	0	32	36.04	9.54	158	0.60 (0.16)	2.28 (0.19)	1.22
		R	44	6	26	36.04	13.13	186	0.84 (0.17)	3.52 (0.23)	1.77
		L	−32	−8	52	19.06	6.10	185	0.45 (0.17)	1.60 (0.17)	0.76
	MFG	R	28	4	52	24.34	6.95	199	0.40 (0.16)	1.57 (0.16)	0.83
	SFG	R	14	8	62	15.51	5.37	168	0.56 (0.15)	1.75 (0.20)	0.72
	SMA	L	−2	10	50	36.04	12.47	174	0.68 (0.25)	3.74 (0.24)	1.54
		R	4	12	50	35.12	10.14	170	0.77 (0.26)	3.82 (0.26)	1.40
Ins	INS	L	−30	20	8	36.04	10.99	177	0.42 (0.16)	2.58 (0.20)	1.40
		R	32	24	0	36.04	10.88	215	0.58 (0.17)	2.96 (0.21)	1.40
Fro	MFG	L	−38	28	24	8.50	3.74	87	0.44 (0.19)	1.24 (0.23)	0.49
		R	48	28	26	29.20	7.94	227	0.74 (0.22)	2.80 (0.28)	1.04
		L	−36	50	28	10.12	4.75	103	0.43 (0.27)	2.15 (0.37)	0.59
		R	30	46	28	13.32	5.23	235	0.28 (0.26)	1.90 (0.31)	0.65
	OrG	R	28	52	−14	9.32	4.04	56	0.22 (0.15)	1.16 (0.22)	0.50

For each lobe (Occ = occipital, Par = parietal, Fro = frontal, Ins = insular) and in posterior–anterior order, peaks in the most relevant brain areas are presented for the left (L) and right (R) hemisphere (Hem). Means (M) and SEs of the difference β-values (face − random) in phase (Ph) 2 and Ph 3 were extracted from gray matter voxels exceeding logBF = 5 in spheres with a radius of 8 mm around the peak voxel coordinates. Effect sizes (ESs; Cohen's d) were computed using the mean difference β-values. Area abbreviations in alphabetical order: FG = fusiform gyrus, IFL = inferior parietal lobule, INS = insula, MFG = middle frontal gyrus, mvOcC = medioventral occipital cortex, OrG = orbital gyrus, PoG = postcentral gyrus, PrG = precentral gyrus, SFG = superior frontal gyrus, SMA = supplementary motor area, SPL = superior parietal lobule.

EEG

In the EEG data of the final sample, an average of 2.38 (SD = 1.09) channels were interpolated and 23.23% (SD = 4.37%) of trials were excluded because of distractor target presentations and artifacts, leading to an average of 82 trials for face stimuli (intact faces only) and 103 trials for random stimuli per phase for statistical analyses. Compared with the study by Shafto and Pitts (2015), the fewer total trials in favor of fMRI-compatible ISIs and scanner-related artifacts resulted in about half as many trials for statistical analysis. However, this limitation was countervailed by the doubled sample size. Grand-averaged ERPs and the results of the awareness and task relevance contrast are presented in Figures 4 and 5, respectively.

Figure 4.

Electrophysiological results. ERPs in the electrodes of interest (N170: P7, P8; VAN: O1, O2, P7, P8; P3b: Pz) in each phase separated by group (blind vs aware in phase 1). Colored, shaded areas around waveforms depict 95% bootstrap confidence intervals. Difference-wave scalp topographies (face − random) were plotted in the respective intervals of interest (N170, 160–230 ms; VAN, 210–270 ms; P3b, 400–600 ms).

Figure 5.

Electrophysiological results. A, The awareness contrast compared difference waves [face (F) – random (R)] between aware and blind subjects in phase 1. Awareness modulated the N170 and VAN but not the parietal P3b component. B, The task relevance contrast compared difference waves (F – R) of all subjects between phase 3 (aware, task-relevant) and phase 2 (aware, task-irrelevant). Thus, pairwise differences (phase 3 – 2) are depicted. Task-relevant face processing enhanced early negativities and elicited a pronounced P3b component. Thin lines represent single-subject difference waves, and thick lines the grand average. Gray areas indicate the intervals of interest for each ERP in which statistical analyses were calculated. Scatter plots present the values entered into the tests. Error bars denote SEs. Topographies show the respective contrast in the interval of interest.

Awareness

The awareness contrast demonstrated a significantly larger N170 in aware compared with blind subjects in phase 1 (t_(49.88) = −2.39, p = 0.010, BF₁₀ = 5.38, d = −0.66). Importantly, this effect was not driven by group differences in latency jitter within subjects (BF₀₁ = 3.51) or between subjects (BF₀₁ = 3.19). Testing the average difference amplitudes against zero confirmed that the N170 was absent in blind subjects in phase 1 (BF₀₁ = 8.54) and present in aware conditions (all: p < 0.011, BF₁₀ > 4.64). For the VAN, the awareness contrast confirmed larger negativities in aware compared with blind subjects in phase 1 (t_(45.12) = −2.06, p = 0.023, BF₁₀ = 3.08, d = −0.58). Furthermore, these negativities were absent in blind subjects in phase 1 (BF₀₁ = 8.13) and present in aware conditions (all: p < 0.016, BF₁₀ > 3.58). For the P3b, the awareness contrast indicated no difference between blind and aware subjects in phase 1 (t_(49.97) = −0.97, p = 0.170, BF₀₁ = 1.52, d = 0.27). Tests against zero showed substantial evidence for the absence of a P3b in blind subjects (BF₀₁ = 4.71), while the result for aware subjects was inconclusive (t₍₂₆₎ = 1.36, p = 0.092, BF₀₁ = 1.19). Exploratory cluster-based permutation tests did not reveal any significant additional awareness effects, either across all electrodes and time points or in posterior or anterior electrodes within the IOIs (all: p > 0.104). Importantly, this also applied for a noteworthy late occipital positivity. Finally, Bayesian post hoc tests indicated no differences in vertical or horizontal eye movements between the groups in any IOI (all: BF₀₁ > 2.97).

Task relevance

The task relevance contrast revealed increases in amplitude in phase 3 compared with phase 2 in the N170 (t₍₅₁₎ = −2.89, p = 0.003, BF₁₀ = 12.02, d = −0.43) and VAN (t₍₅₁₎ = −2.24, p = 0.015, BF₁₀ = 2.92, d = −0.43). Moreover, as hypothesized, the P3b massively increased with task relevance (t₍₅₁₎ = 6.37, p < 0.001, BF₁₀ = 5.07 × 10⁵, d = 1.21).

EEG-fMRI correlations

Electrophysiological and hemodynamic signatures of awareness and task relevance in face perception were correlated across subjects. Note that negative correlations of a negative component indicate a positive relationship with the BOLD response. Concerning awareness effects in phase 1 (Fig. 6, Table 3), there was substantial evidence for a small to moderate correlation of the N170 with the right FG cluster. A smaller correlation was observed between the N170 and right IFJ activation. However, robust skipped correlations suggested an amplifying influence of outliers in the N170 correlations. VAN amplitudes were weakly correlated with the activations in the right FG and IFJ. In these cases, skipped correlations indicated stronger relationships, particularly between the VAN and right IFJ activation. The left LOcC voxel was not conclusively correlated with any ERP. Regarding task relevance, there were no conclusive correlations of the P3b effect with task-related BOLD effects (all: r < 0.18, p > 0.108, and BF₀₁ > 0.90). Only skipped correlations indicated a weak link between the P3b and right IPL activation (r′ = 0.23, p = 0.029). Since none of the correlations survived Bonferroni correction, the results should be interpreted with caution.

Figure 6.

EEG-fMRI correlation results. The scatter plots present awareness-related differential electrophysiological and hemodynamic responses (face − random) across subjects in phase 1. Small correlations were found between the N170 and VAN and the right FG and IFJ. Note that negative correlations between the negative ERPs and positive BOLD responses indicate a positive relationship.

Table 3.

Correlations of awareness-related differential (face − random) ERPs and BOLD responses across subjects in phase 1

ERP	Cluster	Pearson correlation				Skipped Pearson correlation
		r	p	BF10	BF01	r′	p
N170	Left FG	–0.18	0.108	1.10	0.91	–0.08	0.384
	Right FG	–0.30	0.017	4.68	0.21	–0.15	0.324
	Right IFJ	–0.22	0.058	1.79	0.56	–0.04	0.351
	Left LOcC	–0.07	0.324	0.46	2.18	0.05	0.435
VAN	Left FG	–0.12	0.206	0.66	1.51	–0.25	0.076
	Right FG	–0.25	0.039	2.41	0.42	–0.33	0.055
	Right IFJ	–0.24	0.041	2.33	0.43	–0.35	0.018
	Left LOcC	–0.07	0.300	0.49	2.05	–0.09	0.245

Discussion

The present study used simultaneous EEG-fMRI to disentangle the neural correlates of consciousness in visual perception from those of task relevance. In an inattentional blindness paradigm, awareness of faces covaried most strongly with hemodynamic responses in the fusiform gyrus, occipital cortex, and inferior frontal junction as well as the early N170 and visual awareness negativity. Task relevance, on the other hand, led to a strong and extensive activation of occipitotemporal, frontoparietal, and attentional networks, increased early negativities and elicited the late parietal P3b component. These findings suggest that conscious visual perception occurs during early sensory processing (Koch et al., 2016), but may also involve certain subregions of prefrontal cortex (Brown et al., 2019). In contrast, they challenge theories that posit a central role of late positivities and strong, widespread frontoparietal activation (Mashour et al., 2020), which are likely to primarily reflect task-related processes (Aru et al., 2012).

Our EEG data replicated the results of the study by Shafto and Pitts (2015). The N170, a well established marker of face processing (Rossion, 2014), was present in all aware conditions, and substantial Bayesian evidence for its absence during IB concurs with previous studies in which attention was diverted from faces (Axelrod et al., 2015). Importantly, the awareness effect in average N170 amplitudes was not driven by differences in single-trial latency variability (Navajas et al., 2013). Corroborating the role of the VAN in conscious visual perception (Förster et al., 2020), occipitotemporal negativities 210 ms after stimulus onset were enhanced in aware processing, but absent during unaware processing. In accordance with previous studies (Pitts et al., 2012; Shafto and Pitts, 2015; Cohen et al., 2020; Schlossmacher et al., 2020, 2021; Sergent et al., 2021) and contradicting GNWT predictions (Dehaene, 2014; Mashour et al., 2020), the parietal P3b was primarily linked to task relevance and not awareness. However, awareness elicited a weak and insignificant more occipital late positivity, which also appeared in the findings of Shafto and Pitts (2015) and should be scrutinized in future studies. It is important to emphasize that late positivities may reflect the depth of conscious processing, memory encoding, and the degree of reportability (Pitts et al., 2018). From a GNWT perspective, the P3b may signal access consciousness involving the amplification of a given percept and inhibition of competing stimuli (Mashour et al., 2020). Deeper processing may also account for the task effects we observed on N170 and VAN amplitudes (Axelrod et al., 2015; Jimenez et al., 2020). Taken together, conscious face processing does not necessarily evoke a P3b, whereas early negativities are reliably modulated by awareness.

In fMRI, Bayesian PPMs (Friston and Penny, 2003) demonstrated that conscious face perception in the absence of task relevance was predominantly linked to activation in the left and right FG in the occipitotemporal cortex, which strongly differed between aware and inattentionally blind participants in phase 1 and was present in all aware conditions. A smaller activation was found in left inferior LOcC. These findings perfectly conform with the well established core system of face perception in humans (Haxby et al., 2000) and face-specific inferotemporal NCCs in monkeys (Hesse and Tsao, 2020). In frontoparietal regions, only a localized effect in the right IFJ met the current NCC criteria. This finding is approximately consistent with a key role of the lateral PFC in conscious perception in humans (Dehaene et al., 2006; Brown et al., 2019) and monkeys (Panagiotaropoulos et al., 2012; van Vugt et al., 2018; Kapoor et al., 2020). However, it could also be associated with the role of the IFJ in cognitive control and task switching (Brass et al., 2005) and/or action withholding (O'Connor et al., 2015); whereas inattentionally blind subjects were not distracted from the dot task, aware subjects had to actively inhibit the irrelevant faces to perform the distractor task adequately. In fact, the unheralded faces may have acquired some implicit relevance (Block, 2019), since synchronized IFJ and FG activity has specifically been shown to represent object-based attention to faces (Baldauf and Desimone, 2014).

Overall, the strong concentration of awareness effects in occipitotemporal brain regions accords with RPT (Lamme, 2003, 2006, 2010) and other theories positing a dominant role of sensory areas in conscious perception, such as integrated information theory (Koch et al., 2016). Furthermore, our results stress the role of task-related confounds in previous studies on conscious face perception (Rodríguez et al., 2012). It is important to note, however, that, despite the relatively large sample size, Bayesian inference did not provide decisive evidence for or against awareness effects in frontoparietal regions. Conclusions regarding the role of these areas should therefore be drawn cautiously. For example, the relative weakness of awareness effects in frontoparietal compared with visual networks may be related to the former's pronounced interindividual heterogeneity (Marek and Dosenbach, 2018). Nevertheless, while our data provide tentative evidence for a role of specific frontal areas in conscious perception, we failed to find evidence for a strong, all-or-none frontoparietal “ignition” being necessary for visual awareness (Dehaene, 2014; Mashour et al., 2020).

On the contrary, the task relevance contrast demonstrated that strong frontoparietal BOLD activations were primarily linked to task-related processes instead of awareness (Tsuchiya et al., 2015; Koch et al., 2016). Specifically, the observed frontal effects can be explained by the role of the dorsolateral PFC in introspection, self-monitoring, and perceptual decision-making (Heekeren et al., 2008), of the frontal pole in metacognition (Fleming et al., 2010), and of the SFG in movements of the eyes (Paus, 1996) and hands (Grefkes et al., 2008). Effects in the inferior and superior parietal lobe can be related to its role in motor action (Rozzi et al., 2008) and maintaining internal representations (Wolpert et al., 1998), respectively. Moreover, strong task relevance effects were observed in the ventral attention network (Menon and Uddin, 2010), dorsal attention network (Corbetta and Shulman, 2002), and in face-sensitive areas in the mvOcC and FG (Haxby et al., 2000).

Correlating the electrophysiological and hemodynamic signatures of conscious face perception across subjects yielded a weak correlation of the N170 and right FG activity, which agrees with some previous EEG/fMRI studies (Horovitz et al., 2004; Sadeh et al., 2010). However, such correlations do not permit the localization of ERP generators, and other N170 sources have been suggested, for example, superior temporal sulcus (Sadeh et al., 2010) and gyrus (Horovitz et al., 2004) or even prefrontal areas (Babiloni et al., 2010). The VAN (ERP) was also correlated with right FG activation, which coincides with EEG source localizations suggesting that the VAN originates in occipitotemporal regions (Förster et al., 2020). Furthermore, the VAN was linked to right IFJ activation, which may be explained by a role of the PFC in visual awareness (Dehaene and Changeux, 2011; Brown et al., 2019) or spontaneous object-based attention to the faces, during which the IFJ may direct stimulus-specific sensory responses (Baldauf and Desimone, 2014). This result is particularly interesting as no previous study has isolated the VAN from later task-related ERPs while also measuring activations with spatial precision (Förster et al., 2020). Moreover, it concurs with the decoding of awareness from lateral PFC in monkeys earlier than 300 ms after stimulus onset (Panagiotaropoulos et al., 2012).

Concerning task relevance, there were no positive correlations between the P3b effect and any single task-related activation in fMRI. This observation may be explained by an inhibitory rather than an excitatory mechanism underlying the P3b (Polich, 2007; Dehaene and Changeux, 2011) or the widely distributed rather than focal sources of the component (Polich, 2007; Volpe et al., 2007).

The decisive advantage of the present paradigm is its power to isolate awareness-related neural activity from task-related processes. However, it is limited in several ways. Most notably, the delayed reports preclude precise single-trial assessments of awareness, which raises two distinct concerns. First, in the aware conditions in which the faces were task-irrelevant, it is unclear exactly how many of the 120 face stimuli were consciously perceived. This might lead to a slight underestimation of NCCs but should not affect our main conclusions. The fact that we found a VAN, a typically small component, underlines the sensitivity of the current design. Second, one could argue that putatively unaware participants may have perceived, but swiftly forgotten the faces (inattentional amnesia; Wolfe, 1999). However, it has been demonstrated that the inability to report stimuli during IB stems from perceptual and not memory deficits (Ward and Scholl, 2015), and the high memorability of unexpected faces renders the amnesia explanation particularly unlikely in the present context. Finally, fMRI sensitivity in some brain regions may have suffered from interindividual differences in neuroanatomy. Future studies should aim to identify frontoparietal NCCs on a single-subject level.

In summary, our results provide the first direct insights into the interplay of consciousness and task relevance using a unique combination of simultaneous EEG and fMRI and a delayed report inattentional blindness paradigm on face perception. These results corroborate the notion that subjective visual experience is most strongly associated with early activity in sensory brain areas, but suggests that subregions of prefrontal cortex may also play a critical role. In contrast, late, strong, and widespread activation involving frontoparietal networks more likely reflects task-related processes.