Intracranial Markers of Conscious Face Perception in Humans

Abstract

Investigations of the neural basis of consciousness have greatly benefited from protocols that involve the presentation of stimuli at perceptual threshold, enabling the assessment of the patterns of brain activity that correlate with conscious perception, independently of any changes in sensory input. However, the comparison between perceived and unperceived trials would be expected to reveal not only the core neural substrate of a particular conscious perception, but also aspects of brain activity that facilitate, hinder or tend to follow conscious perception. We take a step towards the resolution of these confounds by combining an analysis of neural responses observed during the presentation of faces partially masked by Continuous Flash Suppression, and those responses observed during the unmasked presentation of faces and other images in the same subjects. We employed multidimensional classifiers to decode physical properties of stimuli or perceptual states from spectrotemporal representations of electrocorticographic signals (1071 channels in 5 subjects). Neural activity in certain face responsive areas located in both the fusiform gyrus and in the lateral-temporal/inferior-parietal cortex discriminated seen vs. unseen faces in the masked paradigm and upright faces vs. other categories in the unmasked paradigm. However, only the former discriminated upright vs. inverted faces in the unmasked paradigm. Our results suggest a prominent role for the fusiform gyrus in the configural perception of faces, and possibly other objects that are holistically processed. More generally, we advocate comparative analysis of neural recordings obtained during different, but related, experimental protocols as a promising direction towards elucidating the functional specificities of the patterns of neural activation that accompany our conscious experiences.

Graphical abstract

Introduction

In the last couple of decades, the relationships between brain activity and the contents of perceptual consciousness have been investigated using a variety of experimental techniques operating at different spatial and temporal scales, from single-unit, multi-unit and local field potential recordings in monkeys (Logothetis and Schall, 1989; Leopold and Logothetis, 1996; Wilke et al., 2006; Maier et al., 2007; Wilke et al., 2009), to non-invasive neuroimaging techniques such as EEG, MEG and fMRI in humans (e.g., (Tong et al., 1998; Grill-Spector et al., 2000; Dehaene et al., 2001; van Aalderen-Smeets et al., 2006; Liu et al., 2012; Schurger et al., 2015)) (see (Rees et al., 2002; Tononi and Koch, 2008; Dehaene and Changeux, 2011; Boly et al., 2013; Panagiotaropoulos et al., 2014) for reviews).

The scientific investigation of perceptual states presents a unique challenge, since it requires the objective measurement of subjective states. In particular, accuracy in reports of subjective states is a critical prerequisite for this investigation. With sufficient amount of training and careful experimental design (Leopold et al., 2003), monkeys (and potentially other animals) can be trained to report their perceptual states in a reliable manner (see, for example, (Leopold and Logothetis, 1996)). However, the investigation of the neural correlates of conscious awareness in human subjects constitutes a great advantage, since they can provide accurate reports of their perceptual states with minimal training following verbal instructions from the experimenter. This is critical, especially if graded levels of perceptual awareness are considered, as in the current study.

In humans, non-invasive neural recordings have been extensively employed in the search of the neural correlates of consciousness. Here, we recorded electrocorticography (ECoG) from subdural electrodes implanted on the ventral and lateral surface of the temporal lobes in five epileptic patients undergoing pre-surgical seizure monitoring while they engaged in visual perception tasks. Intracranial recordings from human subjects undergoing pre-surgical monitoring constitute a precious opportunity to advance our understanding of the neural correlates of conscious perception (e.g. (Kreiman et al., 2002; Gaillard et al., 2009; Fisch et al., 2009; Aru et al., 2012a; Willenbockel et al., 2012; Quiroga et al., 2014), see (Lachaux et al., 2003; Engel et al., 2005; Jacobs and Kahana, 2010; Mukamel and Fried, 2012) for reviews), due to the direct measure of electrophysiological responses as well as their high spatial and temporal resolution in comparison with non-invasive modalities. In particular, the high temporal resolution afforded by ECoG recordings enables one to accurately assess high-frequency broadband neural activity (>70 Hz), which has been shown to be highly informative and more related to neuronal spiking activity than lower frequency bands (e.g., (Manning et al., 2009; Ray and Maunsell, 2011; Miller et al., 2014)), while much less contaminated by non-neural (e.g., muscular) activity in comparison with non-invasive recording modalities.

Several techniques have been proposed to investigate the neural correlates of conscious visual perception (Kim and Blake, 2005). These techniques enable the dissociation between retinal images and subjective perception.

Previous intracranial recording studies have investigated the neural correlates of conscious visual perception using stimuli that are perceptually degraded by a technique known as Backward Masking (BM) (Gaillard et al., 2009; Fisch et al., 2009; Quiroga et al., 2008). In a typical BM paradigm, a target image is presented briefly, followed by a masking image after a variable delay, known as Stimulus Onset Asynchrony (SOA). Short SOAs prevent the target image from being consciously perceived, while long SOAs allow the target image to emerge to consciousness reliably. At intermediate SOAs, conscious visibility fluctuates across trials. While most BM studies investigated the neural correlates of consciousness by comparing trials that differed markedly in either stimulus configuration (e.g., (Dehaene et al., 2001)) or other covariates, such as subject training (e.g., (Grill-Spector et al., 2000)), some recent studies aimed to more subtle contrasts that could more specifically expose the neural correlates of consciousness (Gaillard et al., 2009; Fisch et al., 2009). However, even these latter studies compared visible and invisible conditions in response to similar, but not identical, input stimuli, due to the experimental difficulty of adjusting SOA at perceptual threshold (but see (Quiroga et al., 2008; Del Cul et al., 2007) for examples where the contrast between seen and unseen targets at threshold SOA was possible for a subset of subjects). Thus, studies using BM may generally confound neural activity related to different perceptual outcome with neural activity related to different visual stimulation.

Here, we employed a different masking technique, known as Continuous Flash Suppression (CFS). This technique is based on the presentation of rapidly changing Mondrian patterns to one eye, while a static image (the target) is presented to the other eye ((Tsuchiya and Koch, 2005), see (Yang et al., 2014; Sterzer et al., 2014) for recent reviews). Depending on the contrast of the target image and the Mondrian masks, the target image can be completely invisible, clearly visible, or visible only in a subset of the trials. The latter condition is of special interest, since the contrast between neural activity corresponding to trials with different visibility outcomes, in conditions of equal stimulus contrast, enables us to assess the neural correlates of visibility in the absence of any change in the physical properties of the stimulus.

Even when comparing trials corresponding to identical physical stimuli, but different perceptual outcomes, the resulting differences cannot be unambiguously considered as core neural correlates of phenomenal conscious perception, or NCC-core (Aru et al., 2012b; de Graaf et al., 2012; Miller, 2007; van Boxtel and Tsuchiya, 2015; Tsuchiya et al., 2015), because they likely reflect additional processes that also differ between the conditions. In fact, the comparison between seen and unseen trials can also reveal brain states that facilitate (e.g., attentional mechanisms) or hinder (e.g., mind-wandering) the perceptual awareness of threshold stimuli. In addition to this, conscious visual perception of target stimuli can trigger a cascade of neural processes related to memory formation, generation of associations and motor preparation for the ensuing response. Aru et al. and de Graaf et al. conveniently termed the potential confounds belonging to the former category as NCC-prs, or prerequisites, and the latter as NCC-cos, or consequences, of the conscious perceptual experience.

In this work, we take a step towards the dissociation between the neural correlates of core aspects of conscious visual experience (NCC-core) and their prerequisites and consequences by combining different, albeit related, experimental protocols. In particular, we considered a masked visual task, where stimuli were made partially invisible by Continuous Flash Suppression (CFS); and an unmasked visual task, where stimuli were clearly visible. Each of these tasks expose different cognitive processes: while the partially masked visual targets in the CFS task seemed to require some effort to be seen, unmasked images were clearly seen without effort.

Importantly, some of the stimuli used in the unmasked task (photographs of human faces) belong to the same category as the target stimuli in the masked task. Human faces constitute a stimulus category of exceptional behavioral and ecological relevance, and are known to be processed in specific circuits in ventral and lateral regions of the temporal lobe, most evidently in the Fusiform Gyrus (FG) and in the Superior Temporal Sulcus (STS) (Allison et al., 1994; Puce et al., 1995; Kanwisher et al., 1997; Haxby et al., 2000; Kanwisher and Yovel, 2006; Tsuchiya et al., 2008; Kawasaki et al., 2012).

Critically, the image categories used in the unmasked task comprise inverted faces in addition to upright faces and non-face objects such as houses and tools. The comparison between neural activity in response to upright versus inverted faces is expected to reveal features of neural processing that are specific to configural or holistic perception, that is, a gestalt perception where the whole face is perceived in a qualitatively different manner from the sum of its parts (e.g., (Rossion and Gauthier, 2002)). This phenomenon can be measured behaviorally, for example via reaction times in recognition (e.g. same/different judgment) tasks.

The comparison between specific neural markers in the masked and unmasked tasks enables one to discard neural markers that could otherwise be considered as putative NCC-core if only the contrast between visible and invisible trials at threshold were considered. More generally, this work paves the way for a new promising set of methodological approaches in consciousness research based on the comparison between similar experimental protocols, which differ in specific aspects that expose the key differences that enable one to disentangle the different aspects of the conscious visual experience.

Materials and Methods

Data Acquisition

We recorded intracranially with electrocorticographic (ECoG) electrodes from 5 epilepsy patients undergoing pre-surgical monitoring. Sampling rate for the ECoG signal was 2034.5 Hz. Electrode location was based solely on clinical criteria. Patient age, gender, handedness, ocular and language dominance, and locations of seizure foci are reported in Table 1. We did not record for 12 hours after any generalized seizure event. We did not perform any explicit artifact rejection. Voltage traces from each electrode were visually inspected and none presented obvious signatures of artifactual or epileptiform activity. In-tracranial recordings exhibit very little contamination from non-neural (e.g., muscular) sources (with the exception of specific regions of the temporal pole that can exhibit activity related to eye movements (Jerbi et al., 2009; Kovach et al., 2011)). Isolated interictal epileptiform discharges could have occurred; however, their timing, frequency and amplitude are expected to be unrelated to our variables of interest (visibility ratings in the masked protocol and object category in the unmasked protocol). The University of Iowa Institutional Review Board (IRB) approved the study (approval number 200112047), and written informed consent was obtained from each patient. Further details are reported in (Tsuchiya et al., 2008; Kawasaki et al., 2012).

Table 1.

Demographic information for each subject.

sub ID	age	gender	handedness	ocular dominance	language dominance	seizure focus
147	29	male	left	left	left	left temporal lobe, neocortex
153	31	female	right	left	left	right anterior medial temporal lobe
154	40	male	right	left	left	right medial temporal lobe
168	24	male	right	left	left	left anterior lateral temporal cortex
178	47	male	right	right	left	no diagnostic seizure (likely left medial temporal lobe)

Electrode Localization

For each subject, we obtained structural T1-weighted MRI volumes (pre- and post- electrode implantation), CT scans (post-implantation) and digital photos of the electrodes (during surgery, only for the lateral temporal grid electrodes). Coronal MRI slices were obtained with 1 mm slice thickness, 0.78 × 0.78 mm in-plane resolution. Axial slices of the CT scans were obtained with 1 mm slice thickness, 0.45 × 0.45 mm in-plane resolution. Post-implantation CT scans and pre-implantation MRI were rendered into 3D volumes and co-registered using AFNI (NIMH, Bethesda, MD, USA) and/or ANALYZE software (version 7.0, AnalyzeDirect, KS, USA) with mutual information maximization. Cortical surfaces for each subject were automatically parcellated using Freesurfer and the “Desikan-Killiany-Tourville” (DKT) brain atlas (Desikan et al., 2006). Then, each electrode was assigned to an anatomical location based on the closer point on the cortical surface (Fig. S1). Similar results were obtained using the Destrieux atlas (Destrieux et al., 2010). Electrode locations are also shown on a standard Montreal Neurological Institute (MNI) brain rendered with Fieldtrip (Oostenveld et al., 2011) in Fig. 9 and in Fig S7.

Figure 9. Conjunction of significant decoding results in the masked and unmasked paradigms across all subjects.

Electrode locations are shown as circles on standard MNI brain images. The color code indicates the conjunction of significant (p<0.01) decoding results for each electrode as indicated in the legend. C3vB: c₃ vs. blank; FvO: upright face vs. other categories; FvI: upright vs. inverted face; ¬ is the negation symbol. Please note that the category C3vB & v & FvO & ¬ FvI also includes some electrodes from subject 147 for which the FvI decoding analysis was not possible. Corresponding plots are shown for each subject separately in Fig. S7.

Behavioral Tasks

In order to assess and characterize neural activity related to conscious visual perception, we used two different sets of tasks: one involving masked images of faces, and another involving unmasked images of faces and other objects (images available at https://drive.google.com/open?id=0BykWihLWnaYRLVZwXzhWVU40Vnc). The masked and unmasked tasks differ in cognitive requirements such as those related to attention, memory and report; hence their combined analysis can more specifically highlight neural activity directly related to the core mechanisms of conscious vision than could be possible if only the masked task were considered. In both sets of tasks, images were presented at fixation on a 19″ ViewSonic VX922 LCD display (refresh rate: 60 Hz) and subtended about 7.5 × 10 deg in visual angle. Behavioral responses were collected using key presses on a USB keypad. We presented the stimuli using Psychtoolbox (Brainard, 1997) version 2.54 and MATLAB version 7.8 or higher on a PC running Windows XP. In order to ensure maximal precision in the temporal alignment of neural signals and visual stimuli, we displayed a small rectangle on the top-left corner of the screen, which changed in luminance in synchrony with the stimuli displayed at fixation, and recorded the response of a photodiode directly attached at that corner. The output from the photodiode was recorded along with the electrophysiological responses in the same recording system and used for segmenting the raw ECoG traces (see subsection “Data Analysis”).

Masked Visual Task (CFS)

In each trial, subjects were presented with a fixation cross displayed at the center of the screen and initiated a trial by pressing the space bar. The beginning of a trial was reflected on the screen by a 45-degree rotation of the fixation cross. Each trial consisted of two 200 ms intervals. After a variable period (uniformly distributed in [500,700] ms), the first interval was presented on the screen. Subsequently, after a variable period (uniformly distributed in [900,1100] ms), the second interval was presented on the screen.

In both intervals, three distinct Mondrian patterns were flashed within a frame composed of black and white squares to the dominant eye to suppress a visual input to the non-dominant eye (Continuous Flash Suppression (Tsuchiya and Koch, 2005)). Each Mondrian pattern was presented for 67ms and updated without any blank between the different patterns. In one of the two intervals, a face image was presented to the non-dominant eye, while in the other interval a blank field was presented instead within the corresponding area of the black and white frame (Fig. 1A).

Figure 1. Experimental Protocols.

(A) Masked task (CFS). (B) Unmasked task.

Following the termination of the second interval, after a variable period (uniformly distributed in [900,1100] ms), a first response screen appeared, asking for the interval which included the face (two-Interval Forced Choice, 2IFC). After the subject response, a second response screen appeared, asking for a face visibility rating, expressed according to the Perceptual Awareness Scale (PAS, (Ramsøy and Overgaard, 2004; Overgaard et al., 2006), four-Alternative Forced Choice). PAS is a scale for measuring the clarity or vividness of a subjective perceptual experience, which uses a 4 point scale with the following categories: “I did not see a face” (1), “I had a weak glimpse of a face” (2), “I saw a face almost clearly” (3), and “I saw a face clearly” (4). We clearly explained the meaning of each category in the PAS to each subject before each experiment, and we encouraged them to use all points on the scale.

While it is possible that slightly different results could have been obtained with confidence judgments (that is, judgments on the 2IFC task performance (Sandberg et al., 2010; Overgaard and Sandberg, 2012; King and Dehaene, 2014)), visibility ratings are more directly relevant to our primary concern (i.e., conscious face perception) than confidence judgments. Moreover, they have been shown to correlate more closely with objective performance and to yield a lower unconscious performance in identification tasks (Sandberg et al., 2010; Overgaard and Sandberg, 2012), suggesting that they might be generally more trustworthy than confidence judgments.

We used four different face identities with either neutral or fearful emotional expression (Ekman and Friesen, 1976) to reduce low-level perceptual learning (e.g., (Fahle, 2009)). Face images were presented at three logarithmically spaced contrast levels (with the exception of subject 178, for whom a different set of contrast values was used). Ideally, low contrast trials (c=1) would result in an objective performance in the 2IFC task near chance level, high contrast trials (c=3) would result in an objective performance around 90% or above, while intermediate contrast trials (c=2) would result in an objective performance around 75%. However, time constrains (due to, e.g., clinical requirements, non-clinical experimental sessions, and other reasons) did not enable us to adjust contrast levels individually for each subject. Hence, a set of contrast levels were used for the first session. If the behavioral performance was too high or too low (e.g., objective performance above 85% or at chance level at intermediate contrast), contrast levels were scaled geometrically in successive sessions.

All subjects reported in this study performed between 2 and 4 sessions of the masked task, each comprised of 48 trials. Since subjective visibility (as reported by subjects' visibility ratings) is the main focus of this study, we considered an exclusion criterion for each subject and session based on variability in visibility ratings: a session was considered to be “invalid” if 40 or more trials were assigned to the most populated visibility rating class. A subject was included in the analysis if she/he performed at least 2 “valid” sessions. Based on this criterion, one subject was excluded because 4/4 sessions were invalid, and two other subjects were excluded because 1/2 sessions were invalid. None of the included subjects performed any “invalid” session. Subject IDs are consistent with prior studies from the patient population tested in the University of Iowa (e.g., (Tsuchiya et al., 2008; Kawasaki et al., 2012; Haun et al., 2016)).

Unmasked Visual Task

Subjects were presented with images from different categories, presented at fixation for a duration of 500 ms (Fig. 1B). The stimuli used in each experiment, the behavioral response required and the number of sessions and trials for each subject are reported in Table 2.

Table 2.

Stimuli, tasks and number of trials in each class for each subject in the unmasked paradigm. Stimulus set A comprises 9 images per category for upright faces and inverted faces; 20 images for houses; 45 images for tools; 200 for Mondrian patterns. Set B comprises 15 natural images for each category: faces (including upper half body), animals, landmarks, vehicles, flowers. Set A comprises grayscale photographs of upright and inverted faces (Ekman faces (Ekman and Friesen, 1976)) and houses, line drawings (tools) and digitally synthesized color images (Mondrian patterns). Set B comprises natural color photographs, downloaded from the internet. Images were scaled to have the same approximate size and contrast on screen. Tasks: CD, Change Detection task on the fixation cross; OB, One-Back task for the stimulus category. In each trial, subjects reported a change in the fixation cross (in the CD task) or a repetition of stimulus category (in the OB task) in a time window of duration 0.5 s or 1 s, respectively, plus a random interval uniformly distributed in [0,500] ms, immediately following stimulus offset. Following the termination of the response period, the next trial began immediately in CD sessions, or after the presentation of a feedback screen in OB sessions, which lasted 500 ms.

sub ID	Stimuli	# sessions (CD, OB)	U Face vs. Other	U vs. I Face
1471	Face, Checkerboard	2	320/80
153	Set A	2,2	118/455	118/113
154a	Set A	1,1	61/239	61/56
154b	Set A	1,1	64/236	64/57
154	Set A	2,2	125/475	125/113
168	Set A	2,0	41/159	41/43
178	Set B	2,2	100/400

¹this subject performed gender and emotion discrimination tasks as described in (Tsuchiya et al., 2008); the number of trials indicated refer to a “face” vs. “checkerboard” decoding analysis. In the case of subject 154, the set of recorded electrodes differed across sessions: “154a” and “154b” indicate the sets of electrodes that were recorded in only a subset of the sessions (shown in Fig. S5), while “154” indicates the set of electrodes that were recorded in every session.

Data Analysis

Behavioral Analysis

For each subject and face contrast value, we calculated the objective performance (defined as the ratio of the number of correct trials over the total number of trials) in the 2IFC task and counted the number of trials corresponding to each visibility rating. As expected, increasing face contrast values generally corresponds to an improvement in both objective performance and subjective visibility rating (Fig. 2A-B). Please note that the physical face contrast values used differed across subjects, since we aimed to obtain an objective performance around 75% at the intermediate contrast level for each subject.

Figure 2. CFS behavioral results.

(A) Objective performance (probability of correct response) at each face contrast value for each subject. (B) Mean visibility rating at each face contrast value for each subject. In A and B, filled circles indicate threshold face contrast c_thr, downward triangles indicate low contrast c_L, upward triangles indicate high contrast c_H. Downward triangles indicating c_L coincide for subjects 147, 153 and 168 in panel A. (C) d′ (circles) and meta-d′ (diamonds) at each face contrast value for each subject. (D) Objective performance at each visibility rating for each subject. For each subject, only visibility ratings that were reported in at least 8 trials are shown. Filled circles indicate threshold visibility rating v_thr, downward triangles indicate low visibility v_L, upward triangles indicate high visibility v_H. (E) Number of trials for each visibility rating for each subject. (F) Number of trials for each visibility rating at threshold face contrast for each subject. The number of trials for each visibility rating at each face contrast value for each subject is reported in Fig. S2. White horizontal lines in E, F indicate median splits.

We measured the degree to which visibility ratings were predictive of objective performance (i.e., metacognition, or the ability to introspect on the accuracy of one's own perceptual judgements) using a recently introduced measure from signal detection theory known as meta-d′ (Maniscalco and Lau, 2012; Barrett et al., 2013). In most cases, meta-d′ has the same sign and, often, similar amplitude as d′, indicating that high visibility ratings are predictive of correct objective performance, and vice versa, to an extent which is compatible with objective performance and visibility rating originating from a common (or largely redundant) internal signal (Fig. 2C, see also Fig. 2D for a similar analysis performed collapsing across contrast values). This relationship between visibility rating and performance justifies our treatment of visibility ratings as useful representations of subjects' perceptual states. We note, however, that this relationship did not hold for subjects 168 and 178 at the highest contrast value we investigated, suggesting the possibility of a metacognitive impairment or poor understanding/execution of the task for these subjects (see also Discussion).

In this work, we aim to assess the neural correlates of subjective conscious perception, hence we compared trials that resulted in different perceptual outcome as reported by subjects, i.e. different visibility rating. In order to increase sample size and to compensate for individual biases in visibility ratings, we grouped trials into a high visibility and a low visibility class using a median split of the data (Fig. 2E,F). The median split was determined independently for each classification considered, for each subject and, in the case of one subject where different electrodes have different numbers of trials (subject 154), for each electrode. The numbers of trials for each visibility rating for subject 154 indicated in Fig. 2E,F (and the corresponding median splits) correspond to those electrodes that were recorded in every session (i.e. those electrodes with the maximum number of trials for this subject). The number of trials for each visibility rating for the sets of electrodes that were recorded in only a subset of the sessions are reported in Fig. S2.

In order to investigate the neural correlates of conscious visual awareness in the absence of changes in physical properties of the presented stimuli, we compared trials with the same face contrast value but different visibility rating. First, this analysis was conducted using face contrast values that resulted in the face image being detected in roughly 75% of the trials. As it is clear from Fig. 2B,D, different subjects adopted different criteria when declaring their degree of perceptual awareness: some subjects responded with high visibility ratings even when performing the task at chance level (e.g., subject 178), while others were much more conservative and responded with low visibility ratings even when performing the task with high accuracy (e.g., subject 147). To compensate for individual biases in visibility ratings, we adopted a definition of “threshold face contrast” based on the objective performance in the 2IFC task: threshold face contrast c_thr was defined for each subject as the lowest contrast value investigated that resulted in objective performance above 64% (indicated with filled circles in Fig. 2A; mean across subjects: 76%; range: 65% - 94%). Then, we also considered a low contrast c_L, defined as the highest contrast lower than c_thr that resulted in objective performance below 60% (mean across subjects: 54%; range: 53% - 59%); and a high contrast c_H, defined as the lowest contrast higher than c_thr that resulted in objective performance above 90% (this condition was only realized in three subjects; mean across subjects: 95%; range: 92% - 100%).

Analogously, we investigated the neural correlates of changes in physical contrast in the absence of changes in the reported visibility rating. To this end, we compared trials with the same visibility rating but different face contrast. This analysis was conducted using trials with “threshold visibility rating” v_thr, defined for each subject as the lowest rating with sufficient number of trials (see subsection “ECoG decoding analyses” for details) that resulted in objective performance above 64% (only realized in four subjects; indicated with filled circles in Fig. 2D; mean across subjects: 79%; range: 72% - 87%). We also considered a low visibility rating v_L, defined as the highest rating lower than v_thr that resulted in objective performance below 60% (only realized in four subjects; mean across subjects: 55%; range: 52% 58%); and a high visibility rating v_H, defined as the lowest rating higher than v_thr that resulted in objective performance above 90% (only realized in two subjects; mean across subjects: 95%; range: 91% 100%).

Data preprocessing

ECoG signals were recorded with reference to the electrode placed under the scalp near the vertex of the skull. We bipolar re-referenced the original signals along the vertical and horizontal directions to remove low spatial frequency components and hence obtain a more localized signal and better exploit the fine spatial resolution of ECoG recordings. Bipolar channels were anatomically assigned to a given brain area (e.g., the fusiform gyrus) if at least one of the constituent unipolar electrodes was located in that area, according to the DKT brain atlas. We removed 60 Hz line noise from the photodiode trace using a linear combination of sinusoids estimated using the MATLAB function rmlinesmovingwinc.m (included in the Chronux data analysis toolbox (Mitra and Bokil, 2007), http://chronux.org). Onset times for the two intervals in each trial were estimated as threshold-crossing times of the de-noised photodiode traces. Then, the onset times were used to segment the data in time windows comprising [-500,1500] ms relative to interval onset.

ECoG spectrogram analysis

We used the Chronux data analysis toolbox to estimate the spectrograms of the bipolar ECoG signals using a multi-taper method (Mitra and Bokil, 2007). We used 3 tapers and a time window of 100 ms (which corresponds to a half bandwidth of 20 Hz), slided in steps of 50 ms. To improve visualization and yield a distribution that is closer to normal, the logarithm of the power spectrum was considered for plotting and subsequent analyses. Other transformations (e.g., cubic root) were also considered and yielded comparable results.

ECoG decoding analyses

We estimated the amount of information conveyed by neural signals using binary Regularized Least-Square Classifiers (RLSC, (Rifkin et al., 2003)) with regularization parameter λ = 10⁶. Regularized Least-Square Classification is a machine learning technique that estimates the linear separability between patterns according to their class. Here, we aim to assess the amount of information conveyed by a spectrotemporal representation of the ECoG signal in each trial about the presented physical stimulus or the reported phenomenal experience. In particular, we considered log power at 10×11 (time,frequency) points for each trial, sampled from a uniform grid in the interval [100,600] ms after stimulus onset × [0,200] Hz, as the input to the classifiers (Tsuchiya et al., 2008). This decoding scheme is a way of quantifying the information that neural signals convey about physical properties of stimuli or perceptual states, and it does not imply or suggest any mechanisms by which these spectrotemporal patterns would be read-out by other brain regions or influence behavior.

A set of weights that optimally separate trials according to their class is determined using a subset of the available trials, denoted as training set. The performance of the classifier is defined using a different set of trials, denoted as test set, as the area under the Receiver Operating Characteristic (ROC) curve, which we refer to as A′ (A prime). We report the average A′ values over N_iter cross-validations. In each cross-validation, we randomly chose a set of 0.7 × min(N₁, N₂) (rounded to the nearest integer) trials of each class as the training set, where N₁ and N₂ are the number of trials in class 1 and 2, respectively. As the test set, we chose min(N₁, N₂) – round(0.7 × min(N₁, N₂)) trials of each class among those that are not already included in the training set. Before being fed to the classifier, inputs were z-transformed: the mean and standard deviation of log power at each time-frequency point in the training set was calculated, and used to transform both training and test sets. Then, optimal RLSC weights were estimated using training trials, and their capacity to separate test trials according to their class was measured as the area under the ROC curve (A′). The number of cross-validations N_iter was set to 100 for all the decoding analyses, except v@c and c@v decoding analyses, where N_iter = 1000 was used in order to decrease the greater sampling variability that results from decoding analyses on smaller samples.

Significance of A′ values was estimated via a permutation-based statistics. For each classification considered, the class labels were randomly shuffled. Then, the average A′ value over N_iter realizations of training and test sets was calculated as described above. This procedure was repeated N_perm=1000 times, yielding a probability distribution of average A′ values corresponding to the null hypothesis of lack of linear separability between the two classes. An empirical average A′ value was considered significant at level p if it exceeded the p-percentile of the corresponding null distribution (p=0.05, p=0.01, p=0.001).

Significance thresholds at p=0.05 and p=0.01 were estimated separately for each classification considered. In order to improve the estimation of the significance threshold at p=0.001, null A′ values were pooled across electrodes, and the corresponding significance threshold was calculated from the resulting null distribution. For each subject and electrode, each analysis was only considered if at least 10 trials were available in the least populated class.

We did not apply any statistic control for multiple comparisons for the decoding accuracies shown for each face-responsive electrode on the anatomical maps. We aim to characterize both the most informative electrodes, which display high and very significant decoding accuracies, as well as the distributed patterns of informative responses observed over many electrodes, which display low but significant decoding accuracies, and tend to be arranged in clusters, suggesting that at least some of them are unlikely to arise by chance. As opposed to other neuroimaging recording techniques such as fMRI, EEG or MEG, ECoG does not sample neural activity in a spatially homogeneous manner. In addition, adjacent electrodes can record fairly different signals, a consequence of the high spatial resolution of ECoG recordings and the very small contribution of volume conduction effects. Since the spatial density of electrodes varies across brain region and across participants, and correlations between adjacent electrodes can be very low, especially across certain anatomical boundaries, cluster-based statistical control for multiple comparisons is problematic.

Uncorrected and FDR-corrected (over the number of tested electrodes, using the Benjamini-Hochberg procedure (Benjamini and Hochberg, 1995)) p-value thresholds are reported for the cumulative probability density estimates of decoding accuracies (Fig. 6C and D).

Figure 6. Subjective visibility can be decoded better than physical contrast.

(A) Decoding accuracies A′ for a selected group of electrodes from an example subject are shown color-coded on the ventral (top) and lateral (bottom) brain images. A′_c are indicated in the left half of the bisected disks, A′_v in the right half. Only decoding accuracies that are significant at p<0.05 (uncorrected for multiple comparisons) are shown, with larger symbol size indicating higher significance. Brain images show the areas that are enlarged in the main panels. Summary disks indicate the average A′ among the face-responsive electrodes in the selected area. (B) As in A, but with A′_c@v indicated in the left sectors of the pie charts, A′_v@c in the right sectors, as shown in the legend. (C), (D): Cumulative probability density functions of decoding accuracy over the populations of ventral and lateral electrodes for an example subject (C), and pooling over all subjects (D). Only results from decoding analyses with at least 10 trials in the least populated class are shown. Symbols indicate threshold A′ at several significance levels for each classification considered, obtained by permutation. Symbols are only shown if there is at least one electrode that is significant for the corresponding decoding analysis and significance level. Triangles: p=0.01; Diamonds: p=0.001; Stars: p=0.0001; Hexagons: p=0.00001. The FDR-corrected (over the number of tested electrodes and decoding analyses) p-value thresholds for these 4 levels of significance are the following: C, left panel: 0.023, 0.003, 0.0003, 0.00003; C, middle and right panels: 0.08 (n.s.), 0.012, 0.002, 0.0003; D, left panel: 0.036, 0.007, 0.001, 0.0002; D, middle and right panels: 0.2 (n.s.), 0.051, 0.014, 0.003. Single-subject cumulative probability density functions of decoding accuracy for the remaining subjects are shown in Fig. S6.

Face responsiveness

Our purpose is to identify brain loci that are part of the neural correlates of conscious face perception, hence we restricted our analysis to electrodes that are responsive to unmasked faces. Face responsiveness was defined by comparing the post-stimulus interval (comprising [100,300] ms after stimulus onset) of upright face trials in the unmasked visual task with the pre-stimulus interval (comprising [-200,0] ms relative to stimulus onset) of trials from any category in the same task. The linear separability between these two sets of trials was estimated using RLSC over spectrotemporal representations of the ECoG signals as described above, considering log power at 4×11 (time,frequency) points for each trial, sampled from a uniform grid in the interval [100,300] ms (for the post-stimulus set) or [-200,0] ms (for the pre-stimulus set) relative to stimulus onset × [0,200] Hz, as the input to the classifiers. An electrode was considered to be face-responsive if its decoding accuracy A′ was significant at p<0.01. The purpose of this electrode pre-selection is to collect those electrodes that show (not necessarily specific) responses to upright faces, which we then characterize in detail with respect to the information they bear regarding subjective visibility (using the masked data) and object category (using the unmasked data), since we aim to characterize both the most informative electrodes as well as the distributed patterns of informative responses observed over many electrodes.

Comparison between different decoding analyses

In this article, we consider several decoding analyses on neural activity that either contrast an upright face with an inverted face or a non-face stimulus (in both masked and unmasked tasks), or a more visible face with a less visible face (in the masked task). We hypothesized that the brain loci that are responsible for the generation of conscious experiences of upright faces would exhibit similar levels of discriminability across these different decoding analyses. In order to assess the degree of similarity between different decoding analyses, we computed the Pearson correlation ρ between A′ values for every pair of decoding analyses over face-responsive electrodes, separately for those implanted in the ventral and lateral temporal cortex.

For each pair of decoding analyses and for each brain region, we performed two different statistical tests. First, we tested whether the correlation was significant (against the null hypothesis of nonsignificant correlation) using a permutation-based method: a null distribution of correlation values was constructed by shuffling electrode identity independently for each decoding analysis, calculating the resulting correlation among A′ values and repeating this procedure N_perm=1000 times. An empirical correlation value was considered to be significantly positive (negative) at a significance level p (p=0.05, 0.01, 0.001) if it exceeded (preceded) the 1-p (p) percentile of the corresponding null distribution.

Second, we tested for a significant effect of region label (against the null hypothesis of no effect of region label, i.e., ventral or lateral), again using a permutation-based method. A null distribution of pairwise correlations was constructed by iterating N_perm=1000 times the following procedure: we randomly chose N_x (x=ventral, lateral) of (A′_i,A′_j) pairs from the pooled set of (A′_i,A′_j) pairs (comprising both ventral and lateral electrodes), where N_x is the number of (A′_i,A′_j) pairs for region x, and the corresponding Pearson correlation was computed. Then, each correlation coefficient was considered to be higher (lower) than expected by chance (that is, if region labels were irrelevant) at a significance level p (p=0.05, 0.01, 0.001) if it exceeded (preceded) the 1-p (p) percentile of the corresponding null distribution.

In order to visualize the patterns of similarity between different decoding analyses, we performed multidimensional scaling (MDS) on the correlation tables, using D = 1 – ρ as a measure of dissimilarity between pairs of decoding analyses. MDS enables one to represent the original, high-dimensional data (corresponding to one dimension for each decoding analyses considered) in a lower dimensional space (here, two-dimensional) while approximately conserving the relative distances (here, similar patterns of decoding accuracy across electrodes) between data points (Cox and Cox, 2000).

Results

Neural responses to objective and subjective attributes of visual stimuli

Our electrophysiological data set comprised 1071 bipolar channels (187, 171, 219, 228, 266 from subject 147, 153, 154, 168, 178) from the ventral and lateral temporal cortex of 5 subjects. Out of these, 271 channels were face-responsive (82, 57, 53, 53, 26 from subject 147, 153, 154, 168, 178, see subsection “Face responsiveness” in Methods for the definition of face responsiveness).

A subset of face-responsive channels exhibited spectral power responses that differed between categories in the unmasked task, and between intervals that contained a face image and those that did not in the CFS task. Most of these channels exhibited spectral power responses to CFS face intervals that were modulated by both physical aspects of presented stimuli (i.e. the contrast of a target face), as well as by subjective perception of those same stimuli (i.e. the visibility rating), as in the case of the example electrode in the fusiform gyrus shown in Fig. 3.

Figure 3. Spectral power responses in the unmasked and CFS experiment for an example electrode in the fusiform gyrus.

(A) Location of the example electrode. (B) Spectral power responses to some stimulus categories in the unmasked task. (a) Average spectrograms for Mondrian patterns (left), inverted faces (middle) and upright faces (right). (b) Band-limited power (BLP) signals obtained by averaging the corresponding spectrograms in a) over the frequency range delimited by white dashed lines. (C) Spectral power responses in the CFS task. (a) Average spectrogram in “blank” intervals. (b) [d]: Average spectrograms in “face” intervals with different face contrast values [visibility ratings], increasing along columns [rows]. (c) [g]: BLP signals obtained by averaging the corresponding spectrograms in b [d] over the frequency range delimited by white dashed lines. (e) Average spectrograms in “face” intervals corresponding to a given face contrast value (c, varying over columns) and visibility rating response (v, varying over rows). Only (c,v) pairs resulting in at least 5 trials are shown. (f) [h]: BLP signals obtained by averaging the corresponding spectrograms in e) over the frequency range delimited by white dashed lines. BLP signals with the same visibility rating [face contrast value] but different face contrast value [visibility rating] are shown in the same panel, with visibility [contrast] increasing along rows [columns]. (i) Decoding accuracy A′ for some decoding analyses. C3vB: c=3 trials vs. blank trials. c: c=1 vs. c=2,3. v: v=1,2 vs. v=3,4. c@v2: (c=1,v=2) vs. (c=2,3,v=2). v@c2: (c=2,v=1,2) vs. (c=2,v=3,4). Cyan bars indicate significant decoding accuracy (p<0.001). Blank bars indicate non-significant decoding accuracy (p>0.05). The number of trials averaged for each condition is indicated on top of each spectrogram. Dash blue rectangles indicate the time-frequency region that has been considered for the decoding analyses. Differences in log power with respect to a pre-stimulus baseline ([-500,0] ms) are shown for the ease of visualization. The green dash vertical lines indicate interval onset. Shaded areas in the BLP signals indicate s.e.m. across trials.

For this electrode, the presentation of an upright face evoked a stronger response than the presentation of an inverted face, which in turn evoked a stronger response than the presentation of a Mondrian pattern in the unmasked task (Fig. 3B). The spectral response exhibited an increase in power located mostly in the high-gamma band (70-150 Hz) and in the alpha-beta band (4-40 Hz). In the CFS task, increasing face contrast values resulted in increases in power mostly in the high-gamma band (70-150 Hz) and in the time window from 200 to 500 ms after stimulus onset, with a complex spectral power response in the alpha-beta band characterized by an early attenuation in power decrease followed by a late increase (Fig. 3C;b,c). A very similar pattern of spectral responses could also be observed for increasing visibility ratings (Fig. 3C;d,g), which prompts the question of whether spectral changes due to increasing contrast can be explained, and to which degree, by changes due to increasing visibility rating.

Indeed, averaging spectral responses separately for each pair of contrast value c and visibility rating v shows that spectral changes due to increased contrast at a fixed visibility rating are negligible, while spectral changes due to increased visibility rating at a fixed contrast value can still reliably distinguish different visibility ratings (Fig. 3C;e,f,h,i). This result suggests that, for this electrode, the pattern of spectral changes in response to increasing contrast values can be explained almost completely by changes in visibility ratings, i.e. by changes in subjective perceptual state.

While the electrode shown in Fig. 3 exhibited spectral power responses in both the high-gamma and the alpha-beta bands, other electrodes exhibited different spectral responses. For example, one electrode from the inferior parietal cortex of the same subject exhibited spectral responses to face images of increasing contrast that were almost completely confined to the high-gamma band (Fig. S3), while another electrode from the fusiform gyrus of another subject showed mostly spectral power increases in the alpha-beta band (Fig. S4). The variability in the relative information content across frequency bands that we observed, even within the same cortical location, is consistent with previous studies of visual ECoG responses (e.g., (Vidal et al., 2010)).

In the present study, we aimed to assess neural responses that are informative of the presented stimuli or the resulting perceptual state regardless of the spectral range in which they are observed. Hence, in the next section, we will present a systematic quantification of the information about stimuli or perceptual states using a multivariate decoding analysis that combines spectral power sampled from a uniform grid in the interval [100,600] ms after stimulus onset × [0,200] Hz, indicated by a blue dash rectangle in the spectrograms of Fig. 3.

Neural correlates of visual consciousness

In order to investigate the neural correlates of objective physical stimulation, and compare them with the neural correlates of subjective phenomenal experience, we performed a set of decoding analyses for each face-responsive electrode. In particular, we evaluated the linear separability, as measured by the performance of a set of binary RLSC classifiers (see Methods for details), between pairs of subsets of CFS intervals. We considered the following contrasts: i) face intervals with face contrast level equals three vs. blank intervals (c₃ vs. blank); ii) high visibility face intervals vs. low visibility face intervals, grouping across all contrast levels (v); and iii) high visibility face intervals vs. low visibility face intervals, considering only trials with a fixed level of face contrast, namely the “threshold face constrast” as defined in section “Behavioral Analysis”, corresponding to intermediate objective performance (v@c₂ or v@c₃ depending on the subject), which we denote as c_thr.

These three contrasts constitute a gradual shift from a criterion defined exclusively by extrinsic factors (c₃ vs. blank), to a criterion defined exclusively by the intrinsic, subjective phenomenal experience, in the absence of any change in the physical property of the stimulus (v@c_thr).

The results from these three decoding analyses for an example subject are shown in Fig. 4A. The comparison between a decoding analysis specified by extrinsic factors (c₃ vs. blank) and one specified by subjective perception (v) reveals a great degree of overlap between the brain areas that discriminate these two pairs of conditions: the electrodes that differentiate high contrast face intervals from blank intervals are the same ones that differentiate clearly seen face intervals from poorly seen or unseen face intervals. Plotting A′_c3vs.blank versus A′_v reveals a strong correlation between the two decoding analyses, with stronger correlation observed among the most discriminant electrodes (Fig. 4B, top panel). These results are not trivial, since the c₃ vs. blank decoding analysis compares face vs. blank trials, while all trials included in the v analysis are face trials. Hence, these results suggest that the high-level visual areas considered in this study follow subjective visibility, rather than physical aspects of the stimuli, given that they similarly distinguish face from blank trials on the one hand, and clearly seen vs. poorly seen face trials on the other hand.

Figure 4. Distilling the neural correlates of conscious visual perception: from physical attributes of stimuli to subjective phenomenology.

(A) Decoding accuracies A′ for each electrode in the example subject 153 are shown color-coded on the ventral (left) and lateral (right) brain images. The decoding analyses shown progress from a contrast specified by physical stimuli (c₃ vs. blank, top row), to a contrast defined by subjective phenomenology, but contaminated by external factors (i.e., face contrast, v, middle row), to a purely subjective contrast (v@c₂, bottom row). Only decoding accuracies that are significant at p<0.01 (uncorrected for multiple comparisons) are shown. Face-responsive ventral electrodes that are located in the fusiform gyrus are indicated with blue contours in the top left panel. Dark gray indicates non face-responsive electrodes; light gray indicates electrodes that are face-responsive but do not exhibit significant decoding accuracy. Decoding accuracies A′ for each electrode and for each remaining subject are shown color-coded on the anatomical images in Fig. S5. (B) Relationships between pairs of decoding analyses for each electrode that exhibits a significant A′ in either one of the two decoding analyses. There is a strong correlation between the decoding accuracy for c₃ vs. blank and v, and between v and v@c₂. v_sub indicates the decoding accuracy obtained when decoding visibility using the same number of trials for training and test (for each round of cross-validation) as when decoding v@c₂. The vertical (horizontal) black line indicates the p=0.01 significance threshold for the decoding analysis corresponding to the x (y) axis, averaged over electrodes. The vertical (horizontal) gray line indicates the p=0.001 significance threshold for the decoding analysis corresponding to the x (y) axis. Inset shows lateral brain image, with the area enlarged in the main panels indicated with a rectangle. (C) Venn diagrams showing the number of electrodes that are significant (p<0.01) in one or more of the decoding analyses considered, separately for ventral (left) and lateral (right) electrodes. The sets of ventral electrodes that discriminate c₃ vs. blank and v (red and green circles) completely overlap for this subject. c₃ vs. blank, red; v, green; v@c₂, blue.

The comparison between otherwise unselected high and low visibility trials, even though defined on the basis of a purely perceptual category, will typically include the contribution of different levels of face contrast, since higher face contrast results in higher visibility ratings (Fig. 2B). Hence, a further distillment of the neural correlates of conscious face perception can be achieved by contrasting high vs. low visibility trials at threshold contrast c_thr. This analysis reveals a smaller set of electrodes that is almost completely included in the set specified by the v decoding analysis. In particular, the best v@c_thr discriminant electrodes correspond with those that best discriminate v (Fig. 4B, middle panel). In order to quantify the effect of the different number of trials that enter the v and v@c_thr analyses, we performed a decoding analysis discriminating visibility using the same number of trials for training and test (for each round of cross-validation) as when decoding v@c₂ (v_sub for subsample). The results are very similar as when decoding visibility without subsampling (Fig. 4B, bottom panel), suggesting that the decrease in decoding accuracy generally observed when discriminating v@c_thr (with respect to discriminating v) is a genuine result of the decreased discriminability between the two classes, and is only minimally affected by the different number of trials. We expected lower decoding accuracies in the v@c_thr analysis as compared to the v analysis, since it is reasonable to expect a larger visibility-related modulation of neural activity when the stimulus contrast varies from completely invisible to clearly visible (that is, from c=1 to c=3) than when the stimulus contrast is fixed at an intermediate value and variability in subjective phenomenology is only determined by fluctuations in the subjects' state. For the subject depicted in Fig. 4, the degree of overlap between the sets of discriminant electrodes specified by these three decoding analyses (c₃ vs. blank, v and v@c₂) is very high for both ventral and lateral electrodes, albeit higher for ventral electrodes (Fig. 4C).

The whole subject population exhibited a similar trend (Fig. S5 and 5A). In particular, 3/5 subjects presented a cluster of c₃ vs. blank and v discriminant electrodes in the fusiform gyrus, with some discriminant electrodes located in an adjacent area in subject 153 (inferior-temporal cortex). A cluster of c₃ vs. blank and v discriminant electrodes has been observed in the lateral-temporal/inferior-parietal cortex (close to the STS and possibly recording activity from this area) in 4/5 subjects (superior-temporal for subject 147, inferior-parietal for subject 153, between inferior-temporal and middle-temporal for subject 154, middle-temporal for subject 168). Subject 147 additionally presented a cluster of c₃ vs. blank and v discriminant electrodes in the lateral-occipital cortex, which exhibited the highest decoding accuracy for this subject. He is the only subject with multiple electrodes located in this region (6 unipolar electrodes). In addition, subjects 154 and 178 presented one unipolar electrode each in the lateral-occipital cortex at the border of the lateral-temporal grid, which exhibited either weak (for subject 154, low but significant discriminability for c₃ vs. blank) or no discriminability (for subject 178), possibly also because the most face-discriminant loci of the lateral-occipital cortex were not appropriately sampled in these subjects.

Figure 5. Distilling the neural correlates of conscious visual perception: from physical attributes of stimuli to subjective phenomenology.

(A) As in Fig. 4B, pooling electrodes across subjects. (B) As in Fig. 4C, pooling electrodes across subjects.

The great majority of v@c_thr discriminant electrodes sites were observed in subject 153 (see also Discussion and Fig. S8). It is worth noting that the number of trials that enter a v@c_thr analysis is much smaller than the number of trials that enter a v analysis for a given subject, given that only about a third of the trials were presented at threshold contrast. A lower number of trials results in weaker statistical power, as reflected by the significance thresholds shown in Fig. 4B. Hence, it is not surprising that most of the v@c_thr discriminant electrodes are observed in a single subject (subject 153) that both exhibited good metacognition and was tested with a higher number of trials than other subjects. By combining a powerful but generic visibility analysis (v decoding) with a less powerful but more specific visibility analysis (v@c_thr decoding), we could provide a global picture of the neural correlates of conscious face perception in a heterogeneous population of 5 subjects, illustrating both weak and distributed effects that are consistent across most subjects (Fig. S5), as well as strong and specific effects that have been observed in a single subject (Fig. 4A). Importantly, the strong correlation between decoding accuracies among the best discriminant electrodes in the c₃ vs. blank and v decoding analyses, and in the v and v@c_thr decoding analyses, was conserved when electrodes were pooled across subjects (Fig. 5A). The substantial degree of overlap between the three sets of discriminant electrodes, with higher overlap for ventral electrodes, was also conserved (Fig. 5B). We will quantify the degree of similarity between different decoding analyses further below (“Comparison between different decoding analyses”).

To further characterize the modulation of neural responses attributable to physical attributes of stimuli (i.e., face contrast), and compare it with the modulation attributable to subjective visibility, we also performed a decoding analysis that contrasted low contrast trials vs. high contrast trials (c decoding), after grouping adjacent contrast levels using a median split of the data.

All subjects, and in both ventral and lateral cortices, presented several electrodes that reliably differentiated low vs. high contrast trials (c decoding, shown in Fig. 6A for the example subject 153, left half of the bisected disks), as well as low vs. high visibility trials (v decoding, Fig. 6A, right half of the bisected disks). Decoding accuracy is generally higher for decoding visibility than contrast. In the case of the example subject 153, the cumulative probability density of decoding accuracy among face-responsive electrodes is consistently lower for v than for c in both ventral and lateral electrodes (p<3.2 · 10⁻⁴ in both cases, Kolmogorov-Smirnov test), which corresponds to a probability density shifted tothe right (that is, towards higher decoding accuracies) when decoding v in comparison with decoding c(Fig. 6C, left panel). More importantly, the highest decoding accuracy is obtained when decoding visibility, rather than contrast (max(A′_v)=0.917 among ventral electrodes, max(A′_v)=0.911 among lateralelectrodes; max(A′_c)=0.825 among ventral electrodes, max(A′_c)=0.82 among lateral electrodes). Also, 93.75% (19.51%) of face-responsive ventral (lateral) electrodes have a v decoding accuracy that is significant at p<10⁻⁵, as compared to 56.25% (9.76%) of face-responsive ventral (lateral) electrodes that have a c decoding accuracy with the same level of significance (Fig. 6C, left panel; note that similar results hold if considering less conservative significance thresholds). For this subject, decoding accuracy among ventral electrodes is higher than among lateral electrodes for both c and v decoding (p<1.9 · 10⁻¹³ in both cases, Kolmogorov-Smirnov test), although the highest accuracy observed in each region is similar.

The cumulative probability density functions of decoding accuracies for the remaining subjects are also consistent with the hypothesis that decoding visibility results in higher accuracy than decoding contrast (Fig. S6). In particular, the cumulative probability density of decoding accuracies among ventral electrodes is significantly lower for v than for c in 3/5 subjects (p<0.007, Kolmogorov-Smirnov test), yielding non-significant results in subject 168, while the relationship is inverted in subject 178 (i.e., the cumulative probability density of decoding accuracies is significantly lower for c than for v, p=2.8 · 10⁻⁴, Kolmogorov-Smirnov test). Note, however, that subject 178 only presents one ventral electrode that is marginally significant when decoding c (p<0.01), hence the cumulative probability densities of decoding accuracies among ventral electrodes for both c and v are composed mostly of non-significant A′ values for this subject (Fig. S6). Similarly, among lateral electrodes, the cumulative probability density of decoding accuracies is significantly lower for v than for c in 3/5 subjects (p<8.7 · 10⁻⁴, Kolmogorov-Smirnov test), yielding marginally significant results in the other two subjects (154 and 168, p=0.019and p=0.028, respectively).

Importantly, similar results are obtained when pooling electrodes across subjects: the cumulative probability density (cpdf) of decoding accuracy among face-responsive electrodes is consistently lower for v than for c in both brain areas, and it is consistently lower in ventral than lateral electrodes for both decoding analyses (Fig. 6D, left panel). These relationships reach statistical significance in the case of the v cpdf being lower than the c cpdf among lateral electrodes (p=4 · 10⁻¹², Kolmogorov-Smirnov test), and in the case of cpdfs being lower among ventral than among lateral electrodes for both c and v decoding (p=1.4 · 10⁻⁸ for the former, p=0.0068 for the latter, Kolmogorov-Smirnov test). The v cpdf is consistently lower than the c cpdf among ventral electrodes as well, but the relationship does not reach statistical significance (p=0.12, Kolmogorov-Smirnov test). Note, however, that the maximum difference between the v and the c cpdfs is observed for low, non-significant values of the decoding accuracy in the set of lateral electrodes, while it is observed for high and strongly significant values of the decoding accuracy in the set of ventral electrodes (for A′∼0.83, Fig. 6D, left panel). Also, 11.54% (7.25%) of face-responsive ventral (lateral) electrodes have a v decoding accuracy that is significant at p<10⁻⁵, as compared to 1.28% (1.55%) of face-responsive ventral (lateral) electrodes that have a c decoding accuracy with the same level of significance (Fig. 6D, left panel).

To further dissociate the modulation of neural responses attributable to face contrast and subjective visibility separately, we performed a set of more specific decoding analyses for each electrode. In particular, we calculated the linear separability A′ between low and high contrast trials corresponding to a fixed value of subjective visibility (c@v decoding, for each of the four visibility ratings), and the linear separability A′ between low and high visibility trials corresponding to a fixed face contrast value (v@c decoding, for each of the first three contrast values). In all cases, we performed binary classifications after grouping adjacent contrast values (for c@v decoding analyses) or visibility ratings (for v@c decoding analyses).

The results of these analyses for a set of visibility and contrast discriminant electrodes from an example subject are shown in Fig. 6B. When considering a fixed value of subjective visibility, face contrast was no longer decodable, and decoding accuracies dropped to non-significant or marginally significant levels (c@v decoding, left sectors). Conversely, when considering a fixed value of face contrast high enough to enable above chance performance (c=3 or c=2), subjective visibility was still decodable with very high accuracy (v@c₃ and v@c₂, first two right sectors from the top).

In the example subject, the remarkably stronger modulation of neural activity due to visibility at fixed contrast, rather than due to contrast at fixed visibility, is also reflected at the electrode population level in the cumulative probability density functions (cpdfs) of decoding accuracies for ventral and lateral electrodes shown in Fig. 6C (middle and right panels). While several electrodes in both ventral and lateral areas display decoding accuracies that are significant at p<10⁻⁵ for both v@c₂ and v@c₃, v@c₁ and c@v decoding analyses only result in at most one electrode that is significant at 10⁻³<p<10⁻², as expected by chance. This trend is also observed when pooling electrodes across the population of subjects (Fig. 6D, middle and right panels). However, the accuracy of c@v_thr and c@v_H decoding is also high, especially among lateral electrodes, mostly driven by the contribution of c@v₄ from subjects 154, 168 and 178 (Fig. S6, see also Discussion).

It is worth noting that subjects 154, 168 and 178 were overconfident in their visibility ratings. In particular, their performance when reporting the highest visibility rating v=4 was not greater than 91% (91%, 87% and 72%, for subject 154, 168 and 178), as opposed to 100% for subjects 147 and 153 (Fig. 2D). Hence, it is possible that they responded with the highest visibility rating even if their perception of the face was not completely clear. If this were the case, the high accuracy in c@v₄ decoding they exhibit might reflect different degrees of subjective face perception, which is expected to covary with face contrast in v=4 trials in overconfident subjects. In accordance with this interpretation, most electrodes with significant accuracy in c@v₄ decoding also display significant v or v@c_thr decoding accuracy.

Overall, the convergence of the decoding results from different, but related analyses in the masked task strengthens the validity of our results and further suggests that neural activity in the high-level visual areas considered in this study is more closely related to subjective phenomenology than to physical aspects of the stimuli.

Distilling the neural correlates of visual consciousness by combining masked and unmasked presentation of similar images

The comparison between seen and unseen trials, even in conditions of equal physical stimulation, is not guaranteed to reveal the core neural correlates of a specific conscious perception. As pointed out previously (Aru et al., 2012b; de Graaf et al., 2012; Miller, 2007; van Boxtel and Tsuchiya, 2015; Tsuchiya et al., 2015), such comparison is also expected to expose neural activities that can facilitate or hinder the perceptual experience of a faint stimulus, as well as neural activities that are related to the motor act of perceptual report.

Hence, in addition to the masked paradigm, we also considered an unmasked paradigm, where images of faces and other categories were shown at the fixation point for 500 ms without presenting any competing stimuli or any perceptual mask, guaranteeing an effortless and vivid perception of the presented objects. As opposed to the masked paradigm, where identical stimuli could elicit different perceptual outcomes, the unmasked presentation is expected to result in a one-to-one correspondence between physical stimuli and perceptual states.

In order to identify the brain loci that discriminate between images of upright faces and other categories, we performed two decoding analyses for each electrode: i) a more generic analysis, contrasting upright face trials with trials where other categories were presented (FvO, 5 subjects), and ii) a more specific analysis, contrasting upright face trials with inverted face trials (FvI, 3 subjects).

The decoding accuracies resulting from these analyses, for the same electrodes shown in Fig. 6A, are color-coded in the two left quarters of the pie charts in Fig. 7A, with the more generic analysis (FvO) shown on top, and the more specific analysis (FvI) shown on bottom. The two right quarters show the decoding results from the masked paradigm for comparison, with a more generic analysis (v) shown on top, and a more specific analysis (v@c₂) shown on bottom.

Figure 7. Comparison between decoding visibility in the masked paradigm and decoding image category in the unmasked paradigm: both ventral and lateral areas can discriminate visibility, but only ventral areas can discriminate upright from inverted faces.

(A) Decoding accuracies A′ for a selected group of electrodes from the example subject 153 (the same as shown in Fig. 6A,B) are shown color-coded on the ventral (top) and lateral (bottom) brain images. Accuracies in upright face decoding using the unmasked protocol are shown in the left quadrants (top: upright face vs. other categories; bottom: upright vs. inverted face), accuracies in visibility decoding using the masked protocol are shown in right quadrants (top: v; bottom: v@c₂), as indicated in the legend. Only decoding accuracies that are significant at p<0.05 (uncorrected for multiple comparisons) are shown, with larger symbol size indicating higher significance. Brain images show the areas that are enlarged in the main panels. Summary disks indicate the average A′ among the face-responsive electrodes in the selected area. (B) v decoding accuracies in the masked paradigm are plotted against the corresponding decoding accuracies when discriminating upright faces in the unmasked paradigm for each electrode from the example subject that is significant at p<0.01 in at least one of the decoding analyses for each pair of decoding analyses. A more generic unmasked decoding analysis is shown in the left panel (upright faces vs. other categories), a more specific in the right panel (upright vs. inverted faces). The vertical (horizontal) black [gray] line indicates the p=0.01 [0.001] significance threshold for the decoding analysis corresponding to the x (y) axis, averaged over electrodes.

While both ventral and lateral regions can discriminate upright faces versus other categories (top-left quadrant), with higher accuracy in ventral than in lateral regions, only ventral loci can discriminate between upright and inverted faces (bottom-left quadrant).

This remarkable difference is also evident at the electrode population level when comparing decoding accuracies in the masked paradigm (v and v@c_thr) with those in the unmasked paradigm (FvO and FvI) (Fig. 7B and Fig. 8). While both ventral and lateral electrodes can discriminate visibility (v) in the masked paradigm (even when the decoding analysis is restricted to the threshold contrast (v@c_thr), see Fig. 4, 5 and 6), as well as upright face vs. other categories in the unmasked paradigm (with greater accuracy in ventral than lateral regions), only ventral electrodes can discriminate upright vs. inverted faces.

Figure 8. Comparison between decoding visibility in the masked paradigm and decoding image category in the unmasked paradigm across all subjects.

As in Fig. 7B, pooling electrodes across subjects. While both ventral and lateral regions can discriminate visibility in the masked paradigm and upright faces from other categories in the unmasked paradigm, only ventral areas can discriminate upright from inverted faces.

The results of the decoding analyses across our subject sample are summarized as conjunction plots on a standard MNI brain in Fig. 9 (electrodes from all subjects) and in Fig S7 (separately for each subject). The color code indicates the set of condition pairs conjointly discriminated by each electrode: red electrodes discriminate c₃ vs. blank, v, FvO and FvI; green electrodes discriminate c₃ vs. blank, v, FvO but not FvI; blue electrodes discriminate c₃ vs. blank, v, but not FvO nor FvI. While responses that discriminate c₃ vs. blank, v and FvO are observed in both ventral and lateral regions, the only electrodes that discriminate c₃ vs. blank, v, FvO and FvI are located in the fusiform gyri.

It is worth noting that our results are not critically dependent on the precise definition of the generic face discriminability analysis. As control analyses, we consider two additional generic face discriminability analyses for those subjects that were administered with stimulus set A (subjects 153, 154 and 168): one that discriminates upright faces vs. inverted faces, houses and Mondrian patterns (FvIHM), and another one that discriminates upright faces vs. houses (FvH), for each face-responsive electrode. The results are similar to those obtained with the FvO analysis, and show high correlation between the decoding accuracies in either of these control analyses and FvO for both ventral and lateral electrodes (Pearson correlation coefficients ρ equal 0.96 [0.8] for FvIHM vs. FvO, 0.91 [0.5] for FvH vs. FvO, for ventral [lateral] electrodes, p<8 · 10⁻⁸ in all cases), especially among the most discriminating electrodes.

In all three subjects for which the upright vs. inverted face decoding analysis is possible, several ventral electrodes display high and very significant (p<0.001) decoding accuracy, and only one lateral electrode shows marginally significant decoding accuracy (0.001<p<0.01), as would be expected by chance considering the number of electrodes that comprise our data set. Ventral electrodes that reliably discriminate upright vs. inverted faces are observed in both right (subjects 153 and 154) and left (subject 168) fusiform gyri.

As shown in the average spectrograms and band-limited power signals in Fig. 3 and Fig. S3, upright faces often elicit the strongest response in both ventral and lateral electrodes. However, the great variability across trials in lateral electrodes prevents accurate discrimination between different categories, and critically, between upright and inverted faces, on a trial-by-trial basis. Conversely, ventral electrodes record stronger activations that are more reliable across trials of any given category, hence enabling accurate discrimination of upright faces.

When searching for the neural correlates of conscious perception, we assume that areas belonging to the NCC-core will exhibit activity that discriminates conditions corresponding to different perceptual experiences. While both upright and inverted faces are recognized as “faces”, the corresponding phenomenology is quite different. Thus, we expect brain areas corresponding to the NCC-core of upright faces to display neural activity that discriminate these two classes of stimuli accordingly. The high discriminability for high versus low visibility trials observed in some ventral electrodes (which is preserved even when only trials at threshold contrast are considered), and most prominently in those located in the fusiform gyrus, together with the high discriminability between upright versus inverted faces observed in those same electrodes, suggest that the corresponding brain loci are likely to be part of the core network that generates conscious experiences of upright faces.

Comparison between different decoding analyses

All the decoding analyses we considered in this work (with the exception of c@v) contrast a condition of conscious upright face perception versus a condition of less conscious upright face perception or non upright face perception. Thus, it is reasonable to hypothesize that the brain loci that are responsible for the generation of conscious experiences of upright faces would exhibit similar levels of discriminability across these different decoding analyses. Hence, we systematically compared 7 different decoding analyses (5 from the masked paradigm, 2 from the unmasked paradigm) by calculating the Pearson correlation ρ among every pair of decoding accuracies, for the set of ventral and lateral face-responsive electrodes separately (Fig. 10A).

Figure 10. Similarity structure across different decoding analyses.

Correlation analysis between A′ values obtained in different decoding analyses for ventral (top) or lateral (bottom) electrodes. (A) Pearson correlation coefficients for each pair of decoding analyses are shown in the upper right part of the table, p-values for the null hypothesis of no correlation are shown color-coded from dark green (p<0.05) to pale green (p<10⁻⁵). Gray corresponds to p>0.05. Correlation coefficients are printed in red (blue) if higher (lower) than what would be expected by chance if region labels were irrelevant, at p<0.05 (*: p<0.01, **: p<0.001, uncorrected for multiple comparisons). Entries in the lower left part of the table show the number of electrodes that enter the corresponding correlation analysis. Diagonal entries indicate the number of face-responsive electrodes for which the assessment of the corresponding decoding accuracy A′ was possible. Correlation analyses were performed separately for ventral (top) and lateral (bottom) electrodes. (B) MDS representations of the correlation tables shown in A. C3vB: c₃ vs. blank; FvO: upright face vs. other categories; FvI: upright vs. inverted face.

Consistently with the analyses shown in Fig. 4 and 5, we observed high and significant correlation between c₃ vs. blank and v decoding (0.88 and 0.58 among ventral and lateral electrodes, respectively) as well as between v and v@c_thr (0.52 and 0.36) in both ventral and lateral regions. As expected from the results shown in Fig. 7 and 8, we found rather low but significant correlation between decoding v in the masked paradigm and decoding image category in the unmasked paradigm in ventral regions (upright face vs. other categories FvO or upright vs. inverted face FvI: 0.30 and 0.31). However, the corresponding correlation among lateral electrodes was significant only for FvO (0.27), while for FvI it was equal to 0.02 and not significantly different from zero. In general, correlation coefficients between decoding analyses within the same paradigm (from 0.35 to 0.88 in ventral regions and from 0.21 to 0.67 in lateral regions) tended to be higher than those across different paradigms (from 0 to 0.66 in ventral regions and from -0.05 to 0.38 in lateral regions).

To assess if correlation coefficients were significantly affected by anatomical location (i.e., ventral versus lateral), we performed a set of permutation-based significance tests (see Methods for details). This second-order analysis shows that pairs of decoding analyses applied to the population of ventral electrodes often yield correlation coefficients that are significantly higher than what would be expected if the brain region was irrelevant (4 among 21 pairs tested, indicated with red text in Fig. 10A, top panel).

For example, the accuracy for decoding c (a discrimination specified in terms of a physical property of stimuli) is highly correlated with the accuracy for decoding v@c_H (a purely subjective discrimination) among the set of ventral electrodes (ρ=0.85, significant at p<10⁻⁵). Hence, ventral electrodes tend to exhibit similar decoding accuracy for the two discriminations and, in particular, ventral electrodes that discriminate c well also tend to discriminate well v@c_H, and vice versa. The same analysis applied to the set of lateral electrode yields a much lower, albeit highly significant, correlation (ρ=0.36, significant at p<10⁻⁵). Testing for a significant effect of region label reveals that the correlation among lateral electrodes is significantly lower than what would be expected by chance (i.e., if region labels were irrelevant, p<0.001, permutation test). Conversely, the correlation among ventral electrodes is significantly higher than chance (p<0.05), suggesting that decoding accuracies for c and v@c_H are more similar among the set of ventral electrodes. In addition, a significant positive effect of region label at p<0.001 has been observed among ventral electrodes in the comparison between c₃ vs. blank and v, and at p<0.05 between c₃ vs. blank and c.

Conversely, pairs of decoding analyses applied to the population of lateral electrodes generally yield correlation coefficients that are significantly lower than what would be expected by chance (17 among 21 pairs tested, indicated with blue text in Fig. 10A, bottom panel). In particular, a significant negative effect of region label at p<0.001 has been observed in the comparison between c₃ vs. blank and c, between c₃ vs. blank and v, between c₃ vs. blank and v@c_H, and, critically, in almost every pair comprising v@c_H or upright vs. inverted face (FvI). Hence, electrodes in ventral regions tend to show similar decoding accuracy between two different decoding analyses, while those in lateral regions tend to show more variable decoding accuracy depending on the specific type of decoding analysis performed.

This pattern of results could originate from a large population of lateral electrodes, out of which only a few are part of the core NCC network for face perception and most contribute with non-significant decoding accuracies, and a smaller population of ventral electrodes, out of which many or most are part of the core NCC network. In fact, similar second-order analyses performed on the set of electrodes that are significant in both decoding analyses at different levels of significance yielded correlation coefficients that still tend to be higher among ventral electrodes for most pairs of decoding analyses, but significant effects of region labels are sparser and not always consistent among regions. Future studies with larger samples will be needed to rule out this possibility with greater confidence. It is notable, however, that pairs of decoding analyses including a purely subjective analysis in the masked paradigm (v@c_thr and v@c_H) and a face discriminant analysis in the unmasked paradigm (upright face vs. other categories FvO or upright vs. inverted face FvI) still yield much higher correlation coefficients among ventral electrodes, which sometimes reach high levels of significance in the effect of region label (e.g., v@c_thr and FvO, and v@c_H and FvO, both yield significant correlation among ventral electrodes with a positive effect of region label which is significant at p<0.001 if only electrodes that are significant at p<0.01 in both analyses for each pair are considered). This suggests that ventral electrodes actually discriminate different, but similarly related, pairs of conditions in a more similar manner than lateral electrodes.

In order to provide an intuitive representation of the patterns of similarity between different decoding analyses, we performed a multidimensional scaling (MDS) analysis on the correlation tables, considering D = 1 – ρ as a measure of dissimilarity between pairs of decoding analyses (Fig. 10B). This analysis shows that c₃ vs. blank, c and v analyses are very similar to each other in both brain regions, with c₃ vs. blank and v particularly close for ventral electrodes. The two analyses of visibility at fixed contrast, v@c_thr and v@c_H, are also located in the same region of the MDS space, with v@c_H especially close to c among ventral electrodes. The two decoding analyses from the unmasked paradigm, upright face vs. other categories and upright vs. inverted face, are also similar to each other, especially among the set of ventral electrodes. The upright vs. inverted face decoding results are especially close to v@c_H among ventral electrodes.

Discussion

Traditionally, the closely related aspects of conscious face perception (e.g., contrasting a visible with an invisible face) and holistic face perception (e.g., contrasting an upright with an inverted face) have been assessed separately. However, the search for the neural correlates of consciousness consists in assessing the patterns of neural activation that covary with the phenomenal experience, regardless of the precise experimental protocol employed. By combining similar, but different, experimental protocols, each of which is tailored to expose specific aspects of the conscious visual experience of faces in the same subjects, we could provide a more refined description of the relative roles of key high-level visual areas in the generation of conscious face perception than what could have been possible on the basis of a masked protocol alone.

In particular, we presented a progressive distillment of the neural correlates of conscious face perception using ECoG recordings in humans during the presentation of both masked and unmasked images of faces and other categories. In the first part of the article, we focused on the masked paradigm and presented a series of decoding analyses that progressed from the identification of the brain loci that discriminate the presence versus the absence of face images (c₃ vs. blank decoding), to those that discriminate clearly seen vs. poorly seen or unseen faces regardless of face contrast (v decoding), to those that discriminate clearly seen vs. poorly seen or unseen faces in conditions of equal face contrast (v@c_thr and v@c_H). In particular, we identified brain areas in both ventral and lateral cortices that reliably differentiate seen versus unseen faces even in conditions of equal face contrast. We observed that subjective visibility is better decodable than physical contrast. Importantly, visibility is still decodable even when only trials at a fixed value of face contrast are considered. On the other hand, contrast values are in general not decodable when considering only trials with a given visibility rating.

In the second part of the article, we also considered a different, albeit related, experimental protocol, where images of faces and other categories were presented unmasked and could be perceived effortlessly. The comparison between the two protocols revealed a critical difference between ventral and lateral brain loci: whereas certain ventral electrodes located in the fusiform gyrus (FG) could reliably discriminate upright versus inverted faces, electrodes located in face responsive areas of the lateral-temporal/inferior-parietal cortex could only discriminate upright faces versus other categories, but lacked the information required to discriminate upright versus inverted faces.

To relate the different measures of neural discriminability in both masked and unmasked procols, we performed a correlation-based similarity analysis across 7 different decoding analyses (5 based on the masked paradigm, 2 based on the unmasked paradigm), each of which constitutes a contrast between an upright face or more conscious face perception versus a non upright face or less conscious face perception. This second-order analysis revealed greater similarity between different decoding analyses in ventral than in lateral regions. Since brain loci that are more closely related with the generation of conscious visual experiences of upright faces are expected to exhibit similar levels of discriminability across these different decoding analyses, this result is also consistent with ventral areas being more closely involved in conscious experiences of upright faces.

Relationship with previous studies

Taken together, our results suggest a prominent role for ventral areas of the brain, and in particular for the fusiform gyrus, in the generation of conscious face percepts. Our results are consistent with a broad body of literature that assessed the neural correlates of face perception using several methodologies, in both clinical and healthy populations.

Prosopagnosia, a neurological condition where the recognition of faces (and other stimuli that involve configural internal templates) is impaired, is associated with lesions in the fusiform gyri (Damasio et al., 1982). While most clinical cases are associated with bilateral damage, cases of prosopagnosia associated with lesions confined to the right hemisphere have also been reported, suggesting a prominent role of the right fusiform gyrus for normal face processing (Landis et al., 1986; De Renzi et al., 1994). However, more recent studies demonstrated the importance of a broader network for normal face perception (Atkinson and Adolphs, 2011), with prosopagnosia sometimes occurring in patients with intact right FG but lesions in the right inferior occipital cortex (Rossion et al., 2003; Schiltz et al., 2006).

A broad wealth of studies, mostly based on fMRI, support a dissociation between the STS and the FG, with the STS being involved mostly in processing changeable aspects of faces, such as emotional expressions, and the FG with more stable attributes of faces, such as identity (see, for example, (Andrews and Ewbank, 2004; Kanwisher and Yovel, 2006)). However, the results from several recent studies challenge this traditional view. It has been shown that information on emotional expression is also present in the FG, and it is actually better decodable from ECoG electrodes located in this area rather than in the STS (Tsuchiya et al., 2008). Also, some studies reported that face identity is better (Kriegeskorte et al., 2007) or equally well decodable (Nestor et al., 2011; Anzellotti et al., 2014) from anterior regions of the inferotemporal cortex, rather than from the FG (but see (Axelrod and Yovel, 2015) for a study that found higher decoding accuracy for famous face identity in the FG than in the anterior inferotemporal cortex). Furthermore, FG activity is also affected by several lower-level stimulus features, such as contrast, size, orientation and position (Yue et al., 2011; Xu et al., 2009), while neural activity in the anterior inferotemporal cortex is mostly invariant to identity-irrelevant properties (Anzellotti et al., 2014; Anzellotti and Caramazza, 2015).

It has been argued that any difference between neural activity in STS and FG, as measured by ECoG electrodes, is necessarily confounded by different signal-to-noise ratios (SNRs) in the two areas, since cortical surface electrodes are less well suited to recording activity from within a sulcus (e.g. the STS) than from the surface of a gyrus (e.g. the FG) (Said et al., 2011). While we believe that differences in SNR between these two regions are possible and might have affected our results (in particular, the upright vs. inverted face decoding results in the unmasked paradigm, Fig 7 and 8), we believe it is unlikely that they can completely explain them, since some electrodes in both lateral-temporal/inferior-parietal cortex (close to the STS and possibly recording activity from this area) and FG exhibit similar SNR when decoding visibility in the masked paradigm (Fig. 3, S3 and 4). However, we believe that this is an important caveat of the current study and one that deserves to be investigated in detail. The question could be assessed by using depth macroelectrodes or, preferably, through simultaneous ECoG and laminar microelectrode recordings (e.g., (Ulbert et al., 2001; Csercsa et al., 2010; Chan et al., 2011)), which could unveil the contribution of lamina-specific neural populations to the signals recorded from macroelectrodes placed on the cortical surface in the different brain areas.

Many studies aimed to identify the brain regions whose activity most closely matches the behavioral effects of specific face manipulation. Among these, the face inversion effect (FIE) is one of those that have been most thoroughly studied. This phenomenon consists in the perceptual impairment in response to inverted vs. upright faces that is disproportionally large if compared with the inversion of most other visual stimuli (reviewed in (Rossion and Gauthier, 2002; Maurer et al., 2002)), and is widely considered as evidence for the holistic or configural (as opposed to featural or part-based) perception of upright, but not inverted, faces (Van Belle et al., 2010) (but see (Richler et al., 2011) for a study that suggests holistic perception also for inverted faces, albeit with longer latency and possibly greater processing requirements).

Our unmasked tasks were designed to maintain subjects alert while they were viewing images from different categories, hence we did not acquire specific behavioral measures (e.g., the accuracy or latency of a same/different face identity judgment task) that could enable the assessment of the neural markers that best correlate with a FIE behavioral measure. However, the fusiform gyrus, which is the brain area that is most specifically informative of upright faces (as revealed by an upright vs. inverted face decoding analysis), is also the brain area whose activity most closely correlates with the behavioral FIE and other perceptual modifications as investigated in previous studies. In particular, an fMRI study reported that the BOLD activity in the FG, but not in the STS, correlated with the behavioral FIE across subjects, and showed greater sensitivity to face identity when faces were presented upright vs. inverted (Yovel and Kanwisher, 2005). In accordance with these results, an fMRI study employing face morphing reported that activity changes in the FG, but not the STS, tracked perceptual, rather than physical, stimulus changes (Rotshtein et al., 2005). Indications of a prominent role of the FG in the perception of faces also come from a study that used Rubin's vase ambiguous images and reported that face percepts correspond to higher activity in the FG as compared to vase percepts (Hasson et al., 2001), in spite of only minimal and peripheral stimulus changes in the two conditions. Even more direct evidence for a causal involvement of the FG in the generation of face perception comes from intracranial electrical stimulation studies that reported distortions of face perception with electrical stimulation of face selective regions of the fusiform gyrus (Puce et al., 1999; Parvizi et al., 2012; Rangarajan et al., 2014).

One study (Rosburg et al., 2010) reported changes in intracranial ERP amplitude and latencies related to the face inversion effect in the lateral occipital cortex, but not in the ventral temporal cortex as we observed. The discrepancy could be due to the different neural markers examined: while we considered the whole spectrotemporal response of each electrode, which critically comprises the broadband high-gamma range which is considered a highly informative spectral band on the basis of ECoG and LFP studies (e.g., (Manning et al., 2009; Ray and Maunsell, 2011; Bansal et al., 2012; Miller et al., 2014)), Rosburg et al. considered the ERP waveform, which has been shown to convey different and sometimes complementary information to that signalled by broadband high-frequency spectral changes (Vidal et al., 2010; Miller et al., 2016). Furthermore, some LFP studies showed that broadband high-gamma activity correlates with neuronal spiking (Manning et al., 2009; Ray and Maunsell, 2011), an observation which is consistent with the prediction of a computational model (Miller et al., 2009). ECoG signals are expected to differ from LFP signals because of the greater size and lower impedance of the electrodes, and their position on the cortical surface, which is expected to result in sampling neural signals over a larger and more superficial region (Buzsáki et al., 2012). Nonetheless, a higher correlation between neuronal spiking and broadband high-gamma activity (in comparison to other frequency bands) is also expected for ECoG recordings (Miller et al., 2014).

In spite of the lack of discriminability between upright and inverted faces in the lateral cortex, and the greater diversity observed across decoding analyses in the set of lateral electrodes, our current results cannot be interpreted as evidence of a less critical involvement of the lateral cortex in the generation of conscious face perception.

In fact, it is possible that specific regions of the lateral temporal cortex (in particular, the face selective STS) also play an important role in the generation of conscious face perception. For example, it is conceivable that STS loci contribute to aspects of face perception that are invariant to orientation. It is also possible that the information about face orientation is present in the STS, but could not be resolved by our analysis due to the coarseness of ECoG recordings, which lump the contribution of large neural populations. In fact, informative neuronal responses can go undetected when recorded as a mass signal, especially when informative neurons are sparse and/or weakly clustered (Dubois et al., 2015). However, our results pose a constraint on theories of the relative roles of ventral and temporal loci in the generation of conscious face perception.

While it is now established that the FG and the STS respond more vigorously to faces than to most other visual stimuli, the strict face-specificity of these regions is still being debated (see, for example, (Yue et al., 2011; Bilalić et al., 2011; Joseph and Gathers, 2002; Caldara et al., 2006; Haist et al., 2010; Shultz and McCarthy, 2012)). Hence, it is possible that the results we reported generalize to other, non-face stimuli, especially those that are perceived as a holistic gestalt, possibly as a result of repeated exposure and hence development of configural internal templates (Diamond and Carey, 1986; Gauthier et al., 2014).

Heterogeneity across subjects

It is important to note that our subject population is heterogeneous, especially with respect to the behavioral performance and the neurophysiological markers of conscious visibility in the masked paradigm (Fig. 2 and S8). While all subjects present at least some electrodes that discriminate visibility (v decoding, collapsing across contrast levels), the great majority of v@c_thr or v@c_H discriminant electrodes are observed in a single subject (subject 153).

Anatomical differences in electrode location across subjects are likely to underlie some of the observed diversity. In particular, only subjects 153 and 154 present electrodes in the right fusiform gyrus, which is known from several lesion, neuroimaging and electrical stimulation studies to be more markedly involved in face processing than its left homologue (e.g., (Landis et al., 1986; De Renzi et al., 1994; Tsuchiya et al., 2008; Rangarajan et al., 2014)). In addition, face-specific responses are known to be produced in small regions of the fusiform gyri, which vary in location between individuals (Allison et al., 1994; Frost and Goebel, 2012). However, these factors are unlikely to be the only source of heterogeneity. In fact, heterogeneity across subjects when decoding stimulus category in the unmasked paradigm, or when decoding c₃ vs. blank in the masked paradigm, is much smaller than when decoding subjective phenomenology in the masked paradigm (Fig. S8 and S9). In particular, all subjects display at least one electrode in both ventral and lateral regions that is significant at p<0.001 when decoding upright face vs. other categories. Among the three subjects that were presented with inverted faces, all of them present several ventral electrodes that discriminate upright vs. inverted faces at high levels of significance (p<0.001), indicating that all of them had electrodes in regions that are highly face selective. In the masked paradigm, when considering a discrimination that is specified in terms of physical properties of stimuli such as c₃ vs. blank, every subject displayed at least one electrode that is significant at p<0.001 in both ventral and lateral regions. However, heterogeneity across subjects increases for discriminations that are specified by subjective factors, with only subjects 153 and 168 presenting ventral electrodes that discriminate v at p<0.001. Heterogeneity is even higher when considering discriminations that are determined by purely subjective factors, with subject 153 presenting 9 (13) ventral electrodes that are significant at p<0.001 for decoding v@c_thr (v@c_H), as opposed to none (1) for the other subjects. This suggests that at least some of the observed heterogeneity is likely to be due to different levels of task comprehension and execution, possibly resulting in visibility ratings being a poor indicator of subjective state for some subject.

Future studies could assess the extent to which intersubject variability can be accounted for by alignment of functionally corresponding brain areas across subjects. For example, one could perform preoperative fMRI recordings during a passive protocol (e.g. natural movie viewing) and use functional alignment methods to project electrode locations onto a standard brain (Sabuncu et al., 2010; Haxby et al., 2011; Conroy et al., 2013; Frost and Goebel, 2013; Dubois and Adolphs, 2016). While functionally-based alignment has been shown to greatly reduce intersubject variability during passive tasks (Haxby et al., 2011), the extent to which functional alignment methods can account for intersubject variability in more complex and demanding cognitive tasks is an open question.

Our limited subject population does not enable us to draw definite conclusions on the origin of the observed heterogeneity. However, it is worth noting that the degree of metacognition across subjects is also variable, with 2/5 subjects (subjects 168 and 178) exhibiting negative meta-d′ at the highest contrast value tested (Fig 2C), which corresponds to higher visibility rating resulting in higher probability of incorrect response. Also, 3/5 subjects (those we just mentioned with the addition of subject 154) show considerably less than perfect objective performance (91%, 87% and 72%, for subject 154, 168 and 178) at the highest visibility rating, which corresponds to clear visibility. Remarkably, these three subjects are the only ones that exhibit electrodes with significant c@v decoding (in particular, c@v₄, Fig. S6), that is, with responses that discriminate face intervals with different contrast values that resulted in the same visibility rating. The correspondence between overconfident behavioral performance on the one hand, and significant c@v₄ decoding on the other, suggests that these subjects might have responded with the highest visibility rating even if their perception of the face was not completely clear. As a consequence, in those trials where the highest visibility rating was reported, different contrast values would have resulted in different perceptual outcomes for these subjects. It is also worth noting that the most face selective electrodes in subject 154 were only recorded in a single session of the CFS paradigm, which did not result in enough trials for meaningful estimation of v@c_thr and v@c_H decoding accuracies.

While heterogeneity in behavioral performance and metacognition has also been reported in healthy subject populations and has been related to neuroanatomical metrics describing the local structure of gray and white matter (Fleming et al., 2010; Kanai and Rees, 2011) and to dopamine signalling (Van Opstal et al., 2014), the relationship between objective performance and metacognition on the one hand, and electrophysiological data on the other, has received less attention. The importance of progressing beyond group-level analyses and towards the assessment of individual differences in behavior and corresponding patterns of neural activity has been long acknowledged, and has been advocated as a necessary step for deepening our understanding of the relationships between brain activity and subdomains of human cognition (e.g., (Miller and Van Horn, 2007; Van Horn et al., 2008; Dubois and Adolphs, 2016)). However, most endeavors in this direction have employed fMRI recordings, which are affected by poor signal-to-noise ratio and contaminated by non-neural artifacts due to movement, respiration, heart-beat and other potential confounds. Conversely, the high signal-to-noise ratio and resolution of ECoG recordings can yield highly significant results in individual subjects and sometimes unveil informative features of neural activity even at the single-trial level (e.g., (Lachaux et al., 2012)). In addition, it is reasonable to suppose that the heterogeneity among epilepsy patients undergoing pre-surgical monitoring might be even higher than among healthy subjects, due to variable degrees of fatigue, sleep deprivation, motivation to perform the task, possible cognitive deficits associated with long-term severe epilepsy, and possibly lingering effects from anesthesia and/or from surgical pain. The higher heterogeneity among epilepsy patients can potentially expose individual differences more clearly in comparison to the healthy and fairly homogeneous population which typically constitutes the subject sample in non-invasive neuroimaging research studies. Hence, cognitive neuroscience research in implanted epilepsy subjects constitutes a precious opportunity for levering the excellent signal-to-noise ratio and high temporal and spatial resolution of intracranial recordings towards elucidating the relationships between cognitive and electrophysiological heterogeneity.

Comparison between CFS and other masking techniques

In this study, we presented images partially masked by CFS to highlight neural markers that differentiate seen versus unseen trials in the absence of any changes in sensory inputs. Several other techniques allow the presentation of images at perceptual threshold, including masking and crowding. These techniques rely on different properties of the visual system and exhibit different characteristics (e.g. (Izatt et al., 2014; Faivre et al., 2014; Faivre et al., 2012; Peremen and Lamy, 2014; Kaunitz et al., 2014; Fogelson et al., 2014; Almeida et al., 2008; Tsuchiya et al., 2006), see (Kim and Blake, 2005; Macknik, 2006; Kouider and Dehaene, 2007) for reviews), hence it is possible that slightly different results could have been obtained if a technique other than CFS was used to partially mask face images.

In order to yield a deeper understanding of conscious visual perception, it is important to assess the neural markers of conscious visual experiences that are independent from the specific experimental paradigm employed, and those that are specific to a particular class of masking techniques (Dubois and Faivre, 2015). While the latter can increase our understanding of the physiological mechanisms underlying perceptual suppression (and the breaking thereof) in specific experimental paradigms (e.g., dichoptic versus monoptic masking), only the former can be considered as putative NCC-core. Many recent studies investigated the differences and similarities between masking paradigms (Izatt et al., 2014; Faivre et al., 2014; Faivre et al., 2012; Peremen and Lamy, 2014; Kaunitz et al., 2014; Fogelson et al., 2014); however, most of them employed purely psychophysical measures (Izatt et al., 2014; Faivre et al., 2012; Peremen and Lamy, 2014; Kaunitz et al., 2014), or non-invasive neuroimaging methods (Fogelson et al., 2014). Hence, the comparison between neural markers of conscious visual perception under different masking paradigms is an important topic for future research, and one that can greatly benefit from the high temporal and spatial resolution of intracranial recordings.

Distilling NCC-core - caveats and potential confounds

It is important to recognize that the combination of masked and unmasked paradigms can greatly contribute towards a more accurate assessment of the core neural correlates of conscious visual perception, but also presents some caveats. For example, some electrodes can be upright face discriminant and visibility discriminant, but still not NCC-core. For example, areas related to memory formation of faces would be both face discriminant and visibility discriminant, but not NCC-core, in the sense that a hypothetical patient with a localized lesion in that area might still be able to experience faces consciously, albeit incapable of creating long-lasting memories of face identities (Postle, 2009).

More generally, an electrophysiological feature that distinguishes both high vs. low visibility trials in the masked paradigm and upright faces vs. other stimuli in the unmasked paradigm does not necessarily constitute an NCC-core marker, since it could represent a correlate of the same confound in both scenarios. For example, one electrode from the temporal pole of subject 178 exhibited significant A′ both in a CFS visibility decoding analysis and in an unmasked face decoding analysis (Fig. S10). While face selective effects have been reported also in anterior temporal and frontal regions, especially in the anterior ventral temporal cortex, inferior frontal gyri, frontal operculum and lateral prefrontal cortex (Allison et al., 1999; Avidan et al., 2005; Chan et al., 2006; Kriegeskorte et al., 2007; Tsao et al., 2008; Engell and Haxby, 2007; Said et al., 2010; James et al., 2013), the anatomical location of the face discriminant electrode in the temporal pole of subject 178 is also compatible with discriminant activity in this region being due to eye movements towards the eyes of the target face both in high visibility CFS trials and in unmasked trials containing a face (Jerbi et al., 2009; Kovach et al., 2011). Future work can disambiguate the influence of eye movement confounds by simultaneous recordings of eye movements.

It is also worth noting that our results are not incompatible with the possibility that visibility and upright face discriminability in the FG do not constitute signatures of the NCC-core of face perception, but are rather NCC-cos, i.e. they just or mostly reflect a consequence of the conscious experience of face images. For example, one could hypothesize that conscious face perception might induce gain changes in early visual areas (e.g., in the thalamus and/or in primary visual cortex), for example via top-down mechanisms (Gilbert and Li, 2013), which in turn would result in increased bottom-up drive to high-level visual areas. In fact, our observational approach and sparse neural sampling do not enable us to draw conclusions on the causal nature of the effects we describe. An exhaustive assessment of all the brain regions that could potentially contribute to the NCC-core of face perception, and their functional relationships, would require the simultaneous recording of early and high-level visual and multisensory areas with high temporal and spatial resolution. Invasive electrophysiological coverage of both early and high-level visual areas in humans is extremely rare, hence we expect that, at least for the near future, progress in this direction is more likely to arrive from invasive studies in non-human animals (Bastos et al., 2015; Yanagawa et al., 2013; Tajima et al., 2015) combining observational and interventional approaches (Panzeri et al., 2017) across different, but related, experimental protocols.

One likely possibility is that conscious perception arises from coordinated activity across large and spatially distributed neural ensembles. In this framework, it would not be necessary to postulate the existence of a single brain region that “ignites” the distributed pattern of neural activity that constitutes the NCC-core and can hence be considered the “first cause” of NCC-core patterns, since causality would necessarily be shared across several neural populations. Thus, the NCC-core would be best envisioned as a “causal web” of interacting brain areas (see also (Schurger and Uithol, 2015) for similar arguments in the context of self-initiated movement).

In fact, it has been suggested that conscious awareness relies on functional coupling across neural populations (Engel et al., 1999; Thompson and Varela, 2001; Melloni et al., 2007; Godwin et al., 2015); more specifically, on a network of irreducible causal interactions across neural populations (Seth et al., 2011; Oizumi et al., 2014; Tononi and Koch, 2015; Haun et al., 2016). The development of improved methods for the assessment of causal interactions in neural data (Tajima et al., 2015; Oizumi et al., 2016), as well as advances in brain imaging and large-scale neural recording and stimulation techniques (Duyn, 2012; Ahrens et al., 2013; Shobe et al., 2015; Buzsáki et al., 2015; Yang et al., 2016), are expected to greatly foster the investigation of multidimensional patterns of neural interactions as putative neural correlates of consciousness.

Conclusion

We have combined a set of decoding analyses of either physical properties of stimuli or subjective phenomenology in a masked task, with a set of decoding analyses of stimulus category in an unmasked task. Our results in the masked task show that subjective phenomenology is better decodable than physical properties of stimuli (such as contrast) in the high level visual areas considered in this study, i.e. the ventral and lateral sides of the temporal lobe.

While we were able to decode subjective visibility in the masked task with high accuracy in both ventral and lateral loci, the inclusion of a stimulus category decoding analysis in the unmasked task revealed an important difference between the two loci: while both ventral and lateral areas discriminate upright faces from other stimuli, only ventral areas discriminate upright from inverted faces.

Our results suggest a critical role for ventral brain areas, and in particular for the fusiform gyrus, in the conscious configural perception of faces and possibly other objects that are perceived holistically. More generally, this work points towards a promising direction in consciousness research based on the combination of similar protocols tailored to expose specific aspects of the conscious visual experience.