Category Selectivity for Face and Scene Recognition in Human Medial Parietal Cortex

Summary

The rapid recognition and memory of faces and scenes implies the engagement of category specific computational hubs in the ventral visual stream with the distributed cortical memory network. To better understand how recognition and identification occur in humans, we performed direct intracranial recordings, in a large cohort of patients (n = 50), from the medial parietal cortex (MPC) and the medial temporal lobe (MTL), structures known to be engaged during face and scene identification. We discovered that the MPC is topologically tuned to face and scene recognition, with clusters in MPC performing scene recognition bilaterally and face recognition in right subparietal sulcus. The MTL displayed a selectivity gradient with anterior, entorhinal cortex showing face selectivity and posterior parahippocampal regions showing scene selectivity. In both MPC and MTL, stimulus specific identifiable exemplars led to greater activity in these cortical patches. These two regions work in concert for recognition of faces and scenes. Feature selectivity and identity sensitive activity in the two regions was coincident and they exhibited theta phase locking during face and scene recognition. These findings together provide clear evidence for a specific role of subregions in the MPC for the recognition of unique entities.

Introduction

The ability to identify previously encountered people and objects, requires an interplay of activity between sensory and memory regions of the brain. To identify a unique object requires recognizing that it has been encountered before and to be able to retrieve information about it [1-4]. Traditionally, the hippocampus and parahippocampal gyri are implicated as the principal loci subserving these processes [1,5-8]. Distinctive entities such as faces and scenes rely upon specific, category selective constituents of the ventral visual stream and medial temporal lobe (MTL) for recognition.

In recent years it has become clear that the extended memory network underpinning recognition extends beyond the MTL to other regions such as medial parietal cortex (MPC) [9-14]. Prior functional neuro-imaging studies have shown that the MPC is engaged in processing complex scenes [12,15]; path integration [16-18]; spatial memory of landmarks [19,20]; egocentric to allocentric transformation during navigation [21,22] and spatial imagery. It has also been shown to be active in assessments of face familiarity [23-27]. This engagement of MPC in both spatial and facial processing brings up obvious questions regarding its functional organization and its role in recognition. There is some evidence that the MPC engages with known areas of category selectivity within ventral visual pathways [28-33] and the MTL [8,34-37], with known scene and face areas differentially connected to different regions of MPC [27]. Given the role that the MPC is thought to serve as a hub in memory processing, could it also play a role in coordinating category-specific memories alongside the ventral stream?

These questions have been unanswerable till the current time owing to the relative inaccessibility of the medial parietal lobe to time-resolved, electrophysiology (M/EEG), and the infrequency of focal lesions impacting this region [38]. To resolve these questions, we performed intracranial recordings, in a large human cohort, during face and scene identification tasks, using both surface and penetrating electrode arrays. Spatially precise and highly-temporally resolved recordings across a broad region were used to evaluate if MPC is tuned to faces and scenes, and if so, what is its role in recognition and identification relative to the ventral visual stream and the MTL.

Results

Broadband gamma activity (BGA; 70-150Hz) was recorded using intracranial electrodes in 66 participants (30 male, 18-56 years, 13 left-handed) undergoing intracranial electrode placement for the localization of intractable epilepsy - 18 patients had subdural grid electrodes (SDEs) and 48 had depth recordings using stereotactic EEG electrodes (sEEGs). Participants viewed visual images of famous faces, famous landmarks (scenes) or scrambled images and were asked to name the person or landmark (Figure 1).

Figure 1. Task and Patients.

(A) Representative stimuli for each stimulus category. (B) Spatial coverage represented as a heat map on a standardized N27 brain surface showing extensive and consistent coverage of the medial parietal cortex and medial temporal lobe (n = 50 patients that performed both face and scene identification correctly). (C) Individual electrode locations (9,016 total electrodes) highlighting electrodes in the two ROIs defined using a parcellation derived from the Human Connectome Project [79]. Responsive electrodes are shown in MPC (yellow, n = 142) and MTL (green, n = 216). Electrodes within these ROIs not responsive to either faces or scenes are in maroon and electrodes outside these two ROIs are in white

Behavioral Performance

Mean identification accuracy (± SD) was 34 ± 19% for faces and 51 ± 20% for scenes. 16 patients who identified less than 20 face or scene stimuli were excluded from further analysis. Data from the remaining 50 participants, who identified faces and scenes with an overall accuracy of 39 ± 17% for faces and 56 ± 16% for scenes, were used for further analysis. Articulation latencies for this cohort were; faces 1502 ± 346 ms and scenes 1479 ± 321 ms.

Category selective regions in MPC: Individual data

Data from two representative patients are shown in Figure 2, each with electrodes in multiple locations within MPC. Each displayed loci with greater activation for one stimulus class over all other categories. TA603 had an SDE grid spanning the extent of the entire right MPC (Figure 2A). Three electrodes, all located over the subparietal sulcus [39], showed prominent BGA increase (>200% above baseline) for faces, with smaller responses (<50%) to other stimulus classes (Figure 2C). Electrode X15, located more posteriorly, responded to both faces and scenes over scrambled stimuli but was not significantly selective to either category. No other electrodes were significantly responsive. A trial-by-trial analysis of activation (Figure 2D) showed a consistent onset time of activation relative to stimulus onset in face selective electrodes that was not strongly coupled to response latency.

Figure 2. MPC category selectivity – individual analysis.

(A,D) Electrode locations in two representative patients - plotted electrodes are in yellow. (B) Time course (mean ± SEM) of BGA at each electrode in the MPC SDE grid in response to each stimulus category. (C) Raster plots of single trial BGA in response to faces (left) and scrambled faces (right) for two electrodes. Trials are organized by response latency with black dots representing time of articulation onset. (E) Time course of BGA at four sEEG electrodes. Vertical dashed lines indicate time windows used for selectivity analyses. (F) Selective responses for faces and scenes, 500-1000 ms post stimulus onset, at the same four electrodes. ** p<0.01, *** p<0.001.

TS106 had sEEG electrodes in medial parietal and occipital cortex (Figure 2B). The right subparietal sulcus in this case also showed prominent selectivity for face stimuli. Electrodes located in the parieto-occipital fissure were more active for scene stimuli and, in this case, were scene selective. This separation of preferential activation suggests that MPC processes faces and scenes via spatially distinct neural populations rather than a single, generic cognitive process.

Face/Scene selectivity in MPC: Population analysis

All 236 electrodes (99 LH, 137 RH), in 39 patients, located within MPC were evaluated for responsivity and selectivity to faces and scenes (Figure 3; Video S1; Video S2; Figure S1), in the 500-1000 ms window. 142 (60%) were responsive, showing significantly greater activation to either faces or scenes than their scrambled versions. Of these, 21 (15%) were face selective, 60 (42%) were scene selective and the remaining 61 (43%) showed no significant differences between these conditions (Figure 3A). There was no significant correlation between the amplitude of face-scene differences and greyscale-color scrambled differences (r = −0.10, p = 0.24; Figure S2). Face responsive electrodes showed significantly greater activity for whole faces over face parts (Figure S3). In both hemispheres there were a substantial number of electrodes with preferential activation to faces or scenes. Thus, faces and scenes appear to be processed via distinct and only partially overlapping neural substrates in MPC.

Figure 3. Category selectivity in MPC and MTL – population analysis.

Selectivity of responses in (A) MPC and (B) MTL for faces (blue) and scenes (red) in the time window of 500-1000 ms after stimulus onset (See also Figures S1 - S3). Non-selective electrodes (q>0.05) are black. (C) A population-level, surface-based MEMA contrasting (faces vs. scrambled faces) – (scenes vs. scrambled scenes) revealing regions of significant stimulus selectivity (p<0.01 corrected) in the ventral visual stream (See also Figure S4) and medial parietal cortex (See also Videos S1 - S3). Regions in black did not have consistent coverage for reliable MEMA results. Between-patient averaged BGA (mean ± SEM) in MPC (D,F) and MTL (E,G) at face selective (FS) or scene selective (SS) electrodes. Face (D,E) and scene (F,G) identification sensitivity in FS electrodes and SS electrodes.

To further investigate this spatial separation, we used a mixed-effects multilevel analysis (MEMA) [40]. This method creates an effect size and precision estimate on the cortical surface modelling each electrode’s ‘recording zone’ as an exponentially decaying geodesic radius. A mixed effects analysis of the population is then performed at each node on the cortical surface. This method accounts for sparse sampling, outlier inferences and intra- and inter-subject variability to create a population-level, surface-based representation. This analysis also showed bilateral scene selective cortical clusters superior to the parieto-occipital fissure and in posterior precuneus. In the right hemisphere a well-defined face selective cluster was seen, anatomically localized to the subparietal sulcus in the right hemisphere, with a small scene selective cluster superior to it (Figure 3C; Video S3).

Identification sensitivity within category selective regions in MPC

To evaluate the sensitivity of cortical patches in the MPC to the patients’ abilities to identity the faces, we compared activation for correctly vs incorrectly answered face naming trials within face selective electrodes and within scene selective electrodes with a significant face response. Of the 21 face selective electrodes, 15 (71%) were face identification sensitive – they showed greater responses for correctly identified faces over incorrectly identified ones (Figure 3D). By contrast, 10 (36%) of the 28 scene selective electrodes with significant face responsiveness showed face identification sensitivity. Of the 60 scene selective electrodes, 22 (37%) were scene identification sensitive (Figure 3F) and only 5 (24%) of the 21 face selective electrodes showed scene identification sensitivity. A multiple linear regression (r² = 0.23), utilizing all responsive MPC electrodes, revealed that face identity sensitivity was associated with face selectivity (β = 0.94, p < 10⁻⁸) and scene identity sensitivity was associated with greater place selectivity (β = −0.28, p = 0.046). This means face and scene selective areas in MPC show preferential activation for known, identifiable exemplars of stimuli they are tuned to. An important question these findings raise is whether this response pattern of selectivity for faces vs. scenes is present in other regions of the cortical memory network. If so, which region shows this phenomenon first?

Face/Scene selectivity in MTL is similar to MPC

We applied identical analyses to characterize category selectivity and identification sensitivity in the MTL. Of 347 MTL electrodes (192 LH, 155 RH) in 42 patients, 216 (62%) were responsive. Of these, 21 (10%) were face selective, 59 (27%) were scene selective and the remaining 136 (63%) were unselective (Figure 3B). The ratio of face to scene selective electrodes was not significantly different from that in MPC (χ² (1, N = 161) = 0.002, p = 0.96). An anterior-posterior separation between face and scene selectivity was noted, with anterior, rhinal areas more likely to show face selectivity and posterior regions showing greater scene selectivity (r = 0.56, p < 10⁻⁵). As in MPC, there was no significant correlation between face-scene and greyscale-color differences (r = −0.11, p = 0.12; Figure S2) and face responsive electrodes showed significantly greater activity for whole faces over face parts (Figure S3). At the population level, parahippocampal place area was discernable in the right hemisphere (Figure 3C), consistent with previous studies of MTL and ventral visual category selectivity (Figure S4). Of the 21 face selective electrodes, 10 (48%) showed significant face identification sensitivity (Figure 3E). By contrast, 8 (24%) of 34 scene selective, face responsive electrodes were face identification sensitive. As in MPC, a multiple linear regression (r² = 0.14) revealed that stronger face identity sensitivity was associated with greater face selectivity (β = 0.44, p < 10⁻⁵) whereas scene identity sensitivity was associated with greater place selectivity (β = −0.32, p < 0.001).

MPC and MTL show comparable identification responses to Faces and Scenes

Overall, the statistics for category and identification sensitivity were highly comparable between MTL and MPC. A MEMA contrasting responses during correct and incorrect naming trials for each stimulus type showed sites of identification sensitivity within MPC and MTL ROIs (Figure 4; Figure S5). We also saw a prominent focus of activation for correct scene and face identity in mid-fusiform cortex, an area we and others have previously shown to be a crucial component of the lexico-semantic retrieval system [41-43]. Activation of the supplementary motor area and anterior cingulate cortex was greater for correct trials, corresponding to speech motor planning and name production.

Figure 4. All-patient maps of identification sensitivity.

MEMA representation of selectivity to correct over incorrect naming trials of faces (A) and scenes (B). Regions in MPC and MTL show significant identification sensitivity (p<0.01 corrected) for each stimulus category (See also Figure S5). Regions in black did not have consistent coverage for reliable MEMA results.

Of the 142 responsive electrodes in MPC, 53 (37%) showed only face identification sensitivity compared to 45 (32%) with only scene identification sensitivity. 18 (13%) showed significant identification responses to both categories (Figure 5A). Of the 216 responsive electrodes in MTL, 80 (37%) showed face identification sensitivity and 37 (17%) showed scene identification sensitivity. 14 (6%) showed a significant identification response for both categories. The MTL was less engaged in representation of scenes relative to face identification as compared to MPC (χ² (1, N = 215) = 4.62, p = 0.032). The right MTL was more specialized for face rather than scene identification than the left MTL (RH 3.64 vs. LH 1.65; χ² (1, N = 126) = 3.96, p = 0.046), implying a right MTL lateralization for face recognition. In distinction, MPC showed no lateralization in scene vs. face selectivity (RH 1.56 vs. LH 1.07; χ² (1, N = 102) = 0.845, p = 0.36). The time courses of BGA responses of electrodes that were face or scene identification sensitive were remarkably conserved across stimulus categories within each ROI (Figure 5B,C), even though these electrodes were spatially separated. Both regions also showed a beta band (11 – 20 Hz) suppression that reached its trough just after the peak BGA response (Figure 5D).

Figure 5. Identity sensitivity in MPC and MTL.

(A) Co-localization of identity selective electrodes (q<0.05) for face (blue) and scene (red) stimuli, and their overlap (purple), in MPC and MTL. Non-selective electrodes are black. Averaged BGA time courses (mean ± SEM) for face (B) and scene (C) identity sites. Grouped spectrograms showing broadband changes in face identity electrodes for correctly (D) and incorrectly (E) identified faces. (F) Bootstrap distribution determining the initial latency when it is possible to distinguish a face from a scrambled face or a known face from an unfamiliar one based on the BGA in both MPC and MTL. (G) Distribution of paired latency difference between regions.

Face identification responses in MPC and MTL occur virtually simultaneously

A remaining question is how MPC and MTL interact to enable identification and whether this process occurs earlier in one of the regions. To test this, we evaluated a subset of patients (n = 16) with at least one face identity sensitive electrode in both MPC and MTL. We evaluated two time points within each region; the first time point at which gamma traces were significantly different between faces and scrambled faces and the first point at which they were different for correctly vs incorrectly identified faces. In MPC, median latency for face discrimination was 329 ms (306 - 386 ms, 95% CI) and the latency for identification was 430 ms (403 - 523 ms). In the MTL the latency of face discrimination was 340 ms (311 - 395 ms) and the latency of identification was 425 ms (325 - 488 ms). We generated bootstrapped distributions (Figure 5F) from this analysis to show that peak latencies for each contrast occur near simultaneously in both MPC and MTL. Figure 5G shows the distribution of latency differences in trial-paired MPC and MTL responses. The median difference was −20 ms (−98 – 46 ms) for face determination and 10.5 ms (−80 – 137 ms) for identity. Neither difference was significant (p = 0.77; p = 0.93 respectively). In the subset of patients with at least one scene identify selective electrode in each region (n = 9 patients) the analysis did not show any time points with a significant difference between correct and incorrect trials.

To directly probe the interactions between these regions we measured changes in the phase locking value (PLV) of theta oscillations (3 – 5 Hz) [44-47] between MPC and MTL during these tasks. Of the 1719 within-patient MPC-MTL electrode pairs (n = 33 patients), 108 pairs showed a significant increase in PLV only to faces, 47 pairs only to scenes and 18 pairs significantly increased to both (Figure 6A,B). Face selective pairs showed greater PLV during correct trials than incorrect (p < 0.001) or scrambled (p < 10⁻⁴) trials. Scene selective pairs also showed greater PLV during correct trials than incorrect (p < 0.01) or scrambled (p < 0.001) trials. Neither face (p = 0.99) or scene (p = 0.77) selective pairs showed significantly greater PLV for correct over incorrect trials for trials in the category they were non-selective for. Within category selective pairs, the latency for the time points when PLV first became significantly greater for correct vs. incorrect trials was 485 ms for faces (Figure 6C) and 785 ms for scenes (Figure 6D). These findings validate the role of MPC in recognition and imply the existence of category specific functional connectivity between MPC and MTL.

Figure 6. Theta Phase Locking between MPC and MTL.

Localization of electrodes significantly involved (q<0.05) in inter-regional theta phase coupling in the faces (A) and scenes (B) tasks. Non-significant electrodes are black. Average time courses (mean ± SEM) of PLV changes from baseline between the significant face (C; 14 patients) and scene (D; 11 patients) pairs.

Discussion

Our work provides clear evidence of a role of MPC in the recognition of unique entities. MPC is topologically tuned to faces and scenes, especially in the right hemisphere, and co-activates and interacts with the MTL. These observations provide evidence that MPC is an important node during recognition – a process it likely helps accomplish via interactions with the hippocampus and MTL. Our thesis is supported by recent imaging studies [48], by lesional analysis of human MPC that result not only in spatial disorientation, but also in concurrent anterograde and retrograde amnesia [11,49-51], and the impact of lesions of the hippocampus resulting in hypoactivity and loss of plasticity of MPC [13,52,53]. The loss of recognition – manifested as the inability to attach name to a face is a common manifestation of normal aging, but is prominently impaired in dementia. The role of the MPC in aging is hinted at by hypometabolism that occurs as a prodrome of Alzheimer’s disease [54,55]. Understanding the neurobiology of recognition in the extended cortical memory network is therefore of crucial importance.

The timing of the identification distinction here from neural population recordings (MPC 430 ms; MTL 425 ms) is not dissimilar to the familiarity distinction found from single neuron recordings of human hippocampus (461 ms) [56] suggesting that MPC may play a role in familiarity determination. Our findings are supported by differences seen in activation during fMRI studies of recall of personally familiar over famous people and places [27]. The phase locking between MPC and MTL via theta frequency oscillations [44-47] further provides evidence for the existence of face and scene selective pathways between these regions. That the MPC and MTL are not strongly separable in latency or function is consistent with a strong connectivity between these two regions both anatomically and functionally [57,58], so much so that these regions can be viewed as a distinct, dissociable cortical memory system [12,59], as part of a distributed memory network [60,61].

MPC, particularly retrosplenial cortex [58,62,63], has been most commonly investigated in the context of scene and spatial processing during navigation. However, fMRI studies suggest that it also activates during face presentation [23-26,64]. Our study supports these findings, showing strong responses to faces in MPC. Further, we show regions robustly selective to faces, with a subset of these regions being more engaged during face identification. This selectivity is consistent with previous fMRI studies of category specific recall [27,65], that show an anatomically comparable separation of person and place information processing. Importantly, by comparing MPC selectivity to MTL we found no difference in the relative ratios of face to scene selective sites – neither of these two regions preferentially represents one stimulus type relative to the other. In both regions we observed a high ratio of scene to face selective electrodes. Given the semi-random sampling within these ROIs this implies that a greater cortical area of both ROIs is dedicated to processing the scene stimuli. This property has previously been observed in fMRI studies of ventral temporal cortex where parahippocampal place area shows larger activation clusters than fusiform face area [66,67].

A well-documented property of the MPC is its involvement in the brain’s default network (DN), an interconnected network of brain regions that deactivate during externally directed tasks and activate while constructing internal representations. Recently, the DN has been suggested to be segregable into multiple sub-networks [68-72]. Within MPC there is a repeatable, superior-inferior divide between memberships in these sub-networks that approximately aligns to segregation of our face-scene clusters. The inferior, parieto-occipital fissure scene clusters, can be more broadly considered as part of the network associated with scene processing, navigation and memory-based imagery [68,69,72]. The face cluster, is linked to the network linked to social knowledge, theory of mind and moral decision making [68,71,73].

It could be argued that some of the cortical responses to the faces and scenes stimuli may be confounded by differences in color vs greyscale images. Color is more crucial for scene identity than to face identity [74,75] and was the reason for our choice of these stimuli. However, we find no significant correlation between face-scene and greyscale-color differences in MPC or MTL. Additionally, our results are entirely concordant with previous studies that show regions of MPC selectively activating during text-cued recall of person and place information [27,65]. Also, a number of other episodic memory studies have elicited responses in MPC with text based cues [10,76,77] implying that this is a higher-order region that is mostly invariant to the visual features and perhaps even the modality in which the object is presented.

Given the identification sensitivity seen in this study it appears clear that MPC plays an important role in recognition of faces and scenes, however, it is yet unclear what the unique contributions both MPC and MTL make to these processes. This can be probed using the production of transient lesions using microstimulation, time locked to the activation of each of these regions [78] and may help move us to a true causal understanding of their roles in recognition. Given the timings of activation and the close ties to the activity of the MTL, MPC is most likely involved in some aspect of familiarity. However, an inability to name, as assessed by this task, could be either a lack of familiarity or recollection so this will need further study.

Conclusions

We have shown, in a large patient cohort, that MPC shows sensitivity to recognition of both faces and scenes in spatially distinct regions that display robust and clear selectivity to either category. We also show, using within-individual comparisons, no distinction in the latency of these selectivities between MPC and MTL and transient, low frequency interactions between the two regions. Given the lack of strong functional or temporal separability between MPC and MTL, future work should investigate the individual and distinct contributions of each of these nodes of the memory network. Our findings shed new light on the importance of areas outside of those traditionally though to be important to category representation, suggesting the distributed nature of such coding within the human brain.

STAR Methods

RESOURCE AVAILABILITY

Lead Contact

Further information and requests for resources should be directed to and will be fulfilled by the Lead Contact, Nitin Tandon (nitin.tandon@uth.tmc.edu)

Materials Availability

This study did not generate new unique reagents.

Data and Code Availability

The datasets generated from this research are not publicly available due to them containing information non-compliant with HIPAA and the human participants the data were collected from have not consented to their public release. However, they are available on request from the corresponding author. The custom code that supports the findings of this study is available from the corresponding author on request.

EXPERIMENTAL MODEL AND SUBJECT DETAILS

Participants

66 patients (30 male, 18-56 years, 13 left-handed) participated in the experiments after giving written informed consent. All participants were semi-chronically implanted with intracranial electrodes for the clinical purposes of seizure localization of pharmaco-resistant epilepsy. Participants with significant additional neurological history (e.g. previous resections, MR imaging abnormalities such as malformations or hypoplasia, or those with prosopagnosia) were excluded. All experimental procedures were reviewed and approved by the Committee for the Protection of Human Subjects (CPHS) of the University of Texas Health Science Center at Houston as Protocol Number: HSC-MS-06-0385.

METHOD DETAILS

Electrode Implantation and Data Recording

Data were acquired from either subdural grid electrodes (SDEs; 18 patients) or stereotactically placed depth electrodes (sEEGs; 48 patients). SDEs were subdural platinum-iridium electrodes embedded in a silicone elastomer sheet (PMT Corporation; top-hat design; 3mm diameter cortical contact), and were surgically implanted via a craniotomy following previously described methods [80-82]. sEEGs were implanted using a Robotic Surgical Assistant (ROSA; Medtech, Montpellier, France) [83]. Each sEEG probe (PMT Corporation, Chanhassen, Minnesota) was 0.8 mm in diameter and had 8-16 electrode contacts. Each contact was a platinum-iridium cylinder, 2.0 mm in length and separated from the adjacent contact by 1.5 - 2.43 mm. Each patient had 12-20 such probes implanted. Following surgical implantation, electrodes were localized by co-registration of pre-operative anatomical 3T MRI and post-operative CT scans in AFNI [84]. Electrode positions were projected onto a cortical surface model generated in FreeSurfer [85], and displayed on the cortical surface model for visualization [82]. Intracranial data were collected during research experiments starting on the first day after electrode implantation for sEEGs and two days after implantation for SDEs. Data were digitized at 2 kHz using the NeuroPort recording system (Blackrock Microsystems, Salt Lake City, Utah), imported into Matlab, initially referenced to the white matter channel used as a reference for the clinical acquisition system and visually inspected for line noise, artifacts and epileptic activity. Electrodes with excessive line noise or localized to sites of seizure onset were excluded. Each electrode was re-referenced to the common average of the remaining channels. Trials contaminated by inter-ictal epileptic spikes were discarded.

Stimuli and Experimental Design

Stimuli were presented using Python v2.7 at a size of 500 x 500 pixels on a 2880 x 1800, 15.4” LCD screen positioned at eye-level, 2-3’ from the patient (~7.5° visual angle). Each stimulus was displayed for 2000 ms with an inter-stimulus interval of 6000 ms. All 66 participants performed famous face and famous landmark (scene) recognition [28]. Participants who correctly identified less than 20 stimuli in either the faces or scenes task were excluded from analysis, 16 participants were excluded this way. 31 patients also performed a face parts classification task, identifying images of eyes, mouths etc. Additional details of each experiment are below:

Famous Face Recognition

Participants were presented with greyscale photos of famous faces and asked to verbally name them. Stimuli were presented in one recording session, containing presentation of 90-200 images, consisting of a mix of coherent images and their spatially scrambled versions in a pseudorandom order with no repeats. In trials with scrambled faces the patients were asked to respond with “scrambled”. Data were collected over an eight year period so additional face stimuli were gradually added or replaced throughout the collection period. Stimuli were matched for luminance and contrast using Adobe Photoshop.

Famous Scene Recognition

Participants were presented with color photos of famous landmarks (scenes) and asked to verbally recall their location. Stimuli were presented in one recording session, containing presentation of 140-160 images, consisting of a mix of coherent images and their spatially scrambled versions in a pseudorandom order. In trials with scrambled scenes the patients were asked to respond with “scrambled”.

Face Parts Classification

Participants were presented with greyscale photos of parts of faces and instructed to verbally name them. Face parts included ears, eyes, mouths and noses. Stimuli were presented in one recording session, containing presentation of 90-100 images, consisting only of coherent images in a pseudorandom order with no repeats.

Audio Recordings

Continuous audio recordings were carried out during all experiments with an omnidirectional microphone (30-20,000 Hz response, 73 dB SNR, Audio Technica U841A) placed within 2 feet of the patient, and adjacent to the presentation laptop. These recordings were analyzed offline to manually isolate articulatory onsets and transcribe patient responses. Incorrect responses included either an absence of speech, stating they did not recognize the object or incorrectly identifying the object.

QUANTIFICATION AND STATISTICAL ANALYSIS

A total of 13,212 electrode contacts were implanted, 4,196 of these were excluded due to proximity to the seizure onset zone, frequent inter-ictal epileptiform spikes or line noise. The remaining 9,016 electrodes were deemed usable. All electrodes within ROIs in MPC and MTL (parahippocampal, perirhinal and entorhinal cortices), based on a brain parcellation derived from the Human Connectome Project [79], were used for further analysis. The MPC ROI comprised of the areas RSC, 7m, POS1, v23ab, d23ab, 31pv and 31pd. The MTL ROI comprised of the areas H, PHA1, PHA2, PHA3, EC, PreS and PeEc. An ROI in ventral occipitotemporal cortex (vOTC) was also used, encompassing all ventral regions not included in the MTL mask (Figure S4).

Raw data recordings from each electrode were bandpass filtered to isolate broadband gamma activity (BGA; 70-150Hz) following removal of line noise (zero-phase 2nd order Butterworth band-stop filters). A frequency domain bandpass Hilbert transform (paired sigmoid flanks with half-width 1.5 Hz) was applied and the analytic amplitude was smoothed (Savitzky - Golay FIR, 3rd order, frame length of 151 ms; Matlab 2017a, Mathworks, Natick, MA). BGA percentage change from baseline, defined as the period 500 to 100ms before visual stimulus presentation in each epoch, was computed.

Electrodes were then evaluated to determine if they were either Responsive or Selective to the presented stimuli in the 500-1000ms post stimulus onset time window. This window was selected based on the activation latencies of MPC and MTL in previous intracranial studies [44,56]. Face Responsiveness (FR) or Scene Responsiveness (SR) was determined by computing the z-score of trial-by-trial activity, averaged over the analysis window, for each task against its scrambled control condition. Electrodes with a z-score corresponding to a confidence greater than the Benjamini-Hochberg False Detection Rate (FDR) threshold of q<0.05 (calculated across all electrodes) were deemed stimulus responsive. Selectivity to Faces/Scenes within these responsive electrodes was determined by testing whether Faces showed a greater response than Scenes. Electrodes with q<0.05 and a positive z-score were termed Face Selective (FS) and those with a negative z-score were termed Scene Selective (SS). Identification Discrimination (ID) was determined as electrodes that were more active (q<0.05) for correctly identified stimuli against incorrectly identified stimuli. To compute grouped responses across subjects, while minimizing the impact of each individual, within-subject averages of all electrodes within each ROI were calculated and then averaged for the population.

The latency of FR (correct vs. scrambled faces) and Face ID (correct vs. incorrect faces) across ROIs was determined using the latency of the onset of broadband gamma significance that was above threshold for 50 ms. This was computed using a one-tailed Wilcoxon signed rank test applied to within-subject trial averages, tested between 200-700ms post stimulus onset (FDR q<0.01 across time points). To validate the separation of these values across regions, a bootstrap analysis with resampling, that used a random selection of 75% of trials from each patient was rerun 1000 times. Trial selection for each repetition was linked in both MPC and MTL to allow a paired analysis of latency differences between regions.

Multiple linear regressions were used to find the association between face and scene identity selectivity and the category selectivity of the electrodes within each ROI. Face vs. scene z-score was modelled as a function of two variables; correct vs. incorrect faces and correct vs. incorrect scenes. Adjusted r² values are reported.

Phase locking value (PLV) was calculated for patients with electrode coverage in both ROIs (n = 33 patients, 1719 electrode pairs). Phase information was extracted from the down-sampled (200 Hz) and low theta-band filtered signal (3 – 5 Hz; 3rd order, zero-phase Butterworth band-pass filter) [44-46] using a Hilbert transform. PLV was calculated as the circular mean (absolute vector length) of the instantaneous phase difference between each electrode pair at each time point [44] and baselined to the period −500 to −100 ms before stimulus presentation. Statistics were calculated using the mean PLV of correctly answered trials between 500 to 1000 ms after stimulus onset, comparing against a bootstrapped null distribution where MPC and MTL trials were randomly re-paired (2000 iterations). Statistical significance was accepted at an FDR corrected threshold of q<0.05. PLV time courses are presented as the between-subject average of all significant electrode pairs within each patient.

To provide statistically robust and topologically precise estimates of BGA, a population-level representation was created using a surface-based mixed-effects multilevel analysis (sb-MEMA) [40,43,81,86,87]. For the MEMAs, results were calculated for the time window 500 to 1000 ms after stimulus onset representing regions with a significant difference between the tested stimuli. A surface-based, geodesic Gaussian smoothing filter (2mm FWHM) was applied. Significance levels were then computed at a corrected alpha of 0.01 using family-wise error rate corrections for multiple comparisons. The minimum criterion for the family-wise error rate was determined by white-noise clustering analysis (Monte Carlo simulations, 1000 iterations) of data with the same dimension and smoothness as that analyzed [40]. Displayed activation clusters were restricted to regions with at least three patients contributing to coverage and BGA difference exceeding 5%. For the MEMA videos, results were calculated for 150 ms windows in 10 ms steps centered from −100 to 1200 ms from stimulus onset. Clusters were restricted to regions with at least three patients contributing to coverage and BGA exceeding 10%.

Supplementary Material

Movie 1

Video S1. Spread of Face-evoked Activity across the Cortical Surface. Related to Figure 3. Contrast MEMA video of the time course of broadband gamma activation across the cortical surface, displaying areas more active when viewing faces than scrambled faces. Regions in black did not have consistent coverage for reliable MEMA results.

Movie 2

Video S2. Spread of Scene-evoked Activity across the Cortical Surface. Related to Figure 3. Contrast MEMA video of the time course of broadband gamma activation across the cortical surface, displaying areas more active when viewing scenes than scrambled scenes. Regions in black did not have consistent coverage for reliable MEMA results.

Movie 3

Video S3. Temporal Evolution of Category Selectivity across the Cortical Surface. Related to Figure 3. 4D MEMA video of the contrast between the activations to faces and scenes across the cortical surface. Blue regions show preferential activation to faces and red represents preferential scene activation. Regions in black did not have consistent coverage for reliable MEMA results.