Corresponding ECoG and fMRI category-selective signals in human ventral temporal cortex

Abstract

Functional magnetic resonance imaging (fMRI) and electrocorticography (ECoG) research have been influential in revealing the functional characteristics of category-selective responses in human ventral temporal cortex (VTC). One important, but unanswered, question is how these two types of measurements might be related with respect to the VTC. Here we examined which components of the ECoG signal correspond to the fMRI response, by using a rare opportunity to measure both fMRI and ECoG responses from the same individuals to images of exemplars of various categories including faces, limbs, cars and houses. Our data reveal three key findings. First, we discovered that the coupling between fMRI and ECoG responses is frequency and time dependent. The strongest and most sustained correlation is observed between fMRI and high frequency broadband (HFB) ECoG responses (30–160hz). In contrast, the correlation between fMRI and ECoG signals in lower frequency bands is temporally transient, where the correlation is initially positive, but then tapers off or becomes negative. Second, we find that the strong and positive correlation between fMRI and ECoG signals in all frequency bands emerges rapidly around 100ms after stimulus onset, together with the onset of the first stimulus-driven neural signals in VTC. Third, we find that the spatial topology and representational structure of category-selectivity in VTC reflected in ECoG HFB responses mirrors the topology and structure observed with fMRI. These findings of a strong and rapid coupling between fMRI and HFB responses validate fMRI measurements of functional selectivity with recordings of direct neural activity and suggest that fMRI category-selective signals in VTC are associated with feed-forward neural processing.

INTRODUCTION

Humans rapidly and accurately categorize visual objects and scenes from very brief presentations (Grill-Spector & Kanwisher, 2005; Thorpe, Fize, & Marlot, 1996). This ability is thought to depend on neural computations performed along a hierarchy of cortical areas in the ventral visual stream extending from primary visual cortex to high-level visual regions in ventral temporal cortex (VTC) (Ungerleider & Mishkin, 1982). Lesions to the VTC can cause various forms of visual agnosia depending on the location and extent of the lesion (Farah, 1990; Konen, Behrmann, Nishimura, & Kastner, 2011; Rossion et al., 2003; Schiltz et al., 2006), suggesting a causal role of VTC in visual recognition. The functional properties of high-level visual regions in human VTC and their role in the recognition of various categories of stimuli has been examined by a large body of functional magnetic resonance imaging (fMRI) research, as well as intracranial electrophysiology (electrocorticography – ECoG and stereotaxic electroencephalography – SEEG) research.

fMRI has been instrumental in revealing the functional organization of category-selective responses in VTC within individual subjects (Grill-Spector & Weiner, 2014; Kanwisher, 2010; Levy, Hasson, Avidan, Hendler, & Malach, 2001; Malach, Levy, & Hasson, 2002; Weiner & Grill-Spector, 2010), because it is an non-invasive method that allows imaging the entire brain in millimeter resolution while the subject is performing a task. FMRI research discovered that focal regions in human VTC show higher responses to ecologically-relevant categories such as faces, bodyparts, and places compared to other stimuli (Epstein & Kanwisher, 1998; Kanwisher, McDermott, & Chun, 1997; McCarthy, Puce, Gore, & Allison, 1997; Peelen & Downing, 2005; Schwarzlose, Baker, & Kanwisher, 2005) and that these regions have a consistent spatial organization (topology) relative to the cortical folding and relative to each other (Nasr et al., 2011; Weiner et al., 2014; Weiner & Grill-Spector, 2010, 2013; Witthoft et al., 2014). Furthermore, responses in these category-selective regions are correlated with perception as fMRI responses are higher when stimuli of the preferred category are perceived than when they are not perceived (Andrews, Schluppeck, Homfray, Matthews, & Blakemore, 2002; Grill-Spector, Knouf, & Kanwisher, 2004; Hasson, Hendler, Ben Bashat, & Malach, 2001; Moutoussis & Zeki, 2002; Tong, Nakayama, Vaughan, & Kanwisher, 1998). Interestingly, fMRI experiments also showed that the VTC contains consistent distributed spatial patterns of response to a large array of visual categories (Connolly et al., 2012; Cox & Savoy, 2003; Haxby et al., 2001; Huth, Nishimoto, Vu, & Gallant, 2012; Kriegeskorte et al., 2008; Martin, Wiggs, Ungerleider, & Haxby, 1996; Spiridon & Kanwisher, 2002; Weiner & Grill-Spector, 2010; Weiner, Sayres, Vinberg, & Grill-Spector, 2010), even to those that are not associated with a ‘category-selective’ region, such as cars. These consistent spatial topologies in VTC can be captured in representational similarity analyses (Kriegeskorte, et al., 2008), manifesting as higher correlations among VTC distributed responses to images of the same category compared to images of different categories.

Likewise, ECoG and SEEG (referred to as ‘ECoG’ henceforth) research, which measure local electrophysiological neural activity from electrodes on or within the cortex, also uncovered category-selective responses in VTC (Allison, McCarthy, Nobre, Puce, & Belger, 1994; Allison, Puce, Spencer, & McCarthy, 1999; Bastin et al., 2013; Davidesco et al., 2014; Engell & McCarthy, 2011; Fisch et al., 2009; Liu, Agam, Madsen, & Kreiman, 2009; Murphey, Maunsell, Beauchamp, & Yoshor, 2009; Nobre, Allison, & McCarthy, 1994; Privman et al., 2007; Vidal et al., 2010). A set of influential studies was the first to report larger event-related potentials (ERPs) in electrodes placed in the VTC to specific categories such as faces compared to other categories (Allison, et al., 1994; Allison, et al., 1999; Halgren et al., 1994; Nobre, et al., 1994; Puce, Allison, Spencer, Spencer, & McCarthy, 1997). These ERPs peak around 170–220ms after stimulus onset and can also be recorded on the scalp (Bentin, Allison, Puce, Perez, & McCarthy, 1996; Jacques & Rossion, 2011), although the relationship between scalp and intracranially recorded face-selective ERPs is still debated (Rosburg et al., 2010). Later ECoG studies of VTC also reported increases in high-frequency broadband (HFB, >30Hz) power peaking around 150–400ms after stimulus onset (Bastin, et al., 2013; Davidesco et al., 2014;; Engell & McCarthy, 2011; Parvizi et al., 2012; Privman, et al., 2007; Vidal, et al., 2010). For example, several studies reported increases in HFB to images of faces compared to other stimuli (Engell & McCarthy, 2011; Parvizi, et al., 2012; Privman, et al., 2007), as well as to images of houses compared to other stimuli (Bastin, et al., 2013; Davidesco et al., 2014; Vidal, et al., 2010). These category-selective responses measured with ECoG are also correlated with perception, as they display larger amplitudes when subjects perceive the stimulus, compared to when they do not (Fisch, et al., 2009). Moreover, electrical stimulation of category-selective VTC electrodes produces category-specific deficits (Allison, et al., 1994; Allison, et al., 1999; Chong et al., 2013; Jonas et al., 2012; Jonas et al., 2014; Megevand et al., 2014; Parvizi, et al., 2012; Rangarajan et al., 2014), providing strong evidence for their causal role in perception.

These findings from fMRI and ECoG studies provide strong evidence for category-selective responses in human VTC and their role in perception. However, fMRI only provides an indirect measure of neural activity in VTC with a temporal resolution of seconds because it is based on a blood oxygen level dependent (BOLD) signal. While ECoG does provide a direct measure of local electrophysiological neural activity in millisecond resolution, prior ECoG studies did not report the topology of category-selectivity in VTC due to sparse electrode sampling of the cortex in a given brain. As a result, two key gaps in knowledge remain: (1) What is the anatomical origin and the spatial organization of ECoG category-selective responses across the cortical sheet of the VTC? (2) What is the neural origin of the BOLD category-selective signals measured in fMRI over the VTC?

The nature of the relationship between BOLD and electrophysiological signals in human VTC is poorly understood, even as studies in other parts of the brain suggest that the BOLD signal is correlated with the local field potential (Goense & Logothetis, 2008; Logothetis, Pauls, Augath, Trinath, & Oeltermann, 2001; Shmuel, Augath, Oeltermann, & Logothetis, 2006), especially in the HFB range (Conner, Ellmore, Pieters, DiSano, & Tandon, 2011; Hermes et al., 2012; Mukamel et al., 2005; Nir, Dinstein, Malach, & Heeger, 2008; Ojemann, Ojemann, & Ramsey, 2013; Winawer et al., 2013). First, the relationship between BOLD and neural responses signals varies across the brain (Conner, et al., 2011; Ojemann, et al., 2013), thus the nature of the relationship across measurements found in one part of the brain may not generalize to VTC. Second, the relationship between BOLD and electrophysiology has been studied most thoroughly in non-human primates (Goense & Logothetis, 2008; Logothetis, et al., 2001; Shmuel, et al., 2006), but it is premature to extrapolate across species given that homologies in high-level visual cortex are yet to be determined. Third, there has been no systematic investigation of the relationship between BOLD signals and direct neural responses in human VTC within the same individuals, as prior studies either compared signals across a handful of electrodes (Jonas, et al., 2014; Mundel et al., 2003; Murphey, et al., 2009; Parvizi, et al., 2012; Puce, et al., 1997) or across different groups of subjects (Bastin, et al., 2013; Privman, et al., 2007).

In investigating the neural origin of BOLD signals in VTC, comparing ECoG and fMRI signals within individuals provides a particularly appealing approach as both measurements have access to similarly sized population-level neural responses. ECoG electrodes are about 2mm in diameter, which is commensurate with the size of fMRI voxels (1–3mm). This similarity in the size of the probed neural population is advantageous over studies where BOLD signal is compared to single neuron or multi-unit recordings with microelectrodes (Boynton, 2011; Issa, Papanastassiou, & DiCarlo, 2013; Tsao, Freiwald, Tootell, & Livingstone, 2006). Comparing BOLD signals to the firing of single neurons is informative only in scenarios where the single neuron’s response is correlated with larger-scale population response (Logothetis, et al., 2001; Nir, et al., 2008). Furthermore, linking fMRI and ECoG signals has the potential to provide important insights regarding the evolution of category-selective responses across VTC in both time and space. This will enable linking the knowledge about the spatial organization of category-selective responses across the VTC gained from fMRI studies with the temporal signature of VTC responses revealed by ECoG studies (Fisch, et al., 2009; Liu, et al., 2009).

To directly examine the relationship between fMRI and ECoG signals in the human VTC we used a rare opportunity to measure both ECoG and fMRI signals in the same six individuals while they viewed images of various categories. We first examined the topology of ECoG category-selective responses in each individual brain and then related these responses to fMRI responses in the same subjects. We asked: (1) What is the spatial organization of category-selective ECoG responses in VTC relative to the cortical folding? (2) What is the representational structure of distributed ECoG responses to object categories across the VTC? (3) How do the spatial organization of category-selectivity and the representational structure of distributed responses in VTC compare across ECoG and fMRI measurements? (4) Does the relationship between ECoG and fMRI signals vary across time and frequency bands of the ECoG signal?

2. Materials and Methods

2.1. Participants

Six participating patients (3 females) ages 22–57 took part in ECoG and fMRI experiments. Prior to ECoG recordings, participants were implanted with intracranial electrodes for clinical purposes as part of pre-surgical evaluation of refractory epilepsy (Supplementary Table 1). All participants provided written informed consent to participate in the study. The procedure was approved by the Stanford Institutional Review Board. Patient 2 did not undergo any cortical tissue resection, and the cortex resected in the rest of the patients was outside the anatomically defined VTC (Supplementary Table 1).

2.2. Anatomical localization of electrodes

Participants were implanted with flexible strips or grids of subdural platinum electrodes (AdTech Medical Instrument). Electrode placement was guided by the clinical evaluation of each patient. Electrodes were 2.3mm in diameter of exposed surface and 5–10mm center-to-center inter-electrode spacing. The anatomical location of each electrode was determined by co-registering CT images collected post electrode implantation with a high-resolution anatomical MRI of each participant’s brain and further corrected to account for minor surgical brain shift (Foster & Parvizi, 2012; Hermes, Miller, Noordmans, Vansteensel, & Ramsey, 2010). We collected ECoG data from 7 hemispheres (4 right and 3 left hemispheres) in six patients, as one had bilateral electrode implantation.

We focus on ECoG measurements in electrodes located over VTC including the posterior portion of the inferior temporal sulcus (ITS), inferior temporal gyrus (ITG), lateral (LFG) and medial (MFG) fusiform gyrus, collateral sulcus (CoS), parahippocampal gyrus (PHG) and the anterior part of the lingual gyrus (LG). In all patients, the epileptic focus was determined to be outside our VTC regions of interest.

One hundred VTC electrodes were classified into four anatomical regions of interest (ROIs) in each participant’s native anatomy (Fig. 1): (1) ITG (6 participants, right hemisphere: 15 electrodes, left hemisphere: 16 electrodes); (2) LFG: FG lateral to the mid-fusiform sulcus (MFS) (6 participants, right hemisphere: 22 electrodes, left hemisphere: 16 electrodes). (3) MFG: FG medial to the MFS (4 participants, right hemisphere: 6 electrodes, left hemisphere: 3 electrodes); (4) CoS, PHG, and LG (4 participants, right hemisphere: 14 electrodes, left hemisphere: 8 electrodes). The posterior end of VTC was defined by the posterior transverse collateral sulcus (ptCoS) and the anterior extent was halfway into the temporal lobe.

Figure 1. ECoG responses averaged across anatomical partitions of VTC.

The figure depicts the average response across all visually responsive electrodes (the number of electrodes included is displayed at the top) in each of 4 anatomical partitions (bottom right inset): inferior temporal sulcus (ITS)/inferior temporal gyrus (ITG), lateral fusiform gyrus (LFG)/occipitotemporal sulcus (OTS), medial fusiform gyrus (MFG), collateral sulcus(Cos)/parahippocampal gyrus (PHG)/lingual gyrus (LG). (A) The average ECoG time-frequency response to images of dataset 1 for each category (rows) and anatomical partition (columns). Solid horizontal black lines indicate stimulus duration. The color-scale was adjusted for each anatomical region (see minimum and maximum values in the top-right corner of the top row plots). Data illustrate that HFB-responses (30–160Hz) encompass most of the power increase in the ECoG response. Lower frequencies (<12Hz) exhibit an initial power increase followed by a reduction of power after 0.2–0.3s relative to stimulus onset (Supplementary Figure S1). (B) ECoG HFB signal averaged over the same electrodes and anatomical partitions. Shaded regions around waveforms represent ± 1 SEM across electrodes, computed on amplitude normalized waveforms in each electrode. Solid horizontal black lines indicate stimulus duration. Insets show the first 0.3s after stimulus onset.

2.3. Stimuli and procedure

ECoG recordings and experiments were performed in the hospital rooms with the patients sitting in their beds. A Macbook pro laptop with a refresh rate of 60Hz placed on a tray table over the bed was used for stimulus presentation. During ECoG recordings, participants viewed grayscale photographs of faces, limbs, cars, and houses, presented in runs of 96 trials. We selected these stimuli as prior fMRI studies revealed both clustered and distributed VTC responses to these categories (Grill-Spector & Weiner, 2014; Weiner & Grill-Spector, 2010). The object depicted in each image appeared in variable position, size, lighting and viewpoint. Limb stimuli included both upper and lower limbs, always included the digits, and sometimes included the arms and the legs. Half of the images used were from a database as used in our previous studies (Grill-Spector, et al., 2004; Parvizi, et al., 2012; Weiner & Grill-Spector, 2010) and half were from another database collected by CJ (personal shots and images taken from the internet). Stimuli subtended approximately 10° × 10° of visual angle. Each stimulus was presented for 1 second, followed by a blank inter-trial interval of variable duration (0.6 – 1.4s). In each run, 48 images (12 from each category) were shown once. In addition, eight separate images, 2 from each category, were presented 6 times each. In each run, trial order was counterbalanced for category and repetition. Images were never repeated across runs and were randomly selected from our database such that each participant saw a unique sequence of images. Each run lasted for 192 seconds and participants were allowed as much time as needed between runs. Each participant completed 5–12 experimental runs. For all except one participant (P1), the whole experiment was performed in a single day (spread over three days for P1). Task: participants fixated a central black cross and pressed a key on an external USB numeric pad when its color changed to red (15 times per run). Due to equipment malfunction we obtained behavioral data in four out of six participants. Performance was near ceiling with 93 – 98% target detection. Only one participant (participant 4) produced four false alarms. Response times ranged from 520 to 580ms.

2.4. ECoG recording and analyses

Continuous ECoG signal was recorded at a 3052 Hz sampling rate using a 128-channel recording system (Tucker Davis Technologies). All recordings were made relative to the most electrocorticographically ‘silent’ ECoG electrode, defined as a noise- and artifact-free (e.g. movement artifact) electrode with minimal to no variance in voltage amplitude for multiple bipolar recordings, and distal from the identified seizure focus. Data processing was performed using custom functions in MATLAB (MathWorks, Natick, MA). Continuous ECoG data were notched filtered to remove line noise, resampled to 1017.3Hz and segmented in −1.8 to 1.8s epochs centered on the onset of each stimulus. Noisy epochs in which signal amplitude at any time-point in a −0.7 to 1s time-window was above or below 4.5 times the across-trial standard deviation were discarded. This resulted in rejecting about 8% of the trials on average (range: 4 – 13%). The signal at each electrode was then re-referenced using the following procedure. We first calculated the average raw signal across all electrodes in a given subject and subtracted it from the signal measured at each electrode (i.e. a common average reference). Because a large number of electrodes were visually responsive in our experiment, this average signal always contained visually-driven responses. Therefore, we next identified the 20% of the electrodes (minimum of 16 non-pathological electrodes) which average-subtracted signal was the most negatively correlated with the signal averaged across all electrodes, indicating that the signal at these electrode is driven by the subtraction process and that they are largely nonresponsive in our experiment. These electrodes were designated as the reference electrodes. Last, we averaged the original signal across these reference electrodes and subtracted it from the original signal of each of the subjects’ electrodes.

2.4.1. ECoG Time-frequency analyses

We used a Morlet wavelet approach (Tallon-Baudry, Bertrand, Delpuech, & Pernier, 1996) to estimate the amplitude of responses as a function of time and frequency. The wavelet transform was applied to each trial over a range of 2 – 160hz (1Hz steps) and the central frequency of the wavelet was adapted as a function of the frequency analyzed (from 2Hz at the lowest frequency to 9Hz at highest frequency). The resulting time-frequency envelope was decimated to 101.7Hz to save space and computation time in further analyses. Response amplitudes were transformed to percentage signal change relative to the mean amplitude in a pre-stimulus time-window (−600ms to −300ms). This was performed independently for each trial and frequency. Responses were then averaged over several frequency bands typically reported in ECoG studies (θ[theta]: 4–8Hz, α[alpha]: 8–12Hz, β[beta]: 12–30Hz) and high frequency broadband (HFB) range (Fig. 1, Supplementary Figure S1), which encompasses the low (30–80Hz) and high (>80Hz) gamma-frequency ranges. Since the ECoG signals in the HBF range are thought to be correlated with the local neuronal population spiking activity (Manning, Jacobs, Fried, & Kahana, 2009; Miller, Sorensen, Ojemann, & den Nijs, 2009; Ray & Maunsell, 2011) and in our data the signals in the 30–80Hz and >80Hz range showed similar characteristics (Fig. 1A), we report the broadest range for the HFB signals (30–160Hz).

We also estimated the temporal smoothing resulting from the time-frequency wavelet analysis by computing the full width at half maximum (FWHM) of the wavelet transform for each frequency band. This revealed that the timing information of ECoG signals is accurate up to 14ms for the HFB frequency range, 38ms for the beta range, 59ms for the alpha range and 81ms for the theta range, which correspond to half of the median FWHM of the wavelet transform across frequencies of a given frequency band.

2.4.2. Data sets

Responses were analyzed separately for two independent datasets. Data set 1 consisted of responses to images that were each shown once, with ECoG responses averaged across all images of a category (70 – 130 trials per participant after trial rejection). Data set 1 was used in analyses associated with Figs. 1, 2, 5, 6, 8, supplementary Figs S1, S3, and S4). Data set 2 consisted of an independent set of 40–80 images per participant containing 10–20 images per category, each shown 6 times. We analyzed the mean response to each image across its second to sixth presentation, therefore excluding the first presentation of repeated images to reduce potential adaptation effects (Engell & McCarthy, 2014b; Grill-Spector, Henson, & Martin, 2006). Reponses to individual images were averaged (average of 2 to 5 trials per image depending on the number of rejected noisy trials) to achieve higher signal-to-noise ratio per image. Data set 2 was used in analyses associated with Figs. 3, 4, 7, and supplementary Fig. S2. Analyses in Figs. 2, 3, 4, 5, and supplementary Fig. S2 were performed using the HFB averaged over a time-window of 100–350ms after stimulus onset.

Figure 2. Spatial distribution of ECoG category selectivity in human ventral temporal cortex is consistent across participants.

(A–B) Localization of electrodes, visual responsivity and category preference in each participant. (A) Spatial distribution of electrode selectivity in an example right hemisphere (ventral view of the temporal lobe), and HFB-responses as a function of time in three example electrodes spaced one centimeter apart. On the cortex, each electrode is represented by a pie chart plotting the relative amplitude elicited by each category (averaged over responses to images of dataset 1) in the HFB range (30–160 Hz) during a time window of 100–350ms after stimulus onset. The diameter of each pie chart is scaled according to its normalized amplitude of response. (B) Spatial organization of ECoG category-selectivity in all other hemispheres. Note that electrodes both within and outside VTC are shown, highlighting the particular sensitivity of VTC to visual object categories. (C) Selectivity in each electrode (d′) in four anatomical regions arranged from lateral to medial across the VTC. Each circle represents an electrode, where filled circles show significant positive category-selectivity (p < 0.005). For each electrode we show its selectivity to each category as well as to the animate vs. inanimate stimuli distinction. Horizontal lines represent the average d′ across electrodes for each anatomical region. Electrodes overlapping the mid-fusiform sulcus (MFS), lateral fusiform gyrus (LFG) and occipito-temporal sulcus (OTS) show face-selectivity. Preferential responses to inanimate stimuli occurred in the medial fusiform gyrus (MFG), collateral sulcus (CoS) and parahippocampal gyrus (PHG).

Figure 5. Overlapping face- and house-selectivity across ECoG HFB and fMRI signals.

A. Unthresholded maps of selectivity for faces (faces > limbs, cars, and houses; t value, voxel level) measured with fMRI are displayed for each participant. Inflated cortical surfaces show ventral visual cortex, including the VTC, with the occipital pole at bottom. Overlaid on these maps are circles indicating the magnitude of selectivity (contrast of faces > limbs, cars and houses, d′) of ECoG electrodes in the HFB range for a response averaged over a 100–350ms time window relative to stimulus onset. Note that the electrode layouts do not appear as perfect strips or grids on the cortical surface due to the cortical surface inflation process used for better visualization of activations inside sulci. The electrodes displayed are only those in regions where we could measure reliable fMRI responses. The diameter of each electrode has been increased for better visibility. Acronyms: OTS: occipitotemporal sulcus, MFS: mid-fusiform sulcus, CoS: Collateral sulcus. B. Same conventions as for A but showing selectivity for houses (house > faces, limbs, and cars).

Figure 6. The coupling between ECoG and fMRI face- and house-selectivity varies across time and frequency bands.

(A) Correlation between category-selectivity measured with fMRI and ECoG as a function of frequency band and time. Each line reflects the spatial correlation between fMRI and ECoG selectivity across time for a specific ECoG frequency band averaged across 6 subjects. Correlations between ECoG and fMRI were computed separately for faces-selectivity and house-selectivity, then averaged in each subject, and last averaged across subjects. Shaded regions: across-subject SEM. Line thickness indicates whether the across-subject correlation is significantly different from zero (two-tailed percentile bootstrap test, uncorrected). (B) Correlation between fMRI and ECoG selectivity across the frequency spectrum (3–160Hz) for two time-windows. For both (A) and (B) the contribution of fMRI signal from each voxel was weighted using a Gaussian kernel of 3.5mm STD around each electrode (Supplementary Figs. S3 and S4).

Figure 8. ECoG HFB response latency across visually responsive VTC electrodes.

Response latencies at each visually responsive VTC electrode are shown for each category and participant (filled circles), along with the average (± STD) across electrodes in each participant (open circles) and the average across all electrodes and participants by category (black diamonds). The number of electrodes included for each category is shown below each group of plots. Some electrodes respond significantly to more than one category. Thus, the same electrode may appear in multiple category bins.

Figure 3. Single image category-selectivity in VTC.

Each cell shows the normalized HFB response amplitude to a single image averaged across 2–5 presentations over a 100–350ms time window in one electrode taken from data set 2. Electrodes are represented in rows, grouped by anatomical region and category-preference and sorted by descending d′ from data set 1 (Fig. 2C). Responses to images are shown in columns, sorted by appearance order for each category. We measured responses to 40–80 images per electrode depending on the number of blocks of data collected. Gray cells represent missing data for a particular subject/electrode. d′ that are significantly higher than zero (p<.005, 2-tailed permutation test) for data set 1, are indicated on the right and color-coded by category preference. Electrodes that were not significantly selective to any of the four categories are on the bottom of each anatomical region and no d′ is reported.

Figure 4. Representational similarity matrices (RSM) of distributed HFB responses across VTC.

RSM of distributed VTC responses between each pairs of images from data set 2 are shown for each participant. Distributed responses were calculated for each participant across all of their VTC electrodes (electrode count indicated at the top of each panel) for HFB-responses averaged over a time window of 100–350ms after stimulus onset. Responses were averaged across 2–5 presentations of an image (excluding the first presentation). The top triangle shows the correlation of the distributed responses to each pair of images (40–80 images per participant), the bottom triangle shows the average correlation across all pairs of images of a category, excluding the diagonal.

Figure 7. Category representation structure from VTC patterns correlate across fMRI and ECoG HFB from 100ms after stimulus onset.

(A) ECoG RSMs: representational similarity matrices (RSM) of distributed VTC responses for different frequency bands at four time intervals. RSMs are computed from dataset 2, averaged across images in each participant, and then across participants S1 – S5, who had at least two fMRI scans. Acronyms: F: faces; L: limbs; C: cars; H: houses. (b) fMRI RSM: Across participants’ average (S1–S5) RSMs of distributed fMRI signals from voxels under ECoG VTC electrodes. (C) Average correlation between fMRI RSMs and ECoG RSMs across time and frequency band. Correlations were computed in each participant and then averaged across participants. Line thickness indicates if the across-participant correlation is significantly different from zero (two-tailed percentile bootstrap test, uncorrected). Shaded regions indicate across-participant SEM.

2.4.3. Defining visually responsive electrodes

Visually responsive electrodes were defined as electrodes with ECoG signal in HFB range (30–160 Hz) that was 4 times larger than its across-trial standard error. This was determined using images of dataset 1, averaging HFB amplitude over a time window of 100–350ms after stimulus onset.

2.4.4. Visualizing single electrode category profile over each participant’s VTC

Fig. 2 shows for each electrode, the relative amplitude elicited by the average HFB responses across images of a category (dataset 1) during a time window of 100–350ms after stimulus onset in the form of pie charts. The diameter of each pie chart was scaled according to the normalized response amplitude in the 100–350ms time-window. The amplitude was normalized using a metric similar to an effect size metric, which is defined as the ratio between the mean response amplitude and the across-trial standard deviation. Normalized amplitude was computed separately for each category and the diameter of each pie chart was scaled according to the highest normalized amplitude across categories.

2.4.5. Quantifying ECoG category-selectivity in individual electrodes

Category selectivity (d′) for dataset 1 was estimated for each electrode and category using ECoG HFB responses averaged over a 100–350ms time window; d′ (j), selectivity for category j, was defined as:

$$d^{\prime }(j)=\frac{{\mu }_j-\frac{1}{N}{\displaystyle \sum _i^N}{\mu }_i}{\sqrt{\frac{1}{2}\left({\sigma }_j^2+\frac{1}{N}{\displaystyle \sum _i^N}{\sigma }_i^2\right)}};i\ne j.$$

Where μ_i is the mean response to category i; σ_i is the across-trial standard deviation of responses to category i; N=3. The statistical significance of d′(j) was assessed using a permutation test, where we randomly shuffled the category label of all trials, calculating d′(j) for the shuffled data and repeating this procedure 10000 times. The p-value was computed as the fraction of the permuted d′ distribution that was greater than the observed d′. We determined the mean selectivity across electrodes in each anatomical ROI and tested if it was significantly above zero using a one-tailed t-test.

2.4.6. Analyses of representational similarity from distributed ECoG data

We used representational similarity analyses (Edelman, 1998; Kriegeskorte, et al., 2008) to explore the representational structure of category information in distributed ECoG HFB responses over VTC. We computed representational similarity matrices (RSM) by first extracting for each participant the distributed pattern of responses across their VTC electrodes (on average 12.7±6.7 electrodes per participant) for each image from data set 2 viewed by the participant. We used ECoG HFB responses averaged over a 100–350ms time-window. These RSMs indicate the mean similarity (indexed by Pearson’s r correlation coefficient) between distributed patterns of VTC responses to each pair of images a participant viewed. The distributed pattern of a response to an image corresponds to the vector containing the amplitude of response to that image across VTC electrodes. To remove between-electrode differences in response magnitudes, each electrode’s response amplitudes across all presented images were normalized by transforming them into z-scores. Then, in a given participant, we compute an RSM by correlating the z-scored amplitude vectors across all images. We then obtain an RSM at the category level by calculating in each participant the average similarity among distributed responses to images of the same or different categories. For within-category comparisons we excluded the similarity among distributed responses to the same image (which by definition are equal to 1).

2.4.7 Estimation of HFB response onset latency

Response latency of ECoG HFB responses in VTC electrodes was defined as the latency at which HFB responses first rose above the baseline level. This was estimated by first fitting a cumulative Gaussian to the initial portion (−100 to 300 ms relative to stimulus onset) of the ECoG response, then calculating the intersection of a line fitted to the linear portion of this Gaussian with the baseline level response. The intersection indicates the earliest time point at which signal rises above baseline. Response latency was estimated for each VTC electrode and each category using the average response across images of a category from dataset 1.

2.5. fMRI measurements and analyses

MRI data were obtained using GE 3-Tesla Signa scanners at Stanford University. Anatomical MRIs were collected before the ECoG grid implantations for all subjects and were used for registering the ECoG electrodes, registering fMRI activations, and for cortical surface reconstruction. Due to logistical issues we could not collect fMRI on all subjects prior to the implantation. fMRI data were collected for two subjects before implantation (P2: 2 month before, P3: 4 days before) and for four subjects after the ECoG recordings (P1: 20 month after; P4: 11 month after; P5: 17 days after; P6: 10 month after).

2.5.1. Anatomy

A high-resolution anatomical volume of the whole brain was acquired with a head coil using a T1-weighted SPGR pulse sequence. Data were aligned to the AC-PC plane and resampled to 1mm isotropic voxels. Both fMRI data and ECoG electrode locations were aligned to this brain volume. This volume was segmented to gray and white matter, and the resulting segmentation was used to reconstruct the cortical surface of each participant.

2.5.2. fMRI acquisition

Participants were scanned using a T2* sequence (TE=30 ms, TR=2000 ms, flip angle=77° and bandwidth=128 kHz). Participants were scanned with voxels ranging from 1.5mm to 3mm as there is a tradeoff between coverage and resolution and some participants were scanned on additional fMRI studies in the same session. When possible, we used smaller voxels as they improve localization of functional activations and reduce the effects of susceptibility artifacts (Weiner & Grill-Spector, 2013). Scanning parameters were kept constant within a participant. We used the following scanning resolutions and parameters: P1, P6: 2.8×2.8×2.5 mm, 30 slices, FOV=180 mm; P2: 1.8 mm isotropic, 28 slices, FOV=192 mm; P3: 3 mm isotropic, 32 slices, FOV=192 mm; P4: 2.4 mm isotropic, 34 slices, FOV=192 mm; and P5: 1.5×1.5×3 mm 12 slices, FOV=192 mm,

2.5.3. fMRI Experiment

Participants performed 1–3 runs of a block design experiment during which grayscale photographs of faces, limbs, flowers, houses, cars, guitars, and scrambled objects were shown in 12s blocks (same as (Parvizi, et al., 2012; Weiner & Grill-Spector, 2010)). Each block contained images of a single category. Each run consisted of 4 blocks of each condition and 6 blank blocks and lasted for 408 seconds. Categories were counterbalanced within each run and images were not repeated across runs. During a block, each image appeared for 0.75 s with an interstimulus interval of 0.25 s. Images subtended a visual angle of 7.1°. The images used in fMRI were from the same database as half of the images used during ECoG and the same as in our prior fMRI experiments (Grill-Spector, et al., 2004; Weiner & Grill-Spector, 2010). In the present study we only considered BOLD responses to the same four categories as in the ECoG experiment (i.e. faces, limbs, cars, houses). Task: Participants were instructed to fixate on a centrally presented dot and respond by button press when two consecutive images were identical. Repetitions occurred randomly and at a low frequency (~6% of the images). This task requires subjects to attend to the individual images independently of their category, and we used a low repetition frequency to avoid adaptation effects. Due to technical problems we were able to collect behavioral data during fMRI in only 4 out of the 6 participants. Three of the participants had performance (accuracy and RT) within the same range as typical adults participants measured in prior identical experiments (accuracy range: 40 – 94%, response time range: 540 – 933ms (Weiner & Grill-Spector, 2010)). One participant (P5) had typical RT but more misses than controls, however her fMRI activations are typical (see Fig. 5).

2.5.4. fMRI data Analysis

Data were analyzed with MATLAB using the mrVista toolbox (http://white.stanford.edu/software) as in our prior publications (Parvizi, et al., 2012; Weiner, et al., 2014; Weiner & Grill-Spector, 2010). Data were motion corrected, detrended, and aligned to the participant’s whole brain anatomy. There was no spatial smoothing of fMRI responses. We fit every voxel’s time course using standard GLM analyses.

2.6. Measuring the correspondence between ECoG and fMRI

2.6.1. Quantifying the correspondence of the spatial pattern of face- and house-selectivity between ECoG and fMRI measurements

Since the most consistent selectivity measurable both in ECoG and fMRI was found for faces (present in all 6 participants) and houses (present in 4/6 participants), we quantified the correspondence of category-selective signals across methodologies using these two categories. Specifically, we tested whether the spatial patterns of face-selectivity and house-selectivity measured with ECoG and fMRI across VTC electrodes were correlated without thresholding either measurement.

We first compared fMRI selectivity to ECoG selectivity in the HFB range averaged over a 100–350 ms time-window as signals in this frequency range correlate with the local population neural firing, and this time-window corresponds to the peak of the initial visual response in VTC. Next, we compared fMRI selectivity to ECoG selectivity across frequency-bands and time to identify which component of the ECoG signal best matches fMRI, addressing the ongoing debate on the nature of the relationship between these signals (Hermes, et al., 2012; Ojemann, et al., 2013; Scheeringa et al., 2011). The temporal and frequency band variability in the correlation between ECoG and fMRI arose from the time-frequency varying nature of the ECoG selectivity, as fMRI selectivity was constant for each electrode.

Face- and house-selectivity in both methods were computed using the d′ metric (face vs. limbs, cars, houses or houses vs. limbs, cars, faces; see equation in ‘Estimating category selectivity’). We measured the correlation (using Pearson’s coefficient) between ECoG and fMRI selectivity across all visually responsive VTC electrode locations in each participant excluding electrodes overlapping fMRI signal dropout regions that have low SNR. An average of 7.7 ± 2 electrodes per participant were used. ECoG selectivity was defined in the following way: ECoG responses from dataset 1 in each frequency band (θ: 4–8Hz, α:8–12Hz, β: 12–30Hz, HFB: 30–160Hz) and each electrode were temporally smoothed using a Gaussian kernel (STD: 20ms) to accommodate for differences in response latencies across electrodes. These responses were used to compute the d′ for faces (vs. limbs, cars, houses) or houses (vs. faces, limbs, cars) in each electrode, frequency band and 10ms time-bins over an interval of −100ms to 1500ms relative to stimulus onset. fMRI selectivity in each voxel was defined in the following way: We first estimated d′ for faces (or houses) vs. other stimuli, only including the same four categories as for the ECoG data (faces, limbs, cars, houses). Then we calculated the weighted average of fMRI d′ across gray matter voxels in the vicinity of each electrode for which the GLM explained more than 5% of their variance. Each voxel’s d′ was weighted by its distance from the electrode in the brain volume using a 3D Gaussian kernel. Since the spread of ECoG signal in the brain volume might depend on the frequency band and the orientation of the electrode relative to the cortical surface, we parametrically varied the spatial extent of the Gaussian kernel from 1–20mm around each electrodes and computed the corresponding ECoG-fMRI d′ correlations.

To estimate the significance of the correlations between ECoG and fMRI responses across participants, we tested if correlations significantly deviated from zero across participants using a 2-tailed percentile bootstrap test (Efron & Tibshirani, 1993). Specifically, we selected six participants with replacement from the pool of participants and the average correlation across this sample was computed and stored. This was done for all possible sampling combinations of participants with replacement (462 possibilities for a sample size of 6) to obtain a distribution of bootstrapped estimates of the correlation. A two-tailed p-value was computed as the fraction of the bootstrapped estimates that were either smaller or larger than zero depending on the sign of the correlation. The minimal attainable p-value was constrained by the number of resamples performed (here the minimal p-value is 2/462 = 0.0043).

2.6.2. Comparison of representational similarity across fMRI and ECoG data

We used representational similarity analyses to compare the representational structure of category information in distributed VTC responses across ECoG and fMRI measurements. For ECoG measurements, we computed RSMs from ECoG responses in 10ms time bins from 100ms before stimulus onset to 1500ms after stimulus onset and for four frequency bands (θ: 4–8Hz, α:8–12Hz, β: 12–30Hz, HFB: 30–160Hz). For fMRI measurements, RSMs were computed as follows. In each participant we extracted the average fMRI signal from a 5mm radius gray matter ROIs beneath each of their VTC electrodes. Based on these fMRI signals we generated a vector of distributed fMRI responses (z-scored amplitudes (Weiner & Grill-Spector, 2010)) under the electrodes for each of the four categories that were used in the ECoG experiment (faces, limbs, houses, and cars). This distributed response was measured separately for the first and second runs in which participants saw different images from these categories. We measured the correlation among distributed z-scored responses to images of the same and different categories, generating an RSM matrix of VTC responses for each participant (participant 6 was excluded as he had only one run of data) and averaged the corresponding upper and lower off-diagonal correlation values (as this is a non-symmetric cross-correlation matrix).

We quantified the correspondence between the category representational structure of distributed VTC responses in each participant by correlating RSMs measured in ECoG (across time and frequency bands) and fMRI. We used a 2-tailed percentile bootstrap test (sampling participants with replacement) to estimate whether correlations significantly deviated from zero across participants.

3. RESULTS

3.1. ECoG responses in the HFB range reveal a robust and reproducible spatial topology of category information in VTC

3.1.1. Consistent spatial organization of HFB category-selective responses relative to anatomical landmarks

To examine the spatial topology of category-selective responses in VTC, we first examined ECoG responses in the HFB range in each individual. Analyses of HFB responses from ECoG electrodes showed that the majority of VTC electrodes (76/100) were visually responsive, with HFB signals emerging 80–100ms after stimulus onset (Fig. 1, Fig. 2a). Seventy percent of visually responsive VTC electrodes (53/76 electrodes) showed a significant preference for one category, and 10% (8/76 electrodes) showed a preference for two categories (p < 0.005; 2-tailed permutation test, Fig. 2c). For example, three electrodes spaced 1 cm apart, arranged laterally to medially over the OTS, FG and CoS, showed a differential preference to limbs (1), faces (2), and houses (3, Fig. 2a). The largest responses typically occurred 100–350ms after stimulus onset (Fig. 1, Fig. 2a), and the rank ordering among responses to different stimuli in a given electrode remained mostly stable across the duration of stimulus presentation (Fig. 1, Fig. 2a). In addition, while most electrodes showed a significant preference for a particular category, they generally also exhibited significant above-baseline responses for other non-preferred categories. Of the 76 visually responsive electrodes, 71 (93%) electrodes showed a significant response above baseline to faces, 66 (87%) to limbs, 54 (71%) to houses and 66 (87%) to cars. Thus, the proportion of the visually responsive electrodes that showed a significant response for 4, 3, 2 and 1 category was 62%, 21%, 11% and 7%, respectively.

Category-selective responses over VTC measured in the HFB range show a consistent topology relative to the cortical folding in individual brains. Face- and limb-selective responses were detected in the lateral VTC, in the lateral fusiform gyrus (LFG) and inferior temporal gyrus (ITG). All participants had face-selective electrodes located on the LFG. In fact 33/38 electrodes in the LFG showed significant preference for faces (average d′ for faces over 38 electrodes = 1.54±1.13, Fig. 2c). These electrodes were near or lateral to the mid fusiform sulcus (MFS) occasionally extending to the occipitotemporal sulcus (OTS) and ITG (8/17 electrodes), which had overall lower face-selectivity compared to the LFG (d′ for faces over 17 ITG electrodes: 0.59±1).

Fewer electrodes displayed preference for limbs over other stimuli. Limb-selective electrodes were located on the ITG (4/17 electrodes) and the LFG/OTS (4/38 electrodes, Fig 2. S1, S4) and were lateral to electrodes preferring faces in the same participants. Medial to the MFS (medial FG (MFG), in the collateral sulcus (CoS) parahippocampal gyrus (PHG) and lingual gyrus (LG)) we generally found stronger responses to inanimate than animate stimuli (11/21 electrodes preferring inanimate stimuli, 3/21 electrodes preferring animate), and particularly to houses (9/21 electrodes; Fig. 2: S1, S2, S5, S6). Notably, there were no electrodes with preference to faces in the CoS, PHG, or LG. Few electrodes showed preferential responses to cars (5/76), and they tended to also exhibit preference for limbs or houses (Figs. 2 and 3) with no clear anatomical distribution. These data also reveal a large-scale lateral-medial VTC gradient of animacy, where electrodes preferring animate stimuli (faces and limbs) are located in lateral VTC, and electrodes preferring inanimate stimuli (houses and cars) are located in medial VTC (Figs. 2 and 3).

This pattern of selectivity was validated in an independent set of stimuli in which we measured responses to individual images of faces, limbs, houses and cars in each visually responsive VTC electrode (Fig. 3). Particularly striking was the selectivity to faces in face-selective electrodes in the LFG, where responses to each of the face images were consistently strong and higher than to nonfaces. In fact for 15/38 electrodes in the LFG, responses to any of the face images were larger than responses to any of the non-face images (Fig. 3 - Lateral FG). While this analysis with individual images also replicated the preference for houses and inanimate stimuli in the MFG and CoS, the observed selectivity for these stimuli was overall lower. For example, electrodes that showed strong HFB responses to houses in the MFG and CoS/PHG/LG did not respond robustly to all house images and in fact they responded robustly to many car images and even to face and limb images (Fig. 3 – bottom 2 rows).

3.1.2. Distributed ECoG VTC responses to single images reveal a categorical structure

Additional insight about the information structure of VTC responses can be gleaned from the analyses of distributed HFB responses across all VTC electrodes. Examination of the representational similarity structure of distributed HFB responses in each participant reveals that distributed VTC response patterns to images of the same category are more similar than distributed responses to images of different categories (Fig. 4). The average correlation among spatial patterns generated by images of the same category was positive (r=0.29±0.09; average between-category similarity) and significantly higher (t(5)=8.4, p<0.0005) than the average correlation among patterns of response to images of different categories (r=−0.11±0.03). This categorical nature of responses was observed even for categories that did not display strong category preference at the level of individual electrodes or for which we did not find selectivity in each participant (e.g. cars). Nevertheless, consistent with the selectivity of individual electrodes (Fig. 3), the highest within-category similarity occurred among distributed responses to face images (r=0.47±0.12), followed by house (0.3±0.09), limb (0.23±0.12), and least among car images (0.15±0.09). Additionally, there was a distinction between distributed responses to animate vs. inanimate stimuli, as they generated anti-correlated distributed response patterns across the VTC (correlation between animate and inanimate images: r=−0.17±0.08). The distributed response patterns differentiating animate and inanimate stimuli are a consequence of the lateral-medial arrangement of animate representations in the lateral VTC and inanimate representations on the medial VTC (Figs. 2, 3). Critically, distributed HFB responses in low-level visual regions, do not display such a categorical structure (Supplementary Fig. S2), indicating that the categorical structure of distributed HFB responses in VTC is not driven by low-level differences among these images. These results reveal that the consistent and reproducible spatial topologies of HFB responses to different categories identified in the previous section generate a consistent representational structure of category information in VTC.

3.2. Examination of the correspondence between fMRI signals and ECoG responses across time and frequency bands

The results described in the prior section illustrate that direct measurements of neural responses in the HFB range largely replicate findings from fMRI research. In the second part of our study we directly compared fMRI and ECoG responses in each of our participants addressing three main questions: (1) What is the spatial correspondence between face- and house-selectivity measured with ECoG HFB to that measured with fMRI? We focused on these categories as they generated selective responses in both ECoG and fMRI in most of our participants. (2) Is the same correspondence found for other ECoG frequency bands and does it vary across time? (3) What is the relationship between the distributed representational similarity structure of category information across the VTC measured with ECoG and fMRI, and does it vary across time and frequency band?

3.2.1. HFB and fMRI category-selectivity measured in the same participant spatially overlap

Fig. 5 illustrates the degree of selectivity in each VTC voxel to faces (top) or houses (bottom) measured with fMRI in each of our participants. Superimposed on this unthresholded fMRI selectivity map are the ECoG electrodes, which are colored by their level of selectivity to faces (top) or houses (bottom) in the HFB range (d′ from Fig. 2). As expected, fMRI responses showed higher responses to faces vs. other stimuli in the FG (Fig. 5A). Importantly, in all participants we found a clear spatial overlap between regions showing preferential responses to faces measured with fMRI and those measured with ECoG in the HFB range (Fig. 5A). Specifically, when electrodes were located over or directly abutting a region showing positive face-selectivity measured with fMRI they always showed positive face-selectivity in the HFB signal (Fig. 5A, compare yellow electrodes with yellow fMRI activations). Mismatches between measurements were restricted to regions where we could obtain ECoG signals but not fMRI signals due to MR signal dropout in the vicinity of the ear canals, generally in the ITG and anterior lateral FG, and to cortex under a single electrode, which showed face-selectivity in HFB but not fMRI (Fig. 5A, S5, right-most electrode). Furthermore, electrodes that displayed zero or negative selectivity for faces overlapped regions with low or negative selectivity to faces in fMRI. Likewise, we found a spatial correspondence between regions that displayed preferential responses to houses across fMRI and HFB (Fig. 5B). Higher fMRI responses for houses than the other stimuli were found in the CoS, PHG and LG. Overlapping these fMRI activations, we found electrodes with higher HFB responses for houses compared to other categories (Fig. 5B, S1, S2, S5, S6). Conversely, electrodes that displayed lower or negative selectivity for houses than other stimuli overlapped with fMRI regions that showed lower or negative responses as well.

3.2.2. The coupling between ECoG and fMRI responses varies across time and frequency bands

We next quantified the spatial correspondence between fMRI and ECoG responses across time and frequency bands. In each participant, we compared the degree of category selectivity to faces or houses across electrodes to the degree of category selectivity measured with fMRI in the cortex under the same electrodes (See Materials and Methods). We found that the highest and most significant correlation was between ECoG selectivity measured in the HFB range and fMRI selectivity. This significant correlation emerged rapidly, about 80–100ms after stimulus onset and persisted for the remainder of the stimulus duration (average Pearson correlation±STD across participants: 0.68±0.05, significantly above zero, p<0.005, 2-tailed bootstrap test, Fig. 6A).

In contrast, for lower frequency bands of the ECoG signal the correlation with fMRI responses varied with time (Fig. 6). While it was significantly positive 100–200ms after stimulus onset (Pearson r±STD θ: 0.61±0.07, α: 0.58±0.06, β: 0.65±0.05, ps<0.005 2-tailed bootstrap test), at later times the correlations decreased, becoming negative 400–800ms after stimulus onset in the θ and α bands. This negative correlation at low frequencies was due to lower than baseline ECoG power in these frequencies at these later times (Fig. 1, Supplementary Fig. S1) that generated a weak negative selectivity (r±STD in 400–800ms window for θ: −0.17±0.13, α: −0.25±0.08, ps<0.005 for α). In this later time-window, correlations in the β range exhibited a transitory profile between low and high-frequencies (r±STD: 0.34±0.09, Fig. 6B). Importantly, across frequency bands, the correlations between ECoG and fMRI signals were highest when considering the local fMRI signal within a radius of 2–5mm around the center coordinates of ECoG electrodes (Supplementary Figs. S3, S4). Interestingly, the spatial coupling between ECoG and fMRI responses tended to be tighter for face-selectivity than for house-selectivity. When comparing face-selectivity across measurements the spatial correlation dropped sharply when fMRI signals were pooled more than 6 mm away from the electrodes (Supplementary Fig. S3). However when comparing house-selectivity across methodologies, the correspondence persisted across a larger spatial range (Supplementary Fig. S4).

Comparing the representational similarity of distributed responses to object categories across the VTC measured with ECoG (Fig. 7A) to fMRI (Fig. 7B) also showed that the coupling between measurements varies across time and frequency bands (Fig. 7C). Distributed fMRI responses across the VTC show a clear categorical structure (Fig. 7B). Distributed responses to images of the same category are positively correlated, and these correlations are higher than the correlation between distributed responses to images of different categories. However, for ECoG responses the representation structure varies across frequency band and time. While the RSM of distributed HFB signals shows a clear categorical structure from approximately 100ms after stimulus onset until stimulus offset (Fig. 7A–HFB), RSMs of distributed signals in lower frequency bands (α, β, θ) display a more transient structure, exhibiting a category structure from about 100ms to 300ms after stimulus onset but little to no category structure in later times (Fig. 7A).

Consequently, HFB and fMRI RSMs are significantly and positively correlated in each participant from 100ms till stimulus offset (p<0.05, two-tailed bootstrap test against zero). This correlation between measurements is maximal from 100 to 300ms (mean correlation between fMRI and ECoG HFB RSM in 100–300ms time-window: Pearson’s r=0.7±0.13; p<0.005; Fig. 7C). In contrast, ECoG RSMs in lower frequencies are significantly and positively correlated to fMRI RSMs only from about 100 to 300ms after stimulus onset (mean Pearson correlation in 100–300ms window: θ[4–8Hz]: r=0.60±0.11, α[8–12Hz]: r=0.64±0.27, β[12–30Hz]: r=0.57±0.12, ps<0.005). At later time points, the correlation decreases and becomes non-significant (Fig. 7C) because responses in the lower frequency bands lose their coherent category structure (Fig. 7A). These data suggest that spatial topologies across VTC which convey categorical information are most similar across fMRI and ECoG HFB measurements, starting within 100ms after stimulus onset and persisting until stimulus offset.

3.2.3. The significant correlation between HFB and fMRI responses emerges together with the initial onset of signals in VTC

Our findings reveal that the initial coupling between ECoG HFB and fMRI responses starts at about 80–100ms after stimulus onset. It is interesting to note that this timing matches the time-point when stimulus-driven HFB signals in VTC electrodes rise from the baseline level, which is at 91ms±18ms after stimulus onset (Fig 8). Since the latency at which the spatial patterns of category-selectivity and representational structure start to correlate between fMRI and ECoG measurements matches the latency of the initial response onset in VTC, our data suggests that these fMRI responses are associated with the initial feed forward responses in VTC.

4. Discussion

Our data reveal three key findings. First, we find that the spatial topology and representational structure of category-selectivity in VTC reflected in HFB ECoG responses mirrors the structure observed with fMRI. Second, we discovered that the coupling between fMRI signals and ECoG in VTC is frequency and time dependent. The strongest and most sustained correlation is observed between fMRI and HFB ECoG responses. In contrast, the correlation between fMRI and ECoG signals in lower frequency bands is temporally transient. Third, we find that the strong and positive correlation between fMRI and ECoG responses emerges rapidly around 100ms after stimulus onset, together with the onset of the first stimulus driven neural signals in VTC. These findings are important because they validate fMRI maps of functional selectivity in individual participants’ brains with a direct measure of neural activity and provide a millisecond timestamp for fMRI activations in human VTC.

4.1 The correlation between fMRI and ECoG responses varies across time and frequency bands

Our data reveal a strong and positive correlation between fMRI responses and ECoG signals in the HFB range from about 100ms after stimulus onset throughout the entire stimulus duration (Figs 6, 7). These results are in line with a growing body of evidence showing a coupling between electrophysiological signals in the HFB range and BOLD responses in human motor cortex (Hermes, et al., 2012; Siero et al., 2014), in early visual cortex (Winawer, et al., 2013), in language-related cortex (Conner, et al., 2011; Hermes et al., 2014; Lachaux et al., 2007), as well as in nonhuman primate visual cortex (Goense & Logothetis, 2008). Here we show direct evidence that this relationship also occurs in the human VTC when measuring ECoG and fMRI in the same participants. These results also extend previous reports that showed correlations between fMRI and ECoG HFB signals in VTC across different groups of participants (Bastin, et al., 2013; Privman, et al., 2007). Since the HFB signal correlates with the aggregate spiking activity across a neuronal population (Manning, et al., 2009; Miller, et al., 2009; Nir et al., 2007; Ray & Maunsell, 2011), our data suggest that the spatial topologies of category information measured via fMRI in VTC reflect, at least partly, the local population neural firing.

In contrast to the sustained and strong positive correlation between fMRI and ECoG signals in the HFB range, our data revealed a time-varying coupling between fMRI and ECoG signals in lower frequencies. Early (100–300ms) ECoG responses across all frequency bands were significantly and positively correlated with fMRI signals, while later (400–800ms) ECoG responses at lower frequencies (<30Hz) showed negative (when correlating face and house selectivity) or lower (when correlating RSMs) correlation with fMRI signals (Figs. 6, 7 and Supplementary Figs. S3, S4). The former negative correlations at later time windows are driven by decreases in power relative to pre-stimulus baseline, which generate negative face- and house-selectivity in lower ECoG frequency bands (Fig. 1, Supplementary Fig. 1). The lack of significant correlation between RSMs measured with fMRI and ECoG at later time windows is a consequence of the loss of a coherent category representational structure of ECoG responses at these lower frequencies. This dissociation of category information in the HFB and lower frequency bands underscores the importance of examining multiple components of electrical signals from intracranial recordings.

Future research will determine if ECoG power modulations in low frequencies measured in early and late time-windows reflect distinct neural phenomena, such as an initial ERP (Engell & McCarthy, 2011; Fisch, et al., 2009; Winawer, et al., 2013) generated by the input to VTC, followed by later power reduction in low frequency bands reflecting a modulatory signal (Pfurtscheller & Lopes da Silva, 1999; Schroeder & Lakatos, 2009). Nevertheless, our data underscore the power of relating ECoG and fMRI measurements in individual brains and highlight the importance of examining both the temporal and spectral components of electrical neural signals to understand the relation between signals recorded in these different methodologies.

4.2 Linking between fMRI and HFB responses in VTC: Methodological considerations

While our data show that the spatial coupling across ECoG and fMRI is tight (2–5mm around the electrodes, Supplementary Figs. S3, S4) we also found that this spatial coupling was better when comparing face-selectivity than house-selectivity across methods. While we cannot determine with certainty why this is the case, several factors may contribute to this difference. First, face-selective responses in VTC are located largely on gyri (FG, ITG), while house-selective responses are found mostly in a sulcus (CoS). Because ECoG grids are in direct contact with cortex on gyri and away from cortex within sulci, face-selectivity provides a better opportunity to measure local coupling between ECoG and fMRI signals. Second, in our sample there were substantially more face- than house-selective electrodes, and face-selective electrodes also had a higher selectivity. This combination of factors produced an enhanced dynamic range of face-selectivity compared to house-selectivity in ECoG responses, perhaps increasing the correlation across methods.

This difference in the spatial coupling across stimuli underscores the importance of considering analysis and acquisition factors when comparing results obtained from different types of measurements. For instance, while the current spatial coupling estimate between ECoG and fMRI measurements provides an indication as to the volume over which ECoG electrodes are sensitive to neural activity in our particular setup, it is likely that the spatial specificity of the ECoG category-selective responses would be higher if we used denser grids of smaller electrodes (Freeman, Rogers, Holmes, & Silbergeld, 2000). Future comparisons of fMRI and ECoG data may examine to what extent these different methodological factors affect the spatial correspondence across measurements. These include: (1) acquisition parameters, such as voxel size, fMRI dropout zone, electrode size, electrode impedance, as well as inter-electrode distance. (2) The point spread function of fMRI and ECoG responses, respectively. The former is estimated to be 2–3 mm (Issa, et al., 2013; Shmuel, Yacoub, Chaimow, Logothetis, & Ugurbil, 2007) and the latter 1.25mm (Freeman, et al., 2000). (3) Errors in the reconstruction of the electrode location relative to the cortex due to brain shifts caused by the electrode implantation procedure. Even though we used an accurate algorithm that corrects for this potential problem, its spatial precision is bounded in the order of 0.5–2mm (Hermes, et al., 2010).

4.3. The spatial organization of category selectivity VTC matches across HFB ECoG and fMRI

By analyzing single participant ECoG data and visualizing responses relative to the cortical folding in individual brains, our data show that the spatial organization of category selectivity measured with ECoG HFB responses is similar to that measured with fMRI in the same participants. With both methods we find pronounced face-selectivity on the MFS, LFG and OTS, and selectivity for inanimate stimuli, and in particular houses, in the MFG, CoS and PHG. In two participants we also found HFB limb-selective responses around the OTS, which were near fMRI selective activations for limbs (fMRI data not shown). In addition, we find a large-scale animacy distinction along a lateral-medial VTC axis, where preferential responses to animate categories occur in lateral VTC and preferential responses to inanimate categories are observed in medial VTC. The location of category-selective ECoG responses reported here are consistent with prior findings from fMRI studies (Connolly, et al., 2012; Epstein & Kanwisher, 1998; Kanwisher, et al., 1997; Konkle & Caramazza, 2013; Martin, et al., 1996; Nasr, et al., 2011; Peelen & Downing, 2005; Schwarzlose, et al., 2005; Weiner, et al., 2014; Weiner & Grill-Spector, 2010, 2013) and with the reported location of electrodes responding preferentially to specific categories in previous intracranial EEG studies (Allison, et al., 1999);(McCarthy, Puce, Belger, & Allison, 1999; Miller, et al., 2009; Murphey, et al., 2009; Privman, et al., 2007; Puce, et al., 1997) (Liu, et al., 2009) (Bastin, et al., 2013; Davidesco et al., 2014; Engell & McCarthy, 2011, 2014a; Fisch, et al., 2009; Rosburg, Ugur, Haueisen, Kreitschmann-Andermahr, & Sauer, 1999; Vidal, et al., 2010). Our data extend these prior ECoG studies that examined localized responses over a few electrodes or pooled data across participants, by providing a direct and comprehensive mapping of the spatial organization of category representations across the VTC in individual participants. We would like to underscore that understanding the relationship between functional responses and the cortical folding in individual brains has important clinical implications because it may allow predicting function from anatomy alone (Weiner, et al., 2014; Witthoft, et al., 2014).

Our finding of a tight spatial relationship between patterns of face- and house-selective responses across ECoG and fMRI are also in line with prior human case studies reporting ECoG-fMRI correspondence in VTC at a local scale of a single to a few electrodes (Jonas, et al., 2014; Mundel, et al., 2003; Murphey, et al., 2009; Parvizi, et al., 2012; Puce, et al., 1997). However, while these prior studies provided evidence for a co-localization of selectivity for faces across ECoG and fMRI, they are limited in their ability to establish a direct spatial correspondence between BOLD and neurophysiological signals. Here, we reduced the spatial uncertainty by measuring ECoG and fMRI from multiple electrodes spread over a large surface of the cortex in multiple individuals’ hemispheres. This allowed us to show that spatial patterns of graded selectivity measured with fMRI tightly correspond with the underlying patterns manifested in neurophysiology.

Finally, although we used different tasks in ECoG and fMRI (detecting a change of color at fixation or a 1-back task, respectively), it is interesting that we find clear correspondence in the topology of category-selectivity across the VTC with fMRI and ECoG measurements. This consistency is in line with fMRI results showing that the task does not change the overall pattern of distributed responses across VTC to object categories or the location of category-selective regions in VTC (Bugatus, Weiner, & Grill-Spector, 2015; Davidenko, Remus, & Grill-Spector, 2012; Weiner & Grill-Spector, 2010), even though the task modulates the magnitude of responses (Harel, Kravitz, & Baker, 2014; Wojciulik, Kanwisher, & Driver, 1998).

4.4. Spatial topologies of fMRI responses across the VTC are associated with feed-forward processing

Likewise, we found that the representational structure of distributed HFB ECoG responses matches the representational structure measured with fMRI in the same participants. By using a data driven approach without pregrouping stimuli into categories, we find that distributed HFB responses to images of a category are more similar to each other than to images of other categories. This categorical structure is observed even for stimuli that do not generate strong selectivity in specific electrodes. Further, our data show that distributed HFB responses across the VTC contain both a distinction between face, house, car and limb categories, as well as a broader distinction between animate and inanimate categories. By examining the representational structure in each participant over time, we are able to show that the category information in VTC measured in the HFB range arises from reproducible and distinctive spatial topologies across categories, and that these topologies are stable over time from 100ms after stimulus onset till stimulus offset. This insight was not possible to obtain from prior ECoG studies (Liu, et al., 2009) because they measured distributed responses from electrodes that were pooled across participants without taking into account the spatial location of specific electrodes. Nevertheless, the timing we report here is compatible with prior studies in humans (Liu, et al., 2009) and macaques (Bell et al., 2011; Hung, Kreiman, Poggio, & DiCarlo, 2005; Kiani, Esteky, & Tanaka, 2005; Tsao, et al., 2006). Moreover, our data show that the representational similarity measured with HFB starting at 100ms, not only matches fMRI representations within the same participants, but also matches the category structure reported in prior fMRI studies (Kriegeskorte, et al., 2008) (Weiner & Grill-Spector, 2011; Weiner, et al., 2010).

Notably, both spatial maps of face- and house-selectivity and RSMs measured with fMRI correlated with ECoG HFB measurements from 100ms after stimulus onset, and this correlation persists for the remainder of the stimulus duration. This provides a much needed time stamp for the many fMRI studies of category representations in VTC, suggesting that they emerge within 100ms. Importantly, we find that the onset latency of the significant correlation between ECoG HFB and fMRI responses matches the onset latency of HFB responses in VTC (Fig 8). Because ECoG measures local electrophysiological responses directly from the cortex, this indicates that the earliest stimulus-driven neural responses in VTC contribute to the measured fMRI signals and in turn, suggests that feed-forward neural processing in VTC contribute to fMRI responses.

Conclusions

In sum, this study presents two important advancements. First, we provide a direct mapping of electrophysiological neural response patterns in the HFB range in the VTC in individual participants. Second, by finding a tight correspondence between ECoG HFB responses and fMRI signals in the same participants, we provide a way to bridge findings about VTC function across the two methodologies with millimeter and millisecond resolution.

Supplementary Material

supplement

Supplementary Figure S1. Time course of ECoG response in theta, alpha and beta frequency bands across anatomical partitions of VTC. The figure depicts the average response across all visually responsive electrodes in each of 4 anatomical partitions (same as Figure 1B) for theta (4–8Hz), alpha (8–12Hz) and Beta (12–30Hz) frequency bands. Shaded regions around waveforms represent ± 1 SEM across electrodes, computed on amplitude normalized waveforms in each electrode. Acronyms: inferior temporal sulcus (ITS)/inferior temporal gyrus (ITG), lateral fusiform gyrus (LFG)/occipitotemporal sulcus (OTS), medial fusiform gyrus (MFG), collateral sulcus(Cos)/parahippocampal gyrus (PHG)/lingual gyrus (LG).

Supplementary Figure S2. Absence of representational structure from early visual cortex responses. (A) Representational similarity matrix (RSM) showing the similarity of distributed responses between each pair of images presented to the participant, based on ECoG responses measured from electrodes located on early visual cortex (EVC) in two participants, as illustrated in (B). Distributed responses were calculated for each participant across these EVC electrodes (4 electrodes for S1; 9 electrodes for S2) using HFB-responses averaged over a time window of 100–350ms after stimulus onset. Responses were averaged across 2–5 presentations of an image (excluding the first presentation). The top triangle shows the similarity of the distributed responses to each pair of images each participant saw. The bottom triangle shows the average responses across all pairs of images of a category, excluding the diagonal. RSMs show an absence of category structure in the distributed pattern of response (compare to S1 and S2 in Fig. 4). This indicates that representational category structure in VTC electrodes is not inherited from early visual areas that process lower-level visual properties of the images. (B) Location of EVC electrodes in both participants. Electrodes were located in the postero-medial half portion of the lingual gyrus (pLG), around the calcarine sulcus (CS) or on the cuneus (Cun).

Supplementary Figure S3. Correlations between fMRI and ECoG pattern of face-selectivity as a function of frequency band, time, and spatial pooling. (A) Time-course of the correlation between spatial pattern of face-selectivity measured with ECoG and fMRI using 3 different amounts of spatial pooling of fMRI voxels around each electrode. Extent of spatial pooling reflects the contribution of each voxel to the BOLD face-selectivity d′ measured for each electrode location. The contribution of each voxel was weighted by its distance to the electrode location using a 3D Gaussian for which we varied its standard deviation. The time courses of ECoG-fMRI correlation are showed for each amount of spatial pooling (1.5, 3.5, 14mm; see (B), middle plot). Each line reflects the ECoG-fMRI correlation across time for one of 4 specific frequency band (see legend in top right) averaged across 6 participants. Error bars indicate across-participants SEM. Shaded vertical bars in the central panel indicate time-windows used in (B). (B) ECoG-fMRI correlation as a function of the parametric manipulation of the spatial extent of pooling of fMRI signals around each electrode for 4 frequency bands and 3 time-windows. Shaded vertical bars in central panel indicate spatial pooling used in the top panels. Error bars indicate across-participants SEM. The magnitude of the correlation was highest for a spatial pooling of ~2.5 – 5mm, peaking at 3.5 – 4 mm for all frequency bands and decreasing sharply for smaller or larger volumes of spatial pooling of BOLD signal. Correlations were not significant when including signals from fMRI voxels that are further than 6mm from the electrodes. The 0.4 – 0.8s time-window (central panel) displays significant positive correlation in the HFB and significant negative correlation in the alpha band (ps<.005). However, in this time-window correlations in lower frequency bands (theta and alpha) were less robust than in the HFB. Examination of the ECoG signal (mean % change from baseline) and noise (inter-trial variability) in the 400–800ms time interval reveals that the lower magnitude of the correlation in the alpha and theta frequencies is related to a lowered correspondence between the fMRI-ECoG selectivity profile due to reduced signal amplitude in these frequencies relative to higher frequencies.

Supplementary Figure S4. Correlations between fMRI and ECoG pattern of house-selectivity as a function of frequency band, time and spatial pooling. Conventions are the same as for Supplementary Figure S3. The magnitude of the ECoG-fMRI correlation peaked when the spatial extend of pooling fMRI voxels was about 4 mm around ECoG electrodes. In contrast to ECoG-fMRI correlations of face-selectivity, the magnitude of the correlation did not vary as much with varying spatial pooling.

Highlights.

• We measured ECoG and fMRI responses to visual stimuli in VTC of the same subjects

• Topology and distributed pattern of category-selectivity match across ECoG and fMRI

• fMRI correlation with ECoG HFB signal is the strongest and most sustained

• fMRI correlation with ECoG low frequency signals is temporal transient

• ECoG-fMRI correlation emerges at ~100ms, simultaneous with neural response in VTC