The dynamics of cortical interactions in visual recognition of object category: living versus nonliving

Abstract

Noninvasive brain imaging studies have shown that higher visual processing of objects occurs in neural populations that are separable along broad semantic categories, particularly living versus nonliving objects. However, because of their limited temporal resolution, these studies have not been able to determine whether broad semantic categories are also reflected in the dynamics of neural interactions within cortical networks. We investigated the time course of neural propagation among cortical areas activated during object naming in 12 patients implanted with subdural electrode grids prior to epilepsy surgery, with a special focus on the visual recognition phase of the task. Analysis of event-related causality revealed significantly stronger neural propagation among sites within ventral temporal lobe (VTL) at early latencies, around 250 ms, for living objects compared to nonliving objects. Differences in other features, including familiarity, visual complexity, and age of acquisition, did not significantly change the patterns of neural propagation. Our findings suggest that the visual processing of living objects relies on stronger causal interactions among sites within VTL, perhaps reflecting greater integration of visual feature processing. In turn, this may help explain the fragility of naming living objects in neurological diseases affecting VTL.

Introduction

Although visual cortex spans much of occipital and posterior temporal lobes in humans and its circuitry in nonhuman primates has been mapped in extraordinary detail, little is known about how neural activity is propagated across this vast functional-anatomic domain during common tasks, such as recognizing and naming objects seen in our environment. We do know that visual processing of objects can occur in neural populations that are separable for object animacy (living vs. nonliving objects) (Connolly et al. 2012), real-world size (Konkle and Oliva 2012), retinotopic eccentricity (Levy et al. 2001), and other factors (Grill-Spector and Weiner 2014). Indeed, functional magnetic resonance imaging (fMRI) and lesion studies have demonstrated category-specific responses in the posterior ventral temporal lobe (VTL), including a medial-to-lateral bias for nonliving versus living objects within ventral temporal cortex (Martin et al. 1996; Chao et al. 1999; Humphreys and Riddoch 2003; Mahon et al. 2009; Wierenga et al. 2009; Chouinard and Goodale 2010; Zannino et al. 2010; Grill-Spector and Weiner 2014). However, patients suffering from neurological diseases, including viral encephalitis and dementia, that affect the temporal lobes more broadly, without regard to category-specific areas, often manifest more impaired naming of living than nonliving objects (Capitani et al. 2003; Humphreys and Riddoch 2003; Kivisaari et al. 2012; Ritchie et al. 2021). An attractive hypothesis to explain this discrepancy is that there is something special about the network dynamics of processing living objects that render this category susceptible to widespread injury or degeneration. For example, stronger propagation of neural activity within and across visual areas may be needed for processing living objects than nonliving objects. Testing this hypothesis, however, requires measurements of task-related changes in neural propagation at spatial and temporal resolutions higher than those afforded by magnetoencephalography and fMRI, respectively.

To compare the fine temporal dynamics of changes in both neural activation and neural propagation during category-specific visual object naming, we recorded electrocorticographic (ECoG) signals from the subdural electrode grids implanted in 12 patients for epilepsy surgery. We measured the task-related neural activation of stimulus-locked high-gamma (HG; 80–150 Hz), which has been shown to closely track changes in the neuronal population firing rates (Ray et al. 2008; Manning et al. 2009; Crone et al. 2011; Buzsaki and Wang 2012). Having identified sites with task-related neural activation, we applied event-related causality (ERC) analysis (Granger 1969; Korzeniewska et al. 2008), a multichannel extension of the Granger causality concept (Granger 1969), to study the dynamics of neural propagation at HG frequencies (Korzeniewska et al. 2011; Flinker et al. 2015; Nishida et al. 2017; Korzeniewska et al. 2020) and to detect differences of the dynamics of HG propagation in brain networks for category-specific object recognition. Moreover, we tested whether object features other than object category (familiar vs. unfamiliar, simple vs. complex forms, and difference of age of acquisition (AoA)—referred and controlled in previous papers; Humphreys et al. 1989; Wierenga et al. 2009; Chouinard and Goodale 2010; Shimotake et al. 2015; Chen et al. 2016) affect visual recognition and influence the pattern of neural propagation.

Although prior studies have used ERC to study neural propagation during speech production tasks, including visual object naming (Korzeniewska et al. 2011; Flinker et al. 2015; Korzeniewska et al. 2020; Korzeniewska et al. 2022), this is the first study to investigate the effect of visual object category and other object characteristics (familiarity, visual complexity, and AoA) on neural propagation in visual association cortices and other cortical regions activated during visual object recognition at the scale of milliseconds.

Materials and methods

Patients

Fourteen patients who underwent invasive presurgical evaluation with ECoG for intractable focal epilepsy since April 2010 through July 2019 in Kyoto University Hospital were recruited for this study. Twelve of these patients (age: 21–61, male/female = 6/6) were eligible for this study according to the below-mentioned criteria (Table 1). Recordings from 7 participants included here (Pt #1–7) were used previously for representation similarity analyses (Chen et al. 2016). The protocol conformed to the Declaration of Helsinki and was approved by the Ethics Committee of the Institute (IRB #C533). Patients provided written informed consent and were recruited to the study. The placement of intracranial electrodes was decided solely according to clinical needs. The recording electrodes consisted of arrays of macroelectrodes (2.3 mm in diameter, 1 cm in interelectrode distance, AD-TECH, WI; although 8 × 2 grids with 0.5 cm in interelectrode distance were included in the parietal lobe in Pt. #10, further analysis did not show any significant activation). The methods for identification of the implanted electrodes and coregistration to the Montreal Neurological Institute (MNI) standard space are described elsewhere (Matsumoto et al. 2004; Usami et al. 2015). The locations of the electrodes were confirmed by a magnetic resonance imaging image after implantation. Then, they were linearly coregistered into a preimplantation image and next nonlinearly into MNI standard space (ICBM-152) with 2-mm resolution using the program of fsl_anat (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/fsl_anat). Anatomical labels of the electrodes in this study were confirmed by both a preimplantation image and MNI standard space in reference to the Harvard-Oxford cortical structural atlas (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Atlases).

Table 1.

Patient profile and analyzed trials/electrodes.

	Age, gender, handedness	Epilepsy classification	Etiology, pathology	Antiseizure medication	Performance (%)	The percentage of analyzed trials (%)	# Of recording electrodes	# Of electrodes used in ERC analysis of the living versus nonliving category/recording electrodes in each region
								LO	VTL	LT	Frontal (F)	Parietal (P)
1	29, M, L	L-TLE	HS, FCD 3Aa	CLB, PHT, ZNS	90.5	17.5	102	7/12	2/6	0/18	0/18	1/22
2	34, M, R	R-TLE + PLE	Traumatic injury, HS	CBZ, LEV, TPM, ZNS	62.5	37.8	81	6/19	0/0	0/8	2/11	0/28
3	38, F, R	L-TLE	FCD 2A	CBZ, PHT, TPM, LEV	96.8	82	85	0/0	4/9	3/14	1/22	1/15
4	41, F, R	L-TLE	FCD 1	CBZ, TPM, PHT	91.3	31.3	94	0/2	7/11	0/14	0/26	0/18
5	52, M, R	L-TLE	Arteriovenous malformation/gliosis/inflammatory infiltration	CBZ, LTG	42.3	49.3	104	3/7	6/17	1/22	0/16	0/15
6	27, F, R	R-TLE	FCD 1	CLB, LEV, fPHT	93.8	24.5	76	3/4	5/15	0/18	0/9	0/6
7	61, M, R	L-PLE	Oligoastrocytoma WHO grade II	CBZ, TPM	51.5	61.8	94	0/4	3/4	0/14	0/27	0/23
8	28, F, R	L-TLE	Nonneoplastic brain tissue	CBZ, TPM, LEV	50.5	46	98	2/9	5/14	0/13	1/27	2/20
9	21, M, R	L-TLE	FCD 3A	LEV, CLB	93.5	41.5	81	1/1	4/8	0/13	2/25	0/4
10	34, M, R	L-TLE	FCD 3A	CBZ, LTG	81.0	47.5	102	3/7	2/6	0/18	4/27	0/18
11	49, F, R	L-TLE	HS + unknown etiology	PHT, PB, LEV	80.0	23.3	119	2/10	2/7	3/12	1/38	1/34
12	22, F, R	L-PLE + TLE	FCD 1C	VPA, TPM, PER, LCM	64.5	34.5	106	6/17	0/3	0/17	1/18	5/32

# Of recording electrodes represents all electrodes without systemic artifacts, including the electrodes in the anterior temporal lobe (ATL), which were excluded for ERC analysis. Abbreviations: TLE, temporal lobe epilepsy; PLE, parietal lobe epilepsy; HS, hippocampal sclerosis; FCD, focal cortical dysplasia; CLB, clobazam; PHT, phenytoin; ZNS, zonisamide; LEV, levetiracetam; TPM, topiramate; LTG, lamotrigine; fPHT, fosphenytoin; PB, phenobarbital; LCM, lacosamide.

^aFCD classification is according to Blumcke’s criteria published in Epilepsia, 2011.

Task and recording procedure

Sitting comfortably on the bed, patients named pictured items aloud in Japanese (without articles or a carrier phrase) as quickly and accurately as possible. Note that the Japanese language does not use articles as a default when referring to a general name. Hundred pictures appeared consecutively every 5 s on a computer monitor (52.5 cm wide × 29.5 cm height; Fig. 1) located 1.0 m in front of the patient. The visual angle of the stimulus items was 7.5°–10.6°. Stimuli consisted of black line drawings (50 living [animals, including insects, e.g. rabbit and butterfly] and 50 nonliving objects [e.g. airplane and guitar]) on a white background as in previous studies (Morrison et al. 1997; please see a full list of items in Supplementary Table S1). Living and nonliving objects were also characterized by other variables derived from the previous norming studies: AoA, visual complexity, familiarity, and word frequency, which were not significantly different across categories according to independent 2-tailed t-test analyses (Chen et al. 2016). The 100 stimuli were pseudorandomly ordered and presented once per object in 1 session. In total, 4 sessions were performed within approximately an hour. We confirmed that the patient watched the monitor through the recording and that no seizures occurred during the recordings by carefully monitoring the video of patient behavior and ECoG signals. Verbal responses were recorded with a microphone placed in front of patient. ECoG signals and time stamps of visual stimuli were digitally recorded and stored in a hard drive in the recording system (EEG 1000, Nihon Kohden, Tokyo, Japan). ECoG signals were referenced to a scalp electrode placed on the skin over the mastoid process contralateral to the side of electrode implantation and were sampled at 1,000 Hz with a band-pass filter of 0.08–300 Hz in Pt. #1–9 and at 2,000 Hz with a band-pass filter of 0.08–600 Hz in Pt. #10–12.

Fig. 1.

ECoG signals data during visual naming task. A) Patients undergoing subdural grid implantation for invasive presurgical evaluation were instructed to name out loud objects presented on the computer screen every 5,000 ms (e.g. living object such a rabbit or butterfly, or nonliving such an airplane or guitar). B) After removing epochs that included artifacts or interictal epileptic discharges within −2,000 to +1,000 ms at visual stimulus, RTs for living (black) and nonliving (gray) objects were computed. On each box, the central mark indicates the median and the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. The whiskers extend to the most extreme data points not considered outliers (within 1.5 times the height of the central box), and the outliers are plotted individually using the unfilled circle. RTs converted to z-scores for living and nonliving stimuli across the patients were compared, showing statistical significance. ^*P < 0.05.

Data analysis

Analytic tools and scripts

Spectral and ERC analyses were performed using a custom-made analysis pipeline (Fortran 77, C++, and R) developed at Johns Hopkins University (Franaszczuk and Jouny 2004; Korzeniewska et al. 2008). Statistical analyses were performed using in-house scripts in Matlab software (version 2019b, MathWorks, Natick, MA).

Selection of channels that show significant neural activities

First, electrodes with systemic artifacts, which cause no effective recoding from disconnection or overlapped multiple grids, were excluded from further analysis. Then, a common average reference was applied to the remaining electrodes to remove common activity, to remove spatial biases due to varying distances from the reference electrode, and to emphasize local activity when analyzing activation and propagation (Korzeniewska et al. 2008). For each epoch within −2,000 to 1,000 ms relative to stimulus onset in each trial, certified neurophysiologists carefully checked the ECoG signals and excluding, from subsequent analyses, the trials that involved interictal epileptic discharges or artifacts. Electrooculograms were not recorded in this study. Because small artifacts from continuous eye movements can sometimes appear as changes in the high-frequency activity in temporal lobe, especially at the anterior temporal pole (Kovach et al. 2011), raw signals in all recording sites in the anterior temporal lobe (ATL) were carefully inspected, and trials contaminated with these artifacts were also removed. Two out of 14 patients were excluded in this process because they showed eye movement artifacts in the ATL throughout their recordings such that we could not preserve enough trials for further analysis.

To detect the cortical activation elicited by visual stimuli, we focused on the HG frequency range (HG: 80–150 Hz; Fig. 2). First, the data epochs (−2,000 ms to ~1,000 ms, with 0 at visual stimuli) were separated into living and nonliving trials and were analyzed using short-time Fourier transform with 50-ms Hann window shifting/sliding by 10 ms. Second, for each of these shifting windows, the spectral power in HG frequency bands (80, 100, 120, and 140 Hz) was calculated and averaged over trials. Third, the z-score of power changes in each poststimulus window was calculated with respect to the mean power over all prestimulus baseline windows. Fourth, a cluster of consecutive windows where the change in HG power exceeded z-score = 3 (z = 3), lasting at least 50 ms between 0 and 500 ms (to focus on visual processing, which occurs within 500 ms, and to avoid artifacts generated by vocalization), were found. Then, the z-scores in the clusters were summed separately for living and nonliving objects in each recording site (in each electrode, please see Supplementary Fig. S1). The largest sum (for living or nonliving objects) was considered as the value that reflected the strength of neural activity evoked by visual stimuli in each recording site (electrode) in this study. This approach ensured the choice of electrodes responding to 1 or the other category, i.e. living or nonliving objects. Next, these values were ordered and the top 10% of all analyzed recording sites in each patient were used for subsequent causality analysis to ensure sufficient number of data points. Selection of recording sites to those which were activated by the task was required by ERC analysis, which uses multivariate autoregressive (MVAR) modeling (please see Event-related causality subsection in Materials and methods). If an electrode was located at a site in the ATL, which is anterior to the plane perpendicular to the long axis of the temporal lobe passing through the fusiform gyrus (FG) at y = −20, z = −30 and superior temporal gyrus at y = 0, z = −5 in MNI space defined by Rice et al. (2015), the electrode was excluded from further analysis because we wanted to avoid spurious causality caused by eye movement artifacts as much as possible. The anatomical regions of interest were defined independently of the results of HG mapping. The focus of this study was the VTL and lateral occipital (LO) lobe. No electrodes in the medial occipital lobe were present in all 12 patients. The VTL (posterior to the plane defined by the border of ATL in the paper by Rice et al. 2015 and anterior to the posterior transverse collateral sulcus) was first extracted based on previous studies emphasizing its importance for visual processing (Zannino et al. 2010; Grill-Spector and Weiner 2014). Then, the LO lobe (posterior to the temporo-occipital sulcus) was demarcated. The lateral temporal (LT) lobe was defined as the area posterior to ATL and lateral to VTL in the temporal lobe.

Fig. 2.

Neural activation during visual category recognition. A) Electrodes selected for visual category recognition (used for the analysis of ERC). At the left, the filled yellow circles or pie charts (for LO in a red circle and VTL in an orange circle) denote electrodes that ranked in the top 10% of elicited HG activities (80–150 Hz) in each patient. All electrodes, from 12 patients, are depicted on the MNI standard brain. Electrodes from the right hemisphere were flipped into the left one. Posteriorly, the electrodes in LO and VTL are clustered compared to the yellow electrodes that are sparsely located. The electrodes in bad condition (black; baseline fluctuation, or severe artifacts that suggest wire breaking) and all the other recorded electrodes (blue square) were also shown. The dotted white lines indicate the border of the ATL defined by Rice et al. (2015). The electrodes anterior to the border were excluded for ERC analysis in advance, although only 5 electrodes were involved across the patients. At the right, an example of spectral representation of responses elicited by living (top) or nonliving objects (middle) (†) in VTL. z-scores of HG (white rectangle) in reference to prestimulus baseline (−2,000 to 0 ms) were further shrunk/collapsed to the line plot (bottom). z-clusters of HG, the area under the waveform satisfying z > 3 within 0–500 ms were used for HG indices as above. The relative magnitude of z-clusters of HG at each electrode (living [magenta] vs. nonliving [green]) are shown using pie chart. Note that HG-wise, no clear demarcation of activities was found between the subregions in VTL. B) The relative magnitude and the time course of mean HG in the posterior cortex (LO and VTL) during the visual recognition of living (magenta) and nonliving objects (green). Each colored line shows the mean of HG (LO—top, VTL—bottom) and their shaded areas show 1.96 × standard error of the mean at each time point. *: P < 0.05 after correction for multiple comparisons by Holm’s method.

Reaction time (RT) was defined as the onset of spoken words/responses, including the tip of the tongue, whether the answer was correct or not. We focused on neural activities and propagation elicited by visual stimuli and not those elicited by semantic processing or vocalization in this study, although correct answer rates were calculated. These RT data were deemed to be solely useful to differentiate visual-induced network dynamics from the subsequent network dynamics. RT in each patient was also converted to z-score to enable the global analysis across patients.

Statistical comparisons between living and nonliving stimuli were performed by a 2-tailed paired t-test for HG activities (as we paired the mean of HG activation evoked by living objects with the mean of HG activation evoked by nonliving objects for each recording site) or unpaired t-test for RT (living and nonliving trials were unpaired), and their effect sizes (r) were also calculated.

Event-related causality

Details of ERC analysis and its applications to visual recognition of objects may be found in previous papers (Korzeniewska et al. 2008, 2011, 2020, 2022). In short, this method is a multichannel extension of Granger causality concept, which states that an observed time series x(t) causes another series y(t) if knowledge of x(t)’s past significantly improves the prediction of y(t). A set of time series, in this study set of ECoG signals unveiling task-related increase in spectral power, is modeled with MVAR to obtain ERC values, which reflect the directions and statistically significant changes in the intensity of neural activity propagation between recording sites, as a function of frequency (80–150 Hz in this study), relative to the baseline activity (−2,000 to 0 ms). MVAR model assumes that the value of the x at a time t depends on its p previous values and the random component e. The MVAR process for vector signal x consisting of multiple ECoG signals can be expressed as

In the frequency domain, the MVAR model is given by

Here, H is a transfer function matrix, f is frequency, and Δt is the sampling interval.

The intensity and spectral content of the causal influence of signal l on signal k (l → k) is estimated by

where is an element of the transfer function matrix (Kaminski and Blinowska 1991; Korzeniewska et al. 2003, 2022), a measure of the “directed” relationship between signal l and signal k, computed in a short time-window starting at time t, while is an element of the partial coherence matrix, a measure of the “direct” relationship between them, in the same short time-window. The combination of the 2 measures results in a measure of direct causal interactions between signals in a multichannel recording. Therefore, shows whether the k-signal component at a given frequency f is shifted in time with respect to an l-signal component of the same frequency and whether the shifted components are coherent and are not explained by components of other signals; takes values from 0 to 1. Zero indicates a lack of direct causal relationships. The non-0 values of are interpreted as a flow of activity from recording site l to recording site k (l → k). To investigate the interactions that occur on brief time scales, such as during the picture naming task, an algorithm using multiple realizations of MVAR processes (or multiple repetitions of a task—n_t) was applied (Ding et al. 2000). To ensure a sufficient number of data samples, this can be estimated by the inequality:

where K denotes the number of signals, p—MVAR model order, and N_s—the length of the time-window (number of samples per 1 repetition epoch). Please note, that to investigate brief time-scale dynamics, a short window N_s is used, which requires a large number of task repetitions n_t, which may be impossible to obtain in a clinical setting. To compensate for an insufficient number of repetitions, a smaller number of signals can be used. However, it is crucial to assure inclusion of most of the recording sites activated during the task.

The dynamics of causal influence can be acquired by sliding the short time-window over the time course t of the investigated process. For each set of trials of 1 category, aligned to stimulus onset, the MVAR model order p was determined using the Akaike Information Criterion (Akaike 1974). As the orders were the same or similar in individual patients (e.g. 5 in 1 patient and 8–9 in another patient), the same model order was used for each ERC analysis in each patient to allow comparisons between categories. MVAR coefficients were computed for a 50-ms shifting window and overlapped by 10 ms over the 2,500 ms trial (including baseline of −2,000 to 0 ms and response period of 0 –500 ms).

Table 2 provides the numbers of directed connections between ROIs for all patients. Please note, that there are 2 directed connections between each pair of recording sites k and l (between 2 electrodes): from k to l (k → l), and from l and k (l → k). For each of these directed connections, the ERC value was calculated in each 50-ms window.

Table 2.

The number of directed connections within channels.

Directed connections	LO	VTL
LO	124	71
VTL	71	148

Statistical group analysis

ERC values were normalized in each patient to maximum =1 to allow for group-level statistics. Normalized ERC values reaching significance (P < 0.05, Bonferroni-corrected) for propagation within and between anatomical regions (frontal lobe, parietal lobe, LT lobe, LO lobe, and VTL) were pooled across patients, separately for living and nonliving trial categories.

Our null hypothesis was that there was no difference in the HG propagation patterns between the living and nonliving categories. A nonparametric permutation test (Maris and Oostenveld 2007) was used for the comparison of ERC values between categories in each time-window separately. It is worth noting that the ERC values were already tested in each patient and represent statistically significant change in neural propagation as compared to the baseline. To evaluate group-level significance, ERC values between 2 variables (e.g. living and nonliving) were randomly shuffled within each directed connection (e.g. LO—> VTL). Next, a 2-tailed paired t-test between the 2 surrogate variables was applied across time points (0–500 ms) to select epochs where consecutive t values satisfied P < 0.05 for at least 3 consecutive 50-ms windows. Then, we calculated the sum of the t values in each cluster and made cluster-level statistics. This procedure was repeated for 10,000 times and the null distribution of the maximum cluster-level statistics was compared with the original cluster-level statistics in each propagation, respectively. A P-value threshold was defined at <0.05. How to quantify the effect sizes in cluster-based permutation results is discussed elsewhere (Meyer et al. 2021), although it remains undefined. We calculated the Cohen’s d at each time point involved within the cluster that proved significant.

To evaluate the group-level significance of HG activation, pooled spectral powers of HG during living or nonliving stimuli were averaged within each region and were compared by 2-tailed paired t-test. A P-value threshold was defined at <0.05. We corrected for multiple comparisons by Holm’s method.

Analysis for the other picture properties

Each picture stimulus was characterized not only by living/nonliving category but also by familiarity, visual complexity, and AoA. The median of category values of each stimulus was calculated. Then, the trials were regrouped into higher and lower halves to analyze HG activation and ERC analysis as well as their group-level significance for the top and bottom halves of trials for each of these stimulus properties.

Results

Behavioral data

Twelve Japanese patients performed a picture-naming task (Fig. 1A) during chronic ECoG recordings (~2 weeks) for the presurgical evaluation of focal epilepsy (Table 1). Subjects were asked to overtly name objects serially presented on a computer monitor in their native Japanese. The objects belonged to 1 of 2 categories, living (animals, including insects, e.g. rabbit and butterfly) or nonliving (e.g. airplane and guitar) and varied within each category as to their familiarity, visual complexity, and AoA. Trials without artifacts and without epileptic discharges were analyzed (17.5–82% of all the trials in each patient; Table 1). The numbers of remaining trials in the living/nonliving categories were not significantly different across patients (P = 0.13, paired t-test). The subjects’ mean verbal response latencies were significantly faster for living objects, although the effect size was small (t(1,541) = 2.365, P = 0.02, effect size r = 0.06, unpaired t-test; living 1,700 ± 982 vs. nonliving 1,728 ± 952 ms; Fig. 1B).

HG activity and its propagation in the posterior cortex

We analyzed the propagation of HG activity among the top 10% of recording sites that showed significant task-related HG activation within 500 ms when the living or nonliving trials were analyzed independently as per our criteria (Fig. 2A. See Materials and methods for details). We tested whether the numbers of the selected electrodes from living or nonliving category were different in LO and VTL in each patient, respectively. However, no significance was found (P = 0.30 in LO; P = 0.14 in VTL, paired t-test). Signals from 103 of 1,142 recording sites in all patients were analyzed. These included 33 recording sites clustered in the LO region (posterior to the temporo-occipital sulcus) and 40 clustered in the VTL (posterior to the plane defined by the border of ATL in the paper by Rice et al. 2015; please see Supplementary Table S2 for the MNI coordinates of each electrode in each patient). VTL was first extracted based on previous studies implying its importance for visual processing. Then, LO was demarcated. Five electrodes in ATL were excluded because HG activities were potentially contaminated with eye movement artifacts (Kovach et al. 2011; Muthukumaraswamy 2013). HG activation within LO was significantly greater for living objects than for nonliving objects across all patients between ~50 and ~250 ms (Fig. 2B). This predominance of HG activity in LO for living objects was in line with previous invasive studies (Kojima et al. 2013; Kadipasaoglu et al. 2016), supporting the validity of our methodology for electrode selection. By contrast, we found no significant difference between categories for HG activation within VTL.

We next used the ERC method to investigate the propagation of HG activity within and between LO and VTL, or patterns of HG activity propagation, elicited by living and nonliving objects (Table 2; see Fig. 3 for the examples of 2 patients; see Supplementary Fig. S2 of the diagram presentation of Fig. 3). The patterns of HG propagation, revealed by ERC, did not differ significantly between language-dominant and nondominant hemispheres (language laterality was defined by handedness and presurgical electrical stimulation mapping).

Fig. 3.

The examples of ERC during recognition of living (upper rows) and nonliving objects (lower rows) in Pt. 6 and Pt. 8. The propagation patterns were calculated for magenta or green electrodes, respectively. In Pt. 6, The red asterisks are the sites of insertion of depth electrodes, which were excluded from this study. The magnitude of propagation is shown by the color and thickness. Note that relatively earlier propagation within the VTL is observed during recognition of living objects, although the propagations that include LO or parietal areas are observed during the recognition of nonliving objects.

Therefore, results from both sides were combined (Fig. 4) for comparison of ERC results for different object categories. We found significantly greater propagation among sites within VTL for living objects than for nonliving objects at 230–270 ms (P < 0.05, nonparametric permutation test, effect size: d = 0.29, 0.29, 0.27, 0.25, and 0.23 at each 50-ms window within the cluster). Propagation within LO, as well as between LO and VTL, was not significantly different between living and nonliving objects. We did not observe significant differences between living and nonliving objects for the magnitude of propagation among the frontal lobe, parietal lobe, or LT areas.

Fig. 4.

The time course of ERC in the posterior cortex (LO—red circle, VTL—orange circle) during the visual recognition of living versus nonliving objects. *: P < 0.05 by permutation test for cluster-level statistics. Note that HG propagation within VTL is significantly larger around 230–270 ms during the visual recognition of living objects as compared to nonliving, suggesting higher interdependency between nodes in this region. The other conventions are the same as given in Fig. 2.

Each picture named by our subjects varied with respect to object familiarity, visual complexity, and AoA, independent of their living versus nonliving category. To investigate the effect of these characteristics on the propagation of HG activity, we reanalyzed the data after regrouping trials within each patient into the higher and lower halves of the values for each of these categories. As we did for living versus nonliving object category, we selected the top 10% of recording sites revealing HG activation within LO and VTL. For familiarity, it was 32 recording sites within LO and 41 within VTL out of 100. One electrode in LO and 3 in VTL were different from those in living versus nonliving categories. For visual complexity, it was 33 recording sites within LO and 42 within VTL out of 102. Two sites within LO and 3 within VTL were different from those in living versus nonliving categories. For AoA, 32 recording sites within LO and 40 in VTL were selected out of 103. One site within LO and 2 within VTL were different from the electrodes selected in living versus nonliving categories. Objects with lower familiarity induced greater HG activation in LO starting ~100 ms until the end of the analysis period (+500 ms; Supplementary Fig. S3). Moreover, objects with higher visual complexity induced slightly greater HG activation in VTL ~300 ms after the stimulus (Supplementary Fig. S4). However, we observed no significant differences in propagation patterns among LO and VTL (therefore, data not shown) for both properties. The distance between 95% confidence intervals may suggest a tendency of greater propagation from LO to VTL for lower familiarity over higher familiarity around 300 ms (Supplementary Fig. S5). Finally, AoA did not affect the HG and propagation patterns significantly.

Differences of HG activation within subregions of VTL for living versus nonliving objects

We also analyzed HG activation within subregions of VTL, i.e. 10 recording sites within the inferior temporal gyrus (ITG), 29 within the FG, and 1 site within parahippocampal gyrus (PHG). ITG in VTL showed larger HG activation for living objects (t(9) = 2.44, P = 0.037, r = 0.77), while FG and PHG did not show differences (t(29) = −0.697, P = 0.491, r = 0.13). The predominance of ITG activation for living objects was in line with previous invasive studies (Kojima et al. 2013; Kadipasaoglu et al. 2016).

ERC analysis was also performed among subregions (within and between ITG and FG/PHG; Table 3), but we did not observe differences in patterns of HG propagation.

Table 3.

The number of directed connections within channels in VTL.

Directed connections	ITG	FG + PHG
ITG	8	29
FG + PHG	29	82

Discussion

Neural activation and propagation

Most prior studies of the neural dynamics of visual object perception and the effects of object category on these neural dynamics have focused on the magnitude of task-related increases in neural activation, which were measured either with blood oxygen level–dependent (BOLD) responses in fMRI or with HG responses in intracranial electroencephalography (EEG; electrocorticography, or ECoG) (Martin et al. 1996; Chao et al. 1999; Mahon et al. 2009; Wierenga et al. 2009; Chouinard and Goodale 2010; Zannino et al. 2010; Kojima et al. 2013; Kadipasaoglu et al. 2016). Like BOLD signals measured at each voxel, HG responses estimate changes in overall population firing rates at each recording electrode, but do so with temporal resolutions on the order of tens of milliseconds, and are thus capable of unraveling the spatial–temporal dynamics associated with human behavior. Here, we used advanced ECoG signal analyses of changes in ERC to go beyond mere changes in the strength of activation at individual sites to measure the degree to which neural activity is propagated between these regions. This measurement is important for understanding the neural mechanisms of object perception, as well as object naming, because of the dependence of downstream processing on upstream processing in cortical networks that span large cortical areas and that have both hierarchical and parallel distributed architectural features (Zeki and Shipp 1988; Van Essen et al. 1992). The presence of a hierarchical processing system was demonstrated in a seminal study that showed that grouping processes can operate at a level at which combined form information, not single-feature form information, is represented in visual search (Humphreys et al. 1989). Indeed, the intensity of propagation by which neural activity is transmitted from 1 node of a cortical area to another may reflect the specific way by which the network processes information and thus could uniquely identify network nodes that are important to task performance. Although prior studies have used ERC to study neural propagation during speech production tasks, including visual object naming (Korzeniewska et al. 2011, 2020, 2022; Flinker et al. 2015), this is the first study of the effect of visual object category and other object characteristics (familiarity, visual complexity, and AoA) on neural propagation in visual association cortices and other cortical regions activated during naming. Because intracranial electrodes were implanted in our subjects for solely clinical purposes, no individual patient was implanted with enough electrodes for conclusive hypothesis testing. Nevertheless, changes in the HG activity were sufficiently robust to test the statistical significance of both activation and propagation within each of our subjects, and this allowed us to aggregate results across subjects in order to test our hypotheses.

Network dynamics for living versus nonliving objects

Clinical-pathological correlations in the past have consistently pointed to the temporal lobe as an important network for semantic knowledge, with greater susceptibility to injury or disease for knowledge of living objects (Martin et al. 1996; Chao et al. 1999; Capitani et al. 2003; Humphreys and Riddoch 2003; Mahon et al. 2009; Chouinard and Goodale 2010; Grill-Spector and Weiner 2014). Therefore, we hypothesized that visual object naming would elicit not only the neural activation in the temporal lobe but also the propagation of neural activity among activated sites. Moreover, we predicted that propagation would be stronger for living than for nonliving objects. Although we did observe robust activation, and propagation of activation, in both temporal and occipital (VTL and LO) areas of visual association cortex, we only observed greater aggregate neural activation for living than for nonliving objects in LO and did not observe a difference in VTL (HG z-values shown in Fig. 2B). Interestingly, we observed that living objects did evoke significantly greater propagation of HG activity (80–150 Hz) within the ventral temporo-occipital area (VTL) ~230–270 ms after stimulus onset than nonliving objects did. These findings suggest, at least in this study, that neural propagation was better than neural activation alone at capturing differences in category-specific processing dynamics within VTL. The number of recording sites, and consequently the numbers of analyzed directed connections between them, was greatest in VTL, which may have increased the sensitivity for detecting differences in this area. We focused on the propagation of HG bands in this study. However, it may be of value in the future to analyze ERC in other (e.g. low) frequency bands, which may also be relevant for cortico-cortical communication.

Implications for neurological diseases affecting temporal lobe

Stronger neural propagation within VTL evoked by living stimuli may help explain why patients who suffer from neurological diseases that affect the ventral part of the temporal lobe, posteriorly to ATL, demonstrate impairments particularly for naming living objects (Capitani et al. 2003; Humphreys and Riddoch 2003; Kivisaari et al. 2012). Our findings suggest that propagation within VTL revealed by ERC analysis reflects an intrinsic vulnerability of this cortical region to dysfunction from degeneration and viral infection because it relies on the shared successive flows of information processing, particularly for living objects. In contrast to the situation that occurred in LO, damage to 1 node will involve dysfunction of another node within the area. In the context of the conceptual-structure account of semantic organization in temporal lobes which has been used to explain the differences in naming deficits for living relative to nonliving objects during neural degeneration (Tyler and Moss 2001), higher correlation between features shared among living objects like faces or legs, relative to nonliving objects, may also help explain the differences we observed in ERC flows within VTL. Or, more interactions among cortical sites within VTL may be needed to disambiguate the unique constellation of visual features, which is needed for the identification and naming of living objects. From an evolutionary point of view, it is possible that living objects are more salient for humans than nonliving. We observed faster RTs for living objects, which may be partly due to more processing to reassure shorter processing latencies. As such, greater impairments for recognition (and naming) of living objects may occur when neurological diseases involve VTL. A reduction in the shared dynamics of information flow may help explain the preferential deterioration of detailed semantic knowledge for living objects in dementias affecting ATL (Lambon Ralph et al. 2010; Chen et al. 2016; Ralph et al. 2017), which is considered to be downstream of the temporal lobe.

Other object properties and network dynamics

We did not find evidence that the degree of visual object familiarity, visual complexity, or AoA affects the strength of network propagation in visual association cortices in this study, although the inclusion of more participants could have shown significant differences in propagation. Higher familiarity and earlier AoA imply that subjects have had more experience with these objects visually or via other inputs than for the ones with lower familiarity or later AoA. Greater experience might have been expected to result in more efficient processing, which is reflected here by stronger neural propagation within visual association cortex, particularly in the ventral stream. However, we did not find support for this expectation. Our observation of stronger propagation in VTL for living objects may reflect the specialization for processing living objects, independent of visual object familiarity, visual complexity, or AoA, which may in turn have survival value to humans. Whether this depends on innate genetic expression (Mahon and Caramazza 2011) or specific experience for living objects after birth is unclear and beyond our scope here.

Although we did not observe difference in neural propagation within LO, or between LO and VTL, or between these areas and other brain regions, we observed greater neural activation for living objects (in LO) and for objects with lower familiarity (in LO) and higher visual complexity (in VTL). Greater activation for objects of low familiarity may have reflected more initial visual processing for object recognition. Moreover, we found a tendency of greater ERC flow from LO to VTL for lower object familiarity around 300 ms. This latency is similar to latencies at which we observed greater HG activation for lower object familiarity. This suggests continued visual processing and propagation of information from LO to VTL. Higher visual complexity had a similar effect, but this effect was observed in VTL instead of LO. This may be consistent with VTL’s serving as a set of filters that extract specific characteristics from complicated visual forms (Mani et al. 2008). Our results imply that some properties of visual objects have a great effect on the magnitude of neural activation, but not on neural propagation, during visual object processing. It may be that the magnitude of neural activation reflects local processing that does necessarily rely on propagation across the large-scale cortical networks that are sampled using ECoG. On the other hand, we observed greater neural propagation, but not greater neural activation, in VTL for living objects relative to nonliving objects. This suggests that neural propagation may sometimes offer a better index of functional specialization than neural activation alone, and it opens the door for pursuing a deeper understanding of how the neural dynamics of brain networks support complex human behaviors such as visual object naming.

Limitations

Our intracranial EEG data were necessarily recorded in patients with focal epilepsy. We carefully excluded trials with abnormal activity that could potentially affect cognition (Liu and Parvizi 2019). Although object category may have effects on HG activity in the frontal lobe, we did not observe significant effects on propagation to between frontal and occipital or temporal regions. Likewise, we were unable to discuss propagation regarding ATL, which is reportedly important for semantic processing, due to contamination by EOG. Generally, the number of electrodes implanted in ventral temporal areas is often low due to neurosurgical factors. Therefore, detailed cyto-architectural investigation of flows between the subregions in VTL and LO was impossible compared to fMRI studies with high spatial resolution (Grill-Spector and Weiner 2014). Nevertheless, we were able to adequately sample this and nearby visual association cortices for the purposes of this study. Lastly, we combined trials regardless of correct or incorrect answers because the limiting number of trials would affect the statistical significance of ERC. Thus, we analyzed data within 500 ms to focus on visual perception and its ERC. To compare correct versus incorrect trials, or to study repetition priming effects (Korzeniewska et al. 2020), we would have needed to increase the number of trials in the task, which was not feasible after the fact.

Conclusion

During the early phase of visual processing of objects, propagation of HG activity within the VTL is more intense for living than for nonliving objects. Potential confounds in this comparison, including the degree of visual object familiarity, visual complexity, or AoA, did not significantly affect the patterns of network propagation in visual association cortices. These results suggest that living objects require stronger causal interactions at least in VTL devoted to visual object processing, making them more susceptible to naming deficits when these networks are affected by disease.

Funding

This research was supported by the National Institute of Neurological Disorders and Stroke (R01NS091139 to N.E.C.), Japanese Society for the Promotion of Science KAKENHI (19H03574 to A.S., 19K21210 and 20K16492 to K.U., and 20H05471 and 22H02945 to R.M.).

Conflict of interest statement

None declared.

Data/code availability

The data and code used that supported this study are available via a reasonable request to the corresponding author.