Category-selective phase coding in the superior temporal sulcus

Abstract

Object perception and categorization can occur so rapidly that behavioral responses precede or co-occur with the firing rate changes in the object-selective neocortex. Phase coding could, in principle, support rapid representation of object categories, whereby the first spikes evoked by a stimulus would appear at different phases of an oscillation, depending on the object category. To determine whether object-selective regions of the neo-cortex demonstrate phase coding, we presented images of faces and objects to two monkeys while recording local field potentials (LFP) and single unit activity from object-selective regions in the upper bank superior temporal sulcus. Single units showed preferred phases of firing that depended on stimulus category, emerging with the initiation of spiking responses after stimulus onset. Differences in phase of firing were seen below 20 Hz and in the gamma and high-gamma frequency ranges. For all but the <20-Hz cluster, phase differences remained category-specific even when controlling for stimulus-locked activity, revealing that phase-specific firing is not a simple consequence of category-specific differences in the evoked responses of the LFP. In addition, we tested for firing rate-to-phase conversion. Category-specific differences in firing rates accounted for 30–40% of the explained variance in phase occurring at lower frequencies (<20 Hz) during the initial response, but was limited (<20% of the explained variance) in the 30- to 60-Hz frequency range, suggesting that gamma phase-of-firing effects reflect more than evoked LFP and firing rate responses. The present results are consistent with theoretical models of rapid object processing and extend previous observations of phase coding to include object-selective neocortex.

Artificial systems have yet to replicate the speed and accuracy of our ability to categorize objects. One challenge has been to understand how the visual system accomplishes such a task, when the speed of recognition (<200 ms) is estimated to accommodate approximately one spike per neuron in a feed-forward “sweep” through the visual system (1–4). Prima facie, such a fast reaction time seems incompatible with the typical rate-coding scheme used to describe neural discrimination of object categories. Given these considerations, the fastest and smallest temporal window for decoding in face/object-selective cells in inferior temporal cortex (IT) would be ∼100–150 ms after stimulus onset. In contrast, rate-based decoding is typically based on windows of 50–500 ms (5, 6), positioned 100 ms after stimulus onset (7, 8). Considering that downstream movement planning is still needed, this suggests that some categorization behaviors are occurring during the neural responses thought to underlie them. As a consequence of this temporal bottleneck, several alternative schemes have emerged to accommodate rapid coding in face- and object-selective cells.

One of these alternative schemes is phase coding, or phase-of-firing coding, involving stimulus coding by placing spikes at different phases of local oscillations. Phase codes can be complementary to rate codes (9, 10) and have the theoretical benefits of a greater potential information content, greater speed and efficiency of information transfer (e.g., communication through coherence) (11), and greater potential for plasticity through precise spike timing (11–13). Indeed, one model of object recognition explicitly uses phase-of-firing to accomplish rapid object processing (14). Nonetheless, direct empirical support for such a model is scarce. Phase coding of visual stimuli has been demonstrated at low frequencies in primary visual cortex (V1) of the anesthetized macaque (15). In the awake macaque, phase of firing in V1 at the gamma frequency is related to a cell’s preferred orientation of visual gratings (16). Finally, phase coding of temporal order was detected at a low gamma frequency band in the prefrontal cortex of a macaque performing a working memory task (17). Evidence of phase coding for object categories is lacking, however.

To examine whether phase coding would be observed in object-selective brain regions in the awake macaque, we simultaneously recorded the local field potentials (LFPs) and single unit activity (SUA) from separate electrodes placed in the upper bank superior temporal sulcus (STS) of two monkeys as they passively viewed images of faces and nonface objects.

Results

We recorded 84 units from two macaques, 51 of which met the criteria for inclusion in this analysis (Methods). Consistent with previous reports of selectivity in area TPO (18), approximately one-half of these cells were visually responsive to face stimuli, object stimuli, or both (Fig. 1; n = 26).

Fig. 1.

Neural responses to face and object stimuli. (A) Spike-density functions showing the average firing rate of responses to faces (black) and objects (red), in example single units. Thin lines indicate 95% confidence intervals, and the 100-Hz rate is indicated on the scale bar at the top right of each plot. (Left) Responses that were greater for faces (top two traces), objects (bottom two traces), and both, but at different times after image onset. In the middle trace, the bracket designates the example shown in Fig. 2. (Right) Units that were responsive to face and object stimuli but were not category-selective. (B) Histogram of spike response latencies for faces and objects. (C) Mean normalized LFP response to faces (black) and objects (red) of the STS recording sites used in this study (n = 26). Thin lines indicate ±1 SEM for each response category. The LFP responses were generally conserved across sites; thus, the mean LFP is a good indicator of the evoked response from any given site.

The category preference and duration of responses varied across units, but commonly included short-latency phasic components and time-limited category selectivity. Visual response onset latencies ranged from 42 to 176 ms (mean, 86 ms) for objects and from 47 to 203 ms (mean, 79 ms) for faces, with no statistically significant difference between the two (Fig. 1B; P = 0.354, Wilcoxon rank-sum test). Like the category-selective SUA, the LFP responses also differed by face/object category, but unlike the SUA, the LFPs for each category were generally similar across sites and sessions (Fig. 1C), as described previously (19, 20). Spike synchronization to neural population activity was measured using the spikes of an isolated unit from one STS recording site and the LFP from the simultaneously recorded signal of a nearby STS recording site. The preferred phases of firing of these STS single units are close to or slightly lagging, –π, over a range of frequencies (19). The timing of eye movements was not randomly distributed during image presentation; however, no difference by image category accounted for the observed timing/phase differences in neural responses (Fig. S1).

In this study, we asked whether the phase of firing would differ according to the face/object image being viewed. For each trial, the phase angles were calculated from the cross-spectra of the spike and LFP signal within a 200-ms window every 5 ms near the time of stimulus presentation, with a ±12.5-Hz frequency resolution. The phase angle differences across the approximately 120 trials per category were tested against a null distribution generated by randomly permuting (n = 10,000 repetitions) the category labels for all trials (Methods).

Fig. 2A shows the category-specific differences in phases of firing at a given time point and frequency. Half of the cells showed phase coding of image category in a <20-Hz band, a >50-Hz band, or both, but varying in the exact time-frequency bin. When considering any given point in time and frequency, 20–40% of all cells exhibited different phases of firing in the following clusters: within the first 200 ms of image onset, at frequencies below 20 Hz, and at frequencies between 50–90 and 100–120 Hz (Fig. 2A and Fig. S2).

Fig. 2.

Phase-of-firing differences by stimulus category. (A) Population average of phase differences between categories, across time and frequency. The average F statistic taken from each single unit’s phase difference between face and object stimuli is displayed. Time on the x-axis indicates the center point of the 200-ms sampling window; frequency of oscillation is indicated on the y-axis. Significance was determined through permutation testing (Methods). Time-frequency points that failed to achieve significance at the population level are masked in gray, and the color map is capped at 10 for direct comparison with Fig. 3A. (B) Single spike–LFP example showing a difference in mean phase for faces and objects across time and frequency. Time-frequency points are masked if they fall within the 95th percentile of the category-shuffled distribution (Methods). The rate response of this example is shown in Fig. 1A, Left (middle trace). (C) Phase-dependent spike histograms (Upper), and the difference between the histograms from the face and object conditions (Lower). For each trial category, spikes were binned according to the phase of gamma in which they occurred for a given time window. Here the spike phase is shown for one of the peaks of phase angle difference seen in B: 46 Hz, 55 ms after image onset ± 100 ms). For clarity, the cycle is plotted twice (i.e., from 0 to 4π) and normalized by the grand maximum bin count across the two conditions.

The phase angles obtained from a single unit are shown in Fig. 2B, and the binned distribution of phases for spikes obtained during face versus object presentation for one of the significant windows in the gamma range is shown in Fig. 2C. This cell’s firing rate was statistically different across categories, although only for brief periods, with switching of preferences over time (Fig. 1A, Left, middle trace).

Because both LFP and spiking activity were modulated by the stimulus at short latencies, we calculated the extent to which the phase differences related to the evoked stimulus-locked response. Following previous studies (17, 19, 21) we removed the stimulus-locked (or “evoked”) component of the response. Phase differences across categories were conserved for all but the large <20-Hz early-response cluster (Fig. 3A and Fig. S2). The otherwise-conserved responses suggest that the field potential was not acting merely as a proxy for marking stimulus onset, and that spike timing relative to stimulus onset accounted for the single initial <20-Hz cluster.

Fig. 3.

Factors contributing to category-specific phase differences. (A) Population non–stimulus-locked phase differences. Conventions are as described in Fig. 2A. Phase was calculated from the mean-subtracted LFP signal (Methods). (B) Phase angle variance explained by firing rate differences across category. The color map indicates the explained variance (R²) across the population of cells (n = 26). Axes and masking conventions are as in A. Several clusters of activation from A are outlined in white to show that the peaks in A differ from those in B.

One mechanism for phase coding is a particular instance of firing rate to current conversion (22–24), in which greater excitatory drive evoked by a preferred stimulus leads to both a protracted increase in firing rate and a phase-lead or advance in the phase of firing. Conversely, a weaker or nonpreferred stimulus, characterized by less excitatory drive and lower firing rates, would take longer to reach spiking threshold, resulting in a lagged phase of firing (16) and lower stimulus selectivity for firing at later phases in the oscillatory cycle (25).

To address whether the degree of firing rate differences between face/object categories would predict greater phase-of-firing differences, we measured the variance of mean phase differences for each time frequency bin explained by firing rate differences obtained from the same time windows. For frequencies ≤20 Hz, a strong relationship was seen between a cell’s firing rate and its phase angle differences shortly after stimulus onset. Firing rate differences accounted for ∼40% of the variance in phase angles at low frequencies, suggesting that soon after stimulus onset, the lower-frequency phase at which a spike occurs may function as an instantaneous indicator of the firing rate. The other phase-coding clusters did not exhibit this strong relationship, suggesting that additional factors play a role in determining phase angle for cells in the STS. Thus, gamma and higher frequencies showed rapid, category-selective phase-of-firing differences that could not be accounted for by evoked responses; low frequencies (<20 Hz) were partially dependent on the evoked response and on firing rate differences across categories.

Discussion

Phase coding—stimulus coding by the timing of spikes with respect to the phase of local oscillations—is an alternative, complementary coding strategy to rate coding. Here we tested for phase coding in an object-selective region of the brain as a possible mechanism for rapid object categorization (14). We found rapid response latencies, with the majority of onset latencies <100 ms (median, ∼80 ms) and some as fast as 42 ms. These STS cells potentially could be part of a rapid-processing network for categorization, with latencies on the order of those described in the frontal eye field region (26), with which the STS shares reciprocal monosynaptic connections (27, 28). In addition to firing rate differences, the oscillatory phase of firing differed across stimulus categories, with phase differences emerging within 200 ms of stimulus onset and in frequency bands of ≤20 Hz and at gamma (40–90 Hz) and high-gamma (100–120 Hz) frequency ranges. Phase differences remained even after accounting for stimulus-locked activity, suggesting that the response for some frequencies in some single units was more than a latency code or a match of the evoked responses of LFP and single units.

These results add to the growing body of literature on coding stimulus attributes by assignment to different oscillatory phases (15–17, 29–39) (16). Much of the existing literature focuses on invertebrate and hippocampal circuits, demonstrating proof of coding principle; however, more recently phase coding has been observed in V1 for visual stimuli (15, 16) and in the prefrontal cortex (PFC) for item order during working memory (17). Overarching these various systems and stimulus dimensions is the idea that oscillations provide a temporal reference frame for spiking activity. This may be increasingly important for neural populations that are many synapses removed from external inputs. Intrinsic activity may function by timing with reference to ongoing oscillations, thereby shaping or temporally gating activity evoked from the sensory periphery (12). STS activity may influence target regions through these oscillations; gamma band interactions between these sites are seen in the STS and in the phase-lagged LFP and spiking output in the auditory cortex (20, 32, 33).

In principle, our results are consistent with the idea that intrinsic oscillations in the STS shape responses to face/object stimuli, although two observations suggest a more nuanced account. First, visual stimulus-evoked responses can include oscillatory activity (34–36). Phase coding is agnostic to the generators of the oscillation; there merely needs to be phases to which spikes align differently. Here the <20-Hz activity was partially accounted for by phase alignment with respect to the evoked LFP response. Given the period of lower frequencies, this could translate to the spike placement in only one or two cycles, which nevertheless is at the level necessary for rapid categorization and shown to be sufficient in models using phase coding for object categorization (14).

Second, simply because oscillations are not locked to the stimulus of interest does not necessarily mean that that they cannot be locked to other behavioral or cognitive events. For example, these same recordings revealed that STS neurons are phase-locked to saccadic eye movements, and that the LFP shows a supra-additive phase concentration when the eye movements occur coincident with image onset (19). Thus, the phase coding in gamma that was not locked to stimulus onset could still be locked to the saccadic command signals, consistent with saccade-related synchronization in other visual areas (37–42). This is just one example of a specialized type of intrinsic activity that interacts with stimulus-driven responses.

Firing Rates into Phases.

One neocortical mechanism for phase coding is that rhythmic inhibition may interact with stimulus-evoked excitation, producing spikes earlier in an oscillatory cycle (typically gamma) for preferred stimuli compared with nonpreferred stimuli (22–24). Thus, the enhanced response for preferred stimuli commonly seen in the slowly evolving rate code also may be coded through differences in spike timing within a single gamma cycle. Theoretically, this would provide a downstream target with a faster readout than would be possible with rate coding. Indeed, this mechanism was tested directly for orientation preferences of neurons in awake macaque V1, revealing a small but consistent phase advance for a cell’s preferred orientation (16). To test whether greater differences in firing rates directly accounted for differences in phase angle, we measured the variance in phase angle explained by firing rate differences, and found that the greatest explained variances (30–40%) occurred during the initial response in the lower frequency ranges, whereas the gamma frequency phase coding clusters had less variance explained by firing rate differences (<20%; Fig. 3B).

These results suggest that the visual category displayed can be extracted from both the oscillatory phase when firing occurs and the difference in spike count in that time window. When similar stimulus attributes are coded by both rate and phase codes, the advantage of the phase code is that the stimulus difference is evident in a narrower time window, and that the phase-limited spikes may exhibit reduced noise correlations, meaning a higher-fidelity signal, as has been shown in V1 (25). In the present study, variability in the phase differences of cells was generally not attributable to firing rate differences, particularly for the clusters of phase coding seen in the gamma band shortly after image onset. This suggests that even when rate codes can be used for decoding, there may be benefits to the neural systems that also use phase coding. In the primary visual (25) and primary auditory cortex (30), the stimulus-driven rate code is among the most robust and best-understood neural responses; nevertheless, temporal codes have been shown to provide independent information about stimuli. Just as phase precession in the hippocampus provides a complementary code to rate-based “place fields,” the processing described here for STS may complement protracted, rate-coded activity in IT that also shows important aspects of object recognition, including exemplar identification and various “invariances” or tolerances to differing visual inputs of the same object, such as size, lighting, and view (reviewed in refs. 6, 43). Even in IT, responses demonstrate several types of temporal information that meets or exceeds the information provided from a straight “mean rate response” (5, 44–47). It would be interesting to explore whether additional information can be obtained in a phase code for any of the many features and classes of stimuli that evoke responses in IT.

For the upper bank STS at least, phase coding may indicate one role for intrinsic or extraretinal rhythms in the coding of extrinsic, or stimulus-driven, inputs. In addition, the phase code observed in the present study may help explain the speed with which we can recognize and categorize objects.

Methods

General Procedures.

Two adult male rhesus macaques (Macaca mulatta) were each surgically implanted with a recording chamber positioned above the left hemisphere STS. This area corresponds to areas TAa and TPO based on anatomic nomenclature (48) and to the STP based on a functional definition (7). A multiple-electrode array was used to record from a combination of sites in the STS and/or auditory cortex. Details of the surgical procedures and the chamber and electrode localization are available elsewhere (19, 20). Experiments were conducted with the approval of local authorities (Regierungspräsidium, Tübingen) and in accordance with the guidelines of the European Community (EU VD 86/609/EEC) for the care and use of laboratory animals.

Behavioral Apparatus and Paradigm.

Experiments were conducted in a double-walled, darkened, sound-attenuating booth. The monkey subjects sat in a primate chair in front of a 21-inch color monitor at a distance of 94 cm. Eye position was measured with either an infrared system (Iview; SensoMotoric Instruments) or a scleral search coil (CNC Engineering). A trial began with the appearance of a central fixation spot. The monkeys were required to fixate on this spot within a 1–2° radius for 500 ms. After successful fixation, a face or object image was presented for another 500 ms, followed by juice reward for maintaining gaze within the image boundary. Fig. S1 illustrates eye movements relative to stimulus onset.

Each image was 9.1° wide and 6.7° high and contained 1 of 12 conspecific monkey faces or 1 of 12 clip-art objects. Stimulus categories and exemplars were sampled at random without replacement, with typically 10 repetitions per exemplar and 120 trials per face or object category.

Data Acquisition.

Electrodes were glass-coated tungsten wire with an impedance of 1–3 MΩ, measured at 1 kHz (Alpha Omega). The stainless steel chamber was used as a reference. Signals were acquired at a 20.3-kHz sampling rate, amplified, and bandpass-filtered with a low cutoff at 1 Hz and a high cutoff at 5 kHz (National Instruments). Recordings were made regardless of the stimulus selectivity of cells, with a distance of 1–2 mm between adjacent electrodes.

Data Analysis.

All analyses were done using Matlab (Mathworks) with routines for spectral analysis from Chronux (49).

Spike Sorting.

Spiking activity was extracted from the digitized recordings, and individual units were isolated offline using Plexon Offline Sorter. Spike sorting was performed using features of the waveform extracted by principal component analysis. For inclusion in this study, a single unit had to have been functionally localized to the upper bank STS and well isolated from the remaining multiunit activity on at least one of the first two principal component analysis scores of the waveforms, its isolation had to be stable across the duration of this experiment, and the interspike interval distribution had to have a clean refractory period.

Spike Analysis.

Peristimulus time histograms are displayed as spike density functions, with each spike smoothed by a Gaussian kernel with a 20-ms SD. Firing rates for the smoothed function are instantaneous and thus may appear higher than the firing rates calculated by summing spikes over long time bins. Our analysis included only units with significantly different responses in the poststimulus and prestimulus periods, for at least one of the two stimulus conditions. Although in principle, phase differences can occur without a change in firing rate, the response requirement ensured a sufficient spike count per analysis window (Spike–Field Analysis). In line with previous reports (18), these units were broadly tuned across exemplars within a category, and our sampling for each exemplar was limited (∼10 trials/exemplar), making it infeasible to assess phase codes for each exemplar. In contrast, differences across category were more common and were sampled equally over a sufficient number of trials, rendering them suitable for the present analysis. The average number of spikes per unit was 1,509 ± 707 for faces and 1,449 ± 704 for objects. Spike latency was calculated as described previously (50)

Spike–LFP Analysis.

We investigated the relationship between LFP, a continuous signal, and spiking activity, represented as time points of spike occurrence, by estimating the spike–LFP coherency. LFPs were extracted from the raw extracellular recordings by low-pass filtering with a cutoff at 300 Hz and resampling to 1 kHz. Each site’s mean LFP response to faces or objects was normalized to its maximal response across categories (Fig. 1C), before the average across sites was taken. To reduce artifacts caused by leakage of spiking activity through the filter into the LFP, which can potentially artificially inflate the spike–LFP coherence estimates (16), we used LFP and spiking activity from different but simultaneously recorded electrodes. We selected the electrode pairs with the highest spike–LFP coherence for both image categories in a frequency range of 5–150 Hz during the initial stimulus period (0–200 ms after stimulus onset). This selection process did not introduce a multiple-comparison problem, given that high overall spike–LFP coherence does not imply large between-category phase angle differences.

Coherency is the cross-spectrum of the two time series normalized by their autospectral densities. The spike-field coherence is given as the absolute value of the complex valued coherency, and the spike phase as its angle. Coherency was computed using multitaper spectral estimators (51–54). Both LFP and spike spectra were estimated over 200-ms windows with a spectral smoothing of ±12.5 Hz, using five Slepian data tapers. The time window was stepped by 5 ms between estimates, and the time index was aligned to the center of the window. Our data include an ample number (>200) of trials per cell, with similar spike counts across conditions. Importantly, we are not concerned with absolute coherence levels per se, but rather with differences in phase angle across these equally sampled categories, where the expected phase angle is independent of spike count. For these reasons, our data are suitable for analysis using this method. (Methods are compared in ref. 55.)

We examined the average spike-phase differences between the two stimulus conditions over all frequencies (1–150 Hz) for all time-frequency points from 500 ms before image onset until 500 ms after onset (56, 57). We used the Watson–Williams test (51, 58), which is a circular analog of the two-sample t test or single-factor ANOVA, to assess whether the mean directions of two or more groups are identical. To correct for multiple comparisons across time and frequency, we performed a nonparametric suprathreshold cluster test, described in detail elsewhere (59). Suprathreshold cluster tests threshold the F statistic at each time-frequency bin at a predetermined primary threshold. Connected suprathreshold regions with a mass exceeding a critical value are considered significant. The critical value is the 95th percentile of a null distribution containing the maximal suprathreshold cluster masses for the given primary threshold. We obtained this chance distribution through randomly reassigning the labels for each trial (face or object), calculating the F statistic for each time-frequency bin, thresholding at F >1.64, and recording the exceedance mass of the largest suprathreshold cluster. The exceedance mass is the integral of the F statistic that lies above threshold in a suprathreshold cluster (59, 60). We performed 10,000 permutations of the trial labels, producing 10,000 exceedance mass values, and then determined the 95th percentile of this chance distribution. Time-frequency clusters in the empirical, nonpermuted data that had a mass exceeding this critical value were defined as significant. We also used another method to calculate significance, using only the category-label permutation of each time-frequency point to generate a null distribution and selecting time-frequency points that exceeded the 95th percentile of the null distribution. This method led to qualitatively similar results (Fig. S2), but did not correct for multiple comparisons across time and frequency, and so we chose to display the more conservative (multiple-comparison corrected) measure.

To illustrate the difference in preferred spike phase between the two stimulus conditions for one time-frequency point, we divided the spike phases for all face trials and again for all object trials into seven bins, equally spaced from 0 to 2π, and plotted the counts per bin. To account for phase differences that were evoked (i.e., phase-locked to the stimulus), we subtracted the mean LFP for that trial category at that electrode site from the LFP of each trial (17, 19, 21, 61). Fig. 3A shows the proportion of cells with phase differences remaining after removal of the evoked component. We also performed within-category trial shuffling of spikes and LFP signals. This was another way of determining the phase effects that were stimulus-locked (which would be preserved even after decoupling spike and LFP) and effects due to endogenous or induced activity that would change independent of stimulus onset (which would be lost after decoupling the spike and field from a given trial). This yielded similar results to the mean-subtraction procedure, but from the opposite direction; that is, the preserved component for the shuffle is the evoked component, and the preserved response for mean subtraction is the nonevoked component.

Phase Angle and Firing Rate Analysis.

Low firing rates can distort spike–LFP coherence values, and even for cells with high firing rates, some theories postulate that firing rate values may be converted into phase values (22–24), such that stimuli evoking maximal firing rate responses will fire at earlier phases of an oscillation compared with stimuli evoking lesser or no responses. Thus, the greater the firing rate difference across stimuli, the greater the phase advantage for the preferred stimuli over nonpreferred stimuli. We measured the phase angle difference through the signed circular F statistic (57, 58). The sign was determined by setting each cell’s overall preferred phase as π and counting the first mean phase encountered after 0 as the leading phase, that is, the first to emerge from the less-excited phase of firing. We then compared the phase difference between leading and lagging phases to subtractive (i.e., linear) differences in mean firing rate for faces and objects, using the same time windows to calculate both phase differences and rate differences (200 ms sliding in 5-ms intervals). We took the R² explained variance value, where R is the Pearson correlation obtained from comparing the phase angle statistic and their respective rate differences in the specified window across cells. Significance was determined by the suprathreshold cluster test described above.