Effects of Long-term Visual Experience on Responses of Distinct Classes of Single Units in Inferior Temporal Cortex

Abstract

Primates can learn to recognize a virtually limitless number of visual objects. A candidate neural substrate for this adult plasticity is the inferior temporal cortex (ITC). Using a large stimulus set, we explored the impact that long-term experience has on the response properties of two classes of neurons in ITC, broad-spiking (putative excitatory) cells and narrow-spiking (putative inhibitory) cells. We found that experience increased maximum responses of putative excitatory neurons but had the opposite effect on maximum responses of putative inhibitory neurons, an observation that helps to reconcile contradictory reports regarding the presence and direction of this effect. In addition, we found that experience reduced the average stimulus-evoked response in both cell classes, but this decrease was much more pronounced in putative inhibitory units. This latter finding supports a potentially critical role of inhibitory neurons in detecting and initiating the cascade of events underlying adult neural plasticity in ITC.

Introduction

Visual perception is a consequence of the concerted activity of neurons throughout the visual system. At the same time, the response properties of single neurons in the visual system depend on visual experience for their proper development (Hubel and Wiesel, 1965). Therefore, to understand visual perception, one must understand the effects of visual experience. Although receptive field properties of cortical neurons in early visual areas become less plastic with age (Hubel and Wiesel, 1970), neurons later in the visual hierarchy exhibit plasticity well into adulthood. In particular, neurons in the functionally mature inferior temporal cortex (ITC) – a collection of areas in the primate brain hypothesized to underlie visual object recognition (DiCarlo and Cox, 2007; Logothetis and Sheinberg, 1996; Tanaka, 1996) – can adapt their responses to the statistics of visual input (Erickson and Desimone, 1999; Li and DiCarlo, 2008, 2010; Miyashita, 1988) and to a behavioral task's perceptual demands (Baker et al., 2002; Freedman et al., 2006; Kobatake et al., 1998; Logothetis et al., 1995; Op de Beeck et al., 2006). Neuronal activity in ITC is thus a joint product of accrued past experience and current input, and its investigation can shed light on the question of how memory and perception interact continuously at the level of single neurons.

Visual experience with a set of objects can be induced experimentally by mere exposure (Anderson et al., 2008; Freedman et al., 2006), by discrimination training (Baker et al., 2002; Freedman et al., 2006; Kobatake et al., 1998; Logothetis et al., 1995; Sigala and Logothetis, 2002) or by explicit memorization (Sakai and Miyashita, 1991). To infer the impact of visual experience on ITC, neuronal responses to familiar or learned stimuli are compared to a pre-exposure baseline (De Baene et al., 2008), to responses in untrained subjects (Kobatake et al., 1998) or, most commonly, to responses to novel or unlearned stimuli (Anderson et al., 2008; Baker et al., 2002; Freedman et al., 2006; Logothetis et al., 1995; Miyashita et al., 1993). The resulting neuronal changes remain a matter of debate. Early studies reported that single neurons in ITC, on average, developed strong responses to a small (and different) subset of learned stimuli, which were larger than the maximal responses across the unlearned set (Kobatake et al., 1998; Logothetis et al., 1995; Miyashita, 1993; Sakai and Miyashita, 1994). Such strengthening of specific responses could amplify the neurons’ impact on downstream areas, which would, in theory, facilitate behavior driven by recognition of well-known objects. However, recent studies have reported no change or even decreased maximal responses to familiar as compared to novel stimuli as well as a concomitant experience-dependent decrease in the overall population response (Anderson et al., 2008; Baker et al., 2002; Freedman et al., 2006; Op de Beeck et al., 2008; Op de Beeck et al., 2007). These divergent findings have been attributed to more unbiased single unit selection procedures, to comparisons within rather than across animals, and to more finely controlled stimulus exposure protocols. Interestingly, while both firing rate increases and decreases can increase single cell selectivity (i.e., narrow the tuning bandwidth), recently reported modulations have been on the order of a few spikes per second (Baker et al., 2002; Cox and DiCarlo, 2008; De Baene et al., 2008; Freedman et al., 2006), leading some to propose that visual experience results only in subtle neuronal plasticity in ITC (Op de Beeck and Baker, 2010). Behavioral data, on the other hand, indicate that the impact of visual experience on recognition behavior can be large (Gauthier and Tarr, 1997; Logothetis et al., 1995; Mruczek and Sheinberg, 2007).

Two factors have impeded progress in our understanding of the effects of visual experience on single unit responses in ITC. First, it is unclear with which stimuli to sample the tuning functions of individual ITC neurons. Advances have been made on this issue (Brincat and Connor, 2004, 2006; Rust and Dicarlo, 2010; Sary et al., 1993; Tanaka, 1996; Yamane et al., 2008), but we are far from predicting responses to arbitrary visual patterns. The lack of increased responses and small selectivity increases to learned stimuli could thus be a result of not selecting the appropriate images to drive individual neurons; using large stimulus sets can partially ameliorate this issue. The second problem has been the averaging of responses over several distinct cell classes. We know that cortex comprises many different cell types (Connors and Gutnick, 1990; Markram et al., 2004; Peters and Jones, 1984), which mediate different functions within circuits. One means of distinguishing cell classes is by the shapes of their extracellularly recorded spikes (Bartho et al., 2004; Mitchell et al., 2007; Niell and Stryker, 2008). Data indicate that neurons which generate narrow spikes correspond primarily to fast-spiking inhibitory cells whereas broad-spiking neurons correspond primarily to excitatory pyramidal cells (Bartho et al., 2004; Henze et al., 2000; Kawaguchi and Kubota, 1997; McCormick et al., 1985; Nowak et al., 2003). No studies to date, however, have probed the potential differential effect of visual experience on distinct cell classes in ITC.

Here, we show that experience caused putative excitatory neurons to respond much more robustly to their best familiar compared to their best novel stimuli. In contrast, familiarity caused a dramatic decrease in the maximum and average rates of putative inhibitory neurons. Together, the results suggest that visual experience can profoundly alter visual object representations in ITC.

Results

To understand how long-term sensory input sculpts the responses of individual ITC neurons, we first familiarized each of two monkeys with 125 color images of real world objects (Hemera Photo-Objects: Vol. 1, 2 and 3) (Figure S1A). The monkeys were trained to both passively fixate the stimuli and to perform a short-term memory task with them. This exposure phase lasted between 3 months (Monkey I) and 12 months (Monkey D), resulting in an estimated number of exposures equal to 1000 (Monkey I) and 3000 (Monkey D) repetitions per image, split roughly evenly between the two tasks. Once familiarization was completed, we recorded the activity of well-isolated single units in ITC (n = 50 from Monkey D, n = 38 from Monkey I) in a passive fixation task (Figure 1A). Each neuron was screened with 125 familiar and 125 novel stimuli. The 125 novel stimuli were picked randomly on a daily basis from the same database as the familiar set (for examples, see Figures S1B – S1D). We recorded all units deemed visual by inspection of online stimulus-locked rastergrams. Both monkeys provided qualitatively similar data so the results have been combined across subjects. Any notable differences are acknowledged (see Figure S3 for the main results split by monkey).

Figure 1.

Experimental paradigm and spike waveform clustering. (A) Passive fixation task during which 10 stimuli were presented for 200 ms each with a 50 ms interstimulus interval. Familiar and novel stimuli were interleaved. (B) All recorded spike waveforms, aligned by their troughs and labeled according to their cluster membership. (C) Distribution of spike widths (trough-to-peak durations) and the two clusters that emerged from the k-means algorithm. The bars above the distributions show mean ± SD of the respective distributions.

As a means of correlating visual response properties with specific cell classes, we characterized the recorded sample of single units by the trough-to-peak widths of their extracellular spike waveforms (Figures 1B and 1C). Consistent with previous studies (Diester and Nieder, 2008; Hussar and Pasternak, 2009; Mitchell et al., 2007), we observed that the distribution of these widths was bimodal, and we thus divided the neurons via a k-means algorithm into two categories, broad-spiking and narrow-spiking (Figure 1C). Previous results have suggested that narrow spikes correspond primarily to inhibitory, fast-spiking interneurons whereas broad spikes correspond primarily to excitatory pyramidal neurons (Bartho et al., 2004; Connors and Gutnick, 1990; McCormick et al., 1985). For clarity, we thus refer to the narrow-spiking neurons as putative inhibitory and to the broad-spiking ones as putative excitatory.

Example cells

Figures 2A – 2G show the activity of seven representative single units. Each unit was stimulated with the same set of 125 familiar stimuli but with a different set of 125 novel stimuli. The top five rows (Figure 2A – 2E) correspond to putative excitatory cells. In general, these units exhibited an enhanced response to the best familiar compared to the best novel stimulus. This advantage, however, was restricted to the highest ranked stimuli (with the notable exception of the unit shown in Figure 2C). Further, note that the best familiar stimulus elicited a robust firing rate that reached a peak level of around 100 Hz in every neuron, suggesting that we were able to find highly effective stimuli for activating these neurons. The increased firing rates of putative excitatory cells to top ranked familiar stimuli compared to top ranked novel stimuli translated directly into increased selectivity (sparseness) for the familiar stimulus set (Figures 2A – 2E, right column).

Figure 2.

Example neuronal responses to familiar and novel stimuli. (A) – (E) Five representative putative excitatory cells. (F and G) Two representative putative inhibitory cells. In all rows, the column on the far left shows both the mean spike waveform of each cell and the cluster to which the waveform was assigned (blue = broad spike, red = narrow spike). In the middle five columns are plotted the spike density functions (SDFs, spike times convolved with a Gaussian kernel with σ = 20 ms) for the top 5 stimuli from the familiar set (black) and the top 5 stimuli from the novel set (green). These rankings were determined not on the basis of the peak value of the SDF but rather from the spike counts in the interval 75 – 200 ms after stimulus onset, which is shown as a light gray bar abutting the time axis. The insets in these graphs show the actual familiar and novel images eliciting the response. The column on the far right shows each neuron's entire distribution of mean firing rates, sorted according to rank. Again, the mean firing rates were computed from the spike counts in the interval 75 – 200 ms after stimulus onset and the rankings were done independently for the familiar and novel sets. The numbers in the top right of the rank plots show the magnitude of the sparseness metric that was used to quantify single cell selectivity.

The bottom two rows (Figures 2F and 2G) correspond to putative inhibitory cells. Putative inhibitory cells nearly always showed a greater response to the best novel compared to the best familiar stimulus, an effect that appeared after the initial visual transient. These units also responded with an elevated rate to a much larger portion of stimuli than putative excitatory cells, regardless of stimulus set (Figures 2F and 2G, right column), and their firing rates could reach high peak values (~200 Hz, see Figure 2F). In addition, note that the reduced firing rates of putative inhibitory cells to familiar stimuli could span the entire range of ranks (Figure 2F, right column). While these experience-dependent firing rate changes could also result in selectivity increases, these were less reliable than those observed in putative excitatory cells (Figures 2F and 2G, right column).

Visual experience increases maximum responses of putative excitatory cells but decreases maximum responses of putative inhibitory cells

We began with a simple question: did experience with a set of stimuli result in the emergence of stronger ITC responses, and if so, did this effect depend on cell class? As neurons in ITC can exhibit marked selectivity, and thus fail to be activated by many stimuli independent of experience, we narrowed the focus of this query to just the maximum responses. In particular, for every neuron, we extracted a pair of mean firing rates, one elicited by the single most effective familiar stimulus and one by the single most effective novel stimulus.

To gain insight into the time course of experience-dependent maximum firing rate differences, we first computed this statistic with a sliding window (step size = 5 ms, window size = 50 ms). In Figure 3A we see that, averaged across the population of putative excitatory cells, the maximum responses to the familiar set were much greater than to the novel set and this difference emerged at about the same time as the onset of the visual response (earliest significant difference = 120 ms, p < .05, permutation test, corrected for multiple comparisons, see Supplemental Experimental Procedures). In contrast, averaged across the population of putative inhibitory cells (Figure 3B), the maximum responses to the familiar set were much smaller than to the novel set and this difference did not emerge until after the initial visual transient (earliest significant difference = 170 ms).

Figure 3.

Visual experience increases maximum responses of putative excitatory cells but decreases maximum responses of putative inhibitory cells. (A and B) Sliding window analyses (step size = 5 ms, window size = 50 ms) of maximum firing rates to familiar (black) and novel (green) stimulus sets, averaged separately for the putative excitatory (A) and putative inhibitory (B) cells. Shaded regions indicate ±SEM. Tick marks denote the time points at which the differences between the maximum familiar responses and maximum novel responses achieved statistical significance according to a permutation test (p < .05). (C) Distribution of individual cells' responses to the best familiar (x-axis) and best novel (y-axis) stimulus during the early epoch (75 – 200 ms). Each data point represents the activity of a single unit. Cells are color-labeled according to cluster membership (blue, putative excitatory; red, putative inhibitory). Error bars represent mean ± SEM across individual repetitions of the best familiar or best novel stimulus. Histogram in the top right shows the distribution of differences for both subpopulations. Shaded bars show individually significant cases (p < .05, Mann-Whitney test). Arrows denote mean maximum response differences across either the putative excitatory (blue) or inhibitory (red) cells. (D) A magnified view of the plot in (C), emphasizing the distribution of effects in the putative excitatory cells. (E and F) Same as in (C and D) but for the late epoch (200 – 325 ms).

We next examined experience-dependent maximum firing rate differences in individual units. We divided the data into two time epochs, an early epoch of 75 – 200 ms and a late epoch of 200 – 325 ms. In Figures 3C – 3F, we plot for each epoch, and at two different scales to emphasize the distribution of putative excitatory units, the magnitude of each cell's response to its single best familiar and to its single best novel stimulus. In the early epoch (Figures 3C and 3D), the majority of putative excitatory cells (blue points) lie below the diagonal line, indicating that for these neurons the best familiar stimulus elicited a stronger response than the best novel stimulus. Averaged across the population of putative excitatory cells, the firing rate to the best familiar stimulus was 16.55 ± 2.22 Hz (mean ± SEM) greater than the firing rate to the best novel stimulus (blue arrow in Figure 3C, p < .001, paired t-test), an increase of nearly 50% (52.69 Hz compared to 36.14 Hz). In the late epoch (Figures 3E and 3F), this difference diminished (blue arrow in Figure 3E, familiar - novel, 4.40 ± 2.41 Hz, p = .07)

Putative inhibitory cells led to a different distribution of maximum firing rate differences (Figures 3C and 3E, red points). In both the early (Figure 3C) and late (Figure 3E) epochs, most putative inhibitory cells were driven to a much higher firing rate by their best novel than by their best familiar stimulus (red points below unity diagonal). In the early epoch, the population-averaged difference in maximum firing rate was 27.63 ± 7.97 Hz in favor of the novel set (red arrow in Figure 3C, p = .004, paired t-test) but significant only in one monkey (compare Figures S3C and S3D), whereas in the late epoch it rose to 53.65 ± 12.11 Hz (red arrow in Figure 3E, novel - familiar, p < .001) and became significant in each monkey.

Visual experience decreases the average stimulus-evoked firing rate in putative excitatory and inhibitory cells

We next asked how neuronal responses to familiar and novel stimuli differ when averaged across the entire ensemble of stimuli. Such an analysis offers a glimpse into ITC neurons’ more typical firing rate modulations, that is, their stimulus-evoked firing rates to a randomly chosen, as opposed to their most effective, stimulus. We computed for each cell its average stimulus-evoked response, which we defined as the average over the mean firing rates to each of the 125 stimuli within either the familiar or novel set (Figures 4A – 4D). Paralleling previous reports that have grouped neurons into two distinct classes based on extracellular spike waveform (Diester and Nieder, 2008; Mitchell et al., 2007), we first note that putative inhibitory units had much larger stimulus-driven activity than putative excitatory units. This can be appreciated by comparing the axes in Figure 4A (putative excitatory) and Figure 4B (putative inhibitory) and by comparing the blue (putative excitatory) and red (putative inhibitory) points in Figures 4C and 4D. To quantify this difference, we compared the average stimulus-evoked firing rates of putative excitatory cells to those of putative inhibitory cells within each unique combination of stimulus set (familiar/novel) and time epoch (early/late). All comparisons were highly significant (mean ± SEM Hz for putative excitatory versus putative inhibitory; familiar early, 8.62 ± .70 versus 35.12 ± 3.24; familiar late, 5.90 ± .60 versus 22.96 ± 3.54; novel early, 9.20 ± .92 versus 44.26 ± 4.21; novel late, 7.79 ± .91 versus 44.00 ± 4.01; p < .001 for every comparison, uncorrected, two-sample t-tests). Because it has been shown that current injections can drive fast-spiking inhibitory units to very high firing rates (McCormick et al., 1985), the higher average responses of narrow-spiking units further supports the labeling of this cell class as putative inhibitory. We observed a similar difference in firing rates when we looked at spontaneous activity, which we took as the last 500 ms of the fixation epoch (putative excitatory, 5.20 ± .68 Hz; putative inhibitory, 15.01 ± 2.87 Hz; p = .004, two-sample t-test).

Figure 4.

Visual experience decreases average stimulus-evoked responses of putative excitatory and inhibitory cells, particularly in the late phase, but the effect is much larger in putative inhibitory cells. Conventions same as in Figure 4 with the notable exception that the metric of interest is the average, not maximum, response across the 125 familiar or 125 novel stimuli. Error bars in (C) and (D) represent mean ± SEM across the 125 familiar or 125 novel (mean) firing rates. Individually significant cases in histograms of panels C and D were determined with a t-test (p < .05).

Notably, we found that in both cell classes the novel set elicited higher average responses than the familiar set (Figures 4A – 4D). Like the maximum response effect in putative inhibitory units, these experience-dependent differences in average firing rate emerged, in both cell classes, after the initial visual transient (Figures 4A and 4B). In particular, in the early epoch (Figure 4C), the population-averaged difference for the putative excitatory cells was small and not significant (familiar - novel, mean ± SEM, −0.59 ± .42 Hz, p = .17, paired t-test), and while the difference was larger and significant in the putative inhibitory subset (familiar - novel, −9.14 ± 2.85 Hz, p = .006), it was only observed in one monkey (compare Figure S3C and S3D). It was in the late epoch (Figure 4D) that population-averaged differences in average firing rate for both classes of cells became significantly different from zero (familiar - novel; putative excitatory, −1.90 ± .67 Hz, p = .006; putative inhibitory, −21.04 ± 4.01 Hz, p < .001; in one monkey the putative excitatory effect was marginally significant, p = .09). Consistent with these observations, we also observed that experience led to decreases in the proportion of stimuli eliciting a significant elevation in firing rate and to increases in the proportion of stimuli eliciting a significant reduction in firing rate (Figure S4). Further, although both cell classes showed reduced average responses to familiar stimuli, this decrease was much larger in putative inhibitory than excitatory cells (early epoch, p = .001; late epoch, p < .001; two-sample t-tests; early epoch effect not significant in the same monkey whose effects tended to arise later), which can be seen by comparing the red and blue arrows in the histograms of Figures 4C and 4D.

Visual experience increases selectivity of putative excitatory cells

To convey information, neurons modulate their firing rates. The greater and/or more reliable this modulation, the more informative the neuron's firing rate becomes about the presence (or absence) of some stimulus. Because we have shown that visual experience not only led to an increase in maximum response (in putative excitatory cells) but also to a decrease in average response, we have already implicated visual experience in sharper stimulus selectivity. Here, we make this idea explicit.

To capture increases in selectivity with a single metric, we computed the value of (lifetime) sparseness (Olshausen and Field, 2004; Rolls and Tovee, 1995; Vinje and Gallant, 2000; Zoccolan et al., 2007) (see Experimental Procedures). Sparseness quantifies how much of a single neuron's total firing rate, across a stimulus set, is concentrated within a few stimuli. A neuron with high sparseness will be quiet most of the time but there will be a few stimuli that elicit robust firing rates. By definition, this is a selective neuron. An unselective neuron, one with low sparseness, will respond with an elevated firing rate to many stimuli. We calculated the sparseness of cells’ responses across the familiar and novel stimulus sets, first with a sliding window (Figures 5A and 5B) and then in the previously defined early and late epochs (Figures 5C and 5D).

Figure 5.

Visual experience increases selectivity (sparseness) of putative excitatory and inhibitory cells. Same conventions as in Figures 3 and 4, except that the metric investigated is sparseness across the 125 familiar or 125 novel stimuli. Individually significant cases in histograms of panels C and D were determined with a permutation test (p < .05).

As with the average response analyses, one of the more conspicuous features of the data was that putative inhibitory units had much lower sparseness than putative excitatory units for every combination of stimulus set and epoch (mean ± SEM putative excitatory versus putative inhibitory; familiar early, .53 ± .03 versus .16 ± .02; familiar late, .65 ± .03 versus .32 ± .04; novel early, .42 ± .02 versus .17 ± .02; novel late, .57 ± .02 versus .24 ± .02; p < .001 for every comparison, uncorrected, two-sample t-tests). The broad tuning of putative inhibitory units is consistent with recent functional data (Kerlin et al., 2010; Liu et al., 2009; Sohya et al., 2007) as well as neuroanatomical data showing that these units can receive highly convergent and heterogeneous input from the surrounding excitatory population (Bock et al., 2011).

Importantly, we found that the sparseness of putative excitatory cells was significantly greater for familiar than novel stimuli, in both the early and late epochs (compare black and green curves in Figure 5A, see blue points and arrows in Figures 5C and 5D; mean ± SEM familiar - novel; early epoch, .11 ± .01; late epoch, .08 ± .02; p < .001 in both instances, paired t-tests).

In the putative inhibitory population, we observed a somewhat different and less conclusive set of results. First, note that the familiar sparseness for this population of cells did not reach its peak value until late in the visual response (black curve in Figure 5B). Averaged across the population of narrow spiking neurons, sparseness for familiar stimuli was significantly greater than for novel stimuli only in the late epoch (compare black and green curves in Figure 5B, see red points and arrows in Figures 5C and 5D; mean ± SEM familiar - novel; early epoch, −.01 ± .01, p = .43; late epoch, .08 ± .04, p = .04; paired t-tests) and only in one monkey (late epoch, monkey D, p = .19; monkey I, p = .01).

The selectivity analyses argue that the sparseness of putative excitatory, and possibly putative inhibitory cells, in ITC is not a static property, but rather one that visual experience can increase. In general, sparseness can be increased either by increasing the proportion of near-zero responses (Tolhurst et al., 2009) or by increasing the response magnitude to a subset of the most effective stimuli. We have already shown that in the early epoch, putative excitatory cells had higher maximum responses to familiar than novel stimuli. Could this difference account for these cells’ increased sparseness? We addressed this question by subtracting for each putative excitatory cell its maximum response across the familiar set from its maximum response across the novel set and then by correlating these differences with the differences between familiar and novel sparseness (Figure 6). Indeed, the experience-dependent increase in maximum response of putative excitatory cells was a good predictor of how much more selective individual cells were to stimuli within familiar compared to novel sets (Pearson’s r = .77, p < .001; r = .80 in monkey D, r = .75 in monkey I). No such relationship was observed in the late epoch (r = 0.00, p = .998) or in the early or late epochs of putative inhibitory cells (early, r = .27, p = .33; late, r = −.06, p = .82) (data not shown). We further confirmed the robust contribution of the differences in maximum firing rates to selectivity changes with a randomization procedure (Figure S6). We conclude that, in the early epoch, experience-dependent increases in the putative excitatory cells’ maximum responses contributed to a sparser (more selective) representation of familiar compared to novel stimuli. It is important to note that this conclusion is different from the more traditional concept of a sparse neuron as an infrequently active neuron (Haider et al., 2010; Rolls and Tovee, 1995; Tolhurst et al., 2009; Vinje and Gallant, 2000). Here, the analyses suggest that increased sparseness resulted in a neuron that fired more spikes to its preferred stimulus. Only in the late epoch (of both putative excitatory and inhibitory cells) did we find that the experience-dependent increases in sparseness could be better accounted for by decreases in the proportion of familiar stimuli eliciting a significantly elevated response (data not shown).

Figure 6.

For putative excitatory cells, the experience-dependent increase in maximum response predicts the experience-dependent increase in selectivity (sparseness). The difference between familiar and novel sparseness is plotted as a function of the difference between maximum familiar and maximum novel responses. Each point represents a single putative excitatory unit. Maximum responses and sparseness values were taken from the early epoch (75 – 200 ms).

Visual experience does not impede the ability of putative excitatory and inhibitory cells to discriminate between novel stimuli

In our experiments, visual experience caused marked differences in neuronal responses to familiar versus novel stimuli. Nonetheless, novel stimuli elicited robust activity from the population of recorded ITC neurons, indicating that neuronal activity in ITC can contribute to the recognition of both stimulus sets. Could ITC neurons discriminate as well amongst members of the novel set as of the familiar set? We probed this question with a receiver operating characteristic (ROC) analysis. In particular, we performed ROC analyses on all possible pairwise combinations of stimuli (within a set), each time summarizing the discriminability of the two firing rate distributions with the area under ROC curve (AUC) (Rust and Dicarlo, 2010). We took the average of the AUC values as a metric of overall discriminability, which captured how well, on average, a single neuron's spike counts could discriminate between the identities of any two arbitrarily chosen stimuli.

We first note that putative inhibitory cells conveyed more information about stimuli, familiar and novel, than did putative excitatory cells (Figures 7A and 7B, compare blue to red points) (mean ± SEM putative excitatory versus putative inhibitory; familiar early, .673 ± .008 versus .702 ± .011; familiar late, .648 ± .007 versus .698 ± .011; novel early, .665 ± .008 versus .729 ± .013; novel late, .682 ± .009 versus .778 ± .009; p = .04 for familiar early comparison, where the difference was not significant in one monkey; p < .001 for all other comparisons, familiar late comparison was not significant in same monkey, uncorrected, two-sample t-tests). This finding is consistent with the broader tuning of putative inhibitory cells, which allowed them to respond in a stimulus selective manner to more than just the top few stimuli.

Figure 7.

Putative excitatory and inhibitory units can discriminate between stimuli within the familiar and within the novel sets. (A) Distribution of individual cells' familiar (x-axis) and novel (y-axis) mean pairwise AUC values during the early epoch (75 – 200 ms). Each data point represents the mean pairwise AUC value of a single unit. Cells are color-labeled according to cluster membership (blue, putative excitatory; red, putative inhibitory). Histogram in the top right shows the distribution of AUC differences for both subpopulations. For clarity, all bars are shaded but this does not indicate significance. Arrows denote mean differences across either the putative excitatory (blue) or inhibitory (red) cell class. (B) Same as in (A) but for the late epoch (200 – 325 ms). (C) Relationship between sparseness and mean pairwise AUC value for familiar (black) and novel (green) stimuli during the early epoch (75 – 200 ms). Putative inhibitory units are indicated by open circles. (D) Same as in (C) but for the late epoch (200 – 325 ms).

Notably, we found that spike counts of both putative excitatory and inhibitory cells could be used to discriminate between novel stimuli as well as, or even better than, familiar stimuli. The only case in which the familiar set fared better was the early epoch of putative excitatory cells, but this difference was small and not significant in either monkey separately (Figure 7A, blue points and arrow; mean familiar AUC = .673, mean novel AUC = .665, p = .046, paired t-test). Further, note that the late epoch of putative excitatory cells more than compensated for this initial difference (Figure 7B, blue points and arrow; mean familiar AUC = .648, mean novel AUC = .682, p < .001). For the putative inhibitory cells, the novel stimuli could be better discriminated in both epochs (Figures 7A and 7B, red points; early epoch, mean familiar AUC = .702, mean novel AUC = .729 p = .004; late epoch, mean familiar AUC = .698, mean novel AUC = .778, p < .001), with one monkey showing much stronger and reliable differences than the other. Visual experience, therefore, did not prevent neurons in ITC from contributing reliably to the encoding of both familiar and novel stimuli.

Given that putative inhibitory cells had lower sparseness than putative excitatory cells but were better able to discriminate between any two arbitrarily chosen images, we wondered whether there was a relationship between sparseness and mean pairwise AUC values. In Figures 7C and 7D, we have plotted individual cells’ sparseness and mean pairwise AUC values for the early and late epochs (putative inhibitory units are indicated by open symbols). For both familiar (Figures 7C and 7D, black points and lines) and novel (green points and lines) stimuli, we observed a strong linear correlation between the two metrics. The correlation held even when we restricted the analysis to just the putative excitatory cells (Figures 7C and 7D, filled circles). This suggests that an increase in sparseness precluded a neuron from discriminating stimuli at the lower end of its firing rate distribution. As visual experience led to a considerable increase in sparseness, we conclude that individual ITC neurons contributed to the encoding of a smaller number of familiar compared to novel stimuli.

Discussion

Here, we asked whether visual long-term experience’s effects on single neuron responses in ITC vary with cell type. We first showed that the best stimulus from the familiar set drove putative excitatory cells much more robustly than the best stimulus from the novel set. This effect was reversed for putative inhibitory cells. We further showed that, on average, both putative excitatory and putative inhibitory neurons responded with a smaller response to a randomly chosen familiar compared to novel stimulus, but this difference was much larger in the putative inhibitory population. We then went on to show that experience increased sparseness in putative excitatory neurons and, to a lesser degree, in putative inhibitory neurons. For the putative excitatory neurons, the experience-dependent increase in sparseness could be well accounted for by an increased firing rate to the top familiar stimulus. Finally, we demonstrated that the experience-dependent modifications have a minimal impact on the ability of ITC neurons to discriminate between the stimuli in the novel set. In Figure 8, we provide a schematic summarizing the observed firing rate changes in both classes of neurons.

Figure 8.

Schematic representation of experience-dependent firing rate changes in putative excitatory and putative inhibitory units. (A) Firing rates of putative excitatory neurons are arranged in descending order of effectiveness. In this cell class, visual experience increased responses to the most effective stimuli, particularly in the early epoch, and decreased responses to moderately effective stimuli, especially in the late epoch. (B) Same as in (A) but for putative inhibitory units. Visual experience caused a much more widespread and noticeable decline in firing rates of these neurons. This change was most prominent in the late epoch.

Methodological approach

Neurons in neocortex can be classified on the basis of morphology, physiology, connectivity, laminar distribution, neurotransmitter content and/or expression of calcium-binding proteins, to name the most common schemes (Markram et al., 2004). In extracellular recording studies, most of these characteristics remain unknown, leading many to simply average results over all recorded cells, potentially obscuring important cell class-dependent differences. However, a growing body of evidence supports the utility of dividing extracellularly recorded spikes into putative excitatory and inhibitory classes based on spike shape (Bartho et al., 2004; Johnston et al., 2009; Tamura et al., 2004). The technique’s foundation rests on results suggesting that fast-spiking, parvalbumin-positive inhibitory interneurons express an abundance of Kv3 voltage-gated potassium channels, which endow them with their unique narrow action potentials (Kawaguchi and Kubota, 1997; McCormick et al., 1985; Rudy and McBain, 2001).

As with any classification scheme, caution should be exercised with this method’s application. Indeed, a recent electrophysiological study from the primary motor cortex of the monkey showed that pyramidal tract neurons can also emit narrow spikes (Vigneswaran et al., 2011). Whether such results will be extended to cortical areas with a less specialized corticospinal projection, a more representative distribution of cell types, and a more typical laminar profile remains an open question, but it is unlikely neuronal classification based on spike waveform alone can represent a one-to-one mapping (Nowak et al., 2003). Nonetheless, the method offers an important first step for dividing a sample of neurons into putatively different cell classes, i.e., it is better than no division at all if functional differences between the two classes can be shown to exist (Diester and Nieder, 2008; Hussar and Pasternak, 2009; Mitchell et al., 2007). For ease of exposition, we thus assume this division in the following discussion.

Putative excitatory cells

Several studies have explored the impact of visual experience on the maximum response magnitude of single ITC neurons. Early work showed that the best familiar stimulus elicits a higher firing rate than the best novel stimulus (Kobatake et al., 1998; Miyashita, 1993; Sakai and Miyashita, 1994). More recent work, however, has revealed that the best familiar and best novel stimuli, on average, evoke equivalent firing rates (Baker et al., 2002; Freedman et al., 2006; Op de Beeck et al., 2007). Here, we have provided data reconciling these disparate results by showing that whether experience increases or decreases the maximum response depends on both cell class and over what time epoch firing rates are computed. In particular, putative excitatory cells responded more strongly to the best familiar stimulus, but only in the early epoch, whereas putative inhibitory cells responded more strongly to the best novel stimulus, particularly in the late epoch. Given that excitatory cells are estimated to outnumber inhibitory cells by a ratio of about 4:1 (Markram et al., 2004), can the averaging across the two cell classes account for the recent absence of maximum response differences? In principle, this is possible because the absolute magnitude of the experience-dependent maximum response modulation was much larger for the putative inhibitory than putative excitatory cells (absolute difference, putative excitatory early phase = 16.55 Hz, putative inhibitory late phase = 53.65 Hz). Indeed, calculating firing rates over the window 75 – 325 ms post-stimulus onset and collapsing across the two cell classes leads to a much reduced and in one monkey a non-significant maximum response difference between the familiar and novel stimulus sets (two monkeys combined, best familiar - best novel = 2.64 Hz, paired t-test, p = .40; monkey D, −.21 Hz, p = .97; monkey I, 6.40 Hz, p = .02; in the monkey in which the difference remained significant, the difference decreased from 11.93 Hz when computing it from early epoch spike counts of putative excitatory cells alone, nearly a 50% decrease).

Another potential explanation as to why some reports have failed to observe an enhanced response to the best familiar stimulus concerns the size of the stimulus sets. In the studies where the best familiar stimulus failed to elicit a stronger response, the familiar and novel sets each consisted of no more than 20 stimuli (Baker et al., 2002; Freedman et al., 2006; Op de Beeck et al., 2007). Conversely, each of the studies that have reported stronger familiar responses used stimulus sets with at least that many stimuli (Kobatake et al., 1998; Logothetis et al., 1995; Miyashita, 1993; Sakai and Miyashita, 1994). With a small and/or relatively homogeneous stimulus set, it is plausible that the lack of enhanced familiar responses is a consequence of exploring only the low response regions of the high-dimensional image space in which ITC responses lie, regions in which responses to familiar and novel stimuli are similar. Consistent with this proposal, when we randomly selected smaller subsets of familiar and novel responses (from our own dataset), and thus were more likely to exclude the response from the best familiar stimulus, we observed that the population level difference in maximum firing rates decreased (Figure S5). Further supporting the suggestion that the differences in maximum firing rate depend on finding the appropriate stimuli, two of the studies that failed to observe an enhanced familiar response reported the firing rates to the best familiar stimuli to be <25 Hz (Baker et al., 2002; Freedman et al., 2006). Because this value presumably included both excitatory and inhibitory neurons, it is likely to be even lower for just excitatory neurons. In the present study, we recorded from putative excitatory cells that had an average maximum response to the familiar set of 52.69 Hz (taken over the epoch 75 – 200 ms) and a peak maximum response, depending on the monkey, of around 70–110 Hz.

What could the increased response magnitude of the putative excitatory cells to the best familiar stimulus reflect in terms of the underlying neuronal circuitry? Because the experience-dependent enhancement was present at the time of visual response onset, the most parsimonious explanation is to posit a potentiated excitatory input from areas upstream of ITC, such as V4 (Seltzer and Pandya, 1978). This hypothesis is consistent with the present conception of ventral visual stream function. In particular, the ventral visual stream is thought to elaborate on the shape, color and texture attributes of visual input (Anzai et al., 2007; Brincat and Connor, 2004, 2006; Gallant et al., 1993; Hubel and Wiesel, 1959; Kobatake and Tanaka, 1994; Logothetis et al., 1996; Pasupathy and Connor, 1999; Rust and Dicarlo, 2010; Tanaka, 1996; Tanaka et al., 1991; Yamane et al., 2008). The gradual increase in optimal stimulus complexity as one traverses the ventral pathway has been interpreted as an increase in sensitivity for particular combinations of local features. This sort of image transformation makes explicit, and thus easier to read out, the higher order correlations present in the visual input. This process is thought to culminate in ITC. Because the local feature responses of neurons at early stages in the visual system can be recombined in a virtually infinite number of ways, there is no need for their experience-dependent modification beyond that observed in the critical period. Indeed, modification of these building blocks of stimulus encoding could dramatically disrupt responses of downstream neurons dependent on a stable foundation of local responses. The particular combinations of local features that the organism learns to recognize, however, will depend on its recent perceptual history. We propose that one of ITC's computational roles is to learn and encode with a higher maximum response those conjunctions that occur frequently and reliably. To do so, neurons in ITC strengthen the influence of those synaptic inputs that have a tendency to frequently and reliably excite them. Such learning can be implemented through classical Hebbian plasticity mechanisms, and in particular, NMDA receptor (NMDAR)-mediated long term potentiation (LTP) (Feldman, 2009). Supporting this hypothesis, stimulus-specific, NMDAR-mediated response potentiation has previously been reported in mouse visual cortex (Frenkel et al., 2006). It will be important for future studies to determine whether the neuronal changes to the stimuli we used can or cannot be detected earlier in the visual system (Rainer et al., 2004; Yang and Maunsell, 2004). Under our proposed scheme, such changes should be minimal.

We showed that a direct result of experience-dependent maximum response increases in putative excitatory cells is increased sparseness (selectivity) for stimuli within the familiar set. This is consistent with earlier work (Kobatake et al., 1998; Logothetis et al., 1995) but stands in contrast to recent data showing that selectivity increases in ITC are a consequence only of decreased responses to stimuli at the lower end of the firing rate distribution (Baker et al., 2002; Freedman et al., 2006). While we were able to replicate the decrease in average stimulus-evoked responses, this effect’s presence (Freedman et al., 2006) as well as its relationship to increased selectivity, held only in the late phase of the visual response. The late emergence of this suppression suggests that experience not only strengthens feed-forward input, but also likely prunes and/or weakens synaptic connections within ITC (Feldman, 2009). Taken together, these results argue that experience steers putative excitatory neurons to contribute to the encoding of only their most effective stimuli at the expense of less effective stimuli. Supporting this assertion, we showed that there is an inverse relationship between the selectivity of neurons and their ability to discriminate arbitrarily chosen pairs of stimuli. We speculate that a smaller population of projection neurons each firing many, very informative spikes may be better at driving downstream neurons and thus have more impact on perceptually-guided behavior compared to a large population of neurons each firing a few, less informative spikes.

Putative inhibitory cells

Putative inhibitory cells also showed average response decreases to familiar stimuli. The magnitude of this effect, however, was much larger in the inhibitory population. This observation adds to recent reports showing that behavioral factors can affect putative inhibitory cells to a much greater degree (Mitchell et al., 2007; Niell and Stryker, 2010). One intriguing possible role for increased inhibitory output is that it serves to detect novelty and initiate the cascade of events that underlie the subsequent plasticity. Research over the past decade has revealed that critical period plasticity within primary visual cortex is closely linked with the maturation of GABA-ergic transmission, with anecdotal reports implicating, in particular, inhibition mediated by parvalbumin-positive interneurons (Hensch, 2005). Indeed, a recent report indicates that interneurons of this class broaden their orientation tuning in parallel with the onset of the critical period (Kuhlman et al., 2011). We thus propose that the increased activity of our putative inhibitory cells is the neurochemical trigger for the robust selectivity changes within the putative excitatory population. If this hypothesis is true, the challenge will be to elucidate what allows the inhibitory cells within ITC to mediate plasticity into adulthood. That is, even though in primary visual cortex critical period plasticity can be prematurely triggered by enhancing GABA-ergic transmission, the plastic window still has a finite duration, and importantly, once it ends, it cannot be reinitiated (Fagiolini and Hensch, 2000). Further work suggests that there is a developmental trajectory intrinsic to inhibitory cells, which allows them to control the temporal specificity of plasticity (Southwell et al., 2010). Whether this maturational program is in some important ways different in inhibitory cells further along the visual hierarchy, where plasticity can extend into adulthood, is a question for future research.

Our observation that putative inhibitory cells were much less selective than putative excitatory cells, regardless of stimulus set and time epoch analyzed, is consistent with a previous result (Zoccolan et al., 2007). In areas where columnar structure with regard to some feature dimension is well defined (e.g., orientation columns in cat and primate primary visual cortex), inhibitory neurons have narrow tuning. In areas lacking such an organization (e.g., primary visual cortex of mice and rabbits), inhibitory neurons have broader tuning. Thus, an emerging view is that the amount of selectivity within the inhibitory population reflects the degree to which excitatory neurons with similar receptive field properties are in spatial proximity to one another (Bock et al., 2011; Cardin et al., 2007; Kerlin et al., 2010; Liu et al., 2009; Sohya et al., 2007). To the extent that this hypothesis is true, our results indicate that columnar organization within ITC, with respect to the stimulus set employed, is moderate at best (Fujita et al., 1992; Tsunoda et al., 2001). Otherwise, we should have seen selectivity values within the putative inhibitory population mirror the selectivity values within the putative excitatory population. Importantly, we can extend this line of reasoning and propose that inhibitory activity serves as a proxy for the amount of surrounding excitatory activity. Viewed in this light, the massive increase in the average response of our putative inhibitory population to the novel stimuli further speaks to the robust effects that experience exerts on neuronal circuitry in ITC. In other words, the increased inhibitory activity is consistent with the hypothesis that novel compared to familiar stimuli activate a much larger number of excitatory cells and/or drive them, on average, to fire many more spikes. It is worth noting that perhaps the reason why putative inhibitory cells are better at detecting the novelty of stimuli is because they “listen” to the summed excitatory output of a fairly large collection of surrounding neurons. In this manner, the massive increase in inhibitory output would serve to not only signal novelty but also to maintain an appropriate level of excitatory to inhibitory balance. In fact, maintenance of this balance could be crucial to the normal operation of this sensory circuit while it undergoes robust remodeling. Alternatively, another non-mutually exclusive hypothesis is that this balance is important for putting the brakes on too much plasticity occurring too rapidly. Answers to these questions await further experimental exploration.

Methods

All experimental procedures were in accordance with the guidelines published in the National Institutes of Health Guide for the Care and Use of Laboratory Animals and were approved by the Brown University Institutional Animal Care and Use Committee.

Two male adult macaque monkeys were used in this study. Standard operant conditioning techniques were used to train the subjects to fixate and to press buttons for a small liquid reward. Eye movements were recorded using the EyeLink II video tracking system (SR Research, Osgoode, Ontario, Canada) running at 500 Hz. When the monkeys were ready for recordings, we implanted custom chambers that allowed for a dorsal access to ITC (Horsley-Clark coordinates, +15 anterior, +20 lateral). Based on reconstructed electrode trajectories, we believe most of our recordings took place from the lateral convexity of ITC, ventral to the lower bank of the superior temporal sulcus (STS) and lateral to the perirhinal cortex (Figure S2). Recordings were obtained with fine tungsten microelectrodes (Alpha Omega Engineering, Alpharetta, GA or Frederic Haer Company, Bowdoinham, ME). Single units were isolated online using a threshold and dual amplitude windows while analog signals were streamed to disk for offline analysis.

All stimuli used were drawn from Hemera Photo-Objects: Vols. 1, 2 and 3 (Hemera Technologies). Both monkeys were familiarized with the same set of 125 stimuli (Figure S1A). During the familiarization phase, the monkeys saw the images in either a passive fixation task or in a delayed match-to-sample task. When the familiarization phase was completed, we began the recordings. As the goal of this experiment was to compare neuronal responses to familiar and novel stimuli, for every recording session, we selected a new set of 125 never before seen stimuli. Although the selection process was random, we used the scale invariant feature transform and the dot product of normalized color histograms to eliminate from this novel set stimuli which looked either too similar to the familiar ones or to one another (see Supplemental Experimental Procedures).

We attempted to record from every well-isolated and visually responsive unit in ITC. To avoid a neuronal selection bias, the vast majority of visually responsive units (n = 40/50, 80% for Monkey D; n = 35/38, 92% for Monkey I) were found and isolated with an independent set of 50 initially novel stimuli that gradually became familiar as the recording sessions accumulated. Thus, the results presented here are not a consequence of selecting units that we knew ahead of time would be responsive to familiar items. All neurons reported were held for at least 5 repetitions of each unique stimulus, but most were held for 10 (n = 46/50, 92% for Monkey D; n = 35/38, 92% for Monkey I).

We divided the sample of neurons into two classes based on the widths (trough-to-peak durations) of their extracellularly-recorded spike waveforms. Clustering was performed with a k-means algorithm. We labeled the broad-spiking class as putative excitatory and the narrow-spiking as putative inhibitory.

Although we recorded the neuronal activity in a rapid serial visual presentation paradigm to allow each one of the large number of unique stimuli to be presented many times while simultaneously maintaining single unit isolation, the stimulus presentation durations (200 ms) and interstimulus durations (50 ms) were long enough to allow for a separate analysis of the early and late components of the neuronal response. The early phase was defined as the epoch 75 – 200 ms and the late phase was defined as the epoch 200 – 325 ms, both relative to stimulus onset. The main firing rate metrics used throughout this study were the maximum response and the average response. The maximum response was defined as the maximum across the mean firing rates to the 125 stimuli in either the familiar or novel set. The average response was defined as the average over the mean firing rates.

To determine, for a single cell, whether the maximum response across the familiar set was significantly different from the maximum response across the novel set, we used the Mann-Whitney U-test (histograms in Figures 3C and 3E). To compare statistically the average stimulus-evoked response across the 125 familiar stimuli to that across the 125 novel stimuli, we used a t-test (histograms in Figures 4C and 4D). To assess whether population-averaged data were different from a null hypothesis, we applied the appropriate (paired or unpaired) t-tests, always two-tailed. As a measure of selectivity, we used the sparseness metric (Olshausen and Field, 2004; Rolls and Tovee, 1995; Vinje and Gallant, 2000; Zoccolan et al., 2007). This metric takes the form $\mathrm{S}=\frac{1-A}{1-1/n}$, where $A=\frac{{\left({\displaystyle \sum _i^n\frac{r_i}{n}}\right)}^2}{{\displaystyle \sum _i^n\left(\frac{r_i^2}{n}\right)}}$, n is the number of stimuli and r_i are the mean firing rates to a set of stimuli. S takes values between 0 and 1. We evaluated the significance of sparseness differences between the familiar and novel sets with a randomization test (histograms in Figures 5C and 5D). We also used randomization test (corrected for multiple comparisons) to determine the time points at which the sliding window firing rates from two conditions, averaged across the population of neurons, were different from one another (see Supplemental Experimental Procedures for more details on the randomization tests). To establish how well a single neuron's spike counts could discriminate between any two randomly chosen stimuli within either the familiar or novel sets, we used the area under the ROC curve (AUC), which measures the discriminability of two spike count distributions (Green and Swets, 1966). In particular, we computed all pairwise AUC values in the set of 125 familiar or 125 novel stimuli, reflected about .5 values below .5 (e.g., .35 became .65), and took their average (Figure 7).