Neuronal representations of stimulus associations develop in the temporal lobe during learning

Abstract

Visual stimuli that are frequently seen together become associated in long-term memory, such that the sight of one stimulus readily brings to mind the thought or image of the other. It has been hypothesized that acquisition of such long-term associative memories proceeds via the strengthening of connections between neurons representing the associated stimuli, such that a neuron initially responding only to one stimulus of an associated pair eventually comes to respond to both. Consistent with this hypothesis, studies have demonstrated that individual neurons in the primate inferior temporal cortex tend to exhibit similar responses to pairs of visual stimuli that have become behaviorally associated. In the present study, we investigated the role of these areas in the formation of conditional visual associations by monitoring the responses of individual neurons during the learning of new stimulus pairs. We found that many neurons in both area TE and perirhinal cortex came to elicit more similar neuronal responses to paired stimuli as learning proceeded. Moreover, these neuronal response changes were learning-dependent and proceeded with an average time course that paralleled learning. This experience-dependent plasticity of sensory representations in the cerebral cortex may underlie the learning of associations between objects.

A number of studies have shown that individual neurons in various brain regions will respond similarly to sensory stimuli that have become associated with one another or with a reward (1–10). These neuronal response similarities are believed to emerge as new associations are formed and might be involved in the long-term storage or retrieval of associations. To evaluate this plasticity hypothesis more directly, other studies have followed the stimulus preferences of individual cortical neurons as monkeys learned to associate stimuli with rewards or motor responses (10–17). Neuronal responses to specific stimuli were indeed observed to change during associative learning, consistent with the idea that these neurons are involved in linking sensory inputs with expected outcomes and/or actions.

Unlike stimulus-reward and stimulus-response associations, which can be learned in minutes, monkeys generally take many days to learn conditional associations between pairs of stimuli (18–21). Thus, until now, the neuronal substrates of conditional stimulus-stimulus associations have been inferred only from patterns of neuronal selectivity recorded after learning has taken place (1, 2, 4–8, 22–27). In particular, Miyashita and colleagues (1, 2, 4, 5) found neurons in the anterior regions of inferior temporal (IT) cortex that responded similarly to members of visual stimulus pairs that had become associated through extensive prior training. We hypothesized that if these similar responses to paired stimuli (and the correspondingly dissimilar responses to unpaired stimuli) were the neuronal basis of the learned conditional visual associations, then such similar responses should arise during the successful learning of new stimulus pairings.

To determine when these expected response changes develop and their precise relation to learning, we adopted an experimental approach that enabled us to follow the stimulus preferences of individual IT neurons as monkeys learned stimulus-stimulus associations. Two features of this approach permitted sufficiently rapid learning: (i) monkeys learned many such pair associations before electrophysiological recording, so that they were familiar with the task requirements and adept at performance, and (ii) monkeys were required to learn only two stimulus pairs during each electrophysiological recording session. Using this approach, we found that the stimulus-dependent responses of many neurons in areas TE and perirhinal cortex changed significantly over the course of single recording sessions. Furthermore, in support of our hypothesis, we obtained direct evidence that responses to paired stimuli become progressively more similar to one another, relative to responses elicited by unpaired stimuli. These neuronal developments depended on and had a similar time course to the monkeys' learning of the stimulus pairs, suggesting that this plasticity is in fact the basis for the learning of conditional associations between visual stimuli.

Methods

Subjects and Surgery.

Two juvenile male monkeys (Macaca mulatta), weighing 8.5–9.5 kg, were used in accordance with procedures approved by The Salk Institute Animal Care and Use Committee and in compliance with U.S. Department of Agriculture regulations and National Institutes of Health guidelines for the care and use of animals. After training, a recording chamber was implanted on each monkey's skull by using standard surgical techniques (29). Magnetic resonance images of each monkey's brain were used to guide placement of the chambers dorsal to the anterior middle temporal sulcus. Monkey M's chamber was over the right hemisphere at anterior-posterior (AP) 16.5 and medial-lateral (ML) 19.0 mm and monkey N′s chamber was over the left hemisphere at (AP) 15.0 and (ML) 21.0 mm.

Behavioral Task.

The monkeys were trained to perform a visual-visual paired-associate task in which the stimuli changed daily (see Fig. 1). Four picture stimuli (A, B, C, and D), subtending 3.4° on their longest side, were introduced each day. The stimuli were grouped into two pairs (AB and CD), which could be determined only by trial and error. Trials began with presentation of a small (0.4°) fixation target at the center of a neutral gray screen. The monkeys fixated this target during presentation of one of the four stimuli (the cue) followed by the simultaneous presentation of two other stimuli, one from the same pair as the cue (the paired associate) and one from the other pair (the distractor). Once the fixation target disappeared, monkeys were rewarded for looking at the paired associate rather than the distractor.

Figure 1.

Visual-visual paired-associate learning task. (a) For each learning session, four physically dissimilar pictures were chosen (A, B, C, D) and randomly grouped into two pairs. Representative pictures are shown. The correct pairings (AB, CD) are indicated by horizontal arrows. (b) Sequence of events on correct trial. The monkeys were required to fixate a small central target, which was first presented alone (500 ms), then superimposed on one of the four possible cue stimuli (1,000 ms), and then in the presence of two peripheral stimuli (500 ms), one from each pair. After the fixation target was extinguished, the monkey selected one of these stimuli by looking at it (indicated here by the arrow). The monkey was given a juice reward for selecting the stimulus from the same pair as the cue (the paired associate) but not for selecting the corresponding member of the other pair (the distractor). Correct responses also resulted in a secondary reinforcing beep and a 650-ms presentation of both members of the correct pair. The location of the paired associate and distractor varied randomly from trial to trial.

The random selection of the cue stimulus on each trial completely determined the identity of the paired associate and distractor (either stimuli A and C or B and D). When presented during a trial, the paired-associate and distractor were arranged in a column at an eccentricity of 5° in the visual hemifield contralateral to the neuronal recordings. Each stimulus occupied the upper or lower position at random on each trial. Thus all four stimuli and both peripheral locations developed an equal association with reward over trials. Furthermore, the temporal association arising from the sequential presentation of each cue and its paired associate was balanced by the sequential presentation of that cue and its distractor.

Erroneous selection of the distractor caused a buzzer to sound and the presentation of just the paired associate for 150–650 ms. A correction trial with the same cue was given after a brief timeout. The volume of juice reward on correct trials (not including correction trials) was computed according to a formula and increased if recent performance was good. This reinforcement schedule fostered learning and discouraged selection biases. Visual feedback of the correct pair was presented at the end of correct trials because it has been shown to speed paired-associate learning (18).

Eye position was monitored by using the scleral search coil technique (30, 31). Trials where eye position deviated from the fixation target beyond an invisible square window (1.7° on a side for monkey M and 1.5° for monkey N) were aborted and not analyzed.

Electrophysiology.

Once animals became familiar with the paired-associate task and demonstrated evidence of learning new pairs in one day, we began electrophysiological recording using standard techniques (32). Each day, a sharpened tungsten electrode (model VEWLGESM3N1G, 3.0 MΩ, FHC, Bowdoinham, Maine) was lowered through a 23-gauge stainless steel guide tube into the anterior ventral portion of area TE or perirhinal cortex. Electrode placement was guided by magnetic resonance images of each monkey's brain and the distribution in depth of spontaneous neuronal activity and was confirmed with x-ray images (33). Electrode depth was controlled manually (guide tube drive, FHC) and then hydraulically (model 607-W, Kopf Instruments, Tujunga, CA).

cortex software (http://www.cortex.salk.edu/) controlled stimulus presentation and data collection. At each recording site, one or two neurons were isolated by using the Multi Spike Detector system (Alpha Omega Engineering, Nazareth, Israel) and screened for differential responses during fixation of 8–16 novel pictures. Four of these stimuli, including the stimuli that elicited the largest and smallest neuronal response, were randomly grouped into two pairs and the paired-associate task was begun.

Analysis.

Animals had to select between two stimuli (the paired associate and distractor) on each trial. The chance probability of selecting correctly was thus 0.5. A one-tailed binomial test was used to determine whether the animal's probability of responding correctly over a string of trials (e.g., a quartile) significantly exceeded chance probability.

We determined whether recorded neurons were in perirhinal cortex or area TE based on electrode placement and whether or not multimodal responses were encountered on a given penetration (34). The baseline neuronal firing rate was determined from −450 to +50 ms, relative to cue onset, and the firing rate in response to the cue stimulus was determined from 50 to 1,050 ms. Neurons were considered visually responsive if, for all presentations during the paired associates task, the firing rate response to any of the cue stimuli differed significantly (paired t test, P ≤ 0.01) from the baseline firing rate during the preceding period of fixation. All other statistical tests were evaluated at the α = 0.05 level and, unless indicated otherwise, were two-tailed.

We developed the Pair Similarity Index (PSI) to compare the similarity of neuronal responses to cue stimuli from the same pair with that of cue stimuli from different pairs:

where r_A, r_B, r_C, and r_D are firing rates in response to the corresponding cue stimuli and max denotes the maximum. Paired stimuli (i.e., cue + paired associate: AB, CD) were not only conditionally associated by the reinforcement regime but were also temporally associated because they were presented sequentially on every trial. We therefore contrasted the similarity of neuronal responses to paired stimuli with that of those unpaired stimuli that were also presented sequentially (i.e., cue + distractor: AD, BC) to remove similarities that develop because of temporal associations alone (3). The PSI thus isolated the influence of conditional associations on response similarity.

The PSI was positive when responses to the paired associates (AB, CD) were more similar than responses to sequentially presented unpaired stimuli (AD, BC), and negative when the converse was true. The PSI was zero when either (i) the similarity of responses to paired associates (AB, CD) was equal to that for sequentially presented unpaired stimuli (AD, BC), or (ii) when responses to all of the stimuli were identical. The value of PSI was computed for each trial by using interpolated firing rates (see Fig. 3). The net change in the PSI over each session, ΔPSI, was computed as the difference in the mean of PSI over the first and last 36 trials of each session. For all sessions, independent trials were used to compute PSI in these two epochs. Increases in PSI over a session (i.e., positive ΔPSI) serve to confirm our hypothesis that similar responses to paired stimuli will develop during paired-associate learning.

Figure 3.

Example of evolution of stimulus selectivity during paired-associate learning. (a) Responses of a perirhinal neuron to the four cue stimuli as a function of the number of trials performed. Each stimulus appeared as cue on roughly one in four trials. The trial-by-trial mean firing rate for each cue was calculated by averaging responses for the eight nearest correct or incorrect trials when that cue was presented. The SEM for these eight rates, averaged across stimuli and trial number, is shown on the right. Gray and white panels indicate session quartiles. Baseline activity (fixation target only) was 2.2, 3.5, 4.3, and 4.5 sp/s by quartile. The corresponding task performance is shown by arrows in Fig. 2. (b) PSI plotted as a function of trials of the task completed. PSI for each trial was computed by using the averaged firing rates shown in a. The error bar shows the error in PSI (averaged across trials) as computed from the SEM for the neuronal responses to the cue stimuli. As learning proceeded, responses to cues A and B became more similar whereas responses to unpaired stimuli diverged, thereby causing the PSI to increase. Between the beginning and end of the session, the net change in PSI (ΔPSI) was 67.8.

Results

We recorded neuronal activity in the anterior IT cortex of two monkeys during the learning of visual-visual associations. The monkeys performed a paired-associate task (Fig. 1) in which one stimulus from a pair (the cue) was presented and the monkey had to select the other stimulus from that pair (the paired associate) to receive a juice reward. Four new stimuli were used each day and the two correct pairings between them had to be ascertained by trial and error. We monitored the monkeys' performance of the task with new stimuli over 54 (24 for monkey M; 30 for monkey N) sessions while recording from one or two visually responsive neurons. During these sessions, we recorded the activity of 51 visually responsive neurons in ventral area TE and 32 in perirhinal cortex for 1–4 h.

Behavioral Performance.

The monkeys completed between 104 and 1,007 trials (mean 476 ± 29, n = 54 sessions) of the paired-associate task during the neuronal recordings. We divided each session into four equal quartiles to normalize for their varying duration. Fig. 2 shows the quarterly task performance of each monkey during the learning of new pairs. Performance on standard trials and correction trials was not significantly different (paired t test) in any quartile for either animal (or for the two combined) so these two trial types were treated equivalently. Across sessions, overall performance of trials in each successive quartile was 50%, 54%, 55%, and 56% correct. Performance was initially at chance level, as was expected for novel stimulus associations, and improved modestly in successive quartiles as trial-and-error learning proceeded.

Figure 2.

Quarterly performance of the paired-associate task. The percentage of correct responses is shown for each quartile of sessions during which new stimuli were introduced and neuronal activity was recorded. Circles depict the performance of monkey M over 24 such sessions and squares that of monkey N over 30 sessions. Gray and white panels indicate session quartiles. Symbols are displaced along the horizontal axis for clarity. Darker symbols represent quartiles where the monkey's performance was significantly better (one-tailed binomial test) than chance level (dashed horizontal line). Performance during the first, second, third, and fourth quartiles was significantly better than chance for 7, 11, 15, and 20 of the sessions, respectively. The solid line shows the overall percentage of correct responses of both monkeys on all trials in a given quartile, and the error bars show the 95% confidence intervals on this performance based on a binomial distribution. Each monkey's average performance increased modestly but significantly over time. Arrows mark the quarterly performance during the session depicted in Fig. 3.

A two-way ANOVA revealed that the increase in average performance across quartiles was highly significant (P < 10⁻⁶). There was neither a significant difference in task performance between the two monkeys (P > 0.2) nor a significant interaction between quartile and subject (P > 0.1). Post hoc testing revealed that, despite the substantial variability between sessions, the average performance of each monkey was significantly better than chance (t test, P < 0.02) in each quartile after the first. Performance over all trials in a session also varied from day to day (range 46% to 64% correct) but was on average significantly better than chance (t test, P < 10⁻⁷). Thus, although the improvements were limited, both monkeys showed significant and comparable evidence of learning across sessions.

Responses of an IT Neuron Recorded During Learning.

One neuron's changing responses to four stimuli during a paired-associate learning session are shown in Fig. 3a. Initially this neuron had a weak preference for cue B. Over the session, mean firing rate responses to cues A and B increased whereas responses to cues C and D remained relatively low. By the end of the session, stimuli in the same pair had developed similar responses (pair AB high and pair CD low) whereas those in different pairs had developed dissimilar responses. We used the PSI to quantify this convergence of neuronal responses to paired stimuli and divergence of responses to unpaired stimuli with training. The firing rates in Fig. 3a were used to compute the time course of the PSI shown in Fig. 3b. Consistent with our hypothesis, the PSI for this neuron increased, reflecting the convergence of responses to paired stimuli A and B and the divergence of responses to stimuli and their corresponding distractors (i.e., A and D, B and C).

Neuronal Population Effects.

The net change in the PSI between the beginning and end of each neuronal recording (ΔPSI) was significantly positive across the neuronal population (t test, P < 0.05). Fig. 4 shows a population histogram of these changes. Twice as many neurons had increases (n = 52) in the PSI than had decreases (n = 26), with five neurons showing no change. On average, ΔPSI was 8.1 ± 3.6 (n = 83 neurons). Thus in anterior IT cortex, responses to paired (unpaired) stimuli became significantly more (less) similar over the several-hour recording sessions.

Figure 4.

Population histogram of changes in the PSI. Differences between the PSI at the end and beginning of each session (ΔPSI) of various magnitudes are plotted on the horizontal axis. The vertical axis indicates the number of neurons for which this change was an increase or a decrease. ΔPSI was positive for two-thirds of both area TE and perirhinal neurons in which PSI changed. Across the population, there was a significant increase in PSI within the daily sessions (t test, P < 0.05), indicating that neurons developed similar responses to conditionally associated stimuli as those associations were being learned.

The ratio of neurons with increases to decreases in PSI was nearly the same in ventral area TE (31 vs. 16) and perirhinal cortex (21 vs. 10). The average value of ΔPSI for ventral area TE recordings was 7.5 ± 4.5 (n = 51 neurons), which was not significantly different (t test, P > 0.5) from the average for perirhinal recordings of 9.0 ± 6.4 (n = 32 neurons). There was likewise no significant difference in ΔPSI between monkey M and N (t test, P > 0.5, n = 36 and 47 neurons, respectively). Thus similarities in the neuronal representation of conditionally associated stimuli developed in both area TE and perirhinal cortex and across animals.

Behavioral Dependence of Changes in Neuronal Selectivity.

Were these changes in neuronal tuning a consequence of learning the conditional associations or merely a result of exposure to the paired-associate task? To address this question, we considered how changes in PSI depended on behavior. To control for possible influences of task exposure (e.g., the visual feedback of the correct stimulus pairs during delivery of reward) on neuronal responses, we considered all trials from the start of a session through the completion of a fixed number of correct trials. Sessions with at least 300 correct trials were grouped based on whether performance through the 300th correct response was or was not significantly better than chance (one-tailed binomial test, significant if 300 correct trials were completed in ≤560 trials). Fig. 5a shows the percentage of correct responses as a function of trial number averaged separately for the seven sessions during which performance was not significant (gray curve) and the 13 sessions during which it was (black curve). Performance was initially at chance for each group. Through the 300th correct trial, performance averaged 50.7% when learning was not significant and 56.2% correct when it was significant.

Figure 5.

Time course and learning dependence of neuronal changes. (a) Learning curves for both monkeys averaged separately for cases where performance up to the 300th correct trial was (black curve, n = 13 sessions) and was not (gray curve, n = 7 sessions) significantly better than chance (one-tailed binomial test). Performance was computed over the 55 trials surrounding each data point and averaged across sessions except for the data at trial number one, which represents performance over the first 25 trials. (b) Time course of the changes in the PSI recorded when performance was (black curve, n = 18 neurons) and was not (gray curve, n = 11 neurons) significantly better than chance. Changes from the value of PSI on trial one (PSI_initial) were averaged across neurons and over the 55 trials surrounding each data point except the first. There was a significant increase in the PSI when the monkeys' performance was significantly better than chance (P < 0.02) and these changes were significantly different from those when performance was no better than chance (P < 0.01).

Fig. 5b shows the changes in the PSI from its initial value averaged on the basis of whether or not the corresponding behavior was significantly better than chance. The initial value of PSI (on trial one) was not significantly different from zero (t test, P > 0.2) for either group of neurons nor did it differ between groups (t test, P > 0.1). When performance up to the 300th correct response was not statistically different from chance (gray curve, n = 11 neurons), PSI decreased from its initial value by an average of −13.7 ± 10.1 over the required trials. Thus mere exposure to the visual feedback of the correct pairs and to the task in general did not cause PSI to increase. In fact, poor learning was found to be associated with responses to paired stimuli becoming less similar relative to responses to unpaired stimuli. By contrast, when performance was significantly better than chance (black curve, n = 18 neurons), PSI increased from its initial value by an amount averaging 15.7 ± 6.0 over the required trials. This value was significantly greater than zero (t test, P ≤ 0.02), indicating that responses to paired stimuli became more similar relative to responses to unpaired stimuli when there was appreciable learning. The increase in PSI, furthermore, proceeded with a dynamic similar to that of the learning curve.

Changes in PSI from its initial value were averaged over all trials up to the 300th correct response for each neuron and tested by using ANOVA with behavioral significance and brain region (area TE vs. perirhinal cortex) as factors. When performance through the 300 correct trials was significantly better than chance, fewer total trials were performed, and yet the change in PSI was significantly greater (P < 0.02) than when performance was not significant. Hence, PSI tended to increase only when the monkeys demonstrated significant associative learning. There was no significant difference in the change in PSI between neurons recorded in area TE and perirhinal cortex, nor any significant interaction between brain region sampled and the dependence on performance (P > 0.2). Thus throughout anterior IT cortex, neuronal responses to conditionally paired stimuli developed similarities in a learning-dependent manner and with a time course that paralleled learning.

Discussion

In corroboration of previous studies, we find that individual neurons in area TE and perirhinal cortex respond similarly to associated visual stimuli (1–5). Furthermore, we have provided direct evidence that these similar responses develop progressively through stimulus-selective rate changes over the course of several hours (e.g., Fig. 3). This neuronal plasticity occurred on the same time scale as the early stages of associative learning and depended on learning having occurred (see Fig. 5). Consistent with these findings, lesions of perirhinal cortex have been shown to impair the learning of conditional associations between pairs of visual stimuli (18, 20). The existing evidence thus supports a model in which individual neurons in anterior IT cortex mediate the acquisition of a behavioral connection between visual objects through the progressive convergence of firing rates elicited by the objects.

Other brain areas also have been implicated in conditional pair association. An interaction of the frontal and temporal cortices was shown to be important for learning and recall of conditional visual associations, as well as for activating their mnemonic representations in area TE neurons in the absence of bottom-up visual input (4, 19, 35, 36). Furthermore, in prefrontal cortex and various parts of the visual system, neurons have been found to respond to both halves of cross-modal stimulus pairs (6–8) or to exhibit delay activity that was anticipatory of an associated stimulus (i.e., that was prospective) (6, 7, 24, 27). Concomitant responses to paired visual stimuli in a single neuron, however, have not been found in large numbers outside of, and may therefore originate in, anterior IT cortex (26, 27, 37).

Neurons in anterior IT cortex have been shown to respond to temporally associated visual stimuli in much the same way as conditionally associated stimuli (1–5, 28). Erickson and Desimone (3) found that perirhinal neurons exhibited both prospective delay activity and similar visual responses to pairs of stimuli that were repeatedly presented in a fixed order. However, their findings differ from those presented here with regard to the time course and the learning dependence of changes in the neuronal representations of associated stimuli. In particular, Erickson and Desimone found that only the responses of neurons recorded on the days after introduction of new stimulus pairings reflected the stimulus associations, whereas we were able to follow the development of similar responses to a new stimulus pair in a single neuron over several hours. They also found that paired stimuli elicited similar responses whether or not the monkeys learned to predict the stimulus sequences. This learning independence suggests that the response similarities arose from simply viewing the stimulus sequence (3). Our results, on the other hand, indicate that the development of like visual responses to paired stimuli was tied to learning and not to passive exposure to the task. These discrepancies may be because of differences in the behavioral tasks used in the two studies or they could reflect differences in the encoding of temporal rather than conditional associations.

We have found stimulus-specific tuning changes in area TE and perirhinal neurons with dynamics and a behavioral dependence that suggest they may be the basis of associative learning. Although we have not documented differences between areas TE and perirhinal cortex, it seems likely that they each serve unique roles in learning and memory (e.g., ref. 38). Our findings demonstrate that the sensory tuning of cortical neurons can be quite plastic on the hour time scale in certain behavioral contexts. The mechanisms of this plasticity are a current subject of research (21, 39) and likely involve the modification of the strength of synaptic connections between neurons that represent the stimuli from the same pair and different pairs. These mechanisms and the roles that the hippocampus and other temporal lobe memory structures may play in effecting these changes are important targets for future studies.

Abbreviations

IT

inferior temporal

PSI

Pair Similarity Index