Learning of anticipatory responses in single neurons of the human medial temporal lobe

Abstract

Neuronal processes underlying the formation of new associations in the human brain are not yet well understood. Here, human participants, implanted with depth electrodes in the brain, learned arbitrary associations between images presented in an ordered, predictable sequence. During learning we recorded from medial temporal lobe (MTL) neurons that responded to at least one of the pictures in the sequence (the preferred stimulus). We report that as a result of learning, single MTL neurons show asymmetric shifts in activity and start firing earlier in the sequence in anticipation of their preferred stimulus. These effects appear relatively early in learning, after only 11 exposures to the stimulus sequence. The anticipatory neuronal responses emerge while the subjects became faster in reporting the next item in the sequence. These results demonstrate flexible representations that could support learning of new associations between stimuli in a sequence, in single neurons in the human MTL.

Introduction

Adapting to our environment involves learning associations between stimuli that frequently occur together. These learned associations may become so firmly established that experiencing one stimulus can vividly evoke the other. Creating associations between different, initially unrelated stimuli relies on the medial temporal lobe (MTL) ¹. Previous studies have shown that single neurons in the monkey MTL signal learned associations by changing their response properties ^2-7. For example, neuronal responses to pairs of associated pictures become more similar to each other as a result of learning ², and significant neuronal activity is also observed during the delay period between presentations of two associated stimuli ⁵. These learned responses are observed both when the animal has to learn a new rule, as well as when learning is incidental.

In humans as observed with functional magnetic resonance imaging (fMRI), rapid sequential associative learning has also been observed in the MTL For example, MTL structures respond more strongly to structured versus unstructured sequences of visual stimuli ⁸, and their activity depends on the statistical regularities of the sequence ^9-11. Fine-grained multi-voxel pattern analysis revealed that hippocampal and parahippocampal representations for associated objects become correlated to each other as a result of learning. Furthermore, while in most MTL regions the representations of associated objects become more similar during learning, in the CA3 region of the hippocampus sequence learning causes anticipatory shifts in activity ¹².

In this study we take advantage of the unique opportunity of recording from single neurons in the human MTL to study sequence learning at the single neuron level. Subjects learned associations between visual stimuli presented in an ordered, predictable sequence. We report that sequence learning causes an increase in neuronal firing rates in anticipation of the neuron’s preferred stimulus (i.e., a stimulus that would elicit a selective neuronal response when presented outside the sequence), within 11 trials of learning. These anticipatory responses could play an important role in predicting future events based on what has been recently learned.

Results

Learning procedure and behavior

Eight subjects learned arbitrary associations between visual stimuli in 27 short associative learning (AL) sessions that lasted 10-14 minutes each. Each AL session consisted of 60 repetitions of a sequence of 5-7 images presented in a pre-determined, predictable order, and subjects had to learn the order of the sequence. The images were displayed for 1500ms each, with an inter-stimulus interval (ISI) of 500ms (Figure 1A, B). We probed the learning process by including 20% of test-trials where subjects had to predict the next image (Figure 1C). Subjects learned the sequence rapidly, achieving greater than 90% performance on test trials within six sequence presentations, while reaction times continued to decrease over a longer period (Figure 1D, E). Each image sequence contained at least one “preferred” image that drove one of our selective neurons (determined in independent screening sessions; see methods). We ensured that the image immediately preceding the preferred image was always non-preferred for the recorded neuron.

Figure 1. Learning Task and Behavior.

A) During the AL sessions we presented a sequence of 5-7 images in a fixed order. The sequence contained one preferred image for a selective neuron identified in prior screening sessions. The image preceding the preferred image was always non-preferred for the recorded neuron. Each image was presented for 1.5s followed by an ISI of 500ms. The sequence was repeated 60 times. B) Each image was presented at the center of the screen while 3 placeholders (empty grey squares) were presented on either side. At the end of the image presentation period a placeholder replaced the central image and the placeholders moved in the clockwise direction for the duration of the ISI. At the end of the ISI the central placeholder was replaced by the next image in the sequence. C) 20% of trials were test trials on which patients (N=8) saw two choice images and had to report which of the two would be the next image in the sequence. D) Average accuracy and E) average reaction times, shown on a log scale (to increase visibility), on test trials averaged across all subjects and sessions, as a function of image sequence presentation number (binned by groups of two sequence presentations). Error bars represent s.e.m. Accuracies reached their maximum by ~8 sequence presentations. Reaction times decreased more progressively with sequence presentation number.

Electrophysiology

We recorded from 635 neurons in eight patients implanted with depth electrodes ¹³. 56 of these neurons were selective to one of the images presented in the screening sessions. 42 of the selective neurons were in the hippocampus and parahippocampal cortex. These neurons are hereafter referred to as MTL neurons. 14 of the selective neurons were located in the posterior temporal lobe. The quality of the recordings and spike sorting is shown in Supplementary Figures 1 and 2.

Anticipatory responses in human MTL neurons

In our first analysis we examined whether learning causes MTL neurons to start responding earlier in the sequence in anticipation of their preferred stimuli. Such an anticipatory response might occur during the ISI preceding the preferred stimulus or even during the presentation of the preceding stimulus itself. Figure 2A,B illustrates two example MTL neurons that increased their firing rates in anticipation of the preferred stimulus during AL. The cell shown in Figure 2A was located in the hippocampus and showed significant activity for the preceding stimulus as a result of learning (right panel; paired t-test, t(30)=5.71; p<0.00001), whereas selectivity for this stimulus was absent during the screening session when the images were presented in a random order (left panel). The neuron in Figure 2B was recorded in the parahippocampal cortex and showed significant anticipatory activity during the ISI (i.e., the −500 to 0ms window) just prior to the presentation of the preferred stimulus (paired t-test, t(26)=5.19; p<0.0001).

Figure 2. Anticipatory shifts in spiking activity during AL.

A, The left panel illustrates a screening session when the stimuli were presented in a randomized order. The right panel illustrates an AL session when the subject learned the order of the stimuli. The darker blue, pink, and grey areas correspond to the presentations of the preceding, preferred and following stimuli. The lighter areas correspond to the ISIs. The dashed horizontal line is the average firing activity in the ISI window prior to the preceding stimulus. The solid black horizontal lines on the x-axes represent time periods during which activity was significantly greater than the mean ISI response (i.e., in the −500 to 0ms window) prior to the preceding stimulus, continuously for at least 100ms (paired t-test, p<0.05; see methods). The spikes fired on individual trials are shown in the blue raster plots. This unit increased activity during the presentation of the preceding stimulus during AL, although this stimulus did not elicit a significant response during the screening session (left panel). B, Another cell with increased activity during the ISI window between the preceding and preferred stimuli after learning. C, Average activity of MTL neurons (N=42). Anticipatory activity before the preferred stimulus occurred during AL, but not during the screening sessions. Shaded area is the standard error of the mean. The dashed horizontal line at 0 is the baseline-corrected average ISI activity prior to the preceding stimulus. The solid black horizontal lines on the x-axes represent time periods with activity significantly greater than in the ISI (−500 to 0ms) prior to the preceding stimulus, for at least 300ms. Note that the actual pictures used in the experiments are not shown here, as they were personal photographs from the patients.

When considering the latency of the response, the activity of individual MTL neurons during screening sessions increased significantly above baseline 258 ± 157ms (mean ± s.d. across individual cells) after the onset of the preferred stimulus. During AL, however, individual MTL neurons showed anticipatory activity that started 297 ± 740ms before the onset of the preferred stimulus (t(32)=−12.2; p<0.00001), or, equivalently, 1703 ± 740ms after the onset of the preceding stimulus. The anticipatory activity was particularly evident when we averaged responses across all MTL cells (Figure 2C; see Supplementary Figure 3 for normalized responses). During AL, the average firing rate increased significantly above baseline 1738ms before the onset of the preferred stimulus, i.e. even during the presentation of the preceding stimulus. These anticipatory effects specifically occurred during the presentation of the stimulus that preceded the neurons’ preferred stimulus and the subsequent ISI, but not for the non-preferred stimuli in the sequence (Supplementary Figure 4). Similar effects were not observed for neurons recorded in the posterior temporal cortex (Supplementary Figure 5). It can also be seen that the peak responses elicited by the preferred stimulus were weaker in the AL sessions than in the screening sessions. This decrease in visually driven activity is presumably caused by an adaptation effect caused by the frequent presentation of a few images during AL, as has also been observed in previous studies ^14,15 (Supplementary Figure 6). Thus, the increase of activity during the preceding stimulus and ISI was accompanied by a decrease in the visual response, as if a fraction of the omitted visually driven spikes now occurred at an earlier point in time.

Time course of anticipatory responses

To assess the statistical significance of the learning-related increased firing rates during the ISI (500ms window preceding stimulus onset) before the preferred stimulus, we compared it to the ISI activity before the preceding stimulus for the MTL cells (Figure 3A and Supplementary Figure 7). A 2-way random effects ANOVA of ISI-type (preferred vs. preceding) × session (AL vs. screening) revealed a significant interaction (F(1,123)=15.6, p<0.0005) because the ISI firing rate before the preferred stimulus was only higher in the AL sessions (post-hoc paired-test, t(41)=3.3; p<0.005). 69% of neurons (chi-square test against 50%, p<0.05) showed an effect in this direction, and over the group of all neurons the average increase in firing was 37 ± 13%.

Figure 3.

Time course of anticipatory responses in MTL cells (N=42)

A) Comparison of firing rate during the ISI periods (i.e., −500 to 0ms prior to stimulus onset) before the preferred and preceding stimuli during AL. 69% of MTL cells (white circles) showed higher firing rates in the ISI before preferred stimuli (paired t-test, p<0.05). The grey circles correspond to the remaining cells.

B) The strength of the predictive activity calculated with a running average of 15 trials, where on each trial n we considered the difference between the preferred and preceding ISIs based on a moving average of trials n−7 to n+7. The solid black horizontal lines indicate which trials showed a significant learning effect, as computed by a non-parametric bootstrap procedure over 100,000 iterations. The black arrow indicates the 11th trial, on which the difference between the preferred and preceding ISIs was first significant (p<0.05, non-parametric bootstrap procedure). See also Supplementary Figure 8.

MTL neurons thus signal new associations by increasing activity to previously non-preferred stimuli and during the ISI, in anticipation of their preferred stimuli. We next asked how these anticipatory changes evolve during learning. We defined the anticipatory learning effect as the difference between the mean activity during the ISI period (500 ms window before stimulus onset) before the preferred stimulus and the mean activity during the ISI period before the preceding stimulus. This anticipatory activity of single neurons on individual trials was variable (since it corresponds to an increase in baseline firing rates, in the absence of any stimulus presentation). Therefore, we used a 15-trial sliding window average in which, for each stimulus presentation n, we considered the difference between the preferred and preceding ISIs, based on a moving average of the n−7th to the n+7th stimulus presentations (Figure 3B). The anticipatory learning effect became significant starting at the 11^th stimulus presentation, (i.e., averaging across presentations 4 to 18; p< 0.05, non-parametric bootstrap procedure). The effect of performing the sliding window average for various numbers of trials is shown in Supplementary Figure 8.

Subjects’ ability to learn the image sequence was evaluated on test trials that were interspersed throughout the AL sessions. On test trials, subjects saw two pictures and had to choose the one that matched the next one of the sequence (Figure 1C). We chose this two-alternative force choice procedure as a sensitive measure of early learning, which might be expected to precede the ability of the subject to actively predict the next item of the sequence (e.g. ref ¹⁶). The accuracy on these test trials increased quickly and reached its maximum after ~8 sequence presentations (Figure 1D), confirming the sensitivity of the two-alternative forced choice test to early learning. Interestingly, the increase in accuracy on test trials preceded the predictive neuronal activity in the MTL, which occurred after ~11 sequence presentations. It seems likely that this lag in predictive activity in the MTL is related to a delay in the subjects’ ability to actively predict the next item of the sequence. Although we did not require subjects to freely recall the next item, we considered the possibility that recall might also improve performance in a forced choice test by decreasing reaction time. Indeed, the subjects’ reaction times in the test trials continued to decrease for several trials after the appearance of predictive activity in the MTL (Figure 1E). Furthermore, the change in behavioral reaction times on the test trials was significantly correlated with the time course of the learning effect at the neuronal level, as determined by a linear regression analysis of the neuronal learning effect (Figure 3B) versus the reaction times (r²=0.18, p<0.0009).

Even though the correlation analysis above suggests that the behavioral and neuronal learning time courses behave similarly, the anticipatory learning effect also appears to peak relatively early, around the 15th trial, before slowly decaying. This might reflect the transient involvement of MTL structures during the early stages of sequence learning, after which learning is consolidated in other brain areas ¹.

To investigate whether the short episode of learning caused longer-lasting changes in neuronal tuning, we investigated whether the enhanced response elicited by the preceding stimulus persisted when the subjects passively viewed the same stimuli presented in random order (Supplementary Figure 9). Passive viewing abolished the enhanced response to the stimulus that had preceded the preferred stimuli during sequence learning, implying that the short training epoch did not have an enduring influence on neuronal tuning outside the context of the sequence-learning task.

Discussion

We here found that when subjects learn an ordered sequence, neurons in their MTL change their activity. The neurons start showing anticipatory changes in firing activity after 11 exposures to the image sequence. This predictive activity may be related to the recall of the next stimulus in the sequence and it implies the formation of a new association between the preferred stimulus and the one preceding it. Interestingly, these predictive responses only occurred within the context of the sequence task but not when subjects saw the same images in a random sequence outside the sequence-learning task.

The predictive responses in the human MTL are reminiscent of anticipatory activity of place cells ¹⁷ in the rodent hippocampus. In a sequentially ordered spatial environment, place cells also start to fire progressively earlier within a few crossings of the track, thereby encoding the future position of the animal ^18-20. These place cell responses have been proposed to play a general role in encoding spatial and non-spatial events in temporally organized sequences ^21-23, thereby endowing the MTL with the ability to code for the temporal structure of longer sequences (e.g., the visual stimuli in our learning task, or spatial locations in a maze). According to this view, the anticipatory responses in the MTL could serve as the “glue” that links successive events together into longer episodes.

The anticipatory responses could arise from at least two mechanisms. First, the new association could cause activity to spread between the representations of the preceding and preferred stimuli (e.g., via pattern completion). Second, as a result of learning, the now associated stimuli could be encoded as a single chunk ²⁴, and learning could cause neurons to acquire a new selectivity for this combination of the preceding and preferred stimuli. The high temporal resolution of single cell recordings provides a unique opportunity to distinguish between these explanations by comparing the latency of the response. A genuine selectivity change predicts that the latency of the responses elicited by a combination of the stimuli would be similar to the typical latencies of MTL neurons, whereas activity-spreading mechanisms (e.g., pattern completion), where one stimulus activates the representation of the next, predicts a delayed activation by the preceding stimulus. For our MTL cells the mean latency during the screening sessions was 258 ± 157ms (mean ± s.d.), consistent with previous estimates of latencies of human MTL cells ²⁵. In contrast, during the AL sessions the mean latency of response to the preceding stimulus was significantly longer (1703 ± 740ms), in support of the hypothesis that new associations cause activity to spread from the representation of the preceding stimulus to the representation of the preferred stimulus. Our results do not exclude the possibility, however, that with prolonged training new chunks emerge and that they could be represented by pair-coding neurons.

Our results are consistent with previous neurophysiological evidence that the monkey MTL signals the formation of new associations between visual scenes and spatial locations ⁷, and the retrieval of well-learned associations between visual stimuli ²⁶. At the same time, the present results go beyond these previous studies in two important aspects: our results demonstrate associative learning in human MTL neurons during the learning of a sequence of non-spatial stimuli, and on a relatively short timescale (in contrast to the typically much longer learning process in the previous electrophysiological studies in animals). In addition to the anticipatory activity during the ISI before the preferred stimulus, we also observed changes in the response elicited by the preceding stimuli themselves. However, these altered responses occurred only in the context of sequence learning (Supplementary Figure 9), consistent with a similar observation for place cells ¹⁸, and in accordance with the hypothesis that hippocampal neurons associate stimuli with the contexts in which they were experienced ²⁷. At the same time, our results thereby differ from previous studies demonstrating changes in neuronal selectivity that persisted outside the task context, as has been demonstrated for pair-coding neurons in the inferotemporal cortex and surrounding regions in monkeys, ^4,26 and with fMRI in humans ¹². It seems likely that differences between brain regions, learning protocols and the type of learning explain these apparent discrepancies. For instance, one critical difference between our study and the studies in monkey pair-coding neurons is that pair-coding neurons are typically found in perirhinal cortex and area TE, whereas our learning effects were observed in the hippocampus and parahippocampal cortex. Furthermore, the monkey experiments used training protocols across many days, which presumably result in enduring changes in neuronal tuning, whereas our learning protocol lasted only a few minutes. There is also a notable difference in the type of learning between our study and the fMRI study mentioned above. We instructed the subjects to learn the sequence, whereas the previous fMRI study focused on incidental learning in a task that did not require them to learn the sequence. It is conceivable that explicit and implicit learning make different demands on MTL structures and result in different time courses of the memory traces. The precise conditions that determine the involvement of the different MTL structures and the time-course of learning remain an exciting topic for future research.

The anticipatory activity observed by us differs from the influence of novelty and familiarity on MTL activity reported in previous studies. In two of these previous studies ^28,29, patients saw a set of novel images in a first recording session, which were then repeated, along with a new set of images, in a second recording session. In the first session, patients were asked to memorize the images, and in the second session they reported if these images were familiar. Single neurons in the hippocampus and parahippocampal cortex changed their response when a stimulus had become familiar, after a single stimulus presentation^19,20. Interestingly, the influence of familiarity or novelty generalized across multiple visual stimuli. In a related study, Cameron and colleagues³⁰ presented patients with word-pairs that had to be remembered and later recalled. They found that the activity of hippocampal neurons during encoding could predict whether subjects would later remember the pairs, whereas entorhinal cortex activity was correlated with recall success. These studies provide support for the important contribution of human MTL neurons to memory processes. The current study provides evidence for a complimentary MTL contribution to memory by showing that neurons show predictive activity during the learning of arbitrary associations between unrelated stimuli that appear in a sequence. Furthermore, the sequence learning effects were specific to the neurons’ preferred images, unlike the more generalized novelty and familiarity effects observed in previous studies. Additionally, our learning effects appeared later, after 11 stimulus exposures, possibly reflecting the additional time needed to accumulate sufficient information to form an association between two unrelated stimuli.

We observed a learning effect in the MTL but failed to observe a similar effect in the temporal cortex. However, we cannot discount the possibility that other brain regions also support associative learning, and could even drive the changes in neuronal response observed here in the MTL. Indeed, as discussed above, several studies have shown that learning-related changes occur in area TE of IT cortex when monkeys learn to associate paired images ² or to classify stimuli into new categories ^31,32 although these changes in tuning occurred after more extensive training. In accordance with these results, human fMRI studies demonstrated that shape learning changes the spatial profile and amplitude of the BOLD response in extrastriate cortex ^33,34,35. Future studies could aim to determine the contributions of the MTL and IT to the learning of shapes and categories and also to the formation of new associations between these shapes. An important difference between IT and the MTL is that IT appears to be mainly concerned with the analysis of complex visual features, whereas the MTL is responsible for the transformation of visual information into mnemonic and conceptual representations.

Several studies of the human MTL have demonstrated the existence of “concept cells” ³⁶. These cells represent the meaning of a given stimulus, in a manner that is multimodal and invariant to different representations of that stimulus ^37,38, and a recent study demonstrated that MTL neurons can even code new associations between a person and a place within a few trials³⁹. We have recently proposed that the ability of MTL neurons to encode associations between stimuli that occur together could account for the formation of concept cells ⁴⁰. Indeed, the statistical regularities of the world ensure that different representations of a given stimulus (e.g., a person’s face and the sound of her name) co-occur frequently. We hypothesize that such statistical structure along with associative learning mechanisms, as revealed here, promotes the linkage of different forms of a given stimulus within the MTL to generate invariant representations.

Another unique feature of MTL neurons is that they are strongly modulated by attention ⁴¹ and are also activated when subjects imagine a stimulus in the absence of any bottom-up visual input ⁴², or bring to mind a past episode, for example, of a recently watched movie clip ⁴³. It seems likely that the influence of mental imagery and active rehearsal on the activity of MTL neurons is related to the predictive activity for the yet-to-be-shown stimuli in a fixed sequence. Thus, the predictive responses in the MTL during sequence learning may contribute to the neuronal mechanisms for the rehearsal of newly learned associations, causing predictive mental images of upcoming stimuli.

A key function of learning is to endow organisms with the ability to predict future events based on what was learned in the past. We suggest that the anticipatory responses of MTL neurons uncovered here may allow the human mind to predict upcoming events based on experiences that occurred only a few minutes ago.

Methods

Participants were eight patients (4 female, age range 18-36 years) with pharmacologically intractable epilepsy undergoing a work up to determine eligibility for surgical therapy of the epilepsy at the SEIN (Stichting Epilepsie Instellingen Nederland)-VU Medical Center epilepsy surgery program, Amsterdam, Netherlands. Patients were implanted with depth electrodes for 7-10 days for chronic seizure recording in order to localize the seizure focus for possible surgical resection ^13,44. All surgeries were performed by J.C.B. The Medical Ethics Committee at the VU Medical Center approved the studies. Written informed consent was obtained from all patients before participation. The electrode locations were based entirely and exclusively on clinical criteria (patient data of any kind necessary to determine a resection with highest probability to render the patient seizure-free). Each electrode consisted of eight microwires from which we recorded single and multi-unit activity and a ninth microwire that served as a local reference. The signal from the microwires was recorded using a 64-channel Neuralynx system, filtered between 1 and 9000 Hz, sampled at 32KHz. On average, each patient was implanted with 34 ± 12 microwires. Participants sat in their hospital room and performed the experimental sessions on a laptop computer. All patients participated in the two types of experimental sessions described below.

Spike Detection and Sorting

Spike detection and sorting were performed with wave_clus ⁴⁵. Briefly, the data were band pass filtered between 300-3000Hz and spikes were then detected with an automatic amplitude threshold. Spike sorting was performed with a wavelet transform that extracted the relevant features of the spike waveform. Clustering was performed using a super-paramagnetic clustering algorithm. As in a previous study ³⁷, the clusters were classified as single- or multi-units (Supplementary Figures 1 and 2). Multi-unit clusters reflect the activity of several neurons that cannot be further differentiated due to a low signal to noise ratio. As in ref. ³⁷, the classification between single- and multi-unit was performed visually based on: 1) the spike shape and its variance; 2) the ratio between the spike peak value and the noise level; 3) the ISI distribution of each cluster; 4) the presence of a refractory period for the single-units; i.e. fewer than 1% of spikes in a 3ms or smaller inter- spike-interval.

Number of neurons and their locations

Over the eight patients we recorded from 635 neurons (single- and multi-units as defined above) in the left and right hippocampi, posterior temporal cortices, parahippocampal cortices, and the left amygdala. Assuming an effect size of at least 0.5 we calculated that we needed 30 cells to achieve a statistical power of 0.75. We determined the selective neurons in the screening sessions (see below) with a paired t-test (p<0.05) that compared the response during the stimulus presentation period (0 to 1000ms following stimulus onset) with the preceding ISI (−500 to 0ms prior to stimulus onset). Consistent with previous reports of selective neurons in the human MTL ³⁷, 80 (12.6%) of all neurons were selective to at least one image. 56 of the selective neurons were targeted during AL, i.e., their preferred stimuli were included in the subsequent AL sessions. 32 of these 56 neurons were located in the right hippocampus, 8 in the left hippocampus, 2 in the left parahippocampal cortex (PHC), and 14 in the right posterior temporal lobe. These 42 neurons in the MTL structures (hippocampi and parahippocampal cortex) were analyzed here (see Supplementary Figure 5 for responses of the 14 temporal cortex neurons). All 42 MTL neurons that were identified as selective during the screening sessions were included in the AL analysis without any further pre-selection. Because of the small number of neurons in the PHC we were not able to statistically examine differences between hippocampal and PHC neurons.

Screening Sessions

On each day that the patient was available, s/he first performed a screening session during which the patient was presented with a large variety of different images (famous people, relatives, animals, landmarks, objects etc.). Each image subtended 1.5 degrees of visual angle and was presented at the center of the screen. Images were presented for 1000ms, followed by an inter-stimulus interval of 500ms. Each image was repeated 8 times in a randomized order. Between 7 and 51 images were used in the screening sessions depending on the time span the patient was available. After the presentation of each image the patients performed a simple yes/no task, for example “Did the picture contain a human face”? The exact question depended on the picture set. This task ensured that patients attended to the stimuli. Data from the screening sessions were rapidly analyzed to determine which images were the “preferred” images of the neurons. A neuron’s “preferred” images were defined as those that elicited a significant (paired t-test, p<0.05) response during the stimulus presentation period (0 to 1000ms after stimulus onset) compared to the preceding ISI (−500 to 0ms before stimulus onset).

Associative Learning (AL) Sessions

Following the screening sessions, 8 patients performed a total of 27 AL sessions, which were usually performed on different days. The AL sessions were designed using information from the screening sessions. In each AL session, subjects were presented with a sequence of 5-7 images, always in a pre-determined order such that a given image, A, predicted the identity of the next image, B, and so on. Subjects were asked to remember the order of the stimuli in the sequence. Each stimulus was presented for 1500 ms with an inter-stimulus interval (ISI) of 500ms, resulting in individual trials of 2000ms (Figure 1). The AL sessions were designed such that in each sequence of 5-7 images, at least one image was the preferred image of one of the recorded neurons. In the rare cases where the sequence consisted of more than one preferred image, only one of the preferred images was used in the subsequent analyses (in other words, the same neuron was never counted twice in an AL session). Additionally, in order to be included in subsequent analyses, the stimulus immediately preceding the preferred stimulus in the sequence was required to be a non-preferred image (i.e., it did not elicit a significant response during the screening session) for this neuron. The sequence was repeated continuously 60 times resulting in experimental sessions of ~10-14 minutes, not including time spent by the subject on test trials. 20% of trials were “test” trials in which, instead of being presented with the next image of the sequence, subjects were shown two images and asked to decide (by pressing one of two keys on the keyboard; Figure 1C) which of the two would be the next image in the sequence. No constraint was placed on response times. Behavioral performance on the test trials is shown in Figure 1D, E. The test trials were excluded from further analysis. There were no test trials in the first stimulus sequence presentation.

To further the impression of a sequence of images we used the following display arrangement: Each image was presented at the center of the screen while 3 placeholders (empty gray squares) were presented to the left and right of the central image (Figure 1B). At the end of the 1500ms presentation period, the central image was replaced by a gray placeholder and all seven grey squares moved one “step” forward in a clockwise direction for the duration of the ISI, such that each placeholder eventually occupied the next placeholder position. At the end of the ISI the placeholder that now occupied the central position was replaced by the next image in the sequence. The viewer’s subjective impression at the end of the ISI interval was that the central image had been hidden, and then moved clockwise, while the central position was replaced by the next image in the sequence.

Data Analysis

For the individual cells shown in Figure 2A, B, spikes were binned into 150ms time bins to obtain the peri-stimulus time histograms. For individual cells, the smoothed response (black curve in Figure 2A,B) was obtained by convolving the spike trains (i.e., in 1ms bins) with a Gaussian smoothing window with a standard deviation of 100ms (similar results were obtained with 50ms and 70ms smoothing windows). To compute the average across all cells (Figure 2C), we smoothed the average response of each neuron with a 100ms smoothing window as above, subtracted the average ISI activity (in the −500 to 0 ms interval before stimulus onset) before the “preceding” stimulus, and averaged across all cells. This analysis corresponds to the baseline-corrected or equivalently, ISI-corrected firing rates. No further normalization was performed to obtain these figures. The only effect of this baseline correction is to set the activity during the preceding ISI to 0 (see Figure 2C). Without this correction the entire plot would simply be shifted upwards to some positive value corresponding to the preceding stimulus ISI activity, but would otherwise be unchanged. We also examined the influence of normalization before averaging (Supplementary Figure 3). To normalize responses we calculated the mean firing rate of each cell for its preferred stimulus, across the screening and AL sessions. Firing rates were then normalized by this value to make all cells comparable. Normalization of responses before averaging gave almost identical results as averaging without normalization.

Latencies

Instantaneous firing rate curves were calculated by convolving the spike trains (i.e., in 1ms bins) for each cell with a Gaussian smoothing window with a standard deviation of 100ms. The latency of response or the time of anticipatory activity was the first time point (in 1ms bins) at which neuronal activity was significantly greater than the ISI (i.e., the −500 to 0ms interval before stimulus onset) prior to the preceding stimulus (paired t-test; p<0.05) for at least 100ms in individual cells (Figures 2A and B) and for 300ms in the averaged response (Figure 2C). Because of the smoothing the absolute latency values have a precision that depends on the size of the window. However, since the same smoothing is applied to the screening and AL responses, the comparison of anticipatory response latencies across these experimental conditions remains justified.

Statistics

Significance of responses was determined by comparing average firing activity across the entire trial window (i.e., −500 to 1000ms for the Screening sessions and −500 to 5500ms for the AL sessions) at each time point to the average firing activity during the ISI (i.e., −500 to 0ms before stimulus onset) before the preceding stimuli, with a paired t-test. The responses marked significant (solid horizontal black lines in Figure 2) correspond to time periods in which firing rates were significantly greater than the ISI activity (p<0.05), continuously for at least 100ms for individual cells. In Figure 2C (average across cells), the significant responses correspond to time periods in which firing rates were significantly greater than the ISI activity (p<0.05), continuously for at least 300ms. Identical criteria were applied to data from the screening and AL sessions. Adjustments for multiple comparisons were made using the method proposed by ref. ⁴⁶ to determine significant differences of time courses.

Two-tailed paired t-tests were only applied when the normality assumption was tenable. In all other situations the t-test was replaced by a non-parametric equivalent test (Mann Whitney U-test). The sphericity assumption was verified for ANOVAs. If needed, we applied the Greenhouse-Geisser correction.