A neural substrate in the human hippocampus for linking successive events

Abstract

Memory formation requires the placement of experienced events in the same order in which they appeared. A large body of evidence from human studies indicates that structures in the medial temporal lobe are critically involved in forming and maintaining such memories, and complementing evidence from lesion and electrophysiological work in animals support these findings. However, it remains unclear how single cells and networks of cells can signal this temporal relationship between events. Here we used recordings from single cells in the human brain obtained while subjects viewed repeated presentations of cinematic episodes. We found that neuronal activity in successive time segments became gradually correlated, and, as a result, activity in a given time window became a faithful predictor of the activity to follow. This correlation emerged rapidly, within two to three presentations of an episode and exceeded both context-independent and pure stimulus-driven correlations. The correlation was specific for hippocampal neurons, did not occur in the amygdala and anterior cingulate cortex, and was found for single cells, cell pairs, and triplets of cells, supporting the notion that cell assemblies code for the temporal relationships between sensory events. Importantly, this neuronal measure of temporal binding successfully predicted subjects’ ability to recall and verbally report the viewed episodes later. Our findings suggest a neuronal substrate for the formation of memory of the temporal order of events.

Memory formation includes knowledge about temporally dated events and the temporal relations of these events to each other. In human episodic memory, for example, events and concepts are linked together as they appear on the “time arrow,” and in the same context in which they appeared (1–4). Many other types of memory (e.g., rote and sequence learning) require the successful encoding of the temporal relationship between events that follow in time. Case studies as well as studies using imaging techniques with human subjects have shown that formation and maintenance of such memories is mediated by different structures in the medial temporal lobe, with the hippocampus playing a major role (1–10). It is now accepted that neurons of the hippocampus are especially bound to reflect associative events, as these neurons sample inputs from a wide range of structures including sensory areas and high-order association cortices involved in recent event processing (2, 3).

To study electrophysiological correlates of memory formation, research in the hippocampus of animals has focused on several paradigms that require memory for the temporal order of events (2, 11). These paradigms include learning of associations between pairs of stimuli (12–15), trace conditioning in which the unconditioned stimulus is well separated in time from the conditioned stimulus (16, 17), delayed nonmatch-to-sample (18), sequences of locations in spatial navigation tasks (19, 20), and order of events by spatiotemporal context (21, 22). Recent findings provide further support for an animal model of episodic memory in the hippocampus: neurons were shown to encode information about different aspects of a memory task (23–25); internally generated reactivation of spike sequences was found during delay periods with no external cues in single cells (26, 27) and in large cell assemblies (28); and neurons were shown to encode gradually the temporal context of events (i.e., “when” the events occurred) during learning (22). In humans, we recently reported selective reactivation of hippocampal and entorhinal neurons during free verbal recall (29).

Here, we sought to examine how neurons can signal the temporal relationships between events and how these patterns evolve during memory formation. The use of single-unit recordings in human subjects [patients with pharmacologically intractable epilepsy (30)] allowed us to use a free-viewing paradigm (29) that, unlike most animal studies, does not involve direct reinforcement, and thus more closely resembles memory formation in real-life scenarios. Moreover, we could relate the recorded unit activity to subjects’ memory as assessed by conscious verbal recollections.

Results

Single and multiunits were recorded (27 sessions in 13 individuals) from the hippocampus (anterior hippocampus, AH, n = 180), amygdala (n = 160), entorhinal cortex (EC, n = 224), and anterior-cingulate cortex (AC, n = 107). Overall, 52% (349/671) of recorded units were classified as single units based on strict criteria, and these units were distributed homogenously between structures. In each session, subjects freely viewed a new series of 10–16 different audiovisual clips lasting 5–10 s each. To allow investigation of gradual formation of memory, each of the clips was repeated six times in a pseudorandomized order (Fig. S1A). This paradigm was used with the hypothesis that, as a clip is viewed repeatedly, there is gradual formation of memory for the temporal order of events presented in the clip and that the neuronal activity in subsequent time segments reflects this memory formation (Fig. S1 C and D).

As an indicator for the formation of temporal relationship between successive time segments of neuronal activity, we first observed and measured the relationship between activity at any given time (t) and the activity following it (t+1). In many neurons, the raw firing rate at time t during clip presentation was not correlated initially with the subsequent firing rate (that at time t+1) but became so after the same clip was viewed a few times [P > 0.1 for first clip presentations (Fig. 1 A, D, and G); P < 0.01 for the sixth clip presentation (Fig. 1 B, E, and H), Pearson correlation coefficient].

Fig. 1.

Neuronal activity during clip presentation and development of temporal relationships. Firing rate in first (A) and sixth (B) clip presentation for one hippocampal neuron is shown in green, overlaid by the same firing rate shifted by 250 ms (in red). The Pearson correlation coefficient between these two traces is marked within each panel. (C) Gradual development of temporal relationships (red) based on MI between spike counts at successive time windows (in % of maximal information; see text and Fig. S1). Dashed line shows a baseline derived from destroying all temporal patterns, with the 95% confidence interval in yellow. Inset shows average firing rate in different clip repetitions. (D–I) The same analysis is shown for two other neurons.

However, the relationship between subsequent activities does not have to be linear (i.e., low activity entails low activity and high entails high activity), and we therefore used a more appropriate measure: the mutual information (MI) (31, 32) between subsequent time bins. This measure allowed us to unveil more complex relationships and more subtle ones. For example, an increase in the linear correlation between successive time segments will result in an increase in the MI, but the reverse is not necessarily true. Importantly, MI measures directly the reduction in uncertainty about activity at time (t+1) by the knowledge of activity at (t). In other words, it provides a measure of how well we can predict the activity at (t+1) by the knowledge of the activity at time (t). It is thus an adequate measure for our hypothesis.

We therefore calculated the MI between successive time segments for each neuron in each clip in each sequential presentation of the clip. Henceforth, we refer to this measure as “temporal relationships.” Temporal relationships are measured in percentage of the maximum possible information (which is log2 = 1 bit; 2 being the number of representations we used for the spike count; Methods). We found units exhibiting an increase in temporal relationships as a function of clip repetition in all structures (Fig. S2 and Fig. 1 C, F, and I; P < 0.001 for all, linear regression). Overall, 54% (362/671) of all neurons showed temporal relationships for at least one clip (P < 0.01, linear regressions over clip repetitions; Bonferroni-corrected for number of different clips). These neurons were distributed relatively homogenously among the different structures (AH, 59%; EC, 57%; amygdala, 48%; AC, 53%; P > 0.05, χ²; Fig. S3A).

There are two possible contributions to the origin of these temporal relationships that are not related to memory formation. First, there could be context-independent correlations, i.e., correlations that are not related to the clips. For example, the correlations could reflect changes in the synaptic activity in the network of which the neuron is a part, or changes in the internal state of the neuron, and these changes can occur gradually within a session and independently of clip repetitions. To estimate the contribution of such context-independent relationships, we calculated the temporal relationships in blank periods preceding clip presentation, when no external stimulus was presented, and compared them with the temporal relationships during the clip. In 22% of all neurons (more than expected by chance, P < 0.01, Fisher's exact test), the temporal relationships during the clips significantly exceeded these context-independent relationships. The proportion of these neurons was highest in the hippocampus (AH, 31%; EC, 21%; amygdala, 15%; AC, 17%; P < 0.01, χ²; Fig. S3B).

Second, there are correlations in the stimulus itself, i.e., in the clips. The simplest example is that the same image/event continues to appear in successive time windows. To estimate this contribution, we shuffled between successive clip segments recorded during different presentations of the same clip. In 34% of the neurons, the temporal relationships were significantly higher than these pure-stimulus relationships (P < 0.01, t tests, Bonferroni-corrected). The proportion of these neurons was much higher in the hippocampus (AH, 49%; EC, 34%; amygdala, 23%; AC, 25%; P < 0.001, χ²; Fig. S3C). We combined the pure-stimulus and context-independent contributions in one model for each neuron (Fisher's combined test). Overall, the distribution of P values for all cells was skewed clearly to the left for 19% of neurons (P < 0.05), showing that a substantial component of the correlations between successive time windows cannot be explained by other contributions (P < 0.01, χ²).

To observe the formation of temporal relationships within the different structures, we averaged the temporal relationships over all neurons over all clips, but within each clip repetition separately (1–6, x axes in Fig. 2) and within each brain structure separately (Fig. 2 A–D). In all structures, the pure-stimulus (solid black line) and the context-independent (green line) relationships were significant for all clip repetitions (P < 0.05, t tests) compared with baseline patterns derived from destroying all temporal structure (dashed lines), with comparable magnitude across structures [P > 0.05, analysis of variance (ANOVA)]. Importantly, the temporal relationships (red line) showed a significant gradual increase in the hippocampus (P < 0.001, linear regression) (Fig. 2D) but not in the other structures (P > 0.05) (Fig. 2 A–C), suggesting that the temporal relationships are not related to possible alterations in behavior with clip repetition (e.g., general arousal). Only in the hippocampus did the temporal relationships significantly exceed both the pure-stimulus and the context-independent patterns in late repetitions (P < 0.001, t tests for repetitions 2–6). This finding was enhanced when analysis was restricted to clip-responsive neurons (n = 189, 28% of all neurons, and n = 51 hippocampal neurons) (Fig. 2E) but occurred in nonresponsive neurons as well (P < 0.01, t tests). The finding also was replicated when standard linear methods as Pearson correlation were used to measure the temporal relationships (P < 0.01, linear regression) (Fig. 2F). We further examined temporal relationships in simultaneously recorded pairs (n = 4,884) and triplets (n = 2,1061) of neurons and found that a significant increase in temporal relationships is evident in the EC as well (P < 0.01 for both AH and EC; P > 0.05 for amygdala and AC, linear regressions; Fig. S4B).

Fig. 2.

Development of temporal relationships. Averaged MI between successive time windows as a function of clip repetition for the amygdala (A), anterior cingulate (B), entorhinal cortex (C), and hippocampus (D). Temporal relationships (red lines) are calculated during the clip presentation; context-independent relationships (green lines) are calculated on a 2-s period of black screen before clip presentation; pure-stimulus relationships (solid black lines) are derived by shuffling corresponding time windows between different repetitions of the same clip; baseline patterns (dashed lines) are derived by shuffling time windows within the same presentation (they are not zero but are an order of magnitude smaller). Temporal relationships (y axis) are presented as percentage of maximum possible MI. (E) Same analysis is shown for the subset of responsive neurons in the hippocampus (n = 51, P < 0.05). Neurons were considered responsive if their firing rate distribution differed from that of the baseline period for at least one clip. P values were Bonferron-corrected for the number of clips presented in that session. (F) Shown are temporal relationships as measured alternatively by standard Pearson correlation coefficient for all hippocampal neurons (red line) (P < 0.01, linear regression), context-independent for hippocampal neurons (green line) (P > 0.05), and temporal relationships for neurons from all other structures (black line) (P > 0.05).

If the neuronal measure we introduce here indeed reflects the gradual formation of memory for the temporal relationship between experienced events during encoding, then this measure should be reflected in the subjects’ actual memory. At the end of each session, subjects were asked to recall freely and report verbally the clips (Fig. S1B), and they did so with above-chance performance (Fig. 3A). We found a highly significant correlation between the level of temporal relationships in hippocampal neurons and success in the spontaneous free-recall task for individual subjects (r = 0.87, P < 0.01) (Fig. 3B). This correlation was not seen in any of the other structures (r = −0.04, P > 0.5 for all; P > 0.1 for each structure separately) (Fig. 3C). We conclude that the level of temporal relationships in hippocampal single neurons during viewing is related directly to the delayed memory performance of individual subjects.

Fig. 3.

Individual memory performance is predicted by temporal relationships in hippocampal neurons. (A) Individual memory performance per subject as measured by the percentage of freely recalled clips out of all clips. (B) The temporal relationships in the last clip repetition averaged over all hippocampal neurons recorded from a subject (x axis) are plotted against this subject's recall performance (r = 0.867, P < 0.001, linear regression). (C) Same analysis averaged over neurons from all other structures (r = −0.04, P > 0.5, linear regression).

We verified that the changes we observed do not reflect changes in overall firing rate, which was stable in the hippocampus (as well as in other structures) across clip repetition and along clip duration (P > 0.2 for repetition and P > 0.3 for duration, two-way ANOVA; Fig. S5A). Similarly, the variability of firing rates of hippocampal cells was stable across clip repetition and along clip duration (P > 0.3 for repetition and P > 0.3 for duration, two-way ANOVA; Fig. S5B). The experimental context and the duration of time spent viewing clips had no effect on the temporal relationships (calculating the temporal relationships in viewings of entirely different movie clips but from the same repetition, P > 0.1, ANOVA). Although temporal relationships could stem from slower dynamics or sustained/tonic bursts that develop as the session progresses, we could not find any evidence for changes in levels of activity across successive time segments (P > 0.2, ANOVA; Fig. S5C) or of changes in the width of the autocorrelations that can indicate development of slow oscillations (P > 0.1, ANOVA; Fig. S5D).

Finally, we received further confirmation that our results stem from a developing relationship between successive time segments rather than from changes in the statistics of the firing rates by separately calculating the two components of the MI: the marginal entropy and the conditional entropy (31) Marginal entropy tests for changes in the statistics of the firing rate, and the conditional entropy tests for the relationship between successive time windows. We found that the increase in the MI was caused mainly by a reduction in the conditional entropy (P < 0.001, linear regression) (solid line in Fig. S5E) rather than by an increase in the marginal entropy (dashed line in Fig. S5E).

Discussion

Our findings demonstrate a hippocampal neuronal correlate for creating temporal relationships between stimuli that follow in time. We hypothesize that this correlate reflects a general mechanism that is used whenever the brain is required to create temporal associations between stimuli. These findings are in line with and provides direct support to the current dogma in memory research stating that neurons of the hippocampus are able to reflect associative events because they receive converging information about recent experienced events from sensory areas and high-order association cortices (2, 3). This neuronal correlate was detected in the hippocampus but not in other areas, probably due to the distinctive properties of the hippocampus: The temporal relationships we observed were context (clip)-specific. In real life, an event/stimulus can lead to many possible events/stimuli in different contexts and so can the specific neuronal activity associated with it; the hippocampus is unique in its ability to integrate and take into account the successive events and the context in which they appeared (3, 4), and thus it is capable of creating such branching representations. Additionally, we found these temporal relationships in pairs and triplets of simultaneously recorded cells. This finding is in line with Hebb's original proposal (33) that neuronal assemblies can be linked temporally and subserve the evolution of internal states in general (34) and episodic memory in particular (28).

Our study provides important information complementing electrophysiological findings in animals. Studying human subjects, we were able to use a paradigm that mimic memory formation under more natural conditions. Unlike most animal studies, and more like real-life scenarios, we used a free-viewing paradigm with no direct reinforcement involved. Moreover, most animal studies use paradigms in which learning takes many trials, and the animals typically are overtrained, but human memory can form following a single experience (1–4). Here, the neuronal changes already were seen during the second viewing, namely after only one viewing, as happens in real life episodic memory formation.

Although our analysis was designed specifically to detect formation of temporal relationships in neuronal activity and therefore to investigate a neural correlate for formation of temporal order in memory, a few other alternative interpretations should be considered. One such possibility is that subjects develop better memory for the individual elements of the clips. In this interpretation, the neural correlate we found reflects a visual familiarity signal that develops over time in the hippocampus. However, it was shown recently that the total number of spikes elicited in a hippocampal neuron actually is reduced in response to repeated presentations of pictures (still images) (35), although visual familiarity increases in such a scenario. It is also possible that the subjects were able to retrieve or recognize more and more information with each repetition. This interpretation also would fit with the remarkable correlation we found between the neuronal measure and the subsequent individual memory, because it has been shown that memory retrieval improves subsequent memory performance (36). Although we cannot rule out these possible contributions completely, we argue that the specific nature and design of our analysis, together with the lack of changes in similarity and nonconditional statistics of firing rates across time segments, and the relatively small role of the hippocampus in familiarity-based object recognition (37), point to a unique signal that develops in the hippocampus and is related to encoding of temporal relationships. Accordingly, a recent functional MRI study found that hippocampal activation is related specifically to retrieval of the temporal order of events and correlates positively with accuracy of sequence recall for naturalistic movie scenes (10).

Another important aspect of our analysis is that it does not assume a constancy of the representation in a single neuron (i.e., linking a reproducible firing pattern to a repeated external event). Rather, it explores the relationship of a neuron to itself in consecutive times, thus taking into account the global state of the neural network at any given time. Indeed, it has been shown that the overall state of the network before stimulus presentation might be an important contributor to the neuronal representation of that stimulus (38), and thus the representation (i.e., firing rate of individual neurons) can change from one clip repetition to the next. Our approach, by not measuring spike counts directly but instead looking at relationships between spike counts in successive time segments, bypasses these dynamic fluctuations and reveals the formation of patterns that otherwise would remain hidden.

To conclude, our results support the conjecture that the brain encodes temporal information as changes in the state of the network (39) and that memories are encoded as stable states of neural networks (40). In this interpretation, the changes we see in the activity are changes in the internal representation of the external stimuli. The formation of memory here means that the representation of an external stimulus (a network state) would now converge internally and lead more reliably to the representation of other external stimuli (other network states) that followed in time. We suggest that the approach developed here of looking at successive time segments, together with approaches for measuring the coactivity of high-dimensionality networks (41–43), will provide a more complete understanding of how the brain forms temporal memories.

Materials and Methods

Subjects and Recordings.

The data were collected in 27 recording sessions from 13 subjects, age 18–54 years, with pharmacologically intractable epilepsy. (Ten subjects were right-handed; seven were males). Extensive noninvasive monitoring did not yield concordant data corresponding to a single resectable epileptogenic focus. Therefore, the subjects were implanted with chronic depth electrodes for 7–10 days to determine the seizure focus for possible surgical resection (30, 44). Here, we report data from sites in the hippocampus, amygdala, entorhinal cortex, and anterior cingulate (Table S1). The same experiments were used recently to show reactivation of neuronal activity during free recall (29). All studies conformed to the guidelines of the Medical Institutional Review Board at University of California, Los Angeles. The electrode locations were based exclusively on clinical criteria and were verified by MRI or by CT coregistered to preoperative MRI. Each electrode consisted of a flexible polyurethane probe containing nine 40-μm platinum-iridium microwires protruding ∼4 mm into the tissue beyond the tip of the probe. Eight microwires were active, recording channels and were referenced to the ninth, lower-impedance, microwire. The differential signal from the microwires was amplified by using a 64-channel Neuralynx system, filtered between 1 and 9,000 Hz and sampled at 28 kHz. One recording channel was used to record simultaneously the signal from a microphone attached to the subject's shirt. All sessions were conducted at the subject's quiet bedside using a standard laptop screen and speakers.

Experimental Paradigm.

Each recording experiment lasted about an hour and was composed of one to three cycles consisting of two parts: a viewing session and a free-recall session.

Viewing session.

In each viewing session, subjects were presented with a series of 10–16 different, new audiovisual movie clips lasting 5–10 s each. Each clip depicted an “episode” featuring famous people or characters engaged in activity (e.g., a segment from the animated television series The Simpsons, President Bush announcing the capture of Saddam Hussein), landmarks photographed from various views (e.g., coastline of New York City, aerial view of the Golden Gate Bridge), animals in motion (e.g., koala climbing on a tree, snake eating an egg), or objects depicted in a dynamic context. Each clip was presented at least six times, and the order of presentation was pseudorandomized. There were six rounds of clips in each session; each round contained all the clips but in a different order; the same clip never was presented twice consecutively; all clips within a single session were of same length; in some of the experiments interleaving blank periods (“blanks”) of 5 s were used occasionally within a group of successive clips, and in other experiments interleaving blanks of 2–3 s were used before each clip. Subjects were asked to watch the clips freely. After the viewing session subjects performed an intervening arithmetic task (1 min) in which they were asked to arrange and read the digits of six-digit numbers in increasing order (example: for the number 285739, read “2-3-5-7-8-9”). Numbers were presented for 6 s with 1-s interleaving blanks. The first three subjects were engaged in a conversation with the people present in the room during the interval between viewing and recall sessions.

Free-recall session.

In the free-recall session that immediately followed the intervening task, subjects (n = 11; for technical reasons we were not able to record the free-recall session in two subjects) were asked to recall freely the clips they had just seen and to report verbally immediately when a clip “comes to mind.” This session was not limited in time and was stopped only when the subject recalled all the clips correctly or when the subject could not remember any more clips. On average, the free-recall session lasted 4.5 min. The ratio of video clips that were freely recalled correctly was significantly above chance level for all subjects and is given in Fig. 3A.

Data Analysis.

Spike sorting for unit isolation.

Spike detection and sorting was applied to the continuous recordings of each session (about 1 h) by using a well-established clustering algorithm (45). After sorting, the clusters were classified into single units or multiunits based on (i) the spike shape and its variance; (ii) the ratio between the spike peak value and the noise level; (iii) the ISI distribution of each cluster; and (iv) the presence of a refractory period for the single units [i.e., <1% spikes within 3 ms interspike interval (ISI)]. Using these criteria, 52% of all recorded cells were classified as single cells and the rest as multiunit. Note our result is strengthened because it occurs in multiunits as well as in single cells: we show that the network state in a given time segment (measured by the combined activity of the few neurons that compose a multiunit) becomes a reliable predictor of the network state in the successive time segment.

Calculation of temporal relationships.

For each neuron (n = 671), spikes were binned into time windows of 250 ms to form vectors [sc(1),sc(2),…sc(t),sc(t+1),…] where sc(t) is the spike count in time window t. We then calculated the MI between sc(t) and sc(t+1) using all possible pairs from each single clip presentation . Please see Robustness below for standard correlation analysis and sensitivity to window size.

MI was calculated with the standard equation (31, 32):

We used MI as a measure for temporal relationship for several reasons.

First, our hypothesis means that sc(t+1) will follow sc(t) more reliably. Thus, sc(t) will be a better predictor of sc(t+1). MI measures this reliability directly: it measures the reduction in uncertainty about sc(t+1) by the knowledge of sc(t) (uncertainty as measured by the entropy of the distribution). In other words, it provides a measure of how well we can predict sc(t+1) by the knowledge of sc(t). [MI gives an upper bound on how well, on average, an oracle predictor that can use all the information available can estimate sc(t+1) from sc(t) (32)].

Second, it is natural to use MI with a small finite discrete set of representations for the spike count (i.e., categorical variables). Here, we used the transformation

(see also Robustness below). This representation makes sense when using time windows of 50–500 ms and taking into account the relatively low (∼4 Hz) firing rate in structures of the medial temporal lobe (Fig. S5A). The actual spike counts in such short time windows are thus usually less than two. Most other measures of correlation are suboptimal in such cases.

Third, MI does not assume any structure for the correlations or of the underlying distribution. For example, an increase in the linear correlation between successive time windows will result in an increase in the MI, but the reverse is not necessarily true. Thus, using MI, we cover more options for increased reliability that would not be detected using other methods (e.g., using only Pearson coefficient; see Robustness below).

Finally, because of its categorical representation, MI has a natural extension for representing patterns in cell assemblies of few neurons. For example, for two simultaneously recorded neurons (Fig. S4A), there are four possible patterns for each window: [0,0], [0,1], [1,0], and [1,1], in which the first number indicates the spike count (0 or 1 or more) in cell 1 and the second number indicates the spike count (0 or 1 or more) in cell 2.

The measure of MI is subject to bias when the distribution is undersampled. Therefore we used an analytical bias correction when looking at cell pairs or triplets (46). Neural assemblies of n > 3 require much larger amounts of data and could not be measured in the current paradigm.

Significance of the temporal relationships.

To test for significance of the MI between successive time windows, we shuffled the time windows within each clip, thus removing any temporal relationship. This procedure was repeated 50 times, and the resultant MI was averaged to obtain the baseline patterns with standard errors. Then these patterns were compared with the temporal relationships in each clip repetition by t tests (comparing all neurons in each region separately), and P values were corrected by a Bonferroni correction for the six clip repetitions.

Several possible factors not related to memory formation may contribute to the origin of the temporal relationships.

First, there are context-independent correlations. For example, the correlations could reflect changes in the synaptic activity in a network of which the neuron is a part, and these changes could occur gradually within a session and independently of clip repetitions. Alternatively, these patterns could be related to changes in the internal state of the neuron (e.g., slow dynamics for integration of synaptic currents). To estimate such context-independent patterns, we calculated the MI in the blank periods preceding clip presentation, when no external stimulus was presented.

Second, there are correlations in the stimulus itself, i.e., in the clips. The simplest example is that the same image/event continues to appear in successive time windows. This possibility is especially relevant here, because the temporal windows were constructed without reference to a specific clip; therefore it is possible for an image that appears in one window to persist and appear in the following time window. To control for such pure-stimulus relationships, we shuffled time windows across different repetitions of the same clip, thus keeping each window in its original place in the clip and therefore maintaining the temporal relationships between the external stimuli. For example, if sc(t) comes from the second clip repetition, sc(t+1) may come from the fourth clip repetition. Thus, it preserves the external stimulus present at times t and t+1. For each clip repetition, we produced 50 shuffles and averaged the result. These results then were compared with the temporal relationships in each clip repetition by t tests (comparing all neurons in each region separately), and P values were corrected by a Bonferroni correction for the six clip repetitions.

Finally, the experimental context (e.g., the room) and the mere passage of time could affect the formation of temporal relationships. To control for these possibilities, we shuffled the clips to use six viewings of entirely different movie clips, but within the same repetition of each clip (e.g., all come from repetition 1; then all come from repetition 2, and so on).

Robustness.

All calculations were repeated by segmenting time into bins of 50, 100, 250, or 500 ms and by using discretization of spike counts to 0/≥1 spikes, 0/1/≥2 spikes, or 0/1/2/≥3 spikes. The results were unchanged for all combinations, and the temporal relationships increased significantly only in the hippocampus (P < 0.01, linear regression).

We also repeated the analysis using a standard noncategorical Pearson correlation coefficient (Fig. 2F). To do so, we calculated the correlation coefficient between the vector [sc(1), sc(2),… sc(t), sc(t + 1),…] and the shifted vector [sc(2), sc(3),… sc(t + 1), sc(t + 2),…] as a measure for temporal relationships. Results were similar and again were highly significant only in the hippocampus (P < 0.01, ANOVA, over repetitions using all neurons). In accordance with the aforementioned description of why this measure is suboptimal, the slope over repetitions was slightly decreased than when MI was used (as in Fig. 2D), but still was significantly positive (P < 0.01, linear regression, t tests).

Responsive neurons.

Responsive neurons were chosen if they had a different distribution of firing rates during the different repetitions of at least one clip viewing compared with the baseline/blank period before the clip when no external stimulus was present (Mann–Whitney nonparametric test, P < 0.05, Bonferroni-corrected for the number of clips presented in a session).

The analysis then was repeated to observe temporal relationships in responsive neurons only (n = 189, 28% of all neurons). The results were qualitatively similar for all structures but they were quantitatively significantly better for the hippocampus (n = 51, P < 0.01, t tests between maximum information at last repetition) (Fig. 2E). However, the effect seen when all neurons are included (Fig. 2D) is not a diluted effect stemming from responsive cells alone, because we observed a similar increase in temporal relationships for the “nonresponsive” hippocampal cells as well (n = 129, P < 0.01, t tests).

Fluctuations in activity statistics.

There was no correlation between changes in firing rate statistics and the temporal relationships, as can be seen in the single-cell examples (Fig. 1 C, F, and I) and the stability of the firing rate as a factor of clip repetitions and clip duration (Fig. S5A). This finding also held true for the stability of the variability of the firing rate (Fig. S5B). We also looked at similarity of spike counts across successive time windows. Such increased similarity, if found, could result from slower dynamics that develops as the session progresses and would result in higher information content between successive time windows. Similarity was measured as spike count in a time window minus the spike count in the successive time window and averaged over all possible time windows (Fig. S5C). We also tested for development of slow oscillations by calculating standard autocorrelations (Fig. S5D).

Although there was only little difference in the average firing rate during the blank periods preceding the clips and during the clip presentation, we used a further “thinning” method (47) to assure that the statistical differences between temporal relationships (calculated on activity during the clips) and the context-independent patterns (calculated on activity during the blank periods) are not caused by the small difference in the number of spikes available. We randomly down-sampled the activity during the clips to match the average number of spikes in the blank periods preceding them and recalculated the temporal relationships. Repeating this process 50 times and averaging the findings yielded similar results, and temporal relationships still were significantly higher than context-independent patterns (P < 0.001, t tests). We used a similar method to equalize the number of spikes along the session (within and before clips): For each neuron, we ordered the clips according to the number of spikes elicited and randomly down-sampled the spikes to match the lowest count. Temporal relationships still were significantly monotonic, increasing and higher than other patterns (P < 0.01, t tests).

Finally and importantly, the MI measure consists of two components: the marginal entropy and the conditional entropy (31, 32). We calculated these two components and found that the increase in the MI came mainly from a reduction in the conditional entropy (P < 0.001, linear regression) (solid line in Fig. S5E) , rather than from an increase in the marginal entropy (dashed line in Fig. S5E). This finding provides further support that the temporal relationships are a measure of temporal binding of activity in successive time segments.