Temporal proximity promotes integration of overlapping events

Abstract

Events with overlapping elements can be encoded as two separate representations or linked into an integrated representation; yet, we know little about the conditions that promote one form of representation over the other. Here, we tested the hypothesis that the proximity of overlapping events would increase the probability of integration. Participants first established memories for house-object and face-object pairs; half of the pairs were learned 24 hours before a functional MRI session, and the other half 30 minutes before the session. During scanning, participants encoded object-object pairs that overlapped with the initial pairs acquired on the same or prior day. Participants were also scanned as they made inference judgments about the relationships among overlapping pairs learned on the same or different day. Participants were more accurate and faster when inferring relationships among memories learned on the same day relative to those acquired across days, suggesting that temporal proximity promotes integration. Evidence for reactivation of existing memories—as measured by a visual content classifier—was equivalent during encoding of overlapping pairs from the two temporal conditions. In contrast, evidence for integration—as measured by a mnemonic strategy classifier from an independent study (Richter, Chanales & Kuhl, 2016, NeuroImage 124, 323–335)—was greater for same day overlapping events, paralleling the behavioral results. During inference itself, activation patterns further differentiated when participants were making inferences about events acquired on the same day versus across days. These findings indicate that temporal proximity of events promotes integration and further influences the neural mechanisms engaged during inference.

Introduction

Events often overlap with one another, sharing content such as people, places, and objects. Overlapping event content may promote interactions among memories triggering one of two mechanisms that determine how new information is represented with respect to existing knowledge. In one case, overlapping content may serve as a retrieval cue to reactivate related memories (Zeithamova, Dominick, & Preston, 2012a). The new event and reactivated knowledge may then be represented by overlapping neural populations, leading to the formation of an integrated memory representation that links related events (Eichenbaum, 1999; Schlichting, Zeithamova, & Preston, 2014; Shohamy & Wagner, 2008; Zeithamova & Preston, 2010; Zeithamova, Dominick, & Preston, 2012a). Such integrated memories can support novel decisions, such as inference, by combining knowledge from events experienced at different times (Schlichting et al., 2014; Schlichting, Mumford, & Preston, 2015; Shohamy & Wagner, 2008; Zeithamova & Preston, 2010). In contrast to this integration mechanism, some theoretical models emphasize that overlapping events are represented by distinct, or pattern separated, neural populations to minimize interference among similar memories (Kumaran, Hassabis, & McClelland, 2016; McClelland, McNaughton, & O’Reilly, 1995; O’Reilly & Rudy, 2000). Empirical data provide evidence for both representational strategies; yet, little is known about factors that promote one mechanism versus another. Here, we test the hypothesis that the temporal proximity of events influences whether overlapping experiences are integrated or separated.

Recent data have shown that events close together in time demonstrate more similar representations in the hippocampus, amygdala, and prefrontal cortex than events separated by longer delays (Cai et al., 2016; Ezzyat & Davachi, 2014; Hsieh, Gruber, Jenkins, & Ranganath, 2014; Rashid et al., 2016). For example, distinct spatial contexts experienced within the same day are represented by a highly overlapping population of neurons within the CA₁ subregion of the hippocampus (Cai et al., 2016). Such integrated representations lead to the generalization of fear responses from one spatial context to another. However, when spatial contexts are experienced one week apart, distinct populations of CA₁ cells represent the two contexts, and the generalization of fear across contexts does not occur. These findings suggest that temporal proximity may be a key factor that promotes integration.

Temporal proximity may drive integration through a memory tagging and allocation mechanism, whereby neurons and synapses recruited to represent a recent episode are more readily engaged for new events that occur within hours of the original episode (Cai et al., 2016; Rashid et al., 2016; Silva, Zhou, Rogerson, Shobe, & Balaji, 2009). Recruitment of the same neural ensembles through tagging and allocation thus results in overlapping population codes for temporally proximal events. At the cognitive level, a similar idea has been expressed by the temporal context model (Howard & Kahana, 2002), which proposes that temporally proximal events are linked by shared context information. Even when events are separated in time, this model suggests that overlapping memory content may serve to reinstate a prior temporal context, leading to the integration of temporally distant events (Howard, Jing, Rao, Provyn, & Datey, 2009). However, it remains an open question how temporal distance constrains the likelihood of memory reinstatement and integration.

Existing research suggests that reactivation of a stored memory may increase its malleability and allow for memory updating (Bridge & Voss, 2014; Diekelmann, Büchel, Born, & Rasch, 2011; Hupbach, Hardt, Gomez, & Nadel, 2008), even when the memory is a day or more old (Hupbach et al., 2008; Rashid et al., 2016). In contrast, the memory tagging and allocation hypothesis (Cai et al., 2016; Rashid et al., 2016; Silva et al., 2009) proposes that recent memories are more likely to be reinstated than temporally distant events, which would result in integration only for temporally proximal events. Furthermore, remote memories that have been stabilized through consolidation are thought to be less susceptible to modification (Frankland & Bontempi, 2005; Squire & Alvarez, 1995). Therefore, even if events encountered on prior days are reactivated during new encoding, a separate memory may be formed for the new information. Here, our goal was to test how the temporal proximity of events impacts reinstatement of prior related memories and integration across events.

Participants underwent fMRI while encoding object pairs that overlapped with previously learned associations. Half of the pairs overlapped with memories (face-object and house-object pairs) acquired on the previous day, while the other half overlapped with memories (a different set of face-object and house-object pairs) acquired within the same experimental session. We predicted that unseen, related memory content (face, house) would be more likely to be reactivated and integrated for the same day than prior day memories. To test the reactivation hypothesis, participants were also scanned while they viewed individual images of objects, faces, and houses during a functional localizer. Data from this localizer were used to train a multivoxel pattern classifier to differentiate the visual content from individual participants’ ventral visual cortex activation patterns; the trained classifier was then applied to the overlapping object pair encoding data to estimate the degree of evidence for reactivation of related face or house information. These reactivation estimates were then compared for events that overlapped with information acquired on the same day and events that overlapped with memories acquired on the prior day.

We assessed how temporal proximity impacted whether participants integrated overlapping events neurally as well as behaviorally. Behaviorally, we assessed the ease of linking information across related events by having participants perform an inference task, in which they were asked to make judgments about the indirectly related elements of two overlapping pairs. Our prior work has shown that reactivation and integration of prior memories with new content facilitates inference (Schlichting et al., 2014; Zeithamova, Dominick, & Preston, 2012a). Here, we tested the hypothesis that inference would be superior for overlapping events experienced on the same day, due to the increased likelihood of integration.

Neurally, we employed a multivariate classification approach to index the degree of integration during encoding of overlapping events, as opposed to simply retrieving old information or encoding the new associations without reference to existing memories. We hypothesized that overlapping events from the same day condition would be more likely to be associated with an integration strategy. This multivariate approach was based on a neural classifier from an independent study, which demonstrated that the process of memory integration evokes a neutrally distinct signature across the whole brain that can be differentiated from simple retrieval or encoding strategies (Richter, Chanales, & Kuhl, 2016).

Specifically, Richter and colleagues presented participants with events that overlapped with previously learned associations, using a paradigm similar to the present study. In the first experiment, participants were instructed during overlapping event encoding to use one of three strategies: retrieve the prior association without encoding the new one, encode the new information without reference to the prior association, or integrate the new association with the prior association. A neural classifier trained on normalized, whole-brain activation patterns was able to discriminate the three memory strategies—retrieve, encode, integrate—from one another on a trial-by-trial basis (Richter et al., 2016). Importantly, the mnemonic strategy classifier trained on the group of participants given explicit strategic instructions was then applied to a new set of participants who were not given explicit strategy cues. The classifier successfully predicted spontaneous integration in this independent group of participants. In the present study, we used the whole-brain, across-participant mnemonic classifier trained on data from Richter et al. (2016) to test our hypothesis that temporally proximal events, i.e., those encoded within the same day, would show greater evidence of memory integration.

A final goal of our study was to determine whether the temporal proximity of events impacts the processes brought to bear as participants make inferences about their relationships. To this end, we also scanned participants as they completed the inference task for which they made novel decisions about the relationships among overlapping events. Memory integration during encoding has been shown to promote novel inference (Schlichting et al., 2014; Shohamy & Wagner, 2008; Zeithamova, Dominick, & Preston, 2012a). However, successful inference may also be achieved through retrieval and joint consideration of pattern separated memories (Greene, Gross, Elsinger, & Rao, 2006; Kumaran et al., 2016; Preston, Shrager, Dudukovic, & Gabrieli, 2004; Zeithamova & Preston, 2010), albeit at the cost of increased reaction time and decreased accuracy (Schlichting et al., 2014; 2015). We hypothesized that the inference strategies may differ depending on whether an integrated memory has been already formed during encoding. Behaviorally, we predicted that participants would be faster and more accurate when making inferences involving same day than prior day events, due to enhanced integration of same day events during encoding. Neurally, we predicted that activation patterns during novel inferences would differ for the same day and prior day conditions, presumably reflecting relative reliance on different inference strategies (retrieval and recombination of separate memories vs. retrieval of integrated memories) across the two conditions. Unlike the hypotheses formulated for reactivation and integration during encoding, we did not have independent data for training of a neural classifier to differentiate between the hypothesized processes during inference. We thus employed a crossvalidation approach in which a neural classifier was trained to discriminate whole-brain activation patterns for inference trials from the two conditions on a within-participant basis.

Materials and Methods

Participants

Thirty-four young adults (age 18–34 years, mean = 23 years; 18 females) participated in the experiment after giving an informed consent in accordance with the University of Texas at Austin IRB policy. Two participants did not complete the scanned portion of the experiment, one due to scanner failure and one due to failure to learn the paired associates during the training before scanning. Data from three additional participants were excluded from analysis due to incorrect slice prescription (1), loss of behavioral responses (1), and failure to follow task instructions (1). Data from the remaining 29 participants were used in the final analyses.

Procedures

Stimuli were grayscale images of faces (F), houses (H), and common objects (O). Images were organized into triads consisting of either a face and two objects (60 face-object-object triads) or a house and two objects (60 house-object-object). Stimuli from each triad were presented as two overlapping paired associates (Fig. 1). Initial associations, referred to as AB pairs, consisted of a face or a house (the “A” item) paired with an object (the “B” item). Overlapping associations, referred to as BC pairs, consisted of the same B objects now paired with novel objects (the “C” item).

Figure 1. Schematic of study design and behavioral results.

a. Session 1: Training on prior day initial pairs. One day prior to scanning, participants were trained on 30 face-object and 30 house-object pairs across 3 study-test repetitions (test not depicted). b–d. Session 2 on the following day had three successive phases: b. Training of same day initial pairs. Participants were trained outside scanner on a novel set of 30 face-object and 30 house-object pairs across three study-test repetitions (test not depicted). c. Encoding of overlapping pairs. During slow event-related scanning, participants encoded 120 object-object pairs. One object from each pair was novel and one was encountered previously during prior day initial training (blue arrow) or same day initial training (green arrow). d. Inference test. During slow event-related scanning, participants were given a surprise test of 120 relationships that required linking information across events. Their task was to infer which house or face was indirectly related to objects that shared a common associate. Half of the faces and half of the houses were encountered on prior day (blue arrow) and the other half on the same day (green arrow). No explicit cues distinguished inference trials from the two temporal conditions during the test phase, nor were participants required to retrieve whether the stimuli were experienced on the same or different day. All tests consisted of 2-alternative forced-choice judgments with the foil stimuli always of the same type and condition as the correct choice. e. Inference accuracy split by the two temporal conditions, prior day (blue bar) or same day (green bar). f. Median reaction times for correct inferences for the two conditions. Error bars denote across-participant standard error.

The experiment took place over two sessions, approximately 24 hours apart (Fig. 1). During Session 1 (not scanned; Fig. 1a), participants learned half of the initial pairs (30 face-object and 30 house-object) across three study-test repetitions (hereafter referred to as ‘prior day’ pairs). During Session 2, participants learned the remaining set of initial pairs (‘same day’ pairs: 30 new face-object and 30 new house-object) over three study-test repetitions, 30–60 minutes prior to scanning (Fig. 1b). For each AB study trial, a pair of stimuli was presented for 3.6 s followed by a 0.4 s fixation. The left-right position of the two items was randomized. During each AB test trial, one stimulus of a pair was presented on the top of the screen (e.g., A cue) and two choice stimuli were presented on the bottom of the screen (e.g., a correct B and a foil B from a different AB pair). On each test trial, the foil stimulus was constrained to match the type and condition of the correct stimulus. In other words, if A was a male face from a face-object-object triad, then the B foil was an object paired with a different male face during the same training session. Each AB test trial was equally likely to be cued by the A stimulus or the B stimulus. The correct choice was presented on the left or right position with equal probability. Each test trial lasted for 3.6 s during which time the participant indicated whether the left or right choice stimulus was paired with the cue stimulus. Each test trial was followed by a 0.4 s fixation. AB training lasted about a half an hour on each day.

The second phase of Session 2 consisted of scanned encoding of the 120 overlapping object-object (BC) pairs (Fig. 1c). Each overlapping pair was studied only once, with half of the BC associates overlapping with prior day AB pairs and the other half overlapping with same day AB pairs. On each BC study trial, a pair of objects was presented for 3.6s followed by 8.4s fixation baseline for a total of 12 s per trial. BC encoding occurred across three fMRI runs, each lasting 8 minutes. All triad types were balanced across runs, i.e., the BC pairs overlapping with AB pairs from a different temporal condition (prior day, same day) and associated with different stimulus content (faces, houses) were evenly distributed across the runs. Presentation order of the pairs within a run was randomized. Participants were not explicitly informed about the overlap between pairs they learned previously and the new pairs, nor were they asked to retrieve the prior AB associates. Instead, participants were informed that they would be challenged to learn associations in a single trial, as opposed to the three trial learning they had previously experienced.

Following the encoding of overlapping events, participants were given a surprise AC inference test that required them to relate elements across overlapping pairs (Fig. 1d). For the face-object-object triads, participants were told that they had initially seen a person (the A image) owning an object (the B image). Later, they saw the same object (B) paired with another object (C). From these pairings, participants were told that they could infer that the second (C) object must also belong to the same person (A). A similar explanation was given for the house-object-object triads, wherein participants were told they could infer that C items could be found in the same house (A) as their corresponding B associate. We refer to this test as “AC test” or “inference test” as participants needed to infer a relationship between two items (A and C) that were never presented together but were both related to a common associate (the B element). Participants performed the surprise test for 120 inferential AC relationships during fMRI scanning. On each AC inference trial, one stimulus (e.g., a C object) served as a cue presented on the top of the screen, and two choice stimuli (e.g., correct A and a foil A from a different triad) were presented on the bottom of the screen. The foil stimulus was constrained to match the triad type of the correct stimulus. Each AC trial was equally likely to be cued by the A stimulus or the C stimulus. On each trial, the cue and choice stimuli were presented for 3.6s followed by 8.4s fixation baseline for a total of 12 s per trial. Inference testing occurred across three fMRI runs, each lasting 8 minutes, with a balanced distribution of triad types (inferences involving a face or house from the same and prior day conditions). Presentation order within each run was randomized.

After the completion of the inference test, participants underwent testing for their memory of the initial (AB) and overlapping (BC) pairs (not scanned). The test procedure was the same as in previous phases, except that trials lasted 4 s (a combined 3.6 s stimulus and response period, 0.4 s fixation). BC pairs were tested before AB pairs.

Session 2 concluded with the collection of two anatomical scans (see MRI data acquisition) followed by a functional localizer task used for the training of a visual content classifier based on multivoxel pattern analysis (MVPA). During the localizer, participants viewed individual objects, faces, and houses in a blocked design, using different stimuli from those presented during the associative inference task. Each block consisted of successive presentation of 9 stimuli (2 s each). Participants performed a 1-back working memory task, in which they pressed a button whenever the current stimulus was a repetition of the one immediately preceding. Within each block, there were 1 or 2 stimulus repeats. The order of face, object, and house blocks was counterbalanced within and across participants. In total, there were three localizer runs, each lasting 8 minutes.

MRI data acquisition

Whole-brain imaging data were acquired on a 3.0T GE Signa MRI system (GE Medical Systems). Functional images were acquired using a multiecho GRAPPA parallel echoplanar imaging (EPI) sequence with 31 3-mm-thick oblique axial slices (0.6 mm gap), 20 degrees off the AC-PC line (TR = 2 s; TE = 30 ms; 2 shot; flip angle = 90°; 64 × 64 matrix; 3.75 × 3.75 mm in-plane resolution, interleaved slice acquisition). For each functional scan, the first six EPI volumes were discarded to allow for T1 stabilization. Head movement was minimized using foam padding. Two structural images were collected: a T2-weighted flow-compensated spin-echo pulse sequence with the same slice prescription as the functional images [repetition time (TR) = 3 s; echo time (TE) = 68 ms; 256 × 256 matrix, 1 × 1 mm in-plane resolution], and a high-resolution T1-weighted SPGR scan was acquired in the sagittal plane using a 1.3 mm slice thickness with 1 mm² in-plane resolution.

Preprocessing of fMRI data

Data were preprocessed and analyzed using SPM5 (Wellcome Department of Cognitive Neurology) and custom Matlab routines. Functional images were realigned to the first volume in the time series to correct for motion and co-registered to the T2-weighted structural image. The T2-weighted structural image was co-registered to the high-resolution SPGR, and the co-registration parameters were then applied to the functional images. Finally, the functional images were high-pass filtered with a 128 s filter and converted to percent signal.

The majority of analyses were performed in the native space of each participant without spatial smoothing. The exception was an analysis for which we assessed memory strategy during overlapping event (BC) encoding by applying the neural classifier from Richter et al. (2016) (see below). Because this analysis used a classifier trained on whole-brain, MNI normalized data from a different group of participants collected in a different lab (Richter et al., 2016), data from each participant in the present study were normalized into MNI standard space using ANTS (Advanced Normalization Tools: http://picsl.upenn.edu/software/ants/). First, ANTs was used to compute co-registration parameters from each participant’s reference functional volume (first BOLD image from first run) to their high resolution T1 anatomy using a rigid affine transformation. Next, each participant’s T1 anatomical scan was normalized to a standard 2mm MNI anatomical template using a diffeomorphic transformation. The transformation parameters were concatenated and applied to the functional data from the overlapping (BC) encoding phase. Finally, the normalized patterns were smoothed with an 8mm FWHM kernel.

Decoding reactivation of prior related content during overlapping event encoding

Our prior work (Zeithamova, Dominick, & Preston, 2012a) showed that memories are reactivated during encoding of overlapping events, with the degree of reactivation predicting participants’ ability to subsequently infer novel relationships among related events. Here, we were interested in whether reactivation would be modulated as a function of temporal condition (i.e., prior day, same day). To test this hypothesis, we first verified that patterns of activation within visually selective ventral temporal cortex differentiated between viewing of face, house, and object stimuli during the localizer task.

An anatomical mask of the ventral temporal cortex was used for this analysis, which consisted of the Freesurfer-defined inferior temporal cortex, parahippocampal cortex, and the posterior portion of fusiform. Within each participant, ventral temporal cortex activation patterns for each timepoint from the localizer were labeled according to which stimulus type was being viewed (face, house, object). To account for the hemodynamic lag, condition labels were shifted back by two scans (4 s) with respect to the functional timeseries. We then ran MVPA using the Princeton MVPA toolbox (http://code.google.com/p/princeton-mvpa-toolbox/) and custom code for MATLAB. The visual content classifier was a regularized logistic regression algorithm (penalty = 50), and we first used a leave-one-run-out cross-validation procedure to verify that the classifier could differentiate the visual content. Two runs from the localizer task were used iteratively for classifier training, and the remaining run was used to assess generalization performance. The comparison between the MVPA classifier prediction and the actual condition on each trial provided the cross-validation accuracy of the classifier. Cross-validated classification accuracy was very high for the localizer, averaging 92.2% correct (sd = 4.9%) across participants.

Next, we turned to estimating the degree of trial-by-trial reactivation of the previously related stimulus (face or house) during encoding of overlapping pairs. To obtain a single activation pattern for every overlapping encoding trial, we averaged data for three time points, corresponding to activation patterns 4 s, 6 s, and 8 s after the stimulus onset. Second, we trained the visual content classifier on all three runs from the functional localizer and then applied the trained classifier to each BC encoding trial pattern. The classifier output for each trial pattern consisted of three numbers representing the similarity to face, house, and object categories. Our critical comparison was between the face and house outputs, which would reflect reactivation of the previously related A item (face or house) during encoding of BC object-object pairs. Specifically, face and house classifier outputs were compared between BC trials related to faces (face-object-object triads) and BC trials related to houses (house-object-object triads). We also computed trial-by-trial reactivation estimates by subtracting z-scored face and house classifier outputs on each trial according to the trial type (face minus house output for trials in which the unseen but related A stimulus was a face, house minus face output for trials in which related A stimulus was a house). To test whether the degree of reactivation differed for prior day events and same day events, the trial-by-trial estimates were summed within each temporal condition and compared using a paired t-test.

Decoding integration during encoding

We hypothesized that the memory strategy participants employed while encoding overlapping events might differ for the same and prior day conditions, with participants being more likely to employ an integration strategy as they encoded events that overlapped with associations acquired on the same day. To assess the memory strategy participants employed during encoding of overlapping events from the same and prior day conditions, we employed a multivariate neural classifier derived from an independent study by Richter et al. (2016). We chose to employ an independent neural classifier from a different data set because it allowed us to make forward inferences about the strategies participants spontaneously employed in the present study, and whether the strategies differed for our two temporal conditions.

In the study by Richter et al., participants were presented with overlapping associations in a similar paradigm to the present work. On each overlapping event trial, they were given one of three explicit instructions for the memory strategy to use to process the overlapping event: (1) retrieve the prior related memory (without encoding the current event), (2) encode the new event (without reference to the prior memory), or (3) integrate the new information with their existing memory. A sparse multinomial logistic regression MVPA classifier (from Princeton MVPA toolbox) was trained on normalized whole-brain activation patterns from all-but-one participant and was then used to predict memory strategy cued on each trial for the remaining participant. The classifier’s performance reliably discriminated the three memory strategies from one another, including on a pairwise basis (e.g., integration was separately decoded from both the simple encoding and retrieval strategies). Importantly, the neural classifier was then applied to data from a new set of participants from a second experiment, who were not given explicit task instructions during overlapping event encoding. The neural classifier was also successful at isolating when the new, uninstructed participants spontaneously employed an integration strategy when encoding overlapping events.

Here, we employed the whole-brain mnemonic strategy classifier from Richter et al. (2016) in a similar manner to the application in their second experiment described above. Similar to the second experiment in Richter et al. (2016), our participants were not provided with specific instructions for how to process the overlapping pairs for either temporal condition. We applied the trained whole-brain classifier from Richter et al. (2016) to trial-by-trial activation patterns (averaged 4–8s after stimulus onset) from the encoding phase of our experiment. The output of the mnemonic classifier consisted of three numbers, indicating the degree of evidence for the three mnemonic strategies during overlapping event encoding (retrieve, encode, integrate). To test our hypothesis that overlapping events experienced on the same day are more likely to be integrated, we first averaged classifier evidence for integration separately for same day condition trials and prior day condition trials for each participant. We then used a paired t-test to compare the integration evidence between temporal conditions.

Decoding temporal condition during inference test

We further predicted that inference judgments about memories acquired on the same day would evoke distinct neural processes from those required for inference about memories acquired across days. This prediction stems from our hypothesis about the strategies brought to bear during encoding of same and prior day overlapping associations. If overlapping pairs from the same day condition are more likely to be integrated with existing memories during encoding, inference could be supported by retrieval of such an integrated representation during test. In contrast, if overlapping pairs in the prior day condition are more likely to be encoded separately from existing memories, we would expect inference would rely on retrieval and recombination of two separate representations.

Unlike the two MVPA applications that we used on the encoding data, we did not have independent data for classifier training. We thus employed a leave-one-run cross-validation approach on the inference data to test whether whole-brain activation patterns differentiate between the two temporal conditions. Similar to encoding, we first computed a single pattern for every inference trial by averaging the three time points from the functional time-series collected 4s, 6s, and 8s after stimulus onset. The resulting inference trial patterns were labeled based on the day when the corresponding initial memory was first encountered (prior day, same day). We implemented a leave-one-run-out crossvalidation analysis within a whole-brain mask using a regularized logistic regression algorithm implemented in the Princeton MVPA toolbox to obtain the day classification accuracy for each participant. A one-sample t-test across participants was used to assess whether classification was significantly above the chance probability of 0.5 at p < 0.05. To assess whether differential neural recruitment in the two temporal conditions aids inference success, we related the day classification accuracy (i.e., disambiguation of inferences involving prior day and same day events) to inference performance across participants using Pearson’s correlation.

Effect of temporal condition on neural mechanisms that support inference

Inference across events has been shown to involve hippocampal and prefrontal regions (Zeithamova & Preston, 2010). Importantly, their recruitment during inference may further differ depending on which strategy—retrieval of integrated representation or retrieval and recombination of individual memories—is employed to support performance (Preston & Eichenbaum, 2013). Thus, we further tested whether inference about the relationships for overlapping pairs from the same and prior day associations evoke distinct patterns of activation in PFC and hippocampus. Within PFC, we focused on IFG given its role in both memory-based inference (Zeithamova & Preston, 2010) and temporal coding (DuBrow & Davachi, 2014; Ezzyat & Davachi, 2014; Jenkins & Ranganath, 2010) and VMPFC based on its hypothesized role in memory integration (Schlichting & Preston, 2015; van Kesteren, Fernandez, Norris, & Hermans, 2010a; Zeithamova, Dominick, & Preston, 2012a). Because prior studies have shown functional heterogeneity within IFG in memory tasks (Badre & Wagner, 2007; Dobbins, 2005; Raposo, Han, & Dobbins, 2009; Schlichting et al., 2015), we considered the subregions of the IFG—anterior (pars orbitalis), middle (pars triangularis) and posterior IFG (pars opercularis)—separately. To identify these five ROIs on individual participants, we used FreeSurfer (Martinos Center for Biomedical Imaging, MGH, Charlestown, MA) to create a subcortical segmentation and cortical parcellation based on the high-resolution T1 SPGR. We then employed the same temporal condition classification analysis as described above, assessing crossvalidated performance on the inference data within each ROI instead of the whole-brain mask. As with the whole brain classifier, we related the temporal classifier accuracy to inference performance across participants. To correct for multiple comparisons across the 5 regions, we adopted a significance threshold of p < 0.01 (p < 0.05/5 ROIs, resulting in Bonferroni corrected p < 0.05).

Finally, to assess to what degree the multivariate findings may be driven by possible differences in the overall response magnitude for the two temporal conditions, we performed a univariate analysis of the inference data under the assumption of a general linear model. The model consisted of four regressors that separated trials based on the temporal condition (prior day, same day) and inference success (correct, incorrect). This model allowed us to test whether overall activation or the magnitude of success effects (correct—incorrect difference) differ between the temporal conditions. Each trial was modeled as a stick function convolved with a canonical hemodynamic response function as implemented in SPM5. The model also included temporal derivatives for each condition and six motion parameters as regressors of no interest. Parameter estimates (beta weights) were extracted within each ROI for each participant and condition. A 2 x 2 repeated measures ANOVA was used to quantify the main effects of temporal condition (prior day vs. same day), inference success (correct vs. incorrect inference), and their interaction.

Summary of decoding analyses

To summarize, we used three different MVPA classifiers to test our hypotheses regarding the effect of temporal condition on overlapping event encoding and inference. First, we used a visual content classifier to test the hypothesis that memories acquired within the same day would be more likely to be reactivated during encoding of overlapping events relative to memories acquired on a different day. Second, we used a mnemonic strategy classifier to test the hypothesis that overlapping events would be more likely integrated with existing memories acquired on the same day relative to those experienced on the prior day. Finally, we used a temporal condition classifier on the inference test data to test the hypothesis that temporal condition would impact neural engagement during inference.

Results

Behavioral performance

To test how the temporal proximity of events impacts the ability to draw inferences about their relationships, we compared inference performance for each temporal condition. As predicted, inference success was greater and reaction times on correct trials were faster when initial associations were learned on the same day as compared to the prior day (inference success: t(28) = 2.25, p = 0.016; reaction time for correct inferences t(28) = 2.47, p = 0.010; Fig. 1e,f).

However, participants also forgot more prior day than same day initial associations (prior day mean = 0.87, same day mean = 0.92, t(28) = 4.80, p < 0.001). To ensure that the behavioral benefit for same day inferences did not solely result from better memory for same day initial pairs, we performed two control analyses. First, we related the behavioral benefit for same day inferences to the forgetting cost for prior day initial memories across participants. If the difference in inference in the two temporal conditions was due to forgetting of prior day initial pairs, participants who forgot the most prior day initial pairs compared to same day initial pairs should also be those that show the greatest difference in inference performance for those conditions. However, the relationship between the same day inference benefit and the forgetting of prior day initial memories was not significant (r = − 0.132, p = 0.49). In other words, participants who showed the greatest behavioral benefit for the same day inferences are not necessarily those who remember same day initial pairs better than prior day pairs. We also performed a second analysis of the inference data limited to the trials for which the initial pair was remembered. Inference performance was still reliably greater and reaction time faster for same day associations relative to the prior day condition (inference success: prior day mean = 0.60, same day mean = 0.65, t(28) = 2.13, p = 0.021; reaction time for correct inference: prior day mean = 2545 ms, same day 2476 ms, t(28) = 1.76, p = 0.045).

We further examined memory for the overlapping (BC) events themselves as a function of temporal proximity to the initial associations. Memory for overlapping events was not affected by the temporal condition (recognition: prior day mean = 0.86, same day mean 0.85, t(28) = 0.73, p = 0.47; reaction time: prior day mean = 1701 ms, same day mean = 1705 ms, t(28) = 0.18, p = 0.86), indicating that the differences in inference performance did not result from differential memory for overlapping events themselves.

Evidence for reactivation during encoding of overlapping events does not differ between temporal conditions

At the neural level, we first tested whether: 1) information about related A items was reactivated during encoding of overlapping BC events and 2) the degree of reactivation differed as a function of temporal condition. We trained an MVPA classifier on independent data from the same participants to differentiate between neural patterns related to face, house, and object categories. We then applied the classifier to index reactivation of the related face or house stimulus from the initial pairs during encoding of each overlapping object-object pair. Replicating our prior study (Zeithamova, Dominick, & Preston, 2012a), we found a significant reactivation of initial memories during encoding of related events (Fig. 2a). Contrary to our prediction, evidence for reactivation did not differ for prior day and same day memories (t(28) = −0.87, p = 0.39; Fig. 2b).

Figure 2. Decoding reactivation and integration during encoding of overlapping events.

a. Evidence for reactivation of faces and houses during encoding of overlapping object-object pairs derived from a visual content classifier. (f)OO: encoding of two objects related to a face. (h)OO: encoding of two objects related to a house. b. Reactivation evidence (normalized difference between house and face output) during encoding of overlapping pairs from the prior day (blue) and same day (green) conditions. c. Evidence for use of an integration strategy during encoding of overlapping events for the prior day (blue) and same day (green) conditions derived from a mnemonic strategy classifier. Error bars on all panels denote across-participant standard error of the mean.

To replicate our prior findings, we also tested whether individual differences in reactivation evidence during encoding track individual differences in subsequent inference success (Zeithamova, Dominick, & Preston, 2012a), irrespective of condition. We found a significant positive correlation between reactivation and inference success (r = 0.40, p = 0.017), indicating that participants who show evidence for reactivation during encoding of overlapping events are more successful at inference. When we assessed the correlation between reactivation evidence and inference performance separately for each temporal condition, the across-participant correlation was numerically but not significantly greater for same day than prior day conditions (prior day r = 0.18, same day r = 0.32).

Same day events are more likely to be integrated during encoding

Prior work showed that is possible to decode, from normalized whole-brain activation patterns, whether participants integrate information across events as they learn about new experiences that overlap with existing memories (Richter et al., 2016). Here, we used the classifier from that prior study, which was trained on an independent group of participants, to test our hypothesis that same day overlapping events would be more likely integrated with existing memories relative to overlapping events from the prior day condition. Consistent with our prediction, we found greater classifier evidence for integration during same day overlapping pairs relative to the overlapping pairs in the prior day condition (t(28) = 2.04, p = 0.026; Fig. 2c). Thus, while we found comparable evidence for reactivation of related memories across temporal conditions based on the visual content classifier, evidence for integration based on the mnemonic strategy classifier was greater for same day overlapping events, mirroring the pattern of behavioral data.

We did not have specific a priori predictions about the other mnemonic strategies indexed by the classifier (retrieve old, encode new) for the two temporal conditions. However, because we observed differences in evidence for integration, and because the relative evidence is coupled across the three strategies, we tested whether the two strategies showed corresponding differences across temporal conditions. We found no differences in evidence for the retrieve strategy (mean evidence for prior day = 0.327, se = 0.006; same day mean = 0.330, se = 0.006; t(28) < 1), but we did observe significantly greater evidence for encoding of new associations for prior day condition (mean = 0.346, se = 0.009) than same day condition (mean = 0.315, se = 0.006; t(28) = 2.24, p = 0.017).

Temporal condition decoding during inference

Inference across events may rely on retrieval of integrated memory representations or retrieval and recombination of separate memory representations (Schlichting et al., 2014; Schlichting & Preston, 2015; Zeithamova, Schlichting, & Preston, 2012b). These two inference strategies are postulated to recruit different neural mechanisms (Preston & Eichenbaum, 2013). Because behavioral and neural evidence suggested greater probability of integration for same day events, we hypothesized that neural processes brought to bear to inference would differ for the two temporal conditions. Consistent with this prediction, we found that a whole-brain classifier could distinguish inference trials from the same and prior day conditions (mean = 0.534, t(28) = 3.59, p < 0.001; Fig. 3a), although nothing on the screen indicated the temporal condition of inference trials. This decoding analysis suggests that inferences involving prior day memories engage different processes than inferences across same day events. We further hypothesized that the ability to make successful inferences across events would be influenced by the ability to engage the process best suited to a given temporal condition. To test this hypothesis, we computed the correlation between day classification accuracy and inference accuracy across participant. However, we did not find evidence for such a relationship (r = 0.10, p > .5).

Figure 3. Temporal condition effects during inference.

a–b. Temporal condition decoding during inference. a. Cross-validated temporal condition classification accuracy in the whole brain (white bar) and each ROI (gray bars). Dashed line represents chance. Error bars denote across-participant standard error of the mean. b. Correlation between IFG pars opercularis day classification and inference performance across participants. c. Univariate temporal condition and success effects. Mean activation during inference split by temporal condition and inference success. Star (*) next to the region’s name denotes significant main effect of success (corrected for multiple comparisons). No region showed main effect of temporal condition or temporal condition x success interaction. Error bars denote across-participant standard error.

Because same day inferences were overall more successful and faster than prior day inferences, we performed two control analyses to determine if the temporal condition decoding was driven by differences in difficulty across condition. First, we correlated the behavioral inference benefit for same day events (both for accuracy and reaction time) with temporal condition decoding accuracy. If temporal condition decoding is driven by differences in difficulty across conditions, it should be easier to decode temporal condition in those participants who show the largest difference in accuracy across conditions. We did not find a significant relationship between temporal condition decoding and the same day inference benefit (accuracy differences r = 0.20, p > 0.3; RT differences: r = 0.11, p > 0.5), suggesting that condition-level differences in difficulty were not driving decoding. Second, we performed decoding analysis limited to correct trials only, running 100 Monte Carlo simulations that randomly sub-sampled trials to equate the number of training trials used for each condition. Cross-validated decoding accuracy remained above chance (mean = 0.545, t(28) = 3.25, p = 0.003).

Prior research suggests that different inference strategies may differentially recruit hippocampus and prefrontal regions (Preston & Eichenbaum, 2013). We tested this hypothesis using the same multivariate approach as we employed with the whole brain classifier but testing classification performance within hippocampus and each prefrontal ROI separately. Mean day classification accuracy was above chance in VMPFC (t(28) = 2.87, p = 0.008) and all IFG subregions (all t > 3.47, p < 0.002; Fig. 3a). Furthermore, IFG pars opercularis classification accuracy positively tracked inference success across participants (r = 0.49, p = 0.006; Fig. 3b), a pattern not present in other regions (all r < 0).

Because same day inferences were overall more successful and faster than prior day inferences, we performed the same two control analyses as we did with a whole-brain classifier to determine if the temporal condition decoding was driven by differences in difficulty across condition. First, we found no correlation between decoding accuracy and behavioral benefit (in accuracy or reaction time) for same day events (accuracy differences: max r = 0.07, p > 0.7; RT differences: max r = 0.18, p > 0.3), suggesting that condition-level differences in difficulty were not driving decoding. Second, decoding analysis limited to correct trials only (averaging 100 simulations with random sub-sampling of trials to equate the number of training patterns across conditions) showed that cross-validated accuracy in IFG pars opercularis remained above chance (t(28) = 2.71, p = 0.01), with a similar pattern in IFG pars triangularis (t(28) = 2.18, p = 0.04) that did not reach the corrected threshold.

Overall activation and inference success effects in hippocampus and IFG are equivalent across temporal conditions

To test whether temporal condition decoding in these regions was driven by overall activation differences between conditions and to test whether they are engaged during inference in the current task, we performed a univariate analysis with temporal condition and inference success as factors. Consistent with prior reports of hippocampal and IFG role in mnemonic inference (Zeithamova & Preston, 2010), we observed a significant main effect of inference success in the hippocampus, IFG pars triangularis, and IFG pars orbitalis (all F(1,28) > 15.9, p < 0.001), with greater activation during correct than incorrect inferences (Fig. 3c). Success effects in IFG pars opercularis (F(1,28) = 2.59, uncorrected p = 0.12) and VMPFC (F(1,28) = 4.05, uncorrected p = 0.054) did not reach significance threshold. No main effects of temporal condition (all F < 1) nor temporal condition x success interactions (all F(1,28) < 1.33, p > 0.26; Fig. 3c) were found in any region, suggesting that mean activation differences were not driving decoding of temporal condition in these regions.

Discussion

In this study, we tested the hypothesis that temporally proximal events would be more likely to be integrated, resulting in enhanced inference for the relationships among those episodes. Consistent with this prediction, participants were faster and more accurate when making novel inferences—our behavioral marker of integration—that involved events experienced within the same day relative to those experienced on different days. Using a mnemonic strategy classifier from an independent study (Richter et al., 2016), we were also able to quantify the neural evidence for integration during new encoding. Similarly to the behavioral findings, we found greater evidence for integration when participants encoded events overlapping with memories acquired on the same day versus those overlapping with prior day events. In contrast, neural evidence for reactivation during encoding was equivalent for prior day and same day related content, suggesting that reactivation does not always lead to integration.

During inference itself, whole-brain activation patterns distinguished when participants were making decisions about the relationships among events experienced on the same day from those about events experienced on different days. Differentiation of the temporal conditions during inference was particularly apparent in IFG pars opercularis, where discrimination accuracy tracked individual differences in inference performance. These findings may indicate that different strategies are brought to bear when inferring relationships among integrated versus separated memories and that deployment of optimal inference strategies aids performance. Collectively, these data indicate that temporal proximity of events promotes memory integration and modulates neural mechanisms brought to bear during inference.

The temporal context model (Howard & Kahana, 2002) proposes that events experienced close in time are linked together via a shared temporal context. Consistent with this notion, recent neuroimaging studies have shown greater neural overlap when events are experienced in immediate succession (Ezzyat & Davachi, 2014; Hsieh et al., 2014). However, the temporal context model further predicts that events do not have to immediately follow one another to become linked in memory. Events that share overlapping content with prior experience may serve to reactivate a previous temporal context, providing a mechanism for integration of even temporally distant memories (Howard et al., 2009). In the present study, however, we demonstrate that there are boundary conditions that may limit the ability to bridge memories across time. We show evidence for reactivation and integration of events experienced minutes apart within the same experimental session. In contrast, events experienced on different days were less likely to become integrated despite comparable evidence for reactivation. Our prior work showed that integration during encoding aids subsequent inference performance (Schlichting et al., 2014; Zeithamova, Dominick, & Preston, 2012a). In cases when integration does not occur, inference can still be performed by retrieving and recombining two individual associations at test, but at a cost of reduced accuracy and increased reaction time (Schlichting et al., 2014; 2015). The current study demonstrates that integration during encoding is less likely for temporally distant events. Consequently, inference performance for the prior day condition is both slower and less accurate than the same day condition because fewer integrated representations are formed for events experienced across days.

At a cellular level, recent activation of a synapse is thought to temporarily increase its excitability and tag it for subsequent long-term potentiation to support memory formation (Frey & Morris, 1997). This synaptic tagging mechanism has also been proposed to guide allocation of neurons during new encoding (Silva et al., 2009). Recently tagged neurons would be more likely to be recruited during new event encoding, which would promote memory integration. Such enhanced recruitment of tagged neurons, however, is thought to be temporally limited, occurring only when events are experienced within minutes or hours of one another. Two recent rodent studies evince this mechanism showing that overlapping neural populations represent events experienced on the same day, while those experienced on different days are representationally distinct (Cai et al., 2016; Rashid et al., 2016).

In one of these studies (Rashid et al., 2016), mice were exposed to two events that were separated by a variable time interval. Two events separated by up to 6 hours recruited overlapping ensembles of amygdala neurons, and memories of the two events became integrated, as evidenced by generalization of fear responses across events. In contrast, events separated by 18 or 24 hours recruited distinct populations of amygdala neurons and did not show behavioral evidence of integration. Similar findings have been observed within the CA₁ subfield of the hippocampus, for which overlapping neural populations represented two events separated by 5 hours but not two events separated by 7 days (Cai et al., 2016). These findings parallel the behavioral and fMRI results observed in the current study and suggest a potential cellular mechanism underlying preferential integration of temporally proximal events. Our study extends these findings to demonstrate how different representational schemes for temporally proximal and distant events impacts individuals’ ability to extract new knowledge and reason about the relationships among memories.

While these converging animal (Cai et al., 2016; Rashid et al., 2016) and human findings demonstrate that temporal proximity is a critical factor determining the likelihood of integration, other factors may play an important role in determining whether memories are integrated or separated. For instance, memory strength may be another boundary condition of integration, as suggested by two sets of findings (Schlichting et al., 2015; Tse et al., 2007; 2011). In rodents, new flavor-location associations are acquired rapidly when experienced within a well-learned spatial configuration acquired across many sessions relative to a spatial configuration that was frequently changed (Tse et al., 2007; 2011). Presumably, such facilitation occurs because well-learned events are more readily reactivated when learning new content that overlaps with existing knowledge. In humans, stronger memories also facilitate integration (Schlichting et al., 2015). When initial events (AB associations) are repeated many times before the introduction of an overlapping event (BC associations), hippocampus and prefrontal cortex form integrated representations of the overlapping events. However, when overlapping events were presented in alternation, hippocampal and prefrontal cortex form pattern separated memories. In the latter condition, the initial event is experienced only once before introduction of the overlapping event; thus, the initial memory is likely weaker and less likely to be reactivated during the overlapping event.

In one aspect, these findings may seem somewhat contradictory to the present data, as they show enhanced integration for events in which the first experience of the initial memory was more temporally distant from the first presentation of the overlapping event. However, temporal distance between the last presentation of the initial memory and the first presentation of the overlapping memory was equated across conditions in both of these studies. The critical events in the strong and weak conditions were both two days apart in the studies by Tse and colleagues. In Schlichting et al. (2015), the last presentation of the initial memory and the first presentation of the overlapping event were a few trials apart in both strong and weak conditions. Thus, when temporal distance from the last experience is similar, stronger memories are preferentially integrated. As time and memory strength are often intertwined, it is not possible to completely rule out memory strength differences between conditions in the present study. Nevertheless, our results do indicate that a benefit for same day inferences was observed in the present study even after forgetting of initial events was taken into account, both within and across participants. In light of this finding, the present results suggest that temporal proximity provides an additional boundary condition on integration across events that may not be explained by strength alone.

Our study further extends prior research on the boundary conditions of memory integration to show how temporal proximity impacts processes brought to bear during inference itself. Specifically, multivariate activation patterns across the brain and in IFG specifically discriminated when participants inferred relationships about events experienced within the same day from inferences about events that occurred on different days. Furthermore, discrimination accuracy in IFG (pars opercularis) was related to superior inference. One interpretation of these findings is that distinct mechanisms may support inference when an integrated representation has been formed during encoding (as was more likely in the same day condition) relative to when separate representations are formed. Integrated representations would support direct retrieval of the indirect relationship between items that share a common associate; inference from separated representations would require successful retrieval of two memories that would then be jointly considered to support inference (Zeithamova, Schlichting, & Preston, 2012b). Prior work has shown that distinct subregions of IFG and VMPFC support integrated and separated representations (Schlichting et al., 2015). Here, prefrontal activation patterns that discriminate between temporal conditions during inference may reflect retrieval of integrated representations for same day events, and retrieval and recombination of separate memory traces for events encoded across days. The relationship between prefrontal discrimination accuracy and performance further suggests that the ability to access the appropriate representations for a given condition may contribute to individual differences in inference success.

Conclusions

Our findings demonstrate that temporal proximity of events promotes the formation of integrated memory representations for overlapping events that facilitate novel inferences about their relationships. These data shed light on an important boundary condition—time—that influences when overlapping events are represented by integrated or separated neural codes, thus converging with recent cellular work in animals. Furthermore, our findings indicate that the nature of the representations formed during encoding influence the processes brought to bear during inference. More broadly, these findings contribute to our growing understanding that temporal structure plays an important role in organizing memories.