Mental simulation of routes during navigation involves adaptive temporal compression

Abstract

Mental simulation is a hallmark feature of human cognition, allowing features from memories to be flexibly used during prospection. While past studies demonstrate the preservation of real-world features such as size and distance during mental simulation, their temporal dynamics remains unknown. Here, we compare mental simulations to navigation of routes in a large-scale spatial environment to test the hypothesis that such simulations are temporally compressed in an adaptive manner. Our results show that simulations occurred at 2.39x the speed it took to navigate a route, increasing in compression (3.57x) for slower movement speeds. Participant self-reports of vividness and spatial coherence of simulations also correlated strongly with simulation duration, providing an important link between subjective experiences of simulated events and how spatial representations are combined during prospection. These findings suggest that simulation of spatial events involve adaptive temporal mechanisms, mediated partly by the fidelity of memories used to generate the simulation.

Introduction

Mentally simulating events is one of our most fundamental cognitive skills, critical to our ability to anticipate and handle future experiences. It underlies flexible goal planning during navigation (Burgess, 2008) and is a central aspect to the constructive nature of episodic memory (Boyer, 2008; Moulton & Kosslyn, 2009; Schacter et al., 2012; Suddendorf, Addis, & Corballis, 2009). Research over the past decade using mental simulation has revealed new aspects of mnemonic processing, including the ability to recapitulate details from past experiences into novel contexts (Hassabis, Kumaran, & Maguire, 2007a; Szpunar, Addis, McLelland, & Schacter, 2013) and how these anticipatory future simulations can motivate and guide behavior (Boyer, 2008; Suddendorf & Busby, 2005). Many of these studies have cumulated into a growing consensus (Buckner & Carroll, 2007; Hassabis & Maguire, 2009; Moulton & Kosslyn, 2009; Schacter et al., 2012; Schacter & Addis, 2007; Szpunar et al., 2013) that mental simulation involves a dynamic neurocognitive system dedicated to encoding experiences, extracting features form those memories, and actively combining those features into representations, or mental ‘scenes’, that are used to optimize behavior. This has led to new perspectives on how aging influences memory, prospection, and mental imagery (Addis, Wong, & Schacter, 2008; Personnier, Kubicki, Laroche, & Papaxanthis, 2010; Schacter, Gaesser, & Addis, 2013), and how these processes are affected by cognitive and neurodegenerative disorders (Addis, Sacchetti, Ally, Budson, & Schacter, 2009; Hassabis, Kumaran, Vann, & Maguire, 2007b; Irish & Piolino, 2015; Kwan, Carson, Addis, & Rosenbaum, 2010).

Although past work has shown the utility of mental simulation in experimental (Hassabis, Kumaran, & Maguire, 2007a; Szpunar et al., 2013), clinical (Addis et al., 2009; Kwan et al., 2010), and real world contexts (Personnier et al., 2010; Schacter & Addis, 2007), critical components of how simulations operate have yet to be empirically evaluated. Early work (Kosslyn, Ball, & Reiser, 1978; Shepard & Metzler, 1971) demonstrated that mental representations based on visually encoded objects retain metric information. This finding has been extended through work with amnesic patients (Hassabis, Kumaran, Vann, & Maguire, 2007b) and brain imaging (Szpunar, Watson, & McDermott, 2007) to suggest that spatial context acts as a framework to organize features from memory to anticipate future situations (Hassabis & Maguire, 2009). Despite the promising implications of studies on spatial aspects of mental simulations, to our knowledge, no study has investigated their temporal dynamics and how this relates to the constructive nature of memory. Past studies that included temporal components of mental simulations have either (a) limited the simulation to one visuospatial scene, omitting the spatially extended nature of daily experience (e.g. (Borst & Kosslyn, 2010; Cui, Jeter, Yang, Montague, & Eagleman, 2007; Personnier et al., 2010), or (b) used temporal extent as an independent variable in task designs, for example, by analyzing detail generation during recall/simulation at different points in the past/future (Addis et al., 2008; Borst & Kosslyn, 2010; Botzung, Denkova, & Manning, 2008). As such, the temporal basis of mental simulations in human remains unknown, despite its critical importance to understanding cognitive processes related to episodic memory and prospection.

Here, we ask three fundamental questions about the temporal dynamics of mentally simulated events and evaluate them across two studies. First, following the research of Kosslyn and colleagues (Kosslyn et al., 1978), we investigated whether mental simulations retain temporal information derived from previous experiences in a spatial environment. Our hypothesis is that simulated episodes contain temporal aspects of the experiences the simulation is recapitulated from, albeit in a compressed form. This hypothesis is based in part on findings from place cell recordings in the rodent hippocampus during pre-play/replay events that show temporal compression of route sequences (Davidson, Kloosterman, & Wilson, 2009; A. Johnson & Redish, 2007; Nádasdy, Hirase, Czurkó, Csicsvari, & Buzsáki, 1999; Skaggs, McNaughton, Wilson, & Barnes, 1996).

Our second question is whether compression of temporal information is a constant or adaptive process. We hypothesize that temporal compression during mental simulation provides an adaptive mechanism to compensate for the speed at which the events used to generate the simulation were experienced. That is, we predicted slower movement speeds during the original experience would lead to greater compression rates during the simulated episode, with faster speeds leading to lower compression rates. Our rationale for formulating this hypothesis originated from theories suggesting that mental simulations offer a form of prospection where features of past experiences are combined into simulations about future events (Schacter, Addis, & Buckner, 2007). We reasoned that an adaptive temporal compression rate would offer an advantage for prospection and prediction by allowing the temporal dynamics of combining past experiences to be adjusted in order to more efficiently simulate future scenarios. We term this the “adaptive” hypothesis of mental simulation.

Our third question pertains to how the temporal flow of a simulation relates to one's ability to construct detailed representations used for mental simulation. If the subjective fidelity of the encoded spatial context from previous experiences facilitates mental simulation, as has been postulated elsewhere (Hassabis, Kumaran, Vann, & Maguire, 2007b; Szpunar et al., 2007), there may be a statistical relationship between the subjective experience of visuospatial aspects of a simulation and the time it takes to imagine them. However, it is currently unknown how detail generation relates to the temporal flow of simulated episodes. It may be that more vivid and coherent events take longer to simulate. Conversely, more vivid and coherent events might result in faster simulation, consistent with the perspective that environments are represented by a manifold of spatial maps that need to be dynamically organized during navigation (Derdikman & Moser, 2010; McNamara, Hardy, & Hirtle, 1989). Under the multiple spatial maps perspective, simulation speeds depend on how quickly memory systems dynamically organize multiple spatial representations into a coherent representation used to guide navigation. In this case, ratings of subjective experience of simulations, such as vividness and spatial/temporal coherence, would provide a measure of how well the multiple maps are integrated into a task-related spatial representation used for navigation, with less vivid and coherent simulations indicating more effortful and piecemeal integration.

Methods

Study 1

We first investigated whether there was a systematic relationship between the time to simulate a spatial episode and the actual time to subsequently navigate the same route. For example, would routes that required more time to navigate be simulated at a proportionately faster rate than one that required less time?

Participants

Sample size was determined by previous research investigating preservation of memory features during mental simulations (Borst & Kosslyn, 2010; Hassabis, Kumaran, & Maguire, 2007a; Schacter et al., 2013). No stopping rule was used. Data from 28 participants were analyzed in Study 1 (13 females, 15 males; mean age = 19.64, SD = 1.87). Data from four other participants were collected but not included in the final sample. Two participants were excluded for responding only on the extreme ends of the post-simulation questionnaire (see Procedure), one participant for mean reaction times (RTs) <1 second on the post-simulation questionnaire, and the fourth did not complete the simulation phase due to feelings of nausea. All participants gave informed consent and the study was approved by the research ethics committee at the University of California Davis.

Procedure

The raw and filtered data for all conditions in this study is available as supplemental material. Additionally, a Jupyter notebook containing the code for the analysis, results, and supplemental materials are available to view and download on Github (http://bit.ly/1Nxok4I) and all test environments used in the study are freely available for research use (http://bit.ly/1OhQVP3). The study consisted of three phases: an exposure phase, an encoding phase, and a simulation phase. All three phases were conducted in a large virtual city designed using Unity (Unity Technologies, San Francisco, United States). The virtual city consisted of five visually salient landmarks located throughout the environment and a number of non-discreet buildings that were variations of three architectural styles designed to provide limited environmental location information (see Fig. 1 for overview of task and city composition). The layout of the city was selected to be slightly asymmetric so that global cues from the city shape would help prevent participants from feeling lost without providing overt geometric cues about their location.

Fig. 1.

Overview of task design and city composition. (A) Outline of the path passively viewed from the first-person perspective during the exposure phase with the location of the five target landmarks. (B) Street level view of the encoding phase showing non-landmark buildings and the facades of the five target landmarks. Note that the green arrow pointing to the target landmark was only displayed on screen if more than 90 seconds elapsed during the route. (C) Task order for a single trial of the simulation phase. Two second interstimulus intervals (ISI) showing a white fixation cross were inserted between each task component (PostSQ: post simulation questionnaire; PostNQ: post navigation questionnaire).

In the exposure phase, participants were shown a video from a first person perspective of movement along the perimeter of the city. See Fig. 1A for a visualization of the exposure path participants were shown. The video stopped for 5 seconds at each target landmark (e.g. the grocery store) to show its location. At this point, the experimenter pointed to the location of the landmark on the screen and verbally confirmed that the participant could identify the landmark. See Fig. 1B for images of the target landmarks. The video ended at the same point it started, which was a randomly selected point along the city perimeter and was identical across participants. The exposure phase was designed to provide a sense of scale of the city, as well as provide a consistent base knowledge of the landmark locations across participants.

After the exposure phase, participants underwent the encoding phase. There were a total of 20 trials in this portion of the study. For each trial, participants were cued with an image of a landmark on the center of the screen and asked to rate on a scale of 1-5 their confidence in knowing the location of the landmark (1: not at all; 5: very well). After, they were instructed to find the target landmark as quickly as possible. To navigate, the participants used the four keyboard arrows, which were mapped to forward/backward movement and left/right rotation. Once the target landmark was located, participants walked toward the face of the landmark, which then prompted the next trial. The 20 trials consisted of each pairwise route between the five landmarks in both directions (e.g. for the diner-café pair, the diner would act as the starting point and destination once).

While it was possible to navigate each route between landmarks using the perimeter path shown in the exposure phase, the city was specifically designed to have numerous shortcuts between landmarks that offered more efficient navigation. Shortcuts included large alleyway openings between series of buildings, as well as occluded narrow pathways. We also included a helper arrow, which displayed onscreen after 90 seconds had passed on each trial if the participant had not located the target landmark (see Fig. 1B). The arrow pointed in the direction of the target landmark, but did not indicate a specific path to take. The helper arrow was included based on results from an initial pilot study that showed trials where participants took longer than 90 seconds frequently resulted in getting lost. Optimal route time between each landmark pair was calculated by taking the quickest possible path between landmarks using available shortcuts (mean optimal time = 24.85 seconds, SD = 7.15; mean number of turns = 5.9, SD = 2.33). These values were identical across all participants.

Once the encoding phase concluded, participants completed the simulation phase. Prior to starting this phase, participants were given detailed instructions on the task, as well as completing two practice trials during which they could ask the experimenter to clarify any of the instructions. Fig. 1C provides an overview of the experimental design for this phase. Participants were told that the simulation portion would be cued with the word ‘Simulation’ on the center of the computer screen. Afterwards, they would be shown two images of the five target landmarks – one on the left and one on the right. Once the landmarks disappeared from the screen, they were instructed to close their eyes and mentally simulate in as much detail as possible moving through the city from the landmark on the left to the one on the right. The experimenter emphasized that it was important to mentally immerse themselves in the city and to take as much time as they needed to properly navigate the route. Participants were instructed to mentally navigate the quickest route between landmarks rather than trying to specifically recall the route they had previously taken.

Of critical importance here, participants were not instructed to simply try and remember their initial route between landmarks in the encoding phase. The reason for this is twofold. First, routes between landmarks in the encoding phase occurred with different levels of environmental familiarity due to their place in the trial order. As such, simple replay of past experiences during simulations are not in all cases representative of the fastest possible routes between two landmarks. Second, we are interested here in predictive simulations rather than memory replay. Simulations allow participants to incorporate spatial information they've learned throughout the experiment rather than trying to recall specific instances of an episode.

Following the mental simulation of a route, participants were placed directly in front of the starting landmark and had to navigate to the target landmark as quick as possible. Employing route navigation after the simulation helped maintain the relative novelty of each simulated route, allowing us to control for variation in differences in environmental knowledge between the simulations and route navigation that otherwise would have occurred if navigation preceded simulation. It also allowed for an immediate assessment by the participant of how well they were able to simulate/predict the route in question. Finally, it allowed us to assess the efficacy of simulation in terms of its direct impact on navigating a route. After the experimenter provided the instructions, participants completed two practice trials of the simulation phase to ensure they understood the instructions and felt comfortable performing the task. Neither of the routes used for practice were included in the actual simulation phase and no data from the practice trials were analyzed.

After the two practice trials were successfully completed, the simulation phase began. As previously outlined, each trial began with the word ‘Simulation’ displayed on the screen for two seconds. Next, images of the starting and target landmark appeared for three seconds. Immediately afterwards, the screen turned to black and the participants closed their eyes to mentally simulate the route. The experimenter monitored performance to ensure that participants completed the simulations with their eyes closed. Once simulated, the participants opened their eyes and pressed a keyboard button that prompted a 14 item post-simulation questionnaire (PostSQ). The PostSQ included items modified from the Memory Characteristics Questionnaire (M. K. Johnson, Foley, Suengas, & Raye, 1988), as well as novel items, and was intended to probe qualitative aspects of the simulation experience. This included questions about spatial and temporal coherence, vividness, fractionation, confidence in knowing the starting/target locations, and perceived accuracy of their memory for the route. Each item was rated on a scale of 1-5. Immediately following the questionnaire, participants were placed within the virtual city facing the starting landmark and navigated to the target landmark as quick as possible. Once there, a post navigation questionnaire (PostNQ) was displayed where they rated two items on a scale of 1-5 assessing how well they were able to simulate the route and how well the simulation matched their navigation experience. Table 1 outlines the wording for each question/response and how they were grouped for analysis. In total, 10 routes were included in the simulation phase. The starting-destination landmark pairs were pseudo-randomly selected such that each of the five landmarks were included as a starting point and destination only once.

Table 1.

Item list for post-simulation questionnaire (PostSQ) and post-navigation questionnaire (PostNQ)

Post-Simulation Questionnaire

Vividness

My memory for this route is (1: sketchy – 5: very detailed)

*My memory for this route is (1: entirely in color – 5: black and white)

My memory for this route involves visual detail (1: little or none – 5: a lot)

Overall vividness of this route is (1: vague – 5: very vivid)

My memory for this route is (1: dim – 5: sharp/clear)

When imagining this route, it was so vivid I felt I was actually navigating it (1: not at all 5: a great deal)

Spatial Coherence

*The relative spatial arrangements of buildings along the route is (1: clear/distinct – 5: vague)

Temporal Coherence

The order of buildings along the route is (1: confusing – 5: comprehensible)

Fractionation

Simulating the route was like watching a movie in my mind's eye (1: not at all – 5: very much)

The route was a collection of separate images (1: very much – 5: not at all)

Simulation Confidence

I have doubts about the accuracy of my memory for this route (1: a great deal – 5: no doubts)

Other

The route seems (1: long – 5: short)

*My memory of the starting location for this route is (1: clear/distinct – 5: vague)

My memory for the destination location for this route is (1: vague – 5: clear/distinct)

Post-Navigation Questionnaire

Post Route Accuracy

My memory for this route matched my experience (1: not at all – 5: very well)

I was able to mentally simulate this route (1: not at all – 5: a lot)

^*indicates response that was inverted prior to analysis

Results

Inspection of the route time histogram from the simulation phase revealed a number of trials in which participants became “lost” (see Figure S1), which skewed the distribution. To control for this, we calculated the difference between the optimal route time and the observed route time (mean difference score = 14.27 seconds; SD = 19.49). The resulting distribution was then used to remove trials in the top 25% of difference scores across participants (75% quartile = 21.66 seconds, 70 trials removed). This filtering strategy allowed for retention of variance in route time, while excluding trials that took approximately double the optimal route time (M = 24.85 seconds, SD = 7.16). See supplemental sections 1.2.3-1.2.4 and Figure S2 for a comparison using an alternative filter of 3 standard deviations. Route performance on the filtered data set of 210 trials was near the optimal route time (mean difference score = 5.93 seconds, SD = 3.54).

Next, we assessed the relationship between simulation time (M = 14.41, SD = 11.21) and route navigation time (M = 35.81, SD = 9.25). Simulation times were first mean-centered for each participant, providing a more precise estimate of coefficients as it minimizes variance in simulation time due to individual differences in overall temporal compression rate. We found a statistically significant positive correlation between the time it took a participant to subsequently navigate the route and the time it took them to mentally simulate it (r(208) = 0.30, p < 0.001, R² = 0.09, Fig. 3C). We also found a significant positive correlation between simulation time and route distance (r(208) = 0.29, p < 0.001, R² = 0.08); however, route time and distance for each trial were highly collinear (r(208) = 0.97, p < 0.001, R² = 0.94). As such, the remainder of the analysis focuses on route time, which accounts for variance in non-movement related processes (e.g. making decisions at turning points) that are not represented in the distance measure. Our correlation reported here between simulation and route time is consistent with past findings from Kosslyn, Ball, and Reiser (1978) who showed a correlation between the time it took participants to mentally scan between different locations on a map of an island and the physical distance between them.

Fig 3.

Regression plots showing the relationship between mean centered simulation times and the predictor variables for each movement condition. (A) Path time for the slow movement condition (b = 0.17, t(191) = 4.78, p < 0.001). (B) Simulation confidence (see Table 1) for the slow condition (b = −0.86, t(191) = −2.02, p = 0.04). (C) Path time for the medium movement condition (b = 0.19, t(207) = 4.34, p < 0.001). (D) Vividness for the medium movement condition (b = −1.12, t(207) = −2.94, p = 0.004). (E) Path time for the fast movement condition (b = 0.14, t(191) = 3.34, p = 0.001). (F) Spatial coherence (see Table 1) for the fast movement condition (b = −0.88, t(191) = −3.06, p = 0.003). Shaded blue area represents 95% confidence interval.

We also tested our first hypothesis that simulation times would preserve the temporal component of the route in a compressed form. We did this by calculating the temporal compression rate as the ratio of simulation time to route time for each trial. This resulted in a mean ratio of 0.42 (SD = 0.33, Fig. 2C), supporting our hypothesis that temporal information is contained in mental simulations in a compressed form. This ratio indicates that participants on average mentally simulated routes at approximately 2.39 (SD = 3.04) times the rate that they subsequently navigate them.

Fig. 2.

Comparison of simulation and navigation performance between movement speed conditions (slow: 4 virtual m/s, medium: 6 virtual m/s, fast: 8 virtual m/s). Error bars represent 95% confidence intervals. (A) All groups showed statistically significant differences in subsequent route navigation time. (B) Simulation times in the slow and medium groups differed from the fast group, but not between the slow and medium groups. (C) The slow movement group had a statistically faster compression rate than the medium and fast groups. Note that p-values use the Bonferroni correction for multiple comparisons.

Study 2

We introduced two new conditions that varied movement speed during the encoding phase. As previously outlined, if temporal compression is adaptive as we hypothesized (our second hypothesis), the compression rate should increase when the original experiences are slowed down and decrease when the original experiences are sped up.

Participants

Data from 26 participants (16 females, 10 males; mean age = 20.19, SD = 1.58) were included in the slower movement condition and from 26 participants (13 females, 13 males; mean age = 20.92, SD = 1.92) for the faster movement condition. For the slower condition, two participants were excluded for answering on the extreme ends for each item in the post-simulation questionnaire (mean RTs < 1s), one for skipping through the simulation portion (mean RT < 3s), and another for not finishing the simulation phase due to nausea. For the faster condition, two participants were excluded for skipping through the post-simulation questionnaire (mean RTs < 1s), one for skipping through the simulation portion (mean RT < 3s), and one for getting nauseous during the encoding phase of the study.

Procedure

The procedure was identical to Study 1 with the exception of movement speed during the encoding phase. Here, two manipulations were introduced. In the slower movement group, we reduced player movement within the virtual city from the original speed of 6 virtual m/s (equivalent to a 4.47x increase over an average real world walking speed of 1.34 m/s) used in Study 1 to 4 virtual m/s (a 2.98x increase over real world walking speed). In the faster movement group, we increased player movement to 8 virtual m/s (a 5.97x increase over real world walking speed). Participants were not informed that there was a movement speed difference between the encoding and simulation phases.

Results

As in Study 1, we independently processed data from both movement groups by removing trials in the top 25% of difference scores. Simulation times were then mean-centered for each participant. Movement groups were independently filtered due to differences in route time between conditions, which would otherwise bias a condition-independent filtering strategy (slow condition: 75% quartile = 41.66 seconds, 65 trials removed; fast condition: 75% quartile = 18.89 seconds, 65 trials removed). The total remaining trials for each movement condition after filtering was 195. Two trials in the slow condition and one in the fast condition were removed due to negative simulation times that resulted from a software error when writing timing values to text files. Next, we conducted a correlation analysis to investigate the relationship between route time and simulation time for each condition. Replicating the results from Study 1, both slow (r(191) = 0.29, p < 0.001, R² = 0.08) and fast (r(191) = 0.24, p < 0.001, R² = 0.06, Fig. 3A and 3E) movement conditions showed statistically significant positive correlations between mental simulation and navigation time.

Next, we conducted a one-way ANOVA to compare performance between the slow, medium, and fast movement groups on route time, and non-mean centered simulation time of the data collected during the simulation phase, where movement speed was constant across conditions. Non-mean centered simulation times were used for group comparisons because mean-centering simulation times resulted in equivalent average mean-centered simulation times of zero. See Fig. 2A for group performance. There was a significant main effect of route time (F(2,593) = 156.38, p < 0.001, η² = 0.35), suggesting that the time it took to travel a route differed between conditions, consistent with expectations. Post hoc comparisons using a Bonferroni correction showed that the three movement groups all differed from one another: slow vs. medium (mean difference = 12.06 seconds, SEM = 1.03, p < 0.001), slow vs. fast (mean difference = 18.04 seconds, SEM = 1.03, p < 0.001), and medium vs. fast (mean difference = 5.44 seconds, SEM = 1.03, p < 0.001). Simulation time also showed a significant main effect (F(2,593) = 9.78, p < 0.001, η² = 0.03), indicating that the speed at participants encountered spatial information during encoding influenced how quickly routes were mentally simulated. Post hoc comparisons revealed a significant group difference in the slow vs. fast comparison (mean difference = 2.88, SEM = 0.97, p = 0.001) and the medium vs. fast comparison (mean difference = 3.94 seconds, SEM = 0.95, p < 0.001), but not between the slow and medium (mean difference = −1.06, SEM = 0.95, p > 0.250).

To test our second hypothesis that movement speed during encoding relates to differences in temporal compression during mental simulation, we compared the compression rates between groups. The mean temporal compression rates were 0.28 (SD = 0.19) for the slow condition and 0.35 (SD = 0.22) for the fast condition, indicating that routes were temporally compressed at a rate of 3.57 (SD = 5.31) and 2.83 (SD = 4.52), respectively (see Fig. 2C). Next, we assessed the normality of the distributions using the omnibus test measuring skewness and kurtosis (D'Agostino, 1971). Distributions in each of the three conditions were non-normal (slow: D = 50.91, p < 0.001; medium: D = 44.31, p < 0.001; D = 50.93, p < 0.001) and log transformed values did not result in normal distributions (slow: D = 46.25, p < 0.001; medium: D = 96.52, p < 0.001; D = 82.01, p < 0.001). As such, non-parametric tests were carried out on the original compression rates. See Figure S11 for both original and log transformed distributions.

Differences in temporal compression rates were compared using the Kruskal-Wallis H test. There was a statistically significant main effect of condition, H(3) = 21.05, p < 0.001. Post-hoc comparisons using a Mann-Whitney U tests revealed statistical differences in temporal compression rates between the slow and medium conditions (U = 15486, p < 0.001, mean difference = −0.32 seconds, SEM = 0.07, r = 0.2), and the slow and fast conditions (U = 14426, p < 0.001, mean difference = −0.30 seconds, SEM = 0.07, r = 0.19), but not between the medium and fast conditions (U = 19544, p = 0.54, mean difference = −0.03 seconds, SEM = 0.07, r = 0.03). These results support the adaptive hypothesis of temporal compression: the temporal compression rate in the slow condition increased to accommodate the slower movement speed that was experienced during encoding relative to the medium and fast movement conditions. This finding suggests that the temporal dynamics of mental simulations can be ‘sped up’ (i.e. increased in compression) to accommodate larger temporal intervals between features within a spatial environment. Importantly, this cannot be attributed to group differences in simulation time between the slow and medium speed conditions, as they did not statistically differ.

To test our third hypothesis that the fidelity of retrieved memories relates to how effectively participants can simulate using them, we developed independent regression models for each movement group to identify features of the route simulation that explained a statistically significant amount of variance in simulation time. Trial level responses to items in the post-simulation questionnaire were compiled into five dimensions of interest: post-route simulation accuracy, spatial coherence, temporal coherence, fractionation, and simulation confidence (see Table 1). We then entered these, along with subsequent route time from the post simulation navigation, as predictor variables in a stepwise regression using simulation time as the criterion variable.

Table 2 presents the final regression models for each movement condition. For Study 1, this resulted in a regression equation that explained a statistically significant amount of variance in simulation time (F(2,207) = 15.11, p < 0.001, R² = 0.13). Both path time (b = 0.19, t(207) = 4.34, p < 0.001) and vividness (b = −1.12, t(207) = −2.94, p = 0.004) were included in the final regression equation. For the slow condition in Study 2, we also found a regression equation explaining a significant amount of variance in simulation time (F(1,191) = 15.54, p < 0.001, R² = 0.14) with route time (b = 0.17, t(191) = 4.78, p < 0.001) and simulation confidence (b = −0.86, t(191) = −2.02, p = 0.04) as significant predictor variables. For the fast condition, we identified a significant regression equation (F(2,191) = 11.48, p < 0.001, R² = 0.11) with route time (b = 0.14, t(191) = 3.34, p = 0.001) and spatial coherence (b = −0.88, t(191) = −3.06, p = 0.003) as significant predictor variables. Vividness and spatial coherence were correlated in both the medium and fast conditions (medium: r(208) = 0.61, p < 0.001; fast: r(191) = 0.42, p < 0.001), suggesting that each variable is approximating similar representational features of the simulation experiences. Considered together, these findings show that the vividness and spatial coherence of mental simulations explained a significant amount of variance in simulation time for the medium/fast movement conditions over and above the variance explained by the time taken to navigate the route. Interestingly, higher vividness and spatial coherence ratings predicted faster simulation times, suggesting that the fidelity of the simulation relates to the efficiency of its recapitulation.

Table 2.

Statistically significant coefficients for final regression equations in each movement condition.

		B	Std Error	Beta	t-value	p-value
Slow	Constant	−4.61	2.84		−1.62	.106
	Path Time	0.17	0.04	0.33	4.78	.000
	Simulation Confidence	−0.86	0.43	−0.14	−2.02	.044
Medium	Constant	−2.46	2.24		−1.01	.273
	Path Time	0.19	0.04	0.28	4.34	.000
	Vividness	−1.12	0.38	−0.19	−2.94	.004
Fast	Constant	−1.22	1.71		−0.72	.474
	Path Time	0.14	0.04	0.23	3.34	.001
	Spatial Coherence	−0.89	0.29	−0.21	−3.06	.003

Lastly, we investigated whether the time taken to simulate a route was related to how efficiently participants subsequently navigated that route. This addresses an important question about whether mental simulations offer a benefit to subsequent navigation irrespective of how well spatial information is reinstated, or whether variance in mental simulation speed relates to navigation efficiency. Specifically, if temporal compression is adaptive, as hypothesized, then we would predict that more compressed simulated routes would lead to faster subsequently navigated routes.

To test this, we analyzed partial correlations from the simulation phase looking at the relationship between mean centered simulation time and excess path time, which was calculated as the difference between the observed and optimal path time for each route. To ensure that the effect was not confounded by the fact that some trials involved shorter vs. longer paths, we controlled for the optimal path time. See Fig. 4 for scatterplots of the results. Across all three movement conditions, there was a statistically significant positive correlation when controlling for the optimal path (r (593) = 0.13, p = 0.001, R² = 0.02). This relationship was also statistically significant for each of the three movement groups independently (slow: r (190) = 0.16, p = 0.026, R² = 0.03; medium: r (207) = 0.23, p = 0.001, R² = 0.05; and fast: r (190) = 0.22, p = 0.003, R² = 0.05). These results show that in all movement conditions, faster simulations relate to how effectively participants subsequently navigate a route. Considered with the results from the regression analyses, this suggests that participants were better at navigating on trials in which they were able to quickly reinstate and combine spatial information during the mental simulation.

Figure 4.

Partial correlation plots showing the relationship between excess path time and mean centered simulation time for each movement condition and all conditions combined.

Discussion

Mental simulation offers a window into the mind, providing scientists with a powerful means to develop mechanistic models of memory by studying the top-down flow of information processing (Lisman, 2015). To date, the temporal dynamics of mental simulations are unknown, despite representing a unique component of memory and prospection that can provide insight into how features of memories are organized and used to anticipate future scenarios. Here, we show that the flow of mental simulations preserves temporal information from previous experiences in a compressed form, adapts its compression rate to accommodate differences in the rate of exposure to visual stimuli during memory formation, and quantifies how memories of previous experiences are flexibly recombined during prospection.

In Study 1, we found a correlation between the time to simulate movement along a route and the time to subsequently navigate it. This finding extends classic findings on mental simulations (Kosslyn et al., 1978; Shepard & Metzler, 1971) by showing that temporal features, in addition to spatial features, are preserved during simulation. Unlike spatial features, temporal extent is recapitulated in a compressed form, occurring at a rate increase of 2.39x. This finding supports theories on the adaptive basis of memory (Schacter et al., 2007) by showing that simulations are ‘sped up’ to allow more predictive simulations per unit of time than simply replaying previous events at the speed they were originally experienced.

In Study 2, we introduced two new conditions that varied movement speed. We reasoned that if temporal compression during simulation is adaptive, the rate should vary to accommodate differences in the speed at which spatial stimuli was encoded. Our rationale for this hypothesis is that during mental simulation, there should be a beneficial aspect to having the compression rate adapt or change in response to how quickly spatial features of an environment were originally experienced. For example, it is advantageous to have faster simulations in the case of slower movement speeds so that a simulation can be experienced, and predictions based on it can be formulated, with a smaller time requirement. Otherwise, slow navigation would take disproportionately longer to mentally simulate than faster navigation.

Our findings, which showed that the compression rate changed from 2.39x to 3.57x in the medium vs. slow speed conditions, despite a lack of difference in mental simulation times between those conditions, supported the adaptive hypothesis. However, the compression rate between the medium and fast condition did not statistically differ despite differences in simulation times. An intriguing possibility for this pattern of results is that a compression rate ~2.5x may provide the optimal balance of visual cohesion, retrieval accuracy, and capacity to make inferences from the simulation given the spatial stimuli and environmental scale used in this study. That is, there may be a cost to the predictive value of a simulation when lowering the compression rate from this point. An important direction for future research is to fully delineate the interactions between spatial features of temporally extended experiences and how they interact with the compression of a related simulation.

The compression rate observed in this study approximates the 4-8x rate during pre-play of place cell sequences in area CA3 of the rodent hippocampus at spatial decision points (A. Johnson & Redish, 2007). This finding suggests a possible common neural mechanism for encoding and simulating spatiotemporal contexts, although it should be noted that the link between hippocampal neural replay/preplay in rodents and visually-guided mental prospection in humans remains unclear. Interestingly, the observed rate differs from a recent report on temporal compression of real world routes (Bonasia, Blommesteyn, & Moscovitch, 2015), which observed a rate of 32.8x. The difference in compression is likely based on the researcher's use of highly familiar real world routes, which had been experienced over several years, resulting in replay of well learned spatial memories that are partially semantic (Moscovitch et al., 2005). Their use of highly familiar routes limits their extension to theories of prospective memory, which focus on how memory traces are combined to make predictions about future behavior rather than memory replay. In contrast, our study employed a novel environment and task design that required subjects to combine features for past experiences into new predictions, allowing us to make more direct inferences about the constructive basis of prospection.

The constructive perspective of memory posits that memory retrieval is not a binary operation (i.e. a memory is either reinstated or not) (Hassabis & Maguire, 2007; Schacter, 2012). Rather, it is viewed as an active process whereby features of past experiences recombine into a task-oriented representation. While spatial detail generation has provided an effective measure to test the constructive functioning of memory (Addis et al., 2008; Burgess, Becker, King, & O'Keefe, 2001; Cui et al., 2007; Hassabis, Kumaran, & Maguire, 2007a), our third hypothesis asked whether temporal information could also inform how memory traces are combined during prospection. We investigated this by identifying experiential aspects of simulations that explained variance in simulation time. Simulation vividness in the medium condition and spatial coherence in the fast condition were predictive of simulation times, with faster simulation times relating to more vivid and spatially coherent simulations. These findings indicate that the fidelity of mnemonic representations relates to how efficiently they can be used for simulations.

Our findings also suggest that retrieval of navigationally relevant memory traces is not necessarily a binary process, but one that occurs by combining spatial features over time. This supports views that environments are represented by a manifold of spatial maps (Derdikman & Moser, 2010; Han & Becker, 2014; Mou & Wang, 2015) rather than one unitary spatial map. In the case of unitary spatial maps, simulation speeds should not vary with spatial coherence and vividness, as the same representation is being used to make predictive inferences in all cases. However, if environments are represented as a manifold of maps, simulation speeds are expected to vary as a function of how well different spatial representations can be flexibly combined. Our data show that simulation speeds decrease when they are more vivid and spatially coherent, indicating that more effective integration of spatial representations results in higher fidelity of mental simulations.

Importantly, we also found a behavioral advantage to faster simulations, showing that the faster people are able to simulate a route, the quicker they are at subsequently navigating it. Considered with the results linking memory fidelity with mental simulations, this suggests that the more readily spatial features of an environment can be integrated into a route representation during mental simulation, the more effective they are at subsequently navigating using that route. Spatial memory fidelity may therefore mediate the relationship between adaptive mental simulations and navigation, where better fidelity during memory reinstatement leads to more efficient mental simulations and wayfinding. An important future direction will be to further investigate how these spatial features are combined during mental simulations, the integration of features across multiple episodes, and how these processes relate to the temporal dynamics of a mental simulation.

Although there is much to learn about the relationship between temporal compression and memory function, our study demonstrates that temporal compression of route memories in humans exists behaviorally, which may have parallels to a similar neural mechanism in rodents. Our findings further demonstrate that participants’ compression rates adapt in response to the speed at which the spatial features of an environment were first experienced. We suggest that variance in temporal compression during mental simulations offers a novel metric to investigate boundary conditions for encoding and representing spatial features in memory and how they combine during memory reinstatement and navigation. Adaptive temporal compression also offers a new metric to understand how memory function – both during the encoding and reinstatement of features from memory – changes in the context of aging, as well as in response to neurocognitive disorders such as mild cognitive impairment, Schizophrenia, Alzheimer's Disease, and forms of topographical disorientation (Iaria et al., 2014).