Episodic recombination and the role of time in mental travel

Abstract

Mental time travel is often presented as a singular mechanism, but theoretical and empirical considerations suggest that it is composed of component processes. What are these components? Three hypotheses about the major components of mental time travel are commonly considered: (i) remembering and imagining might, respectively, rely on different processes, (ii) past- and future-directed forms of mental time travel might, respectively, rely on different processes, and (iii) the creation of episodic representations and the determination of their temporal orientation might, respectively, rely on different processes. Here, we flesh out the last of these proposals. First, we argue for ‘representational continuism’: the view that different forms of mental travel are continuous with regard to their core representational contents. Next, we propose an updated account of episodic recombination (the mechanism generating these episodic contents) and review evidence in its support. On this view, episodic recombination is a natural kind best viewed as a form of compositional computation. Finally, we argue that episodic recombination should be distinguished from mechanisms determining the temporal orientation of episodic representations. Thus, we suggest that mental travel is a singular capacity, while mental time travel has at least two major components: episodic representations and their temporal orientation.

This article is part of the theme issue ‘Elements of episodic memory: lessons from 40 years of research’.

1. Introduction

Mental time travel—the capacity to mentally project oneself to the past and the future—is a highly complex cognitive activity [1,2]. It involves, among other things, the generation of mental images [3], representations of spatial and temporal relations [4], the retrieval of semantic and episodic information [5], metacognitive monitoring [6] and a conceptual understanding of possibility and time [7]. It is no surprise then that there has been considerable debate about the correct cognitive ontology of mental time travel [8]. While it is uncontroversial that mental time travel relies on various other cognitive capacities like working memory and metacognition (e.g. [9]), the extent to which these components are best conceived of as a singular mechanism remains controversial. Is mental time travel really a single cognitive capacity or is it better analysed in terms of various independent capacities working in concert? Most commonly, this question is taken to be asking whether mental time travel should be analysed into two separate natural kinds: episodic memory and imagination.

In particular, recent debate in the philosophy of memory has produced two opposing views about the relationship between episodic memory and imagination: continuism and discontinuism (for review, see [10]). On the one hand, continuism argues that there is no essential cognitive difference between episodic memories about the past and imaginations about the future apart from their differing temporal orientations (e.g. [11,12]). On the other hand, discontinuism argues that episodic memory and imagination are best viewed as two fundamentally different capacities (e.g. [13,14]).

Different versions of (dis)continuism have emphasized different aspects of mental time travel as essential for deciding whether it is best thought of as one or multiple capacities. Process (dis)continuism (e.g. [11,12,15–18]) focuses on the question of whether the retrieval processes involved in memories for the past are different in kind from those involved in imagination. In contrast, attitudinal (dis)continuism is concerned with the question of whether memorial and imaginative forms of mental time travel involve different propositional attitudes (e.g. [19–23]). One way to understand this latter concern is as asking to what extent episodic memory and imagination are subject to different metacognitive processes that allow them to play different functional roles in the mind [24,25]. Thus, in its most prominent guise, the debate between continuism and discontinuism is concerned with the correct analysis of human mental time travel into different component processes.

Here, we will propose an account of the cognitive architecture of mental time travel that suggests an alternative continuist view—namely, representational continuism. Representational continuism claims that—regardless of other differences or similarities between episodic memory and imagination—all forms of mental travel trade in a distinct kind of representation—namely, episodic representations. In what follows, we will first provide an exposition of representational continuism (in §2) before diving deeper into the two main distinctive features of episodic representations: their process of construction (i.e. episodic recombination; §3) and their relationship to spatio-temporal information (§4).

2. What is representational continuism?

Representational continuism is the view that episodic memory and imagination are continuous cognitive capacities in so far as they utilize the same kind of mental representation: episodic representations. Thus, representational continuism is committed to two claims: first, episodic (re)combination (the process underlying the generation of episodic representations) is a natural kind. Second, there is a real distinction between episodic and semantic contents in the mind. Thus, on this view, episodic representations are distinctive both in terms of the process through which they are constructed and in terms of their contents.

First, episodic representations are distinctive in terms of their process of construction because they are the result of the (re)combination of stored elements of previous experiences ([26]; we elaborate below on our conceptualization of this process). Importantly, this point is compatible with the idea that memorial and imaginative episodic construction processes typically function in slightly different ways. For example, episodic combination might be commonly guided by a specific or unique memory trace in the case of episodic memory but not in the case of episodic imagination (e.g. [17]). However, these potential differences do not license a distinction between memorial and imaginative forms of episodic (re)combination. In particular, a given episodic representation might be experienced as a memory or an imagination irrespective of whether the underlying construction process was guided by a memory trace. Thus, we deny that mental time travel should be analysed in terms of a distinction between memory and imagination. Instead, mental time travel is a unified mental capacity in so far as it is concerned with the construction of a distinctive kind of mental content: episodic representations.

Second, episodic representations are distinctive in terms of their contents because they have eventive perceptual elements including spatial and temporal information [27,28]. Crucially, spatio-temporal elements in episodic representations are non-conceptual (e.g. [29]). Only episodic but not semantic representations have temporal and spatial contents of this kind, even though episodic construction processes draw on semantic information [5,30]; and episodic representations can turn, over time, into semantic representations [31]. Of course, as we will argue below, episodic contents are not wholly non-conceptual and their non-conceptual elements are, moreover, available for recombination themselves. While there are likely other differences between episodic and semantic contents, the non-conceptual representation of certain spatial and temporal relations makes episodic contents minimally distinctive.

One crucial corollary of the view we are defending here is that mental travel is essentially episodic but not essentially temporal (or, for that matter, essentially mnemonic [32]). In other words, mental travel constructs event representations irrespective of the time they occupy. The construction of episodic contents is neither necessary nor sufficient for their experience as ‘future’ or ‘past’. Instead, a given instance of mental travel becomes an instance of mental time travel only through the additional involvement of mechanisms for temporal reasoning. In this respect, what we propose is aligned with those who view episodic memory and/or imagination as essentially atemporal (e.g. [12,33–35]). A creature might be capable of mental travel without being capable of mental time travel. This stands in contrast to those who have argued that episodic memory and/or imagination are intrinsically connected to temporal cognition (for review see [33]), for example, by enabling a conceptual understanding of linear time (e.g. [36]) or enabling a conceptually rich form of foresight (e.g. [37]).

Representational continuism is the view that episodic representational contents and the cognitive machinery underlying their construction constitute a natural kind. Thus, mental travel is a unitary capacity in so far as it is concerned with episodic representations. Beyond that, however, we remain agnostic with regard to the question of whether treating episodic representations thus understood as a unitary natural kind suffices for treating memory and imagination as a single natural kind as well or how one should make that decision. Therefore, representational continuism is compatible with various forms of discontinuism about episodic memory and imagination. In particular, our view is compatible with those versions of discontinuism that grant that both episodic memory and imagination rely on the same kind of retrieval process and that accept a distinction between episodic and semantic representations.

First, there might be process discontinuists who accept that episodic (re)combination is a single natural kind but insist that it produces representations of different kinds depending on whether it relies on the retrieval of a memory trace or not (we read e.g. [17] along these lines). Such a view would be compatible with representational continuism in so far as it accepts a distinction between episodic and semantic representational contents and treats episodic retrieval as a unitary process. Second, representational continuism is compatible with attitudinal discontinuism about episodic memory and imagination: episodic representations can be entertained under various attitudes that might determine their functional roles as memories or imaginations irrespective of how they were constructed [21,24,38]. For example, one might hold that episodic remembering and imagining end up being separate natural kinds owing to the differential involvement of various downstream metacognitive processes determining the ‘mnemicity’ of the episodic representation [32]. Again, such a view could be discontinuist about memory and imagination while being continuist about the representational machinery involved in generating their respective representational contents. Finally, on some readings of the (dis)continuism debate, it does not turn on an empirical description of the mechanisms underlying remembering and imagining. Instead, one might think that the debate concerns the normative or prescriptive question of when we should call a mental state a memory or an imagination [39,40]. Since representational continuism is a view about the actual cognitive machinery underlying episodic cognition, it is compatible with the idea that different norms should govern usage of the terms ‘remembering’ and ‘imagining’.

In the remainder of this article, we will proceed as follows. In the next two sections, we will describe and review evidence regarding episodic (re)combination [26]—the cognitive process through which episodic representations are constructed. Then, in the final sections, we will discuss our view of the temporal features of episodic contents as well as their relationship to subjective time.

3. What is episodic (re)combination?

On the view we are proposing, all forms of mental travel rely on a distinctive process of construction: episodic (re)combination. Episodic combination refers to the retrieval of stored (conceptual and non-conceptual) elements of previous experiences and their combination into a coherent event representation, which can result in either a remembered or imagined experience [26]. From this perspective, episodic remembering is not the result of the direct retrieval of entire past experiences stored (e.g. via a memory trace) but of the principled composition of stored elements of such experiences and their relations (see also [41]). In other words, episodic (re)combination is an instance of compositional computation (see also [42]). This formulation of the idea extends earlier ones that emphasized the primacy of episodic memory retrieval for episodic imagination [26] or the primacy of association as the only principle of composition ([12]; see [43] for criticism). Instead, we emphasize that this process of retrieval and composition supports both memory and imagination, but will commonly be guided by different principles in each case. On the one hand, in the case of memory, the composition of episodic elements is likely guided primarily by ‘bottom-up’ processes that do not merely rely on associative strength but also on a hippocampal ‘index’ [44] pointing to a composition of elements encoded during experience and knowledge structures such as schemas and scene grammars (e.g. [45] across a hierarchy of representations [46,47]. In the case of imagination, on the other hand, recombination will commonly primarily be guided by a high-level description of the to-be-constructed event determining what composition of elements satisfies the retrieval target [20].

One virtue of this account is that it explains various apparent discontinuities between episodic memory and imagination. First, it explains why imaginative construction has different phenomenological and neuro-cognitive properties from memorial construction [43]. Second, it explains the fact that imagination but not memory seems to be productive and flexible/systematic (a circumstance that has received surprisingly little attention in the discontinuism debate; though see [48] for relevant discussion). That is, in imagination but not memory, a finite set of composable elements seem to enable an infinite set of possible expressions (productivity) and elements can be recombined to produce new expressions (systematicity). For example, one can imagine but not remember both Napoleon riding a camel on the Moon or a camel riding Napoleon on the Moon. ¹ Productivity and systematicity are commonly pointed to in motivating the idea that human thought shares central features with language [50,51]. Both language and thought, on this view, are systems that produce complex semantic expressions through the composition of simple constituents according to syntactic rules. Similarly, imagination might achieve productivity and flexibility by being the result of the combination of simple constituent episodic elements into complex expressions (see also [42,52]). ²

However, on the account proposed here, the discontinuity between episodic memory and imagination in terms of productivity and flexibility is not due to a discontinuity in their compositional representational structure but due to the principles guiding episodic (re)combination in either case. Both episodic memories and imaginations are the result of the composition of episodic elements according to principles ensuring that the resulting expressions are well formed [56]. Presented in this way, the idea of episodic recombination remains agnostic towards the format of the episodic elements being retrieved as well as the exact algorithms involved in encoding and recombining these elements. Nonetheless, it provides various constraints for explanations of these more specific questions. First, whatever the nature of stored episodic information, it needs to be amendable to principled composition—i.e. consist of discrete, simple elements that can be composed into meaningful, more complex expressions (see e.g. [57]). These expressions need to be of a kind that can be ‘read’ by modality-specific imagery generation processes. We take it that the evidence reviewed below indeed suggests that episodic construction processes include discrete elements such as ‘location’, ‘person’ and ‘object’ and that the same elements can occur in both episodic memories and imaginations. However, the exact nature of these elements (e.g. how many kinds there are and whether they share a representational format) remains unclear so far. Second, in addition to distinct representations for locations, persons and objects that can be composed into meaningful expressions, representations determining the ‘roles’ of such elements within a given expression need to be available. For example, Hafri & Firestone [58] recently argued that perceptual processes explicitly represent physical relations like containment (see also [59,60]), eventive relations like launching, and even social relations like helping (see also [61]). Furthermore, Hafri et al. [62]. showed that people spontaneously and rapidly extract event roles (e.g. ‘agent’ and ‘patient’) from the perception of visual scenes. From the perspective proposed here, it is likely that relations represented in perception are also encoded into long-term memory and thereby available for episodic recombination. More generally, our proposal coincides with recent accounts of compositional computation across cognition [42,52,63,64]. For example, it has been variously suggested that (visual) perception exhibits a compositional semantics and combinatorial syntax [54,59,60,65–67], similar to what we are suggesting here to be the case for episodic representations. Even more closely related is a recent proposal by Kurth-Nelson et al. [42], who argue that hippocampal replay instantiates a compositional semantics and a combinatorial syntax along the same lines we are proposing for episodic thought. On their view, replay sequences are the result of the composition of sensory-specific entities according to abstract roles that are not constrained solely to the spatial domain (see, e.g. [68], for evidence for non-spatial role representations in the human hippocampus). While these proposals remain speculative, we take them to be productive starting points for a research program that explores the extent to which episodic cognition indeed is best described as a form of compositional computation.

4. The experimental history of episodic (re)combination

The concept of episodic recombination emerged from an attempt to integrate research on the role of episodic retrieval processes in allowing individuals to imagine or simulate experiences that might occur in their personal futures with research on constructive processes that result in memory errors and distortions [26]. Neuroimaging studies had revealed striking similarities in brain regions that show increased activity when people remember past experiences and imagine future experiences [69–71]: future imagining recruits a core network of brain regions that overlaps closely with the well known default mode network (for a meta-analysis, see [72]). In addition, behavioural studies pointed to similar phenomenal characteristics associated with remembering the past and imagining the future (e.g. [73]). Arguing that these and related findings indicated a role for episodic retrieval in imagining future experiences, Schacter & Addis [26] also made a conceptual link to research on constructive memory processes dating to Bartlett [74] and others, contending that a constructive episodic memory system ‘can draw on elements of the past and retain the general sense or gist of what has happened. Critically, it can flexibly extract, recombine and reassemble these elements in a way that allows us to simulate … events that have never occurred previously in the exact form in which we imagine them [26, p. 778]’. The downside of such a system, they argued, is that the same flexible episodic recombination process that is adaptive for purposes of simulating future events can produce memory errors that result from miscombining elements of past experiences. Importantly, subsequent attempts to explore and test this constructive episodic simulation hypothesis required a method to gain control over how people recombine elements of past experiences to simulate future and other hypothetical experiences. In this section, we review cognitive and neuroimaging studies that have used controlled methods to examine episodic recombination processes in the context of research on episodic simulation.

(a). The experimental recombination task

In an initial attempt to gain control over episodic recombination processes, Addis et al. [75] developed a procedure they referred to as experimental recombination of event details. The main objective of this fMRI study was to determine whether the striking similarity in brain activity observed when people remembered past experiences and imagined future experiences could also be observed when people imagined event details that were experimentally recombined from different past episodes. In earlier fMRI studies where participants were given open-ended instructions to imagine a future experience in response to word cues [69,71], they might have simply ‘recast’ an entire past episode that actually happened into the future, thus accounting for the similarity in brain activity associated with remembered past and imagined future events. To ensure episodic recombination, prior to scanning Addis et al. required participants to recall past episodes, providing an event title as well as a critical person, location and object from the event. The experimenters then randomly recombined elements across events to ensure that participants simulated a novel event during scanning. So, e.g. during scanning participants might have been asked to imagine a future event involving Cathy (from a memory titled Meeting Cathy), Harvard Yard (from a memory titled Graduation Day) and a ring (from a memory titled Ring for Xmas). Brain activity under these conditions of experimental recombination was compared with activity when participants remembered actual past events in response to person, place and object cues. Importantly, the same core network of regions associated with future imagining in previous studies showed robust activity, as well as extensive overlap with brain activity during the memory task, thus confirming the involvement of episodic recombination processes in eliciting this activity. In addition, Addis et al. identified several brain regions that were differentially associated with remembering past experiences versus imagining future experiences, in line with earlier observations that the two forms of episodic processing can be distinguished, despite numerous similarities ([69]; refer also [8,72]).

The experiment by Addis et al. [75] also addressed a related issue that is central to a conceptual concerns of this article: whether the activity observed during future imagining is specifically linked to the prospective temporal direction of this task or whether the same or similar activity occurs when participants imagine past events, i.e. imagine an event that might have occurred in their personal past in response to person–place–object cues. Imagining past events recruited for the most part the same network of regions associated with imagining future events, thus reflecting the operation of episodic recombination processes independent of temporal direction.

Addis et al. [76] conducted a closely related behavioural study using the same experimental recombination task to examine age-related changes in episodic remembering and imagining. In a previous experiment in which participants were given word cues and asked to remember past events or imagine future events, Addis et al. [77] replicated earlier findings of age-related declines in the number of episodic details produced by older compared with younger adults when remembering past events [78] and extended this finding to imagined future events. Noting that this latter finding might reflect an age-related deficit when ‘recasting’ a remembered past event into the future, Addis et al. [76] documented that older adults show a similar reduction in episodic details when imagining a future event in response to recombined person–place–object cues. Moreover, this age-related deficit occurred to the same extent when young and older adults imagined a past event in response to recombined cues. These findings thus complement the data from the Addis et al. [75] fMRI study in that they document the operation of an episodic recombination process during simulation of imaginary events and dissociate this process from the temporal direction of simulation.

(b). Cognitive studies of episodic recombination during future simulation

Subsequent studies have used either the same experimental recombination task established in the [75,77] research or variants of it to explore cognitive aspects of episodic recombination (for studies that have used the experimental recombination procedure to examine encoding of future event simulations, see [79,80]; for studies that have used the procedure to examine memory errors, see [81,82] for related research on episodic recombination and memory distortion, see [83,84]).

Extending the data on remembering and imagining from Addis et al. [76], Addis et al. [85] examined possible links between remembered past, imagined past, and imagined future events with a standard measure of divergent creative thinking, the Alternative Uses Task (AUT), which participants in the Addis et al. [76] study had completed after the memory and imagination tasks. On the AUT, participants are asked to generate novel uses for common objects (e.g. a brick or a coin). Divergent creative thinking refers to the ability to generate new ideas by combining diverse types of information [86], a process that should overlap to some extent with episodic simulation. Consistent with this idea, Addis et al. [85] reported hierarchical linear regressions that revealed that generation of novel ideas on the AUT is a significant predictor of the number of episodic details in imagined future events, but not in remembered or imagined past events. These findings suggest that the temporal direction of simulation played a role in driving the differential association with AUT performance, perhaps reflecting the fact that both remembered and imagined past events are constrained by what actually happened in a way that imagined future events are not. Thakral et al. [87] replicated the main findings of Addis et al. [85] and also added a condition in which participants were instructed to recast an actual past event into the future and imagine it happening then. This condition potentially dissociates temporal direction from recombination demand because recast future events do not involve an explicit requirement for episodic recombination. Importantly, Thakral et al. [87] found a positive association between AUT performance and both novel and recast imagined future events (but not remembered past events), suggesting that temporal direction of simulation into the future rather than recombination demand is the key factor linking AUT performance and episodic simulation. ³ In a second experiment, Thakral et al. [87] reported the same pattern of results with another divergent thinking task, the Consequences Task [86], in which people are asked to imagine the consequences of various improbable scenarios that do not exist in the real world (e.g. if people could live on without dying). Taken together, these findings suggest that imagining future events, either novel or recast, overlaps with divergent thinking in a way that remembering or imagining past events does not. Further research will be required to more fully understand the underlying processes that support this overlap.

More recent studies have used the experimental recombination procedure to examine aspects of future simulation with applied implications. Acevedo-Molina et al. [88] used the experimental recombination task to examine what they called past and future autobiographical thinking in clinically normal older adults who carry the APOE4 allele, which has been associated with increased risk for the development of Alzheimer’s disease (e.g. [89]). After providing 16 ‘very vivid’ person–place–object memories from the past 5 years (aided by picture cues intended to facilitate retrieval), participants returned for a second session in which they were instructed to ‘either remember the previously generated past event from the past 5 years or imagine a future event happening in the next 5 years involving the newly combined person, place, and object’ [88, p. 197]. Compared with matched older adults who do not carry the APOE4 allele, the APOE4 carriers showed reduced episodic detail when remembering past events and imagining future events, together with no differences in non-episodic details. Acevedo-Molina et al. [88] did not examine imagination of past events, as in the [76,85] studies, or future recast events, as in the Thakral et al. [87] experiments. It seems likely that comparable deficits for the APOE4 allele carriers would be observed in these conditions, too, but future studies could address the issue. In another clinical application, Pollak et al. [90] used a slightly different version of the experimental recombination task to examine future thinking in a group of adolescents who experienced recurrent suicidal thoughts and/or behaviours (STBs), compared with adolescents who did not experience recurrent STBs. Rather than recombine elements of episodic memories that are specific to each individual, as in the aforementioned studies, participants in both groups were given the same set of person–place–object cues, which were constructed to be plausible for adolescents (e.g. My parent/guardian–Central Park–water bottle), and they were instructed to imagine an event occurring in the next year involving the three elements. The STB and non-STB groups generated simulations of novel future events that contained similar amounts of episodic detail. However, the STB group reported subjectively experiencing their future simulations as less detailed and less likely to occur than did the non-STB group.

Other studies have used a more streamlined version of the recombination task to test the constructive episodic simulation hypothesis. For example, Thakral et al. [91] examined whether the subjective quality of key elements of episodic memories—people and places—predict the subjective quality of novel episodic future simulations to which they contribute. Thakral et al. [91] accomplished this objective by having participants rate the vividness of a key person and place in an episodic memory, and then examined the rated vividness of the person and place when those elements were recombined with other elements to form a novel event. For instance, consider a participant who provided one episodic memory titled ‘Picture with seagulls’ that featured a critical person (Janine W.) and place (‘Ocean Beach pier’) and another memory titled ‘Dog training Sunday’ featuring a different person (Otis W.) and place (‘Kitchen in Dallas’). For the episodic simulation task, the experimenter recombined the person and location so that, for example, a participant would be instructed to imagine a novel future event featuring Janine W. and ‘Kitchen in Dallas’. The key finding was that the vividness of the person and place details in the episodic memories covaried with the vividness of those details when they were recombined into episodic future simulations, thus supporting a key tenet of the constructive episodic simulation hypothesis that episodic simulations are built from recombined elements of past experiences.

(c). Neuroimaging studies of episodic recombination during future simulation

Several neuroimaging studies have used versions of the aforementioned tasks in an attempt to test the constructive episodic simulation hypothesis and to specify the contributions of specific regions within the core network to imagining and remembering. Thakral et al. [92] adapted the person–place version of the experimental recombination task used in the aforementioned Thakral et al. [91] behavioural study to an fMRI study using multi-voxel pattern analysis (MVPA), focusing on two core network regions that had previously been implicated in episodic memory and simulation: the hippocampus and the angular gyrus [72]. After providing person–place autobiographical memories prior to scanning, participants performed memory, future simulation, and control tasks during fMRI scanning. As in Thakral et al. [91], for the memory task, participants were provided an event title (e.g. ‘Picture with seagulls’) with instructions to recall the related memory and focus on the person and location (e.g. Janine W. and ‘Ocean Beach pier’), whereas for the episodic simulation task, participants were instructed to imagine a novel future event involving a recombined person and location (e.g. Otis W. and ‘Ocean beach pier’). Participants made vividness ratings for people and places, as in Thakral et al. [91], and they also performed a control task adapted from the Addis et al. [75] study in which they constructed a sentence from cue words.

Thakral et al. [92] assessed neural similarity between memory and simulation by correlating the multi-voxel pattern associated with the memory containing a specific detail (e.g. the location detail ‘Ocean Beach pier’), with the multi-voxel pattern for the simulation containing that detail (the matching correlation). For comparison, Thakral et al. [92] correlated the multi-voxel pattern for that same remembered detail with the corresponding patterns for all the other episodic simulations that did not contain that detail (the mismatching correlation). Thakral et al. [92] reasoned that a significantly larger matching than mismatching correlation provides evidence for reinstatement of a specific detail from memory during episodic simulation (this logic is based on earlier studies of reinstatement of event details during remembering; see [93,94]).

The MVPA analyses showed that for both the hippocampus and angular gyrus, the matching correlations for person and location details were significantly higher than the mismatching correlation, providing neural evidence for the operation of episodic recombination when participants simulated future events. Interestingly, this difference was apparent only for highly vivid details in the hippocampus, whereas in the angular gyrus the difference between the matching and mismatching correlations did not depend on the vividness of the recombined details (for discussion of the theoretical implications of this finding, see [95]). These results, then, parallel nicely the behavioural data reported in Thakral et al. [91].

In an earlier study, Gaesser et al. [96] used the person–place–object version of the experimental recombination task in an attempt to distinguish the role of the hippocampus in constructing novel future simulations based on recombined details from the successful encoding of those representations. As noted earlier, several studies have used the experimental recombination procedure to examine cognitive processes associated with encoding of event simulations. In addition, an fMRI study by Martin et al. [97] revealed that the hippocampus showed greater activity during encoding of person–place–object future event simulations that were later successfully remembered than those that were forgotten on a cued recall test in which two details from the simulated event were presented and participants attempted to recall the third detail. Gaesser et al. [96] showed that, even when encoding success was controlled for, there was still evidence of a hippocampal contribution to what they called ‘the constructive process of recombining disparate details from memory into a coherent scenario when simulating a future event’ [96, p. 1157]. In a related study, van Mulukom et al. [98] showed that repeated simulation of a future event diminished the contribution of the hippocampus and other regions thought to be involved in the constructive process of episodic recombination (for related findings concerning simulation of individual event features, see [99,100]).

In summary, both cognitive and neuroimaging studies have provided strong evidence for a contribution of episodic recombination processes to the construction of imagined events, both in the future and in the past. Moreover, the evidence suggests that episodic (re)combination involves the retrieval and composition of discrete elements such that the elements of particular experiences (like ‘location’, ‘object’ or ‘person’) can be combined in the service of both episodic memory and imagination.

5. The role of time in mental travel

Above, we argued for representational continuism: the view that mental travel is a distinctive and unified cognitive capacity in so far as it is the capacity that handles a distinctive kind of mental representation (namely, episodic representations). Part of this view is the claim that the right unit of analysis is in fact not mental time travel but mental travel as such. In other words, the fact that mental travel often seems to occur ‘in time’ is, on this view, not a necessary part of its operation. In this final section we will specify this claim.

Episodic representations are distinctive in the human mind partly because they display a unique relationship to temporal information. On the one hand, these representations are episodic arguably exactly because they represent events—i.e. they contain non-conceptual information about the temporal order and extension of occurrences. In other words, they have contents similar to what linguists refer to as temporal ‘aspect’ [101]: they contain information about the ‘internal temporal constituency’ [98, p. 3] of an event.

On the other hand, episodic representations also frequently display temporal information corresponding to what linguists call ‘tense’ [102]: an event’s location in time relative to a ‘deictic centre’ (i.e. the present). This, roughly, corresponds to what Tulving [2] termed ‘chronosthesia’ and should be distinguished from an event’s definite ‘date’ in conventionalized time (‘yesterday’, ‘next Christmas’, etc.). We refer to this feature of episodic representations as ‘temporal orientation’ (see also [20,103]).

Relatively little is known about how episodic representations achieve their temporal orientation (for review see [34]) even though (as reviewed above) it can have effects on the neurocognitive profile of episodic recombination. To some extent, the fact that episodic recombination processes are sometimes sensitive to the temporal orientation of the to-be-constructed event representation is unsurprising: thinking about the future differs from thinking about the past in some principled ways—most obviously in being less constrained by our knowledge and experience of the actual past. This is also true for the difference between future-directed imaginations and past-directed counterfactuals. The latter but not the former rely on the retrieval of actual occurrences: imagining what could have happened requires recalling what actually happened [104,105]. This is not the case for imaginations of the future (for discussion of the relationship between episodic future thinking and episodic counterfactuals, see [106]).

Nonetheless, effects of temporal orientation on episodic combination highlight the question how episodic representations come to be oriented in time. Are episodic construction processes necessarily ‘tensed’ so that we should essentially distinguish between future- and past-directed forms of episodic combination? Or does temporal orientation rely on a dedicated cognitive capacity separate from a unitary episodic combination process? The answer to this question is important because it bears on the cognitive ontology of mental travel. If it turned out that we should distinguish past- and future-directed episodic construction processes, mental (time) travel could not be said to be a unitary capacity.

Several considerations suggest that temporal orientation is best viewed as the result of a separate, dedicated cognitive capacity. First, note that it is possible to entertain episodic representations that are not temporally oriented—e.g. imagine your usual way to work or the general outline of your house [107]. Representations such as these have been shown to rely on the core network of brain regions associated with episodic recombination [108] but lack any orientation towards the past, present or future (see also the well known work on atemporal scene construction, e.g. [109]). This observation suggests that the episodic construction process can operate without producing tensed representations. Second, temporal orientation cannot be the result of encoding (and therefore ‘direct’ retrieval) of episodic information itself. When we go through the world, we do not encode experiences ‘as past from a future perspective’. This kind of perspective, however, would be required for temporal orientation of future memories (which encoded experiences are) to be the result of straight-forward retrieval (see also [110] for a similar point). As a result, temporal orientation has to be due to mechanisms that do not rely primarily on the retrieval of previously encoded experiences (e.g. [111]), thereby suggesting that the temporal orientation of both memories and imaginations is not part of the process of episodic recombination but draws on a separate conceptual capacity. Finally, empirical evidence suggests that mental ‘time’ travel and mental ‘space’ travel—while utilizing similar computations—rely on clearly dissociable neuro-cognitive systems [112,113]). For example, in a neuroimaging experiment, Gauthier and van Wassenhove [112] found that mentally projecting oneself in space and in time engaged distinct brain networks. Moreover, in a recent study, Mahr et al. [103] sought to directly test the extent to which retrieval of episodic information predicts participants’ capacity to recall the temporal orientation of events. Results suggested that participants’ accuracy in recalling episodic features of their imaginations only weakly predicted their accuracy in recalling their imaginations’ temporal orientation. Similarly, Mahr & Schacter [38] found that participants were able to recall the ‘content’ of an event (i.e. what happened) while misremembering its temporal orientation. This finding suggests that temporal orientation is not encoded or retrieved alongside other types of episodic information.

How could this capacity for generating temporally oriented episodic representations operate? We have argued that temporal orientation (along with other features such as an event’s status as remembered or imagined) could enter into episodic cognition during either pre-retrieval or post-retrieval processing—as part of a high-level description of the to-be-constructed representation or as the result of metacognitive inference from the features of the constructed representation [20]. More concretely, D’Argembeau [34] has proposed that the temporal orientation of an event relies on its integration with autobiographical knowledge. On this account, only by being related to one’s understanding of the unfolding of one’s life does a given episode achieve its subjective location on a personal timeline. In any case, both the ability to entertain tensed episodic representations and our concept of a ‘personal timeline’ must rely on the prior conceptual capacity for sophisticated temporal reasoning (e.g. [114]). That is, in contrast to those who think that mental travel enables a sophisticated understanding of time (e.g. [36]), our view suggests that a mature concept of time as extended along a linear dimension is required for the human capacity to entertain tensed episodic representations in the first place.

6. Conclusion

We have presented a view of the cognitive ontology—the main cognitive components—composing the human capacity to mentally ‘travel’ to different places and times. On this view, mental travel is a unitary cognitive capacity: the capacity to construct episodic representations through processes of episodic (re)combination. A consequence of this view is that, while episodic representations have temporal contents, they are not necessarily located ‘in time’. That is, mental travel is not necessarily mental time travel. Of course, the fact that mental time travel is best analysed into separable components does not mean that we should therefore not treat it as a natural kind in its own right. It does mean, however, that if we are interested in the cognitive ontology of episodic cognition, we should make a distinction between mental travel and mental time travel.

We believe that this distinction can be useful in both organizing the literature and suggesting directions for future research. We briefly consider three possible future directions here. First, there has been extensive debate over whether mental time travel is uniquely human or is present in non-human animals as well (cf. [9,115–117]). These debates have hinged centrally on questions concerning the capacity of non-human animals to represent time. From the present perspective, an equally important question is whether non-human animals are capable of mental travel as defined here. A positive answer to the question of whether non-human animals can engage in mental time travel would indicate that they can engage in mental travel, but a negative answer to the question of whether non-human animals can engage in mental time travel need not indicate that they are incapable of other forms of mental travel, such as episodic recombination that is not located ‘in time’. While we refrain from speculating about it here, the issue is worth examining in future research.

A second issue concerns the role of the self in mental travel. A shared feature among most studies of mental travel involving episodic imagining is that individuals imagine themselves in hypothetical scenarios, but people can imagine events or episodes that do not involve the self, such as imagining the outcome of a future election or sporting event. Imagined events involving the self are associated with more episodic detail and a stronger feeling of experiencing than are imagined events that do not involve the self [118], and are linked with distinct subsystems of the default mode network [119], but next to nothing is known about how the composition of episodic elements might differ in episodes that do and do not involve the self. We think that research that attempts to fill the gap would be timely and important.

A third issue concerns the extent to which the episodic elements used for mental travel are represented discretely and are thus amenable to episodic recombination processes of the kind proposed here. As alluded to earlier, the general idea that episodic memories are composed of distinct elements represented in separate neocortical regions, linked together by a hippocampal ‘index’, has a long history in memory research (e.g. [44,120]) and can be seen as broadly consistent with the view of episodic recombination proposed here. With respect to imagined episodes, neuroimaging studies of future simulation have provided some relevant evidence. For example, in an fMRI study by Benoit & Schacter [72], participants imagined episodes comprised of individual people and places of varying familiarity, and provided evidence that the dorsal medial prefrontal cortex is preferentially involved in imagining people, the parahippocampal cortex is preferentially involved in imagining places, and the ventromedial prefrontal cortex serves to integrate these distinct elements into a coherent episode. Thakral et al. [92] also provided evidence linking the dorsomedial prefrontal cortex with person (versus place) elements in an imagined episode. Additional neuroimaging and cognitive research will be needed to examine further the extent to which elements of episodes are represented in a manner that makes them suitable targets for recombination.

Ethics

This work did not require ethical approval from a human subject or animal welfare committee.

Data accessibility

This article has no additional data.

Declaration of AI use

We have not used AI-assisted technologies in creating this article.

Authors’ contributions

J.B.M.: conceptualization, writing—original draft, writing—review and editing; D.L.S.: conceptualization, writing—original draft, writing—review and editing.

Both authors gave final approval for publication and agreed to be held accountable for the work performed therein.

Conflict of interest declaration

We declare we have no competing interests.

Funding

Preparation of this manuscript was supported by National Institute of Mental Health grant R01MH60941 and National Institute on Aging R01AG00841 to DLS.