Remembering and Imagining Alternative Versions of the Personal Past

Abstract

Although autobiographical memory and episodic simulations recruit similar core brain regions, episodic simulations engage additional neural recruitment in the frontoparietal control network due to greater demands on constructive processes. However, previous functional neuroimaging studies showing differences in remembering and episodic simulation have focused on veridical retrieval of past experiences, and thus have not fully considered how retrieving the past in different ways from how it was originally experienced may also place similar demands on constructive processes. Here we examined how alternative versions of the past are constructed when adopting different egocentric perspectives during autobiographical memory retrieval compared to simulating hypothetical events from the personal past that could have occurred, or episodic counterfactual thinking. Participants were asked to generate titles for specific autobiographical memories from the last five years, and then, during functional magnetic resonance (fMRI) scanning, were asked to repeatedly retrieve autobiographical memories or imagine counterfactual events cued by the titles. We used an fMRI adaptation paradigm in order to isolate neural regions that were sensitive to adopting alternative egocentric perspectives and counterfactual simulations of the personal past. The fMRI results revealed that voxels within left posterior inferior parietal and ventrolateral frontal cortices were sensitive to novel visual perspectives and counterfactual simulations. Our findings suggest that the neural regions supporting remembering become more similar to those underlying episodic simulation when we adopt alternative egocentric perspectives of the veridical past.

1.1 Introduction

Remembering the personal past, or autobiographical memory (AM), is closely related to our ability to imagine hypothetical episodes that might occur in the future (i.e., episodic simulation; Schacter et al., 2008; Szpunar et al., 2014) or how events could have turned out differently in the past (i.e., episodic counterfactual simulation; De Brigard et al., 2013). According to the constructive episodic simulation hypothesis (Schacter & Addis, 2007, 2009) both remembering and imagining are supported by a constructive episodic memory system that involves access to stored episodic details and the ability to flexibly recombine details from past episodes when memories are reconstructed or when imagining a hypothetical event. Supporting this hypothesis, AM retrieval and episodic simulation recruit a similar pattern of brain regions referred to as a core network, which includes medial temporal lobe, medial prefrontal cortex (PFC), posterior cingulate, retrosplenial cortex, and lateral temporal and parietal cortices (Addis et al., 2007; Benoit & Schacter, 2015; Buckner & Carroll, 2007; Schacter et al., 2015; Spreng et al., 2008; Szpunar et al., 2014). Episodic simulation, however, typically involves stronger neural recruitment in some of these core network areas, as well as additional recruitment of other brain regions attributed to greater constructive demands in recombining details when imagining a novel event (Benoit & Schacter, 2015; Schacter & Addis, 2009; Schacter, et al., 2012). We and others (Hardt et al., 2010; Howe & Derbish, 2010; Newman & Lindsay, 2009; Schacter, 2012; Schacter & Addis, 2007; Schacter et al., 2011) have argued that the adaptive nature of memory, which enables the ability to simulate novel events, also makes it vulnerable to modification (e.g., Carpenter & Schacter, 2017; St Jacques et al., 2013; St Jacques & Schacter, 2013; St Jacques et al., 2017). Yet, to our knowledge, previous functional neuroimaging studies have not fully considered how reshaping the veridical past may also place similar demands on constructive processes as episodic simulation of hypothetical events. Here, we used an fMRI adaptation paradigm (Barron et al., 2016; Grill-Spector et al., 2006; Larsson et al., 2016; Szpunar et al., 2014) to isolate neural regions sensitive to the creation of alternative versions of the past during AM retrieval when adopting novel visual perspectives and episodic counterfactual simulations.

As noted above, episodic simulation typically involves greater neural recruitment than AM retrieval. For example, some studies have shown greater recruitment of the hippocampus when imagining future events (Schacter & Addis, 2009; Schacter et al., 2017), and stronger coupling with the frontoparietal control network when planning plausible future events (Gerlach et al., 2014; Spreng et al. 2010). Although episodic simulations of the future and the past have been shown to recruit similar core network regions (e.g., Addis et al., 2009), there are some reported differences. For example, episodic counterfactual thinking preferentially engages posterior dorsal medial PFC (Van Hoeck, et al., 2012). Further, De Brigard et al. (2013) found that probable or likely episodic counterfactual simulations, in contrast to unlikely episodic counterfactual simulations, recruited a more similar pattern of neural activity to that associated with AM retrieval. A recent meta-analysis by Benoit & Schacter (2015) revealed that episodic simulations, including counterfactual thinking, involved greater engagement of core network areas than AM retrieval within the left inferior parietal cortex, dorsolateral PFC and hippocampus, as well as additional recruitment of precuneus and other brain regions overlapping with the frontoparietal control network. Such differences in the neural regions between episodic simulation and AM retrieval are thought to reflect additional constructive demands involved in recombining disparate episodic details to form novel coherent scenarios (for discussion see Schacter & Addis, 2009; Schacter, et al., 2017; Schacter, et al., 2015). Importantly, however, previous research comparing episodic simulation and memory retrieval has focused on AM conditions that placed minimal demands on constructive processes. For example, participants are typically instructed to recall memories in accurate detail and/or in similar ways to which they were originally experienced. It is well known that AMs can be reconstructed in multiple ways (Hirst & Echterhoff, 2012; Marsh, 2007; Pasupathi, 2001). Yet it is not well understood how the different ways we can retrieve memories influence the neural mechanisms typically associated with additional recombination demands during episodic simulation.

One of the most prominent ways we reconstruct AMs is by retrieving them from multiple egocentric or self-centered visual perspectives. Although we typically experience the world from our own eyes, when we retrieve AMs we can flexibly shift our first person viewpoint from inside to outside the body, seeing ourselves in the memory (Nigro & Neisser, 1983; Rice & Rubin, 2009). Retrieving AMs from visual perspectives that were never experienced is thought to reflect reconstructive processes that can reshape memories (Butler et al., 2016; McDermott et al., 2016; Robinson & Swanson, 1993). Supporting this idea, adopting a particular visual perspective influences the content and phenomenological properties of AM retrieval (Berntsen & Rubin, 2006; McIsaac & Eich, 2002; Robinson & Swanson, 1993) and recruits different neural processes (Eich et al., 2009; Freton, et al., 2014; Grol et al., 2017). We recently provided evidence concerning the neural mechanisms that contribute to changes in memories online and during subsequent memory retrieval when retrieving AMs from novel visual perspectives (St Jacques, et al., 2017). In this functional magnetic resonance imaging (fMRI) study, participants were asked to retrieve AMs during scanning while repeatedly adopting an identical own eyes perspective or shifting to a novel observer perspective. We found that adopting a novel observer perspective reduced subjective ratings of emotional intensity online, and also biased the natural visual perspective memories were subsequently retrieved from. Neural recruitment in the precuneus, bilateral inferior parietal cortex, and lateral PFC supported the ability to adopt a novel versus an identical visual perspective during AM retrieval. However, only the precuneus supported online and subsequent changes in AMs attributable to constructing memories from a novel visual perspective.

In the current fMRI study, we examined whether increasing constructive demands on the veridical past by requiring participants to retrieve personal memories from novel visual perspectives would evoke neural activity in regions associated with constructive processing and that are typically more active for episodic simulation than remembering. Participants generated the titles of AMs in a pre-scanning session. One week later, they repeatedly retrieved or created an episodic counterfactual event while adopting an own eyes or observer visual perspective. We used an fMRI adaptation approach to isolate neural regions that were sensitive to adopting a novel visual perspective during AM retrieval compared to adopting a novel episodic counterfactual simulation. We predicted that the precuneus, inferior parietal cortex and lateral PFC would be sensitive to construction of alternative versions of the past for both AM retrieval and episodic simulation.

2.1. Methods

2.1.1. Participants

In total, there were 39 participants who gave written informed consent. Two participants were excluded due to technical issues. Additionally, eight participants were excluded from the analysis because of excessive movement during fMRI scanning (i.e., maximum absolute movements greater than 2 mm, more than 5 movements greater than 0.5 mm, and/or a slice signal-to-noise ratio less than 99).¹ Thus, the final results were based on 29 participants (16 women, Mean Age in Years = 21.3, SD = 3.4).

2.1.2. Procedure

The study took place across two sessions separated by approximately a week (M =7.0 days, SD = 2.1). In session 1, participants generated 228 memories and provided a unique event title and specific date. They were also asked to provide subjective ratings of reliving, own eyes perspective, observer perspective, emotional intensity, positive valence, and rehearsal each on 7-point scales from 1 = low to 7 = high. In order to minimize demands to shift visual perspective across the study sessions (e.g., St Jacques, et al., 2017), we selected 192 memories that were associated with a variety of own eyes and observer perspective ratings. Memories were randomly assigned to autobiographical memory and episodic counterfactual simulation conditions that were matched in terms of the phenomenological ratings (for means and SD see Table 1).

Table 1.

Session 1 Ratings

	Memories Randomly Assigned to Two Tasks:
	Autobiographical Memory Retrieval	Episodic Counterfactual Simulation
Reliving	4.50 (0.70)	4.47 (0.70)
Own Eyes	5.17 (1.22)	5.18 (1.22)
Observer	2.85 (1.19)	2.85 (1.21)
Emotional
Intensity	3.99 (0.87)	4.01 (0.80)
Positive Valence	4.63 (0.42)	4.59 (0.40)
Rehearsal	2.76 (0.57)	2.76 (0.52)

Mean (Standard Deviation)

In session 2, participants were presented with event titles and asked to retrieve memories again or to simulate a counterfactual episode by imagining an alternative way in which the same event could have occurred (see Fig. 1A). In the counterfactual task, participants were additionally instructed that the simulation should entail a plausible alternative, as well as to maintain the positive or negative emotions associated with the actual event. For example, if the original event involved “the picnic was ruined when it started to rain” participants were instructed to imagine instead that “the picnic was ruined when a bunch of ants got in our food.” In both tasks, participants were asked to think about the event from an own eyes or an observer perspective. Specifically, participants were instructed: “If the perspective is own eyes, mentally reinstate the event as if seeing it through your own eyes. If the perspective is observer, mentally reinstate the event as if viewing it from the perspective of a spectator or observer, watching yourself in the event.” Each event was presented for 7.5 s and was followed by emotional intensity and task difficulty ratings (on 5-point scales from 1 = low to 5=high) for 2.5 s each, the order of which was counterbalanced across participants. Trials were separated by an active baseline consisting of a left/right decision that was variable in length (2.5 – 10 s), distributed exponentially such that shorter inter-trial intervals occurred more frequently than longer inter-trial intervals.

Fig. 1. fMRI Design and trial structure.

A) Participants were presented with the titles of autobiographical events they generated in a previous session and asked to retrieve the memory or simulate an episodic counterfactual event while adopting an own eyes or observer visual perspective, then to rate how difficult the task was, and the emotional intensity associated with the event. We used an active jittered baseline consisting of a left/right decision. B) Immediately prior to scanning participants were asked to retrieve or simulate an episodic counterfactual event for two thirds of the autobiographical events, with two repetitions. During fMRI scanning, some of these events were shown for a third time with the same instruction (i.e., identical repetition), with the instruction to adopt a novel visual perspective or episodic counterfactual simulation (i.e., novel repetition). The other third of the autobiographical events were shown for the first time during fMRI scanning (i.e., initial trial).

In order to examine fMRI repetition suppression effects, two thirds of events had been shown twice outside the scanner before being shown for the third time during fMRI scanning, whereas the remaining third of events were shown for the first time during scanning (see Fig. 1B). In previous research we have observed similar repetition suppression effects irrespective of whether trials are repeated within the same fMRI run or not (Szpunar et al., 2015), and we expected to find the same robust effects here. For the repeated trials, one half were identical repetitions, whereas the other half were novel repetitions. For identical repetitions, participants were instructed to retrieve or simulate the counterfactual of the event in the same way, while holding the visual perspective constant. For novel repetitions, we manipulated whether participants remembered or imagined alternative versions of the past. Specifically, in the novel perspective condition, participants were asked to adopt a different visual perspective compared to the previous two memory retrieval trials (i.e., memories were retrieved from an own eyes perspective on each of the first two trials and then retrieved from an observer perspective on the third trial, or vice versa). In the novel counterfactual condition, visual perspective was maintained but participants were instead asked to adopt a novel episodic counterfactual simulation of the memory (i.e., memories were retrieved on each of the first two trials and then a novel episodic counterfactual simulation was constructed on the third trial). Thus, the novel perspective and novel counterfactual conditions allowed us to isolate the neural regions that support the ability to construct alternative versions of the personal past. There were four functional runs of the task with 48 trials each, for a total of 16 trials per trial type.

2.1.2.1. fMRI Data Acquisition and Pre-Processing

Imaging was conducted on a 3T Siemens Magnetom TimTrio Scanner, equipped with a 12-channel head coil at the Center for Brain Science at Harvard University. A laptop computer running Eprime 1.0 software (Psychology Software Tools, Pittsburg, PA) controlled stimulus display via an LCD projector, which projected onto a screen placed at the head of the MRI bore. Participants viewed the screen through a mirror fastened to the head coil. Cushions were used to minimize head movement and earplugs dampened scanner noise. Participants made responses using a five-button box placed in their right hand.

Anatomical images were acquired using a high-resolution three-dimensional magnetization-prepared rapid gradient echo sequence (MPRAGE; 176 sagittal slices, echo time (TE) = 1.64 ms, repetition time (TR) = 2,530 ms, flip angle = 7 degrees, voxel size = 1 × 1 × 1 mm). Functional images were collected using a T2* gradient echo, echo-planar imaging (EPI) sequence sensitive to blood oxygen level-dependent (BOLD) contrast (TR = 2,500 ms, TE = 25 ms, flip angle = 85 degrees, 3 × 3 mm in-plane resolution). Whole-brain coverage was obtained with 41 contiguous slices, acquired in the oblique coronal orientation. An online correction for distortion in the EPI images was conducted by acquiring two EPI images pre-scan with phase-encoding gradients in opposite directions and then computing a displacement map correcting the distortion in each voxel. Following the functional runs, we included a 6 min 12 sec resting state scan in which participants were asked to keep their eyes open while fixating on a crosshair as part of our standard protocol for an analysis that was not the focus of the current study.

Imaging data were preprocessed and statistically analyzed using SPM8 (Wellcome Department of Imaging Neuroscience, London, UK). First, data were preprocessed to remove sources of noise and artifact. Preprocessing included slice-time correction to correct for differences in acquisition time between slices for each whole brain volume; realignment within and across runs to correct for head movement; spatial normalization to the Montreal Neurological Institute (MNI) template (resampled at 2 × 2 × 2 mm voxels); and spatial smoothing at 8 mm full width at half maximum (FWHM) using a Gaussian kernel.

2.1.2.2. fMRI Analysis

Fixed effects analyses included regressors at the onset of each event title in each condition, which were modeled with a canonical hemodynamic response function with a duration of 7.5 s. An additional regressor of no interest was included at the onset of the first rating with a duration of 5 s (i.e., the total length of the two ratings). A general linear model was then used to examine random effects.

To examine repetition suppression effects, we first compared the linear reduction in the blood oxygen level dependent (BOLD) response across identical repetitions of memory retrieval, or the basic repetition suppression effect (i.e., Initial Retrieval – Identical Retrieval Repetition). Next, we used an fMRI adaptation approach to isolate regions within this retrieval network that were associated with constructing alternative versions of the past. We did this by examining the linear rebound in the BOLD response when an event was altered on repeated trials, by adopting a novel visual perspective or a novel counterfactual simulation of the event, when compared to the identical repeated retrieval of memories (i.e., Novel Repetition – Identical Retrieval Repetition). A conjunction approach was then used to isolate neural regions showing both the linear reduction in the BOLD response across identical repetitions and the linear rebound when an event was altered. To examine potential differences in the neural representations supporting the construction of alternative versions of the past we directly compared fMRI adaptation effects in the novel perspective and novel counterfactual conditions and their interaction with visual perspective using an ANOVA approach. Additionally, we conducted a parametric modulation analysis to examine neural recruitment that correlated with trial-by-trial variation in difficulty, separately within the AM retrieval and episodic counterfactual simulation tasks.

A whole-brain analysis with a primary voxel-level threshold of P = .001 and a minimum cluster-extent threshold of k ≥ 61 voxels was used to correct for multiple comparisons at p < .05 as determined by 10000 Monte Carlo simulations (Slotnick, Moo, Segal, & Hart Jr, 2003). To minimize potential false positives with using cluster thresholding we incorporated the correct smoothing value (i.e. derived from the average FWHM value calculated from the group-analysis in SPM = 14.1) and used a conservative primary voxel-level threshold (Eklund et al., 2016; Woo et al., 2014).

2.1.2.3. Region of Interest (ROI) Analysis

We also conducted a targeted ROI analysis based on the fMRI results from a previous paper in which we found engagement of central precuneus when altering the visual perspective of AMs during retrieval (St Jacques, et al., 2016). Percent signal change was calculated on a 6 mm sphere centered on the peak voxel in the central precuneus (MNI: 0, −60, 44) using MarsBaR. A 3 (Trial: Initial Trial, Identical Repeated, Novel Perspective) × 2 (Visual Perspective: Own Eyes, Observer) repeated measures ANOVA was then used to examine the pattern of percent signal change in the central precuneus. Follow-up analyses employed one-tailed paired t-tests to test the predicted direction of the trial level effects (i.e., initial trial > identical repeated, novel perspective > identical repeated).

Results

Behavioral Results

There were no differences in the subjective ratings of memories assigned to the autobiographical retrieval or episodic counterfactual simulation conditions (for means and SD see Table 1).

To examine subjective ratings and reaction times during session 2, we conducted a 2 (Visual Perspective: Own Eyes, Observer) × 2 (Task: Retrieval, Simulation) × 3 (Trial: Initial, Identical Repetition, Novel Repetition) repeated measures ANOVA, separately for difficulty and emotional intensity ratings (for means and SD see Table 2). First, turning to difficulty rating responses, we found a significant main effect of task, F (1, 28) = 49.85, p < .001, η_p²= .64. Inspection of the means revealed that it was more difficult to construct simulations (M = 2.57, SD = 0.68), than to retrieve memories (M = 1.85, SD = 0.44). There was also a main effect of visual perspective, F (1, 28) = 9.38, p = .005, η_p² = .25, which was reflected by greater difficulty when adopting an observer (M = 2.32, SD = 0.59), than an own eyes perspective (M = 2.10, SD = 0.48). There were no other main effects or interactions, nor differences in reaction time to make difficulty ratings.

Table 2.

Session 2 Ratings

	Difficulty		Emotional Intensity
	Response	RT	Response	RT
Autobiographical Memory Retrieval
Intial Trial
Own Eyes	1.70 (0.54)	0.95 (0.32)	3.15 (0.68)	0.93 (0.28)
Observer	2.01 (0.62)	0.95 (0.31)	3.00 (0.78)	0.93 (0.26)
Identical Repetition
Own Eyes	1.76 (0.50)	0.93 (0.27)	2.95 (0.59)	0.89 (0.23)
Observer	1.92 (0.55)	0.95 (0.29)	2.82 (0.67)	0.94 (0.29)
Novel Repetition
Own Eyes	1.72 (0.52)	0.94 (0.31)	2.94 (0.62)	0.89 (0.28)
Observer	1.97 (0.56)	0.96 (0.31)	2.94 (0.65)	0.93 (0.27)
Episodic Counterfactual Simulation
Intial Trial
Own Eyes	2.56 (0.67)	1.03 (0.26)	2.88 (0.59)	1.06 (0.29)
Observer	2.64 (0.77)	1.01 (0.27)	2.88 (0.57)	1.08 (0.29)
Identical Repetition
Own Eyes	2.41 (0.72)	0.95 (0.26)	2.75 (0.59)	0.99 (0.27)
Observer	2.62 (0.77)	0.95 (0.24)	2.74 (0.64)	1.00 (0.28)
Novel Repetition
Own Eyes	2.47 (0.68)	0.98 (0.26)	2.79 (0.49)	1.00 (0.32)
Observer	2.73 (0.82)	0.96 (0.25)	2.73 (0.59)	1.03 (0.30)

Mean (Standard Deviation)

Turning to emotional intensity, there was a main effect of task, F (1, 28) = 8.95, p = .006, η_p² = .24, which was reflected by higher ratings during autobiographical memory retrieval (M = 2.97, SD = 0.61), compared to episodic counterfactual simulation (M = 2.80, SD = 0.53). There was also a main effect of repetition trial, F (2, 56) = 9.32, p < .001, η_p² = .25. Follow-up tests revealed that emotional intensity was higher on initial trials (M = 2.98, SD = 0.61), when compared to identical repetitions (M = 2.82, SD = 0.55), and novel repetitions (M = 2.85, SD = 0.53), both p’s < .01. There were no other main effects or interactions. However, there were differences between the tasks in reaction times to make emotional intensity ratings, F (1, 28) = 15.93, p < .001, η_p² = .36, which reflected slower reaction times to make emotional intensity ratings for episodic counterfactual simulations (M = 1.03, SD = 0.26), compared to autobiographical memory retrieval (M = 0.92, SD = 0.25).

The behavioral findings from session 2 suggest that episodic counterfactual simulation was a more difficult task and associated with less emotional intensity when compared to autobiographical memory retrieval. There were also differences in difficulty when adopting a particular visual perspective, and between autobiographical and counterfactual tasks, however, critically there were no interactions.

3.1.2. Episodic Counterfactual Simulation and Autobiographical Memory Retrieval

As a first step, we sought to replicate previous findings that showed greater neural recruitment during episodic counterfactual simulation compared to AM retrieval. We examined the average differences in the BOLD response between AM retrieval and episodic counterfactual simulations when collapsed across identical repetitions (see Fig. 2 and Table 3). As expected, we found greater engagement for episodic counterfactual simulations in a number of regions, including bilateral inferior parietal cortices, dorsolateral and ventrolateral PFC, lateral temporal cortex, visual cortex, dorsomedial PFC and posterior precuneus. There were only a few regions that were engaged more during autobiographical memory retrieval compared to episodic counterfactual simulation, including bilateral inferior parietal cortex and anterior cingulate. However, there was no interaction with the particular visual perspective adopted. Thus, these findings replicate previous research showing that episodic counterfactual simulation involves greater neural recruitment of frontoparietal regions under conditions in which minimal reconstructive demands are placed on AM retrieval (Benoit & Schacter, 2015).

Fig. 2.

Neural regions that differed during episodic counterfactual simulations (CF) and AM retrieval, when averaged across identical repetitions.

Table 3.

Comparison of AM Retrieval and Episodic Counterfactual Simulation

Region	Voxels	~ BA	t	MNI Peak		z
				x	y
AM Retrieval > Episodic Counterfactual Simulation
vmPFC	156	10	3.71	−10	28	−10
			3.57	−16	44	−6
inc. Anterior Cingulate Cortex		10/32	3.71	−10	32	0
Posterior Inferior Parietal Cortex	164	40	4.62	−64	−30	38
			3.93	−60	−28	24
	234	40	4.16	54	−28	32
			3.23	66	−24	22
Episodic Counter factual Simulation > AM Retrieval
Lateral PFC, inc. Posterior dlPFC	8924	6,8	7.93	−40	8	48
Posterior dmPFC		9	7.14	−8	30	48
		8	6.96	−6	20	54
dlPFC		9	6.77	−42	26	42
vlPFC		45	6.39	−56	18	12
Posterior dlPFC	1363	8	5.76	38	12	46
		9	5.05	42	30	40
inc vlPFC		44	4.04	56	22	20
Insular Cortex	141	13	3.90	32	22	0
Temporal Cortex	147	21	4.76	62	−38	−6
Posterior Inferior Parietal Cortex	3091	39	8.41	−42	−62	46
	1234	39	5.81	48	−58	46
Posterior Precuneus	117	7	4.13	−4	−68	38
Fusiform Cortex	1441	37	6.49	−58	−44	−6
inc. Temporal Cortex		21	6.28	−48	−34	−2
Visual Cortex	6709	18	8.16	2	−86	−2
		18	6.33	−12	−84	−12
inc. Cerebellum		−	5.97	34	−60	−30
Thalamus	64	50	3.68	−12	−6	4

MNI = Montreal Neurological Institute; BA= Brodmann’s Area; PFC = Prefrontal Cortex; dl = Dorsolateral; dm = Dorsomedial; vl = Ventrolateral; vm = Ventromedial

The behavioral results indicated that subjective ratings of difficulty were higher for the counterfactual than the AM task. To examine whether subjective difficulty contributed to differences in neural recruitment, we conducted an additional parametric modulation analysis that examined regions that were sensitive to trial-by-trial variation in subjective ratings of difficulty, separately within each task. There were no regions that tracked with difficulty in the AM retrieval task. In the episodic counterfactual simulation task, there was greater engagement of left visual cortex on trials that were rated higher on difficulty (~BA = 18; MNI = −22, −94, −10; t = 4.75; voxels = 106), but little to no overlap with the regions that were recruited more during the counterfactual than AM task. Additionally, targeted ROI analyses showed that none of the brain regions that differed between the counterfactual and AM tasks were engaged more on trials with higher subjective ratings of difficulty (see Supplemental Fig. 1). In sum, these findings demonstrate that behavioral differences in subjective ratings of difficulty cannot account for the neural differences between the episodic counterfactual simulation and AM retrieval tasks found here.

3.1.3. Whole-Brain Analysis: Constructing Alternative Versions of the Past

The main goal of the paper was to isolate neural regions that support the ability to reconstruct alternative versions of the personal past when changing the visual perspective of memories or imagining an episodic counterfactual event. As a first step, we examined the neural representations associated with identical repetitions during autobiographical memory retrieval (i.e., reduction in BOLD response from an initial retrieval to an identical retrieval repetition; see Fig. 3 and Table 4). The analysis revealed repetition suppression effects in bilateral ventromedial and dorsomedial PFC, ventrolateral and dorsolateral PFC, lateral temporal cortices, inferior posterior parietal cortices, medial temporal lobe (including posterior parahippocampal cortex and hippocampus), posterior midline regions (including retrosplenial and posterior cingulate cortices), and cerebellum. This pattern of neural activity overlaps with default network and other regions that are frequently engaged during autobiographical memory retrieval (Addis et al., 2016; Andrews-Hanna et al., 2014; St Jacques, et al., 2013). There were no regions where repetition suppression effects significantly differed when adopting an own eyes or observer perspective during autobiographical memory retrieval. Thus, the pattern of repetition suppression effects likely support general retrieval-related processes irrespective of the particular visual perspective taken (also see St Jacques, et al., 2017).

Fig. 3.

Neural regions that were sensitive to the basic repetition suppression effects (i.e., reduction in BOLD response from initial trials to identical repetitions) during AM retrieval.

Table 4.

AM Retrieval (Initial - Identical Retrieval Repetitions)

Region	Voxel s	~BA	t	MNI Peak		z
				x	y
PFC, inc. vmPFC	8844	11	6.29	−2	48	−16
Posterior dmPFC		8	5.75	−10	42	42
vlPFC		45	6.09	−50	30	0
		47	6.08	−52	38	−8
			5.93	−32	24	−20
Posterior dlPFC		8/9	5.60	−42	10	46
vlPFC	539	47	4.50	36	32	−14
			4.10	26	18	−16
			3.96	48	30	−6
		45	4.24	54	26	16
			3.97	56	30	0
			3.64	58	24	8
Posterior dmPFC	74	8	4.07	20	34	42
Temporal Cortex	472	21	5.80	−60	−8	−18
		20	3.31	−40	−14	−24
	532	21	4.41	−60	−36	−2
			4.22	−60	−34	−10
			3.51	−60	−52	2
	259	21	4.95	60	−4	−20
Posterior Cingulate	4850	31	6.79	−4	−56	24
		23	4.30	−6	−42	34
inc. Hippocampus		–	4.18	−28	−26	−16
Posterior Parahippocampal Cortex		30	5.52	−18	−38	−14
		36	4.44	24	−34	−16
Retrosplenial Cortex		29	4.87	14	−46	4
Posterior Inferior Parietal Cortex	1575	39	5.88	−44	−74	32
			5.34	−48	−62	24
	360		4.62	48	−60	24
Globus Pallidus	313	–	4.53	10	−2	−6
			4.06	0	10	−8
inc. Thalamus		–	3.49	2	−12	0
Cerebellum	976	–	5.23	18	−84	−36
			4.90	24	−80	−30
			4.61	14	−78	−28

MNI = Montreal Neurological Institute; BA= Brodmann’s Area; PFC = Prefrontal Cortex; dl = Dorsolateral; dm = Dorsomedial; vl = Ventrolateral; vm = Ventromedial

Next, we examined the sensitivity of the neural representations within these retrieval specific regions to the construction of alternative versions of the past when shifting to an alternative visual perspective during memory retrieval or creating a novel counterfactual simulation of the event (i.e., reduction in BOLD response from a novel repetition to an identical retrieval repetition). If the underlying neural representation is insensitive to changes in visual perspective or counterfactual simulation, then the BOLD response will be reduced similarly to identical repetitions (i.e., no difference in neural activation between novel repetitions and identical retrieval repetitions). However, if the neural representations are sensitive to changes related to visual perspective or counterfactual simulation, then the BOLD response will rebound to the same level as the initial trials (i.e., difference in neural activation between novel repetitions and identical retrieval repetition). The fMRI adaptation findings revealed a number of regions that were sensitive to changes in both the visual perspective and counterfactual aspect of memories (see Table 5). First, changing the visual perspective of memories revealed effects in bilateral ventromedial and posterior dorsomedial PFC, posterior inferior parietal, and posterior cingulate cortices, as well as left ventrolateral and dorsolateral PFC (see Fig. 4A). Second, adopting a novel counterfactual simulation of a memory recruited similar regions including bilateral posterior dorsomedial PFC, ventrolateral and lateral temporal cortices, as well as left dorsolateral PFC, posterior inferior parietal and posterior cingulate cortices, and right cerebellum (see Fig. 4B). A conjunction analysis directly comparing the fMRI adaptation effects in the novel perspective and novel counterfactual conditions revealed that there was overlap in voxels within the left posterior inferior parietal cortex, posterior dorsomedial PFC, ventrolateral PFC, and posterior dorsolateral PFC (see Fig. 5). Further, each of these regions completely overlapped with neural recruitment that differed between episodic counterfactual and veridical retrieval of autobiographical memories, when reconstructive demands where minimized. Critically, a direct comparison showed that there were no regions where fMRI adaptation effects differed when adopting novel visual perspectives or novel counterfactual simulations, nor interactions with the particular visual perspective adopted. Thus, our findings demonstrate that altering memories when adopting a novel visual perspective recruits similar neural regions as those that enable simulation of counterfactual events.

Table 5.

Regions Contributing to Constructing Alternative Versions of the Past

Region	Voxels	~ BA	t	MNI Peak		z
				x	y
Novel Visual Perspectives (Novel - Identical Retrieval Repetition)
vlPFC	107	47	4.40	−52	38	−8
	90	44	3.49	−52	16	30
			3.45	−42	16	24
vmPFC	390	11	4.08	0	38	−20
		10	3.24	−6	62	−2
			3.23	−4	54	−8
Posterior dlPFC	142	6/8	4.13	−38	6	46
Posterior dmPFC	527	8	4.03	−22	30	44
			3.87	−16	30	52
		9	3.56	0	48	48
		8	3.47	−6	38	46
			3.44	−10	46	46
			3.42	−16	46	40
	68	8	4.05	20	34	42
Posterior Cingulate Cortex	592	31	4.08	−6	−56	28
		23	4.13	20	−50	26
			3.38	−10	−42	32
Posterior Inferior Parietal Cortex	362	39	4.26	−40	−56	24
			3.98	−40	−64	34
	220	39	3.98	46	−62	28
Novel Counterfactual Simulation (Novel - Identical Retrieval Repetition)
Lateral PFC inc. vlPFC	4794	47	5.39	−52	38	−8
		44	5.00	−42	14	26
Posterior dlPFC		6/8	5.40	−42	10	46
Posterior dmPFC		8/9	5.27	−10	44	44
			5.36	−6	22	56
			4.87	−28	22	50
vlPFC	65	47	3.59	46	28	−8
			3.59	46	28	−8
Posterior Inferior Parietal Cortex	535	39	4.67	−38	−66	36
			4.36	−42	−60	28
Temporal Cortex	352	21	4.35	−58	−36	−4
Cerebellum	847	−	4.74	20	−80	−30
			4.61	22	−86	−38
			3.58	32	−78	−40
Conjunction of Novel Visual Perspective and Novel Counterfactual Simulation
vlPFC	107	47	4.33	−52	38	−8
	90	44	3.46	−52	16	30
			3.45	−42	16	24
Posterior dlPFC	142	6/8	4.08	−38	6	46
Posterior dmPFC	414	8	3.88	−24	28	46
			3.87	−16	30	52
			3.47	−6	38	46
		8/9	3.44	−10	46	46
			3.31	−4	42	32
		9	3.54	−2	48	48
Posterior Inferior Parietal Cortex	252	39	3.93	−40	−64	34
			3.91	−42	−58	26

MNI = Montreal Neurological Institute; BA= Brodmann’s Area; PFC = Prefrontal Cortex; dl = Dorsolateral; dm = Dorsomedial; vl = Ventrolateral; vm = Ventromedial

Fig. 4.

Neural regions that were sensitive (i.e., rebound in BOLD response for novel repetition versus identical repetition) to A) changes in visual perspective, or B) adopting a novel episodic counterfactual simulation.

Fig. 5.

Common neural regions that were sensitive to changes in visual perspective and adopting a novel episodic counterfactual simulation.

3.1.4. Precuneus ROI Analysis: Constructing Alternative Versions of the Past

Targeted central precuneus ROI analysis revealed that this region also contributed to the construction of novel visual perspectives. There was a main effect of trial within precuneus, F (2,56) = 3.49, p = .044, η_p² = .11 which was reflected by a basic repetition suppression effect for initial trials versus identical repeated retrieval trials, p = .041, and a rebound in the BOLD response when we varied the visual perspective of memories by asking participants to adopt a novel visual perspective on repeated retrieval trials, p = .013 (see Fig. 6). Critically, there was no main effect or interaction with the particular visual perspective taken, suggesting that adaptation effects in the precuneus were similar irrespective of whether an own eyes or observer perspective was adopted. To examine whether adopting a novel episodic counterfactual simulation led to a similar rebound in central precuneus we directly compared identical repeated trials with novel counterfactual trials, collapsed across perspective. The pattern for the novel counterfactual (M = .27, SD = .27) compared to identically repeated trials (M = .22, SD = .26) was in the same direction as above, but it did not reach significance, t (28) = 1.46, p =.08, d = .27. Thus, the pattern of repetition suppression findings suggest that precuneus contributes to the ability to construct alternative versions of the past, particularly when adopting a novel visual perspective.

Fig. 6.

ROI analysis in the precuneus revealed fMRI adaptation effects reflecting sensitivity to changes in visual perspective during AM retrieval, irrespective of whether an own eyes or observer perspective was adopted. One-tailed p-values are shown. Error bars reflect the within-subject 95% confidence interval.

4.1. Discussion

Our findings reveal for the first time the neural regions that are sensitive to constructing alternative versions of the veridical and counterfactual past. Using an fMRI adaptation design, we show that similar neural regions that contribute to the creation of novel episodic counterfactual simulations support the reconstruction of memories from novel visual perspectives. Importantly, this pattern also overlapped with neural regions that are typically more engaged during episodic counterfactual simulations than veridical AM retrieval. Taken together, these results are consistent with the constructive episodic simulation hypothesis, which maintains that both remembering and imagining are supported by similar constructive processes that can modify the veridical past and create novel scenarios of plausible past events that could have occurred (Schacter & Addis, 2007).

Previous research has shown that memory and episodic simulations are strongly related, as indicated by extensive overlap in their neural correlates (Benoit & Schacter, 2015; Schacter, et al., 2015). However, episodic simulation typically involves greater constructive memory processes than remembering because it places more demands on recombination processes that enable episodic memory details to be reorganized in novel ways (Benoit & Schacter, 2015; Schacter & Addis, 2007). Manipulating the different ways that we can retrieve the past, such as adopting novel visual perspectives, however, likely involves additional constructive processes during remembering that would minimize differences between memory and imagination (McDermott, et al., 2016). In a similar vein, neural differences between the imagined and veridical past can be minimized when fewer demands are placed on constructive processes during imagination (De Brigard, et al., 2013). These ideas dovetail with the current findings of a similar pattern of neural activity when adopting a novel visual perspective during AM retrieval and a novel counterfactual simulation, suggesting that when constructive demands are increased retrieval of the past becomes more like imagination. Critically, here we also showed that general differences in task difficulty did not account for neural differences. These findings are consistent with the idea that increasing reconstructive demands during retrieval of AMs, by shifting visual perspective, recruits similar neural regions as those that support the transformation of memories into episodic counterfactual simulations.

Adopting a novel visual perspective during AM retrieval led to similar fMRI adaptation effects compared with novel episodic counterfactual simulations in left inferior parietal cortex, ventrolateral PFC, and dorsomedial PFC. Additionally, a targeted ROI analysis suggested that precuneus also contributed to adopting novel visual perspectives, but the pattern of adaptation effects was less strong when adopting novel episodic counterfactual simulations. A recent meta-analysis found that inferior parietal, dorsomedial PFC and precuneus were all engaged to a greater extent during episodic simulation than AM retrieval (Benoit & Schacter, 2015). Additionally, neuropsychological evidence has suggested that PFC is more crucial to episodic simulation than AM retrieval (Berryhill et al., 2010; de Vito, et al., 2012). For example, patients with unilateral lesions to the PFC are impaired on flexibly recombining elements of past experiences to construct episodic simulations, despite an intact ability to remember the veridical past (Berryhill, et al., 2010). Here we show that manipulating the visual perspective of AMs can also engage neural activity in regions that support constructive processes typically engaged by simulation.

Another explanation of our findings could be that episodic counterfactual simulations are simply more similar to AM retrieval than future oriented episodic simulations (e.g., De Brigard et al., 2013), since both AM retrieval and episodic counterfactual simulations are centered on the past and necessarily more constrained than the open-ended future. While this may be true to a certain extent, we do not think that it could easily explain the current findings, because we found that regions in the left posterior parietal and frontal cortices were recruited depending upon the level of constructive demands rather than the autobiographical or counterfactual nature of the task. Understanding how the flexible restructuring of the constrained past compares with imagining the open-ended future, as well as better understanding the link between AM retrieval and episodic counterfactual simulation, are important areas for future research (De Brigard et al., 2015; Schacter, et al., 2012; Schacter, et al., 2015).

One reason why adopting a novel visual perspective may recruit similar neural processes as those involved in episodic simulation is because both could involve the manipulation of mental images in the service of constructing a novel scenario. According to a prominent neural model of spatial memory and imagery, egocentric frameworks generated during retrieval from long-term memory within the posterior parietal cortex, and, in particular, the precuneus, can be manipulated and updated when people imagine the possible movements they can make within the remembered scene (Byrne et al., 2007). Supporting this model, recent evidence has shown that precuneus and lateral parietal cortices are recruited during imagined changes in self-location in space (Dhindsa, et al., 2014; Lambrey et al., 2012). St. Jacques et al. (2017) recently demonstrated that precuneus and inferior parietal cortices, as well as ventrolateral PFC, contributed to actively shifting the visual perspective of AMs during retrieval. Moreover, St. Jacques et al. (2017) found that neural recruitment in the precuneus also predicted the degree to which shifting visual perspective modified the phenomenological properties of memories, suggesting that memories were reshaped when adopting a novel visual perspective. Here, utilizing an entirely within-subject design, we show that overlapping regions within the left posterior parietal cortex and ventrolateral PFC support the construction of alternative visual perspectives during AM retrieval and novel episodic counterfactual simulations.

The current findings could also provide insight into why previous functional neuroimaging studies of AM retrieval have sometimes found inconsistent results concerning the involvement of the precuneus when adopting a particular visual perspective. For example, during AM retrieval precuneus has been linked to adopting an own eyes perspective (Freton, et al., 2014), an observer visual perspective (Grol, et al., 2017), or to both own eyes and observer (Eich, et al., 2009). St. Jacques et al. (2017) showed that the precuneus was engaged more when adopting a novel visual perspective than maintaining an identical one during AM retrieval. However, one limitation of this study was that shifts in visual perspective were examined in one direction only—from own eyes to observer. Here we replicate the findings from St. Jacques et al. (2017) in a different group of participants, and further reveal that the precuneus is engaged irrespective of the direction of perspective shifting (i.e., from both own eyes to observer AND observer to own eyes). Other research has demonstrated that the precuneus contributes more widely to mental imagery and self-referential processes (for review see Cavanna & Trimble, 2006), which may support the construction of complex and realistic scenes of the personal past (Hassabis et al., 2007; Summerfield et al., 2009). Together these findings suggest that precuneus does not preferentially support own eyes or observer perspectives during AM retrieval, but instead provides a particular perspective from which to inspect and manipulate the mental images that arise during remembering, thereby contributing to the construction of past events.

The ventromedial PFC was sensitive to changes in visual perspective during AM retrieval, but not when changing the counterfactual nature of memories. Ventromedial PFC is frequently involved in AM retrieval and has been linked to self-referential and emotional aspects of memory recollection (Cabeza & St Jacques, 2007; Svoboda et al., 2006). Recently, Lin et al. (2016) showed that ventromedial PFC adds a subjective sense of personal emotional value to the individual elements of memories. Similarly, during episodic simulation ventromedial PFC contributes to the integration of knowledge and affective value (Benoit et al., 2014). Both visual perspective and episodic counterfactual simulation manipulations are capable of altering the affective quality of memories (Berntsen & Rubin, 2006; De Brigard & Giovanello, 2012; St Jacques, et al., 2017), and such manipulations may result in greater sensitivity of the ventromedial PFC when adopting novel visual perspectives and counterfactual simulations. However, in the current study we directly instructed participants not to change the emotion of the event when constructing a novel episodic simulation, which could explain the lack of adaptation effects in the ventromedial PFC found here. Future research should examine how adopting a novel visual perspective affects the value or personal significance attached to individual elements comprising a particular AM.

The current findings present novel evidence that reshaping of the veridical past recruits common neural mechanisms that align memory with imagination. In our previous research, we showed that actively shifting the visual perspective of AMs biased the phenomenological properties of memories and we delineated the neural mechanisms during retrieval that contributed to this reshaping of memories (St Jacques, et al., 2017). Here we revealed that such retrieval-related changes in memories recruit the same neural correlates that enable the construction of hypothetical events that could have occurred in the past. A number of behavioural studies have shown that visual perspective influences the content and phenomenology of AMs (Berntsen & Rubin, 2006; Robinson & Swanson, 1993; Vella & Moulds, 2014), and can also lead to persistent changes in memories (Butler, et al., 2016; Sekiguchi & Nonaka, 2014; St Jacques & Schacter, 2013). Marcotti & St. Jacques (2017) recently found that shifting from an own eyes to an observer perspective during retrieval also reduces the accuracy of subsequent memories (also see Bagri & Jones, 2009; St. Jacques & Schacter, 2013). Similarly, constructing counterfactual simulations of events can lead to subsequent memory distortions (Gerlach et al., 2014). An important avenue for future research will be to better understand the different ways in which restructuring the past, namely changing one’s visual perspective during retrieval and simulating plausible alternatives to past events, may alter and even distort memories from the personal past.

Research Highlights.

• Episodic counterfactual simulation involves constructing alternative versions of the past.

• Shifting visual perspective in memory retrieval requires similar constructive processes.

• Alternative perspectives and versions of the past recruit similar frontoparietal regions.

• Remembering becomes more like imagination when shifting visual perspective.