The ghosts of brain states past: Remembering reactivates the brain regions engaged during encoding

Abstract

There is growing evidence that the brain regions involved in encoding an episode are partially reactivated when that episode is later remembered. That is, the process of remembering an episode involves literally returning to the brain state that was present during that episode. This paper reviews studies of episodic and associative memory that provide support for the assertion that encoding regions are reactivated during subsequent retrieval. In the first section, studies are reviewed in which neutral stimuli are associated with different modalities of sensory stimuli or different valences of emotional stimuli. When the neutral stimuli are later used as retrieval cues, relevant sensory and emotion processing regions are reactivated. In the second section, studies are reviewed in which participants employ different strategies for encoding stimuli. When the stimuli are later retrieved, regions associated with the different encoding strategies are reactivated. Together, these studies demonstrate not only that the encoding experience determines which regions are activated during subsequent retrieval, but also that the same regions are activated during encoding and retrieval. In the final section, relevant questions are posed and discussed regarding the reactivation of encoding regions during retrieval.

An increasingly popular view in cognitive neuroscience is that remembering a particular experience involves the partial reactivation of the widespread network of regions that were active during the episode itself (Buckner & Wheeler, 2001). This reactivation account of memory has been popular in theoretical psychology for over a century. James (1890) argued that remembering involves reinstating the sensory and motor components of the original event in their respective brain regions. Penfield and Perot’s (1963) early stimulation experiments lent support to this notion by demonstrating that stimulation of the temporal association cortex in epileptic patients could elicit auditory and visual memories, though this work later attracted substantial criticism (Bancaud et al., 1994; Halgren et al., 1978). The legacy of this idea was present in the encoding specificity (Tulving & Thompson, 1973) and transfer appropriate processing (Morris, Bransford, & Franks, 1977) accounts of memory retrieval, according to which the encoding context determines what is retrieved and which retrieval cues are most effective. However, the idea itself was not explicitly reintroduced into modern theory until almost a decade later, when it was invigorated by the connectionist revolution (McClelland et al., 1995; Rumelhart & McClelland, 1986) and Damasio’s (1989) retroactivation account of recall and recognition, according to which “representations … are recorded in precisely the same neural ensembles in which corresponding activity occurred during perception” (p. 39). Since then, the reactivation account has become a core component of many models of cortico-hippocampal interaction (Alvarez & Squire, 1994; McClelland et al., 1995; Moscovitch et al., 2005; Rolls, 2000; Shastri, 2002; Squire & Alvarez, 1995). For example, according to McClelland and colleagues (1995), the hippocampus quickly learns sparse representations of episodes and these representations serve to reinstate the episode in cortex during retrieval. While McClelland et al. argue that over time the hippocampus trains the cortex to reinstate these experiences independently, others argue that reactivation of specific episodes is always dependent on the hippocampus (Moscovitch et al., 2005). Reactivation in memory has also been integrated into many theories of embodied and grounded cognition, according to which multimodal neural simulations of previous episodes serve to partially recreate those episodes during recollection (Barsalou, 2008). According to Barsalou (1999, 2008), knowledge is represented in sensory cortex in the form of perceptual symbols, and during episodic retrieval frontal and medial temporal lobe structures exert control over these representations to simulate the past. Mesulam (1998) aptly describes how the reactivation account of memory can be integrated with current theories of hippocampal and prefrontal function:

“A memory has no anatomical boundaries. Its encoding and retrieval involve almost all parts of the association cortex, but with an orderly anatomical distribution of component processes: relevant unimodal and transmodal areas encode the sensory aspects; the limbic systems binds this information into a coherent whole; and prefrontal and other heteromodal areas guide the orderliness of storage and retrieval” (p. 1029).

There is some evidence from research on brain damaged patients that remembering relies on the same regions that are engaged during encoding. Greenberg and Rubin (2003) reviewed neuropsychological data demonstrating that impaired perception in brain damaged patients is often accompanied by impairments in autobiographical memory (for a related review of data demonstrating that impaired perception is often accompanied by impaired mental imagery, see Farah, 1988). Greenberg and Rubin presented evidence that damage that affects one perceptual system can lead to marked retrograde amnesia, supposedly because memory traces are in part stored in that perceptual system. This amnesia is usually modality specific. For example, auditory agnosia, an inability to recognize auditory stimuli, is associated with retrograde impairments in auditory memory. In contrast, visual agnosia is often accompanied by global retrograde amnesia that is not limited to the visual modality, a condition called visual-memory deficit amnesia (Rubin & Greenberg, 1998). Greenberg and Rubin theorize that damage to the visual system in particular can cause pervasive memory deficits because autobiographical memory is highly visual in nature. The forms of amnesia caused by damage to the perceptual systems seem to be predominantly retrograde in nature. Based on their review of the literature, Greenberg and Rubin conclude that “searching for a single neural location of memory is a fool’s errand. Memory is stored everywhere, and at every level of analysis” (p. 690).

As Greenberg and Rubin (2003) point out, “the neuropsychological approach is inevitably messy,” so their findings require further discussion (p. 716). It is exceedingly difficult for neuropsychological studies to distinguish a memory deficit from an attentional deficit or an encoding deficit. For example, it is possible that aphasic patients are able to retrieve the sensory components of the relevant episode but are unable to process or attend to those components. That is, it is possible that there is no memory deficit that cannot be explained by the underlying sensory processing deficit. For this reason, Greenberg and Rubin explicitly express the need for neuroimaging data to provide converging support for the overlap in the regions engaged during encoding and retrieval.

The dawn of functional neuroimaging has offered a new outlet with which to study the theory that brain regions are reactivated during retrieval. Early attempts using positron emission tomography (PET) focused on the what/where distinction in the visual system (Kohler et al., 1998; Moscovitch et al., 1995; Owen et al., 1996). According to Ungerleider and Mishkin (1982), the primate visual system can be subdivided into two major processing streams: a ‘what’ system that runs from the occipital lobes ventrally to the temporal lobes and is concerned with identifying visual objects and a ‘where’ system that runs from the occipital lobes dorsally to the parietal lobes and is concerned with identifying the location of visual objects. These studies found widespread activation in the ventral stream during the retrieval of item information and widespread activation in the dorsal stream during the retrieval of location information (Kohler et al.; Moscovitch et al. (2005); Owen et al.). However, the authors themselves acknowledge that, while these results may be suggestive of reactivation, alternative explanations also exist. As Vaidya et al. (2002) point out, the retrieval tasks themselves were not equivalent with respect to the amount of object- and location-related processing. In these studies, the identity condition required additional object processing during retrieval, and the location condition likewise required additional location processing during retrieval. For example, in the Kohler et al. study, participants were asked to make a forced choice between two arrays, one of which was studied and one of which was new. In the identity condition, the two arrays differed only with respect to the identity of the objects in the array, and in the location condition, the two arrays differed only with respect to the location of the objects in the array. Therefore, the object retrieval task required identity discrimination, and the location retrieval task required location discrimination. That is to say, the differences between retrieval conditions could be caused by differences in how the retrieval cues were processed rather than differences in which regions were activated during encoding.

The recognized limitations of these PET studies informed subsequent, more polished designs for exploring the reactivation of encoding regions during subsequent retrieval. In contrast to the early PET studies discussed above (Kohler et al., 1998; Moscovitch et al., 1995; Owen et al.,1996), these designs relied critically on the association of neutral retrieval cues with different kinds of associative or contextual information, such that the stimuli that engage different kinds of regions during encoding are retrieved but not actually presented during retrieval (Heil et al., 1994; Nyberg et al., 2000; Wheeler et al., 2000). While the exact details differ across experiments, the general structure of the design is presented in Figure 1. During the study session (Figure 1), participants study neutral stimuli (e.g., words) that are randomly associated with different contexts, such as different kinds of stimuli (e.g., a picture or a sound) or different encoding strategies (e.g., verbal vs. imagery). During the test session (Figure 1), brain activity is recorded while participants are presented with the neutral stimuli as retrieval cues and asked to make decisions about their memory of the cue and/or its associates. For example, participants might make a decision about whether the cue was studied or not (old/new decision), about how well they remember encoding the cue (remember/know/new decision), or about how the cue was encoded or what its associates were (source/context decision). These designs rely critically on the random assignment of cues to encoding conditions across participants, such that any given retrieval cue is equally likely to be assigned to any encoding condition. Since only the neutral retrieval cues are presented during test and each cue is randomly assigned to an encoding condition, any regions that are differentially active during the retrieval of information studied in different encoding conditions must be involved in the retrieval of the different kinds of encoding contexts. This design discounts alternative explanations for the differential activation of these regions due to attentional or perceptual processes, but is unfortunately limited to the retrieval of associative and contextual information. This limitation is substantial, as direct evidence for reactivation can only come from associative memory tasks. Because the study stimuli and the retrieval cues are the same in item recognition tasks, such paradigms are incapable of distinguishing reactivation of the regions involved in processing the stimulus at encoding from activation of such regions caused by attentional or perceptual processing of the same stimulus at retrieval.

Figure 1.

The prototypical design for exploring the reactivation of encoding regions during retrieval. STUDY. Neutral items are studied in one of multiple possible contexts. The different contexts could be different kinds of stimulus associations (e.g., pictures or sounds) or different encoding strategies (e.g., verbal vs. imagery). TEST. At test, brain activity is measured while retrieval cues are presented. Participants make some sort of memory decision about the cue such as old/new decisions, remember/know decisions, or context decisions. Strong support for reactivation requires: 1) The engagement of different brain regions during the retrieval of information studied in different encoding conditions, and 2) The engagement of the same brain regions during the different encoding conditions. Many of the reviewed studies provide only the first form of evidence but some, such as Wheeler et al. (2000, data shown, copyright © by the National Academy of Sciences), provide both forms.

Strong support for reactivation requires two forms of evidence (Figure 1). The first form is that different brain regions are engaged during the retrieval of information studied in different encoding conditions, and requires brain activity to be measured at test. The second form of evidence is that the same brain regions that were differentially during retrieval were also differentially active during the appropriate encoding conditions, and requires brain activity to be measured at study (Figure 1). While the majority of studies reviewed herein measure brain activity only during test and therefore provide only the first form of evidence, a subset of the studies reviewed also measured brain activity at study and provide both forms of evidence. While the studies that measure neural activity only during test can provide strong support that the encoding experience alters which regions are activated during retrieval, only the studies that measures neural activity during both study and test can provide compelling evidence for reactivation per se.

This review will focus largely on studies conforming to the design presented in Figure 1. In the course of this review, research will be drawn from a wide array of methodologies used to measure brain activity, including event-related potential (ERP), which measures electrical activity at the scalp, positron emission tomography (PET) and functional magnetic resonance imaging (fMRI), which measure blood flow in the brain, and single cell recording, which records electrical activity directly from neurons in vivo. It is worth emphasizing that inferring localization is less straightforward in ERP studies than in studies using other methodologies. All of the ERP studies reviewed herein infer localization from electrode placement, and while more advanced methods of localization exist (Luck, 2005), none of them are as straightforward as the localization provided by fMRI or PET. The ERP studies reviewed in this paper should be interpreted with this limitation in mind, such that effects arising at the same electrode do not necessarily come from the same source. Furthermore, just because an effect arises at a particular electrode (e.g., a parietal electrode) does not necessitate that it arises from the immediately underlying cortex (e.g., parietal cortex).

In the first part of this review, we will summarize the findings from a large number of studies that garner support for the reactivation account of episodic memory by measuring brain activity during the encoding and retrieval of information studied in different encoding conditions. The majority of these studies have explicit encoding manipulations (as in Figure 1). In this review, studies of mental imagery will only be covered to the extent that they speak to the notion of reactivation during episodic or associative retrieval (but for reviews of the reactivation account of visual imagery, see Kosslyn, 1994; Kosslyn & Thompson, 2003; Kosslyn, Thompson, & Ganis, 2006). This review is divided into three major sections. In the first of these, studies will be reviewed that demonstrate that neutral stimuli that are associated with different kinds of stimuli at encoding activate different sets of regions when they are used as retrieval cues at test. The second part of this review focuses on studies that demonstrate that stimuli that are encoded using different strategies activate different sets of regions when they are used as retrieval cues at test. In both of these sections, a number of studies are reviewed that provide both forms of evidence for reactivation by demonstrating that the regions that are differentially active across encoding conditions are also differentially active during retrieval. In the third and final section of this review, we pose several questions about the reactivation process and draw answers from the existing literature as best we can.

Reactivation of stimulus associations

In this section, we review studies that provide converging evidence that when people retrieve an episode from memory, the brain regions involved in processing the sensory and emotional aspects of that episode are reactivated. We divide these studies into three general categories: 1) Studies demonstrating reactivation of sensory cortex during the retrieval of associations of different sensory modalities (Gottfried et al., 2004; Nyberg et al., 2000; Vaidya et al., 2002; Wheeler et al., 2000; Wheeler & Buckner, 2003; Wheeler & Buckner, 2004; Wheeler et al., 2006), 2) A series of studies demonstrating the reactivation of increasingly specific processing modules within the visual system during the retrieval of specific kinds of visual associations (Gratton et al., 1997; Ishai et al., 2000; Khader et al., 2005a; Khader et al., 2005b; Khader et al., 2007; O’Craven & Kanwisher, 2000; Polyn et al., 2005; Rosler et al., 1995; Woodruff et al., 2005), ultimately concluding with some preliminary evidence for reactivation at the level of single cells (Sakai & Miyashita, 1991), and 3) Studies demonstrating reactivation of different brain regions during the retrieval of emotional associations (Gottfried et al., 2004; Lewis et al., 2005; Maratos et al., 2001; Smith et al., 2004). These three categories of studies are reviewed in separate subsections, below. Together, these studies give weight to James’ (1890) claim that “we have thus…not memory so much as memories. The visual, the tactile, the muscular, the auditory” (Vol. I, p. 684).

Sensory associations

Current evidence for the reactivation of sensory regions during the retrieval of sensory associations comes from a group of studies using PET and fMRI (Gottfried et al., 2004; Nyberg et al., 2000; Vaidya et al., 2002; Wheeler et al., 2000; Wheeler & Buckner, 2003; Wheeler & Buckner, 2004; Wheeler et al., 2006). In Nyberg et al. (2000), words were encoded either alone or with an accompanying sound. In Vaidya et al. (2002), objects were encoded either as a word or as a picture. In a group of studies by Wheeler and colleagues, words (e.g., “dog”) were encoded with either an associated sound (e.g., “WOOF!”) or with an associated picture (e.g., a picture of a dog; Wheeler et al., 2000; Wheeler & Buckner, 2003; Wheeler & Buckner, 2004; Wheeler et al., 2006). In all cases, participants were shown only words as probes during test and had to make some decision about them. In the earliest studies, participants made source judgments about what kind of stimulus was associated with each word during study (sound/no sound in Nyberg et al., 2000; picture/sound in Wheeler et al., 2000), but several different decision rules were adopted in later studies (e.g., old/new in Vaidya et al., 2002; picture/sound/new in Wheeler & Buckner, 2003; remember/know/new in Wheeler & Buckner, 2004). The combined results of these studies demonstrate that activity in visual association cortex is reinstated during the retrieval of visual information (Vaidya et al., 2002; Wheeler et al., 2000; Wheeler & Buckner, 2003; Wheeler & Buckner, 2004; Wheeler et al., 2006) and activity in auditory association cortex is reinstated during the retrieval of auditory information (Nyberg, et al., 2000; Wheeler, et al., 2000). Two of these studies demonstrated that the regions that are differentially active during the encoding of visual and auditory associations are also differentially active during their retrieval (Vaidya et al, 2002; Wheeler et al, 2000). The other studies measured brain activity only during retrieval, and did not provide direct evidence of an overlap between encoding and retrieval. In all the studies, the activated sensory regions were located relatively late in their respective perceptual processing streams. Specifically, the fusiform gyrus was associated with the retrieval of visual information, and the superior temporal gyrus was associated with the retrieval of auditory information. In a mini meta-analysis combining their own data set with that of Wheeler et al. (2000), Wheeler and Bucker (2003) demonstrated that memory-related activation was limited to late processing regions within the visual system, and did not include early visual regions adjacent to and including primary visual cortex (V1-V3). The studies that did not require source decisions are compelling because access to the visual and auditory associations is not necessary to make recognition judgments on the cues (Vaidya et al., 2002; Wheeler & Buckner, 2003), whereas these associations are necessarily the basis for context and recollection decisions (Nyberg et al., 2000; Wheeler et al., 2000; Wheeler & Buckner, 2004). These studies demonstrate that unimodal association areas in visual and auditory cortex appear to be reactivated during the retrieval of visual and auditory associations, respectively.

In a study investigating memory for smells, Gottfried and colleagues (2004) presented images of objects with neutral, positive, or negative-valenced odors¹, and asked participants to form associations between each image and its co-presented odor. At test, they were given an image of an object and had to decide whether it was previously studied with an odor (old) or not (new). Relative to new images, images studied with odors were associated with activity in several regions that typically show old/new effects in recognition memory paradigms, including bilateral posterior parietal cortex and left inferior prefrontal cortex (Buckner & Wheeler, 2001). In addition, more activity was also found in primary olfactory (piriform) cortex for items studied with odors relative to new items. This suggests that the part of the brain that was involved in processing the associated odors is reactivated during retrieval. However, the critical contrast in this study was between old items and new items, and this contrast confounds the encoding context (odor presentation) with whether the image was encoded at all. As is made apparent in Figure 1, a more ideal contrast would involve comparing items studied with odors to items that were studied alone or with some other kind of stimulus. However, because similar studies without odor presentation have found no evidence of old/new effects in piriform cortex (e.g., Smith et al., 2004), Gottfried et al.’s (2004) study remains highly suggestive despite its limitations. Overall, there is substantial evidence in the literature to support that sensory regions are reactivated during the retrieval of sensory associations.

The grain size of reactivation within the visual system

As was discussed in the previous section, parts of visual cortex appear to be reactivated during the retrieval of visual associations (Vaidya et al., 2002; Wheeler et al., 2000; Wheeler & Buckner, 2003; Wheeler & Buckner, 2004; Wheeler et al., 2006). More than any other sensory system, the organization of the visual system is well understood on a number of levels, and can be divided into different functional systems and regions (e.g., Felleman & Van Essen, 1991; Mountcastle, 1997; Ungerleider & Mishkin, 1982). A large number of studies have demonstrated that different processing modules within the visual system are reactivated during the retrieval of specific kinds of visual associations. In the following section, we will describe studies that investigate reactivation effects within increasingly small processing modules within the visual system, starting at the level of the cerebral hemispheres and ending at the level of single cells in inferotemporal cortex (IT).

Hemisphere

Stimuli are best recognized when presented to the same visual hemifield (and therefore cerebral hemisphere) during study and test (Gratton et al., 1997). Gratton et al. (1997) chose to investigate the hypothesis that visual memory was organized in a hemispheric manner in a study using ERP. During study, drawings were briefly presented to the left or right of fixation, and at test participants had to distinguish centrally presented studied items from new items. Gratton et al. (1997) demonstrated that correctly recognized old items evoked a lateralized negativity over the hemisphere contralateral to the initial side of presentation. Interestingly, this asymmetry in the ERP occurs despite empirical evidence that participants are incapable of remembering to which hemifield items were presented during study. Gratton et al.’s findings are worth following up with two methodological changes: 1) recording brain activity during both study and test and 2) using a technique that is better at inferring localization (e.g., fMRI). Replicating Gratton et al.’s results with these changes would provide solid evidence for reactivation of the visual system in the appropriate hemisphere during the retrieval of laterally presented cues.

Ventral and dorsal streams

As described earlier, one of the fundamental principles of organization within the visual system is the distinction between a “what” stream running ventrally to the temporal lobes and a “where” stream running dorsally to the parietal lobes (Ungerleider & Mishkin, 1982). In addition to the early PET studies that were already discussed, there have been several studies by Khader and colleagues (Khader et al., 2005a; Khader et al., 2005b; Khader et al., 2007) comparing the retrieval of location associations with the retrieval of object associations. Additionally, several studies by Heil and colleagues (Heil et al., 1996; Heil et al., 1997; Rosler et al., 1995) tapped the ventral/dorsal distinction by comparing memory for colors, spatial locations, and words. The paradigm used in all of these studies was the modified fan paradigm designed by Heil et al. (1994), in which participants had to decide whether two stimuli with a varying number of associations, or fan (Anderson, 1974), shared a common associate.

Rosler et al. (1995) used this paradigm and measured negative slow wave ERPs during the retrieval of verbal, color and spatial information. Slow wave ERPs are event-locked waveforms in the DC frequency range that are recorded from the scalp, and negative polarity slow wave ERPs have been shown to reflect increased neural activity in underlying cortex (McCallum et al., 1993; Speckmann et al., 1984). Furthermore, there is evidence that the duration and topography of negative slow wave ERPs reflect the duration and modularity of the underlying cognitive processes, respectively (Rosler et al., 1997). During the study session, words or line drawings were associated with 1 to 3 colors, 1 to 3 spatial locations, or 1 to 3 concrete nouns. For any given retrieval cue, all the associated items were of the same type (i.e., all were colors, locations, or nouns). At test, two retrieval cues were presented and participants had to decide whether or not they shared a common associate. This requires participants to retrieve some number of associations until a match is found. Of relevance to the current discussion, the retrieval of color information was associated with increased negative slow waves over occipital and temporal electrodes and the retrieval of location information was associated with increased negative slow waves over parietal electrodes. Relative to the other two conditions, the retrieval of concrete nouns was associated with increased negative slow waves over left frontal electrodes. In a pair of similar studies without a color condition, Heil et al. (1996; 1997) also found that the retrieval of spatial information was associated with increased negative slow waves over occipital and temporal electrodes and the retrieval of concrete nouns was associated with increased negative slow waves over left frontal regions.

Khader and colleagues performed a series of ERP and fMRI studies using Heil et al.’s (1994) modified fan paradigm that measured neural activity at test and in some cases at study (Khader et al., 2005a; Khader et al., 2005b; Khader et al., 2007). In Khader et al.’s studies, participants associated abstract words (e.g., concept) with 1 or 2 visual objects (faces or cups) or 1 or 2 spatial positions. At test, two words with the same kinds of associates were presented and participants had to decide whether or not they shared a common associate. Overall, the findings from the ERP and fMRI data were consistent and provided interesting converging information. The fMRI data provided evidence that the retrieval of object information was associated with widespread activity in the ventral stream in addition to striate and parastriate cortex (Khader et al., 2005a). The retrieval of spatial information was associated with widespread activation in the occipital and parietal components of the dorsal processing stream, including inferior and superior parietal cortex. There were corresponding differences in the topography of negative slow wave ERPs between the object and spatial conditions that were present at encoding as well as retrieval (Khader et al., 2005b; Khader et al., 2007). Together, studies using slow wave ERP and fMRI provide converging evidence that the retrieval of object and spatial information reactivates the ventral and dorsal processing streams, respectively.

Processing modules within the ventral stream

The visual system has been theoretically divided into processing modules at a much finer grain size than that of the ventral/dorsal distinction. There is a body of evidence that different categories of visual stimuli show distinct topographies of activation within the ventral stream (Ishai et al., 1999). Specifically, a portion of the fusiform gyrus called the fusiform face area (FFA) is preferentially activated by faces (Kanwisher et al., 1997; Ishai et al., 1999), a portion of the parahippocampal gyrus called the parahippocampal place area (PPA) is preferentially activated by visual scenes (Epstein & Kanwisher, 1998; Epstein et al., 1999; Ishai et al., 1999), and a portion of the left fusiform gyrus called the visual word form area (VMFA) is preferentially activated by visually presented words (McCandliss et al., 2003). Ishai et al. (1999) also found a region that is adjacent to the PPA and FFA and is more activated by images of chairs than images of faces or places. Early evidence for the reactivation of different regions in the ventral stream from memory comes from fMRI studies of visual imagery for different categories of visual objects (Ishai et al., 2000; O’Craven & Kanwisher, 2000). O’Craven and Kanwisher found that both perception and imagery of famous faces activated the FFA and both perception and imagery of familiar buildings activated the PPA. Ishai et al. (2000) found that distinct regions within the ventral stream were activated by faces, places, and chairs, and that portions of these same regions were activated during imagery for these categories of objects. Both Ishai et al. (2000) and O’Craven and Kanwisher found that the activation during imagery is weaker and has a smaller spatial extent than activation during perception. While these studies provide strong evidence that the generation of mental images from semantic memory relies on the same regions of the brain that process actual images, they do not provide evidence that remembering specific occurrences of images reactivates the regions engaged during those occurrences.

Later studies demonstrated reactivation in specific parts of the ventral stream using episodic or associative memory paradigms (Ranganath et al., 2004; Polyn et al., 2005; Woodruff et al., 2005). Woodruff et al. (2005) had participants study visually presented words and pictures. At test, participants were asked to make remember/know judgments (Tulving, 1985) when presented with new words, words that were previously studied as words, and words that were previously studied as pictures. Woodruff et al. found that recognition of words studied as pictures was associated with activity in a part of the fusiform gyrus implicated in object recognition, while recognition of words studied as words was associated with activity in the VWFA. Ranganath et al. (2004) had participants learn face-house pairs and then perform a delayed paired associate task in which participants were presented a face or house, and had to think of its associate during a delay interval. Ranganath et al. found that when the cue item was a face, there was initially more activity in FFA but more activity in PPA during the delay. Likewise, when the cue item was a house, there was initially more activity in PPA but more activity in FFA during the delay. These activation patterns are indicative of an initial processing of the cue stimulus followed by the reactivation of the brain areas involved in processing its associate. That is, activity during the delay period reflected the kind of stimulus being retrieved. Similarly, Polyn et al. (2005) investigated reactivation during free recall of different categories of visual objects. Participants were scanned using fMRI while they studied photographs of famous faces, famous locations, and common objects and then freely recalled verbal labels of the studied photographs. Polyn et al. found that the patterns of activity during the perception of faces, locations, and objects were reinstated immediately preceding their retrieval. In fact, a neural network pattern classifier trained on the study data was capable of reliably predicting which category of item was being retrieved based on the activation pattern during retrieval. Of particular interest to this discussion, voxels comprising the PPA and FFA were crucial for correct classification, but notably so were non-peak voxels in other areas of the visual system. These studies provide converging evidence that the different parts of the brain that are activated during the perception of different kinds of visual objects are reactivated during when they are retrieved from memory.

Single cells within inferotemporal cortex

Empirical evidence from neuroimaging studies indicates that visual reactivation of encoding context can occur at grain sizes ranging from the cerebral hemispheres (Gratton et al., 1997) all the way down to category-specific processing modules within the ventral processing stream (Ranganath et al., 2004; Polyn et al., 2005; Woodruff et al., 2005). We will conclude by discussing some research that suggests that reactivation may be occurring at the level of single cells in the monkey visual system (Sakai & Miyashita, 1991). This groundbreaking neurophysiology research was critical in motivating some of the imaging research already discussed (e.g., Ishai et al., 2000; Ranganath et al., 2004). Sakai and Miyashita (1991) discovered two different kinds of memory-sensitive cells in inferotemporal cortex (IT) using a paired associated paradigm. In their task, monkeys learned to associate 12 randomly assigned pairs of Fourier descriptors (parametrically defined snowflake-like objects). At test, monkeys were first presented with one member of a pair as a cue, A₁, and then after a delay they had to correctly identify the other member of the pair, A₂, when presented with A₂ and a distractor stimulus, B. The first kind of memory-sensitive cell categorized by Sakai and Miyashita (1991) were pair-coding neurons. Pair-coding neurons respond most strongly when presented with either member of a learned pair as a cue (A₁ or A₂), and these cells are statistically more common than neurons that respond most strongly to two cue stimuli that were not paired during learning (e.g., A₁ and B). The second kind of memory-sensitive cell categorized by Sakai and Miyashita (1991) were pair-recall neurons. Pair-recall neurons respond most strongly to A₂ during the cue period but most strongly to A₁ during the delay period. That is, pair-recall neurons prefer one stimulus, but will fire in response to its associate after a short delay -- as if the preferred response is being retrieved by association.

The demonstration of reactivation at the cellular level requires at least two pieces of evidence: 1) Before A₂ is associated with A₁, a cell responds strongly to A₂ but not A₁, 2) After A₂ is associated with A₁, the cell responds strongly to both A₂ and A₁ (either immediately or after a delay). The current data on pair-coding and pair-recall neurons provides only the second piece of evidence required to support a reactivation account. While there is evidence that both pair-coding and pair-recall neurons are acquired through learning (Erickson & Desimone, 1999; Sakai & Miyashita, 1994), there is a missing link in that there is not yet any evidence that either kind of neuron prefers A₂ before training. Without this evidence there are several possible explanations for the behavior of these cells, only one of which is reactivation: 1) Before A₁ and A₂ are associated, the cell responds to A₂ but not A₁. After they are associated, the cell responds to both (reactivation). 2) Before A₁ and A₂ are associated, the cell responds to neither A₁ nor A₂. After they are associated, the cell responds to both. 3) Both before and after A₁ and A₂ are associated, the cell responds to both A₁ and A₂. Option 3 is unlikely given the evidence that pair coding neurons are statistically more common than neurons that responds to two unassociated stimuli (Sakai and Miyashita, 1991). However, without collecting data on the response profile of these cells prior to training, option 2 remains a compelling alternative explanation to reactivation. The lack of conclusive data on this front is likely due to methodological difficulties in maintaining a cell recording for such an extended period of time. Only future research and technological advances will reveal the learning profile of these IT cells.

Emotional associations

Emotional regions of the brain are activated during the retrieval of emotional associations. A group of studies by Rugg and colleagues have focused on comparing the topography of neural activity during the retrieval of items studied in positive emotional, negative emotional, and non-emotional contexts (Gottfried et al., 2004; Maratos et al., 2001; Smith et al., 2004). In an fMRI study by Maratos et al. (2001), emotionally neutral nouns (e.g., “corn”) were studied in the context of emotionally neutral sentences (“The farm labourers began harvesting the corn”), negatively valenced emotional sentences (“The farmer was shredded when he fell into the corn grinder”), or positively valenced emotional sentences (“The farmer was overjoyed with his bountiful crop of corn.”). Similarly, in the fMRI study by Smith et al. (2004), emotionally neutral objects were studied in the context of emotionally neutral scenes, negatively valenced emotional scenes, or positively valenced emotional scenes. In the fMRI study lead by Gottfried (Gottfried et al., 2004), discussed earlier in the section on olfactory associations, emotionally neutral images of objects were studied in the context of neutral odors, unpleasant odors, or pleasant odors. In all the studies, participants had to decide whether an emotionally neutral retrieval cue was previously studied (old) or not (new).

In these studies, several regions implicated in emotional processing were more active during recognition of items studied in emotional contexts relative to neutral contexts, including the amygdala, the insula, and orbitofrontal cortex (Maratos et al., 2001; Smith et al., 2004). Smith et al. (2004) found that the amygdala was more activated by emotional contexts than neutral contexts at both encoding and retrieval. In all of the studies, differences were also found between negative and positive emotional contexts during encoding (Smith et al., 2004) and retrieval (Gottfried et al., 2004; Maratos et al., 2001; Smith et al., 2004). The most consistent finding across studies was that, compared to positive emotional memories, negative emotional memories are associated with more activity in the parahippocampal gyrus and other visual processing regions in the temporal lobe (Gottfried et al., 2004; Maratos et al., 2001; Smith et al., 2004). Smith et al. (2004) found that the fusiform gyrus was more active during both the encoding and the retrieval of negative emotional information. This suggests that negative emotional memories are accompanied by more robust sensory engagement during encoding and correspondingly strong sensory reactivation during retrieval. However, because in the majority of these studies the negative contexts were rated as more emotional than the positive contexts, it is difficult to ascertain whether this difference in temporal cortex activity between negative and positive contexts is a result of their valence or their degree of emotionality. Aside from temporal cortex consistently being more strongly engaged during the retrieval of negative emotional information, the specific regions that were more active during the retrieval of negative and positive information varied substantially across studies. For example, Gottfried et al. (2004) found that left lateral orbitofrontal cortex and the right hypothalamus were more active during the retrieval of words associated with negative smells, while left medial orbitofrontal cortex and right anterior insula were more active during the retrieval of words associated with positive smells. In contrast, Maratos et al. (2004) and Smith et al. (2004) implicated almost entirely different sets of regions in the retrieval of negative and positive associations. While these studies provide strong evidence that there is a difference in which regions are active during the encoding and retrieval of emotional compared to neutral information, little consistency comes from their results comparing the retrieval of negative and positive emotional information.

There is substantial evidence for mood congruence in memory, such that information with an emotional valence is better remembered in mood states with a matching valence (Bower, 1981; Blaney, 1986; Eich, 1995). For example, it is easier to remember negative events when you are in a sad mood, and it is easier to remember positive events when you are in a happy mood. According to Bower (1981), mood congruence could be interpreted within an associative network of connected nodes, wherein moods and episodes with the same emotional valence share one or more nodes (a positive or negative affect node, for example). In such a network, mood congruence effects can be explained in much the same way as any context effect: a node is activated (in this case by the mood), and this node in turn spreads the activation and boosts the activation of relevant episodes. Lewis et al. (2005) conjectured that such a theory could be probed with fMRI, and predicted that mood congruence would be accompanied by a boost in activation in the parts of the brain involved in the encoding of valence-specific material during retrieval. In their study, Lewis et al. had participants study lists of positive and negative words using a self-referential deep encoding task, and then had participants perform a remember/know task on studied and unstudied words in blocks in which negative and positive moods were induced. Lewis et al. performed a congruence analysis to highlight regions that showed subsequent memory effects during encoding but also showed mood congruence effects during retrieval for positive and negative moods. In line with their predictions, Lewis et al. found that the right subgenual cingulate was more active during the encoding of later remembered positive words compared to forgotten positive words, and also that this region was more active for positive words remembered in a positive mood compared to positive words remembered in a negative mood. Likewise, they found that the left posterolateral orbitofrontal cortex was more active during encoding for later remembered negative words than forgotten negative words, and also that this region was more active for negative words remembered in a negative mood compared to negative words remembered in a positive mood. Lewis et al. propose that the subgenual nucleus and posterolateral orbitofrontal cortex are good candidates for the positive and negative nodes in an associative memory system, respectively, insofar as they are involved in the encoding of valence-specific information and boost in activation during mood congruence. Overall, the studies investigating the retrieval of emotional information demonstrate that the regions generally involved in the encoding of emotional information, such as the amygdala, are reactivated during retrieval. Furthermore, a pair of regions involved in encoding the specific valence of an emotional experience show mood congruence effects during retrieval, suggesting that reactivation of encoding regions may play a role in improving memory performance during mood congruence.

Reactivation of regions involved in strategic processing

In the previous section, we presented evidence from empirical studies that when people retrieve an episode or association from memory, the regions of the brain involved in originally processing that episode or association are reactivated. Reactivation of sensory regions was demonstrated across several modalities (e.g., Gottfried et al., 2004; Nyberg et al., 2001; Wheeler et al., 2000) and could be observed at several grain sizes in the visual system (e.g., Gratton et al., 1997; Khader et al., 2005a; Polyn et al., 2005; Rosler et al., 1995; Sakai & Miyashita, 1991). In addition, during the encoding and retrieval of emotional information, increased activity has been observed in the amygdala (Smith et al., 2004), and mood congruence effects have been shown in valence-specific areas of cortex (Lewis et al., 2005). However, the quality of the encoding experience varies in more ways than the sensory and emotional content of the actual stimuli being encoded. Particularly, individuals exert control and bring different strategies to bear when they are trying to encode information. A number of studies have demonstrated that which regions are activated at test is influenced by what strategic processes were engaged during study. These studies investigated the retrieval of material encoded via several strategies, including enactment, verbal mnemonics, and the mnemonic use of mental imagery (Heil et al., 1999; Johnson & Rugg, 2007; Johnson et al., 2008; Kahn et al., 2004; Lundstrom et al., 2003; Nilsson et al., 2000; Nyberg et al., 2001). In the following section, we review evidence that suggests that brain regions involved in strategic encoding are reactivated during subsequent retrieval. In the final section of this paper, we return to this topic in order to discuss what psychological processes may underlie the reactivation of region involved in strategic encoding.

Johnson and Rugg (2007) performed an fMRI study in which participants were presented with words describing objects and were instructed to either generate a sentence including the word (verbal condition) or imagine the word’s referent in a visual scene (imagery condition). At test, participants made remember/know judgments about studied and unstudied words (Tulving, 1985). The comparison between study conditions was masked with a Remember > Know contrast, such that only regions that were differentially involved in recollection (Remember judgments) across the two conditions were selected. In the study, recollected items studied verbally were associated with more activity in left ventromedial frontal gyrus during both study and test, while recollected items studied using visual imagery were associated with increased activity in the fusiform gyrus and left occipital cortex during both study and test (Johnson & Rugg, 2007). Since the fusiform gyrus and occipital cortex have been implicated in the perception of objects and scenes as well as in the generation of mental images (Ishai et al., 2000; Kosslyn et al., 1997), this result suggests that the retrieval of words in the imagery condition involved the reengagement of sensory regions used to generate mental images during study. What role the left ventromedial prefrontal cortex played in the generation of sentences and the retrieval of words in the sentence condition is somewhat unclear, but it was nonetheless engaged during both study and test in that condition.

Kahn et al. (2004) performed a similar study in which participants encoded adjectives either by imagining a place that could be described by that adjective (imagery condition) or by orthographic-to-phonological transformation (read condition). At test, participants were given an adjective and had to decide whether it was new, studied in the imagery condition (old-imaged), or studied in the read condition (old-read). Correctly identified old-imaged items showed increased activation in the PPA, a region that is associated with perceiving (Epstein & Kanwisher, 1998) and imagining (O’Craven & Kanwisher, 2000) visual scenes, while correctly identified old-read items showed increased activation in left premotor/posterior ventrolateral prefrontal cortex, a region that has been associated with processing language since the landmark neuropsychological work of Broca (1861). Both of these differences were present at both study and test, providing further support that the regions engaged in mnemonic processing during study are reengaged during test.

So far, the reviewed studies on the mnemonic use of mental imagery have compared information studied using mental imagery with information studied using a verbal strategy. All of these studies have found that, relative to verbal strategies, information studies using imagery mnemonics engages more visual regions during retrieval (e.g., PPA, fusiform gyrus). In a study performed by Lundstrom and colleagues (Lundstrom et al., 2003), the critical contrast compared information studied via the generation of mental images with information studied via the perception of actual images. Participants studied words describing objects (e.g., “apple”) either by viewing an image of the object (viewed condition) or by imagining the object (imagined condition). At test, participants were presented with studied and new words and had to make old/new and source (old-viewed or old-imagined) decisions about the presented words. More activation was found in the precuneus during the retrieval of imagined objects compared to viewed objects. The precuneus has been proposed to play a critical role in the generation of mental images (Fletcher et al., 1995). This is particularly interesting because both conditions (viewed and imagined) might be expected to involve the generation of mental images of the word’s referent during retrieval. However, only in the case of the imagery condition were mental images also generated during the original encoding experience. No regions were significantly more engaged in the viewed compared to the imagined condition.

The enactment effect is the phenomenon by which lists of action phrases (e.g., “brush your teeth”, “bounce the ball”) are better remembered when the appropriate actions are actually performed at study (for review, see Engelkamp, 1998). It has been suggested by Engelkamp that this effect is dependent on the reactivation of motor signals during retrieval. A handful of PET (Nilsson et al., 2000; Nyberg et al., 2001) and slow wave ERP (Heil et al., 1999) studies have provided some evidence that motor cortex is reactivated during the retrieval of enacted action phrases. Heil et al. (1999) used slow wave ERPs to investigate the question of motor reactivation. Participants studied action phrases either with or without enacting them (between-subject), and had to later distinguish studied items (old) from unstudied items (new) while scalp recordings were performed. Enhanced negative slow wave ERPs were apparent at fronto-central scalp electrodes in the enacted group, suggesting that enacting action phrases during study caused the reactivation of motor cortex at test.

In a pair of PET studies, Nyberg and colleagues (Nilsson et al., 2000; Nyberg et al., 2001) refined the topography of the motor reactivation. Participants studied action phrases by enactment, by imagined enactment,or without enactment. At test, participants were given a verb and had to provide the appropriate noun (e.g., “bounce the _____”). Relative to action phrases that were not enacted, enacted action phrases were associated with more activity in premotor, primary motor, and primary somatosensory cortex during both encoding (Nyberg et al., 2001) and retrieval (Nilsson et al., 2000; Nyberg et al., 2001). Encoding by imagined enactment evoked more activity than verbal encoding in premotor, motor, and somatosensory cortex during both encoding and retrieval, but less activity than encoding by actual enactment (Nyberg et al., 2001). Nyberg et al. (2001) not only clearly delineated the topography of the motor reactivation effect, but also explicitly demonstrated an overlap between the activation present during the enactment and imagined enactment of action phrases and the later retrieval of these same phrases. The demonstration by Nyberg and colleagues of activation in motor and somatosensory cortex during the retrieval of enacted information complements the results of Wheeler et al. (2000) demonstrating the reactivation of association cortex during the retrieval of auditory and visual information. Additionally, the demonstration of motor and sensory cortex activity during imagined enactment mirrors the demonstration of activity in sensory regions during the retrieval information studied using visual imagery (Johnson et al., 2007; Johnson et al., in 2008; Kahn et al., 2004), and also suggests that retrieval of imagined experiences may engage sensorimotor regions to a lesser extent than the retrieval of actual experiences, although Lundstrom et al. (2003) found no evidence for this in their study comparing visual imagery and perception.

The studies investigating the retrieval of information studied using different strategies demonstrate imagery-related reactivation in the precuneus (Lundstrom et al., 2003) and sensorimotor regions of cortex (Johnson et al. 2007; Johnson et al., in 2008; Kahn et al., 2004; Nilsson et al., 2000; Nyberg et al., 2001), reactivation related to verbal processing in various parts of the left prefrontal cortex (Johnson et al., 2007; Johnson et al., in 2008; Kahn et al., 2004), and reactivation related to actual or imagined enactment in motor and somatosensory cortex (Heil et al., 1999; Nyberg et al., 2001). Together, these studies provide converging evidence that some of the regions engaged in the service of particular encoding strategies are reengaged during retrieval of the encoded information. We return to this topic in the next section, where we theorize about what cognitive processes underlie this reactivation of strategic encoding regions.

Discussion and Open Questions

We have reviewed the substantial evidence that the brain areas that are active during encoding are reactivated during retrieval. Evidence was drawn from studies using ERP, PET, fMRI, and single cell recording. We presented converging evidence that brain regions that are activated during encoding are reactivated during the retrieval of different sensory and emotional associations. For example, auditory and visual association cortex are reactivated when auditory and visual associations are retrieved (Vaidya et al., 2002; Wheeler et al., 2000), distinct regions within the visual system are reactivated during the retrieval of specific kinds of visual stimuli (Khader et al., 2007; Polyn et al., 2005), and the amydala and other regions implicated in emotional processing are reactivated during the retrieval of emotional information (Lewis et al., 2005; Smith et al., 2004). Additionally, some regions that are strategically recruited during encoding appear to be reactivated during retrieval. For example, the precuneus and sensory regions are activated during the retrieval of information encoded using mental imagery (Johnson & Rugg, 2007; Johnson et al., 2008; Kahn et al., 2004; Lundstrom et al., 2003), left prefrontal regions are activated during the retrieval of information encoded verbally (Johnson & Rugg, 2007; Kahn et al., 2004), and motor and somatosensory regions are activated during the retrieval of information encoded via actual or imagined enactment (Heil et al., 1999; Nilsson et al., 2000; Nyberg et al., 2001). All of this research implies that remembering an episode involves partially reconstructing the brain state that was present during encoding. In this section, several questions are posed about this reactivation process. In some cases, there is research to be drawn on to answer these questions, and in others, the questions remain unanswered and will hopefully motivate future research. The questions are discussed in separate subsections, below.

How is reactivation affected by the amount of retrieved information?

One might expect that the strength of the reactivation should scale with the amount of information that is retrieved. In fact, the slow wave ERP and fMRI studies using Heil et al.’s (1994) modified fan paradigm have uniformly provided evidence for this assertion during the retrieval of different kinds of associations (Heil et al., 1996; Heil et al., 1997; Khader et al., 2005a; Khader et al., 2005b; Khader et al., 2007; Rosler et al., 2007). As discussed earlier, in the modified fan paradigm, participants are given two retrieval cues with varying degrees of fan and have to decide if they have a common associate. Crucially, high fan items required an increased number of retrievals relative to low fan items, because participants had to serially retrieve the associates of each item until a match was found². Rosler et al. (1995) and Heil et al. (1996, 1997) found that the negativity of slow waves at parietal electrodes increased as the number of associated spatial locations increased, and that negativity of slow waves at left frontal electrodes increased as the number of associated nouns increased. Khader et al. (2005a, 2005b, 2007) similarly found content-specific fan effects during the retrieval of objects and locations. This demonstrates that neural activation scales with the amount of retrieved information in a content-specific manner.

In order to investigate the differences between remembering real and laboratory events, Cabeza et al. (2004) used a unique paradigm in which participants were given cameras and instructed to take photographs of specific campus locations. Additionally, they came into the laboratory and studied pictures of the same campus locations taken by other participants. In a test session, participants were presented with photographs they had taken, photographs they studied in the laboratory, and new photographs. Relative to photographs studied in the lab, photographs taken by the participants were associated with more activity in visual cortex and the PPA during retrieval. Cabeza et al. (2004) hypothesize that because of the richer encoding context, more visual details were retrieved during the recognition of pictures taken in real life compared to pictures studied in laboratory, and there was correspondingly more activity in the parts of the brain involved in processing visual information. This provides converging evidence that more reactivation occurs when more information is retrieved.

How is reactivation affected by the accessibility of retrieved information?

We just presented evidence that reactivation occurs more strongly when more retrievals are made, but how is the amount of reactivation affected by the accessibility of a memory, as might be manipulated by frequency, recency, or associative strength? Reactivated regions might behave similarly to the left mid/posterior ventrolateral prefrontal cortex (VLPFC), which is more active when a memory is less accessible (e.g., Danker, Gunn, & Anderson, 2008; Thompson-Schill et al., 1997; Wagner et al., 2001). That is, the reactivation signal may increase as more effort or control needs to be exerted over the retrieval process. On the other hand, reactivated regions may show the opposite pattern, with more accessible memories showing more robust reactivation signals because the information is more strongly “re-experienced.” Nyberg et al. (2000) attempted to explore this question by presenting words with sounds either two or six times. Their results were inconsistent and not significant, with the left auditory cortex tending to reactivate more strongly when frequency increased and right auditory cortex tending to reactivate less strongly when frequency increased. Wheeler and Buckner (2003) likewise manipulated frequency by presenting words with corresponding pictures or sounds either once or 20 times. They found that increasing frequency during study reduced activity in the left VLPFC during retrieval, but found no frequency differences in the regions of visual and auditory association cortex that were reactivated during the retrieval of pictures and sounds. It remains unclear whether this null effect represents a true insensitivity of these regions to frequency or whether there was simply not enough power to detect such differences. Based on current research, the relationship between the accessibility of the retrieved information and the amount of reactivation in encoding regions is unclear and should be a subject for future research.

What is the relationship between reactivation and subjective ratings of remembering?

Grounded theories of cognition propose that remembering is accompanied by a subjective experience of reliving the remembered episode, and that the reactivation of encoding regions that occurs during retrieval corresponds to this experience (Barsalou, 2008). If this were the case, one would expect that subjective ratings of remembering should correlate with the amount of reactivation in encoding regions during retrieval. Studies comparing recognition by familiarity with explicit recollection of source information suggest that the amount of reactivation is related to subjective ratings of remembering (Johnson & Rugg, 2007; Johnson et al., in 2008; Wheeler et al., 2004; Woodruff et al., 2005). These studies used the remember/know paradigm (Tulving, 1985), a recognition memory paradigm in which participants must distinguish new items from items that are recollected (remember) and items that are merely familiar (know). These studies find that information that is endorsed as remembered is associated with greater reactivation in relevant encoding regions than information that is endorsed as known.

For example, Wheeler et al. (2004) found that the portion of the fusiform gyrus previously associated with the retrieval of visual information (Wheeler et al., 2000) reactivated for items endorsed as remembered, but not items endorsed as known. Additionally, in their ERP study, Johnson et al. (2008) only found differences between the retrieval of verbal and imagined information for items endorsed as remembered. Similarly, in their fMRI study, Johnson and Rugg (2007) found that the content-specific activations associated with the retrieval of verbal and imaginal information were greater for items endorsed as remembered than items endorsed as known. These findings raise two possibilities: 1) that encoding context is reinstated in the brain only when stimuli are endorsed as remembered, or 2) the encoding context is reinstated more strongly when the stimuli are endorsed as remembered. Since fMRI data is correlational by nature, whether the reactivation drives the remember response or vice-versa is unknown.

Daselaar et al. (2008) collected subjective ratings of remembering in the context of autobiographical memory. In their study, participants were given cue words and had to retrieve a relevant autobiographical memory and hold it in mind. After a delay, participants rated the memory on emotional intensity and on the degree to which they felt like they were reliving the experience. Daselaar et al. found that the degree of activation in auditory and visual association cortex was positively correlated with participant ratings of reliving, and that the degree of activation in a number of regions, including the amydala, was positively correlated with participant ratings of emotionality.This study suggests a continuous relationship between subjective ratings and the amount of reactivation that occurs during retrieval.

Several studies have shown that reactivation correlates with subjective ratings of remembering. However, it is worth noting that reactivation is not always associated with conscious awareness of the encoding experience. For example, Gratton et al. (1997) found hemispheric differences in the ERP for stimuli studied in different hemifields even though their participants were not capable of remembering to which hemifield stimuli were presented. Furthermore, several studies on false memory found that stimuli were endorsed as remembered even when reactivation is reduced (Schacter et al., 1996; Slotnick & Schacter, 2004) or completely absent (Fabiani et al., 2000). This begs the question, is the reactivation signal used to make recollection decisions, and if so, why is the difference in reactivation signal not enough to distinguish true and false memories? Furthermore, what sources of information are used by the system during familiarity decisions and false memories? Future research should be directed at clarifying what aspects of reactivation are consciously accessibly and related to subjective experiences of remembering.

What other factors determine which encoding regions are reactivated?

There is substantial evidence that regions involved in encoding information are reactivated when that information is later remembered. Many of the reviewed studies found that only a subset of the regions active during encoding are reactivated and of those that are reactivated, a smaller proportions of the regions are activated during retrieval than encoding (Ishai et al., 2000; Johnson & Rugg, 2007; Lewis et al., 2005; Nyberg et al., 2000; Nyberg et al., 2001; O’Craven & Kanwisher, 2000; Wheeler et al., 2000). We have discussed evidence that reactivation is greatest when information is endorsed as recollected (Johnson & Rugg, 2007; Johnson et al., in 2008; Wheeler et al., 2004), and that sensory and emotional regions are reactivated more when subjective ratings of reliving and emotionality are greatest, respectively (Daselaar et al., 2008). The results seem to suggest that reactivation occurs when information from the reactivated regions is required to make a source decision (Nyberg et al., 2000; Wheeler et al., 2000), or remembering is accompanied by explicit recollection (Johnson & Rugg, 2007; Johnson et al., in 2008; Wheeler et al., 2004). It makes sense that reactivation would be only partial because attention is limited during encoding and the memory system must reproduce the original episode based on only partial information in the form of retrieval cues. Connectionist memory models have this character of partially reactivating relevant units from retrieval cues (Rumelhart & McClelland, 1986).

Barsalou (1999) made the argument that “a perceptual symbol is not the record of the entire brain state that underlies perception. Instead, it is only a very small subset that represents a coherent aspect of the state” (p. 583). We discussed an explicit analysis demonstrating that reactivation effects were restricted to late perceptual processing regions (Wheeler & Buckner, 2003), but also presented evidence that early perceptual regions, such as primary visual cortex (Slotnick and Schacter, 2004) and piriform cortex (Gottfried et al., 2004) can be reactivated during retrieval. Furthermore, the ERP findings of Gratton et al. (1997) suggest that reactivation may be occurring in retinotopic visual cortex. This begs the question, under what conditions are early perceptual regions reactivated? Kosslyn and Thompson (2003) performed a meta-analysis to directly answer this question with regard to the activation of early visual regions (BA17/18) during visual mental imagery. Kosslyn and Thompson found that across studies a combination of methodological and task-related factors influences whether activation was found in early visual regions. Specifically, they found that studies using more sensitive techniques, such as fMRI, were more likely to find activation in early visual regions than studies using less sensitive techniques, such as PET or single photon emission computed tomography (SPECT). They also found that certain properties of the task predicted whether early visual regions were activated, such as whether the task required attention to high-resolution details and whether the task involved imagining shapes as opposed to spatial relations. Future research should be directed at determining how these kinds of factors influence reactivation of encoding regions during retrieval more generally, and not just during the retrieval of visual information.

What are the temporal properties of reactivation?

It is important not only to understand which regions are reactivated and how much they are reactivated, but also when this reactivation occurs. While many of the studies reviewed herein have used fMRI and PET, which have poor temporal resolution and are not well suited to understanding the temporal characteristics of reactivation, there are still many studies, particularly those using ERP, that have provided information about the temporal characteristics of reactivation. For example, in a follow-up to their fMRI study on the retrieval of information studied using verbal or imagery mnemonics, Johnson et al. (2008) recorded event-related potentials in the same paradigm. They replicated their fMRI results by finding that recollected items studied verbally were associated with more positive ERPs over left frontal locations, while recollected items studied using visual imagery were associated with more positive ERPs over posterior locations. Critically, the left frontal effect occurred as early as 300ms after stimulus presentation, providing the best indication to date of how early brain regions may reactivate during retrieval. We believe it would be valuable for future research to record ERPs during both encoding and retrieval, as this would allow researchers to demonstrate an explicit overlap between encoding and retrieval and would allow researchers to compare the temporal dynamics of activation during encoding to the temporal dynamics of reactivation during subsequent retrieval.

In their single cell recording study, Sakai and Miyashita (1991) found that pair recall neurons responded to the associate of the preferred stimuli only after a delay, while pair coding neurons responded immediately to either member of an associated pair. If both kinds of cells are in fact indicative of reactivation, than pair coding neurons reactivate much earlier than pair recall neurons (but see the earlier discussion of their work for alternative interpretations). Similarly, Ranganath et al. (2004) found that the reactivation of encoding regions occurred substantially later than the activation of regions involved in processing the retrieval cues (several seconds), while Polyn et al. (2005) found that reactivation occurred relatively early and preceded actual recall. This raises the question, what determined the speed of reactivation? One possibility is the presence of a retrieval cue, as Ranganath et al.’s participants were performing cued recall (with delayed recognition) while Polyn et al.’s participants were performing free recall. The time required to process the retrieval cue may have delayed reactivation in Ranganath et al.’s study. Another important factor may be the accessibility of the learned information, which we discussed in an earlier section. Information that is easier to access may reactivate encoding regions more quickly during retrieval. Accessibility could be speeded either by learning the material better (e.g., manipulating frequency or depth of encoding) or providing better retrieval cues (e.g., manipulating associative strength). In order to better understand the variability in when reactivation occurs, future research should manipulate these factors and measure the speed of reactivation.

Does reactivation differ for information that is falsely remembered?

We have seen that some of the brain regions activated during encoding are reactivated during retrieval of studied information. However, what happens when a remembered episode never actually occurred? In this case, the individuals who are falsely remembering must be relying on some sort of neural signal to decide that the information is in fact veridical. This information could come from the areas that would have been activated during encoding, or it could come from some other source, such as parts of parietal cortex that show old/new effects (Wheeler & Buckner, 2003) or parts of the medial temporal lobe (Cabeza et al., 2001). In their study investigating the effect of different encoding strategies on subsequent retrieval, Kahn et al. (2004) found that adjectives falsely endorsed as studied using a certain strategy evoked more activity in the regions associated with using that strategy. Specifically, adjectives falsely endorsed as studied in the imagery condition evoked more activity in the parahippocampal gyrus, while adjectives falsely endorsed as studied in the read condition evoked more activity in left premotor/VLPFC. Because of the correlational nature of fMRI, it remains unclear whether these activations caused the false memory decisions or vice versa.

A number of studies have demonstrated that true memories show stronger neural reactivation during retrieval than false memories (Fabiani et al., 2000; Schacter et al., 1996; Slotnick & Schacter, 2004). These studies focused on the reactivation of sensory cortex during retrieval. A pair of studies by Schacter and colleagues investigated the reactivation of sensory cortex during the retrieval of true and false memories for auditorily presented words (Schacter et al., 1996) and visually presented objects (Slotnick and Schacter, 2004). Schacter et al. performed a PET study using the Deese/Roediger-McDermott (DRM) paradigm. During study, participants listened to lists of words (e.g., bed, pillow, nap, etc.) related to critical non-presented words (e.g., sleep). At test, participants were visually presented with studied words (true targets), critical non-presented words (false targets), and non-presented words unrelated to a studied list (new items). In the DRM paradigm, participants have a strong tendency to falsely endorse the critical non-presented words as studied (Roediger & McDermott, 1995). Schacter et al. (1996) compared neural activity during the recognition of “false targets” with neural activity during the recognition of “true targets.” They found activity in superior temporal gyrus, a part of auditory association cortex, was stronger during the recognition of true targets compared to false targets. They interpreted these findings as suggesting that true memories are associated with the reactivation of the sensory context in which they were studied, while false memories are not. It is crucial that in their paradigm words were studied auditorily, but tested visually, such that an increase in auditory cortex activity during test cannot be attributed to differences in attentional or perceptual processing of the retrieval cue.

Slotnick and Schacter (2004) performed a similar study using event-related fMRI. In their study, participants studied novel shapes. During test, participants were presented with studied shapes (true targets), shapes that were similar to studied shapes (false targets), and new shapes, and had to decide whether these stimuli were previously studied (old) or not (new). As in the DRM paradigm, participants have a strong tendency to falsely remember studying the false targets. Slotnick and Schacter found that early parts of the visual processing stream (BA 17/18) were engaged more strongly during the true recognition compared to false recognition. If this greater activity in the case of true remembering reflects reactivation, these findings challenge Wheeler and Buckner (2003)’s analysis, which suggested that only late visual regions are reactivated during retrieval, and converge with those of Schacter et al. (1996) to suggest that true memories are associated with more contextual reactivation in sensory regions than false memories. Unlike many of the studies discussed in this review, which conform to the design presented in Figure 1, this study involved item recognition rather than the retrieval of associative or contextual information. For this reason, it is also possible that this increase in early visual cortex activation during true recognition could be caused by an increase in attentional or perceptual processing of the cue during true recognition. When this interpretation is adopted, one might theorize that false recognition could be caused by a failure to process the cue thoroughly during recognition rather than a difference in reactivation.

Fabiani et al. (2000) piggybacked on the ERP work of Gratton and colleagues (Gratton et al., 1997) by comparing the sensory signature of correctly and falsely remembered words presented to different visual hemifields. Like Schacter et al. (1996), Fabiani et al. used a version of the DRM paradigm. In Fabiani et al.’s paradigm, entire lists of words related to non-presented words (false targets) were presented to one visual field or the other. For example, a group of words all related to “sleep” might be presented on the right visual hemifield and a list of words related to “chair” might be presented to the left visual hemifield. As is typical in the DRM paradigm, participants had to make old/new decisions to true targets, false targets, and new words, and there were a large number of false alarms to false targets. As discussed earlier, lateralization in the ERP during retrieval is indicative of the visual hemifield in which a stimulus was studied (Gratton et al., 1997). Fabiani et al. found that this sensory signature occurred for true targets but not false targets.

Of the studies investigating the effect of false memory on encoding regions engaged during retrieval, three suggest that false memories evoke less or no activation in relevant encoding regions compared to true memories (Fabiani et al., 2000; Schacter et al., 1996; Slotnick & Schacter, 2004), and one study suggests that increased activity in such regions can cause individuals to falsely remember encoding context (Kahn et al., 2004). The interpretations of these studies present an obvious discrepancy, in that in one case reactivation of encoding regions is influencing individuals’ decisions and causing them to misremember, and in the others individuals are not taking advantage of that information when it could help them. This suggests that there must be other regions, such as parietal cortex (Wheeler et al., 2003) or the medial temporal lobe (Cabeza et al., 2001), at play when individuals falsely remember. Future research should be motivated to discover which factors cause participants to rely more or less on these different sources of information. Additionally, one could expect that in line with the findings of reading studies (e.g., Johnston et al., 1991), activation in encoding regions would be greatest for hits and least for correct rejections, with false alarms and false negatives evoking a moderate amount of activation. This kind of profile would be consistent with all the results presented in this section.

What is the significance of the reactivation of brain regions involved in strategic encoding?

A sizeable number of the studies reviewed in this paper investigated the retrieval of information that was encoded using a number of different strategies, such as enactment, verbal mnemonics, and the mnemonic use of mental imagery (Heil et al., 1999; Johnson & Rugg, 2007; Johnson et al., 2008; Kahn et al., 2004; Lundstrom et al., 2003; Nilsson et al., 2000; Nyberg et al., 2001). Together, these studies found that the regions that may have been engaged in service of these encoding strategies were activated during retrieval. In order to better understand reactivation more generally, it is important to understand what underlies this reactivation of strategic encoding regions during retrieval. One possibility is that participants are recollecting the encoding experience, and are vividly remembering the strategies they employed in service of encoding the cue. Consistent with this proposal, Johnson and Rugg (2007) found that left medial PFC was more active during encoding and retrieval of information studied verbally, while anterior fusiform gyrus and the lateral occipital complex were more active during encoding and retrieval of information studied using imagery, but these differences were only present when stimuli were endorsed as remembered. Likewise, Kahn et al. (2004) found differences in which regions were engaged during the retrieval of information studied using different strategies only when the strategy was explicitly remembered. Another possibility is that participants are using these encoding strategies in service of retrieval or in service of re-encoding the material after it is retrieved. Verbal protocols or self reports would be helpful in determining whether different strategies are actually being employed during retrieval, and whether these strategies correlate with the reactivation of strategic encoding regions. However, none of the studies reported here collected such data. Therefore, while there is evidence that the reactivation of encoding regions may be due in part to remembering the encoding strategy, it remain an open empirical question whether the employment of corresponding strategies during retrieval also contributes to this reactivation.

Can information be abstracted away from the encoding experience?

According to Barsalou’s (1999) perceptual symbol system, all knowledge is embedded in sensorimotor representations. This begs the question, are there memories that are not attached to sensory information? In the course of this review, we have presented substantial evidence that sensory regions are reactivated during retrieval, but we have also presented evidence that information can be endorsed as remembered in the absence of activation in relevant sensory regions (Wheeler et al., 2004). Is it possible that there is part of the brain involved in the storage of truly abstract knowledge, or is all knowledge grounded in real world experiences?

Barsalou has cast much of his argument for perceptual symbol systems as an attack on the empirical inadequacy of “amodal symbol systems.” The earlier behavioral evidence advanced for propositional representations was the evidence that people remembered the meaning of experiences independent of the exact details of the experiences (Anderson & Bower, 1973; Anderson, 1983) or even the modality of the experiences (Rosenberg & Simon, 1977). These early demonstrations have been explained post-hoc (e.g., Anderson, 2005; pp 152-153) as reflecting strategic re-encoding of the cue to contain information other than in the original experience. For instance, participants develop a visual image of a verbal sentence and no longer can remember whether they read the sentence and imaged what it described or saw the situation and described it with a sentence. Neuroimaging could offer us a way to check on the accuracy of such post-hoc explanations. For instance, we should see activation in regions reflecting the modality of the re-encoding.

The issue of amodal representations has been examined in imaging studies of working memory and here the evidence seems to be that there are both modality-specific and amodal systems. In the case of verbal working memory there are studies finding equal activation in prefrontal and parietal regions for verbal material presented visually or aurally (e.g., Buckner et al., 2000; Schumacher et al., 1996) while others have found modality specific effects. In the case of visual working memory (e.g., Crottaz-Herbette et al., 2004). Lehnert & Zimmer (2008) present evidence that object working memory is modality specific while spatial working memory is not. Anderson et al. (2007) found that the same parietal region activated independent of the modality in which participants encoded the material or the modality in which they responded. The reactivation hypothesis is not committed on the issue of whether there are amodal regions involved in the coding of a memory or not. According to the reactivation hypothesis, if a region is not specific to visual or auditory input but is involved in the encoding of a memory, it should be reactivated independent of the form of study or test.

What is the role of the medial temporal lobe in reactivation?

The crucial role of medial temporal lobe (MTL), which includes the hippocampus and surrounding structures such as the perirhinal cortex and parahippocampal cortex, in the retrieval of episodic memories was compellingly demonstrated when patient H.M. suffered severe anterograde and retrograde amnesia after MTL resection (Scoville & Milner, 1957). As mentioned in the introduction, many models of memory posit that the hippocampus plays a crucial role in replaying past episodes in cortex (e.g., Alvarez & Squire, 1994; McClelland et al., 1995; Moscovitch et al., 2005). According to McClelland et al. (1995), episodes are encoded by sparse connections between cortex and the hippocampus such that the sparse representations in the hippocampus are capable of reinstating the episode in cortex. While it is a point of contention whether retrieval can become hippocampus-independent after years of consolidation (McClelland et al., 1995; Moscovitch et al., 2005), most of these models agree that the hippocampus is crucial for the encoding and retrieval of all kinds of episodes and associations, at least initially. This suggests that the hippocampus may play a general role in the encoding of episodes and associations as well as their retrieval. This assertion is supported by the strong correlation between hippocampal activity and the successful encoding of many kinds of associations (e.g., Awipi & Davachi, 2008; Prince, Daselaar, & Cabeza, 2005; Staresina & Davachi, 2008) as well as the subsequent activation of the hippocampus during the retrieval of those associations (Prince et al., 2005; Ranganath et al., 2004). There is evidence that the other regions of the medial temporal lobe are involved in the encoding of associations in a more content-specific manner. For example, perirhinal cortex appears to be involved in the successful encoding of object information and parahippocampal cortex appears to be involved in the successful encoding of scene information (e.g., Awipi & Davachi, 2008; Davachi, Mitchell, & Wagner, 2003). As reviewed earlier, Polyn et al. (2005) and Ranganath et al. (2004) both found that the parahippocampal place area was activated during the retrieval of place associations. It is an important avenue for future research to determine how these MTL regions differ from other regions that are reactivated during retrieval. Their direct connections with the hippocampus as well as visual cortex may allow them to act as relays between the hippocampus and content-specific cortex in the visual system (Awipi & Davachi, 2008; Suzuki & Amaral, 1994). Based on this architecture, one might expect activation to proceed from the hippocampus to parahippocampal cortex to the dorsal stream during the retrieval of location associations and from the hippocampus to perirhinal cortex to the ventral stream during the retrieval of object associations. A crucial step forward would be to investigate how the hippocampus is interacting with reactivated regions during encoding and retrieval, possibly by calculating functional connectivity between these regions with fMRI or measuring electrical activity with intracranial recordings to discover the temporal relationship between activity in the hippocampus, the rest of the MTL, and other reactivated regions.

Conclusions

The theory that the encoding experience is reinstated neurally during episodic retrieval has been around in cognitive psychology for over a century (James, 1890). This paper set out to review neuroimaging and neurophysiology studies that provide evidence for this theory. These studies found that appropriate sensory and emotional regions are reactivated during the retrieval of modality-specific sensory associations and valence-specific emotional associations, respectively. Within the visual system, this reactivation occurs at many grain sizes. Additionally, the parts of the brain deployed during strategic encoding appear to be reactivated during the retrieval of the encoded information. Reactivation is stronger when more information is retrieved, is reduced when information is falsely remembered, and is correlated with subjective ratings of remembering. However, the reviewed research leaves several unresolved questions: When information is more accessible, is reactivation greater or less? Does it occur faster or slower? What factors determine which regions reactivate and which do not? How does the MTL interact with reactivated regions? These questions should motivate future cognitive neuroscience research in episodic and associative memory.