Neural Correlates of Relational Memory: Successful Encoding and Retrieval of Semantic and Perceptual Associations

Abstract

Using event-related functional magnetic resonance imaging, we identified brain regions involved in successful relational memory (RM) during encoding and retrieval for semantic and perceptual associations or in general, independent of phase and content. Participants were scanned while encoding and later retrieving associations between pairs of words (semantic RM) or associations between words and fonts (perceptual RM). Encoding success activity (ESA) was identified by comparing study-phase activity for items subsequently remembered (hits) versus forgotten (misses) and retrieval success activity (RSA) by comparing test-phase activity for hits versus misses. The study yielded three main sets of findings. First, ESA-RSA differences were found within the medial temporal lobes (MTLs) and within the prefrontal cortex (PFC). Within the left MTL, ESA was greater in the anterior hippocampus, and RSA was greater in the posterior parahippocampal cortex/hippocampus. This finding is consistent with the notion of an encoding-retrieval gradient along the longitudinal MTL axis. Within the left PFC, ESA was greater in ventrolateral PFC, and RSA was greater in dorsolateral and anterior PFC. This is the first evidence of a dissociation in successful encoding and retrieval activity within left PFC. Second, consistent with the transfer-appropriate processing principle, some ESA regions were reactivated during RSA in a content-specific manner. For semantic RM, these regions included the left ventrolateral PFC, whereas for perceptual RM, they included occipitoparietal and right parahippocampal regions. Finally, only one region in the entire brain was associated with RM in general (i.e., for both semantic and perceptual ESA and RSA): the left hippocampus. This finding highlights the fundamental role of the hippocampus in RM.

Introduction

When we remember a past event, we typically remember not only the components of the event but also the relationships among these components. For example, when recalling a rumor heard at a party, we may recall not only the statement but also the topic of the conversation (semantic associations), the voice of the speaker (perceptual associations), etc. (for review, see Johnson et al., 1993). Memory for these associations, or relational memory (RM), has been strongly linked to the medial temporal lobes (MTLs) (for review, see Eichenbaum et al., 1994) and the prefrontal cortex (PFC) (for review, see Moscovitch, 1994). In the present study, we used event-related functional magnetic resonance imaging (fMRI) to investigate three fundamental questions regarding the neural correlates of RM.

First, do RM encoding and RM retrieval activations differ within the MTL and the PFC? This is a controversial issue in the functional neuroimaging literature. Within the MTL, one view postulates that anterior regions are more involved in encoding and posterior regions in retrieval (Lepage et al., 1998), whereas another view proposes that anterior regions are more involved in relational memory and posterior regions are more involved in nonrelational memory (Schacter and Wagner, 1999). Within the PFC, one view postulates a hemispheric asymmetry between encoding and retrieval (Tulving et al., 1994; Nyberg et al., 1996), but recent event-related fMRI studies have found left PFC activations both during RM encoding (Davachi and Wagner, 2002; Davachi et al., 2003) and RM retrieval (Ranganath et al., 2000; Giovanello et al., 2004). It is unclear, however, whether successful encoding and retrieval activations actually differ within left PFC.

Second, does RM involve the reactivation during retrieval of process-specific encoding regions? This idea is predicted by the transfer-appropriate processing (TAP) principle (Morris et al., 1977), which postulates that memory performance is a function of the overlap between encoding and retrieval operations and by the source-monitoring framework (Johnson et al., 1993), which proposes that cognitive operations recruited during retrieval vary depending on the nature of the target information. Although a few functional neuroimaging studies have found support for this concept, they focused usually on sensory or motor processes (Nyberg et al., 2000, 2001; Wheeler et al., 2000) and have not directly compared successful encoding and retrieval activity (Vaidya et al., 2002; Kahn et al., 2004).

Finally, is there a brain region critical to successful RM, regardless of memory phase (encoding vs retrieval) and stimulus content (e.g., semantic vs perceptual)? The obvious candidate is the hippocampus, which is assumed to mediate the storage and recovery of memory traces (Teyler and DiScenna, 1985; Squire, 1992) and RM processes in general (Eichenbaum et al., 1992). Although functional neuroimaging studies of RM have reported hippocampal activations in a variety of conditions (for review, see Cabeza, 2005), it is unclear whether a common hippocampal region was activated across these different conditions.

To answer these three questions, we conducted an fMRI study crossing phase (encoding and retrieval) and content (semantic and perceptual), using successful memory activity (remembered vs forgotten) as our critical measure. We defined encoding success activity (ESA) as study-phase activity for subsequently remembered versus forgotten items and retrieval success activity (RSA) as test-phase activity for hits versus misses. To investigate encoding-retrieval differences, we compared ESA and RSA. To investigate transfer appropriate processing, we distinguished ESA-RSA overlaps for semantic versus perceptual RM. Finally, to identify general successful RM regions, we isolated areas involved in both encoding and retrieval and in both perceptual and semantic RM.

Materials and Methods

Paradigm overview. The behavioral paradigm is illustrated in Figure 1. Participants were scanned while encoding and then retrieving associations between words (semantic condition) or between words and fonts (perceptual condition). The retrieval task was an associative recognition test that included identical and recombined pairs. In the semantic condition, recombined pairs consisted of words encoded in separate pairs, whereas in the perceptual condition, recombined pairs consisted of studied pairs written in a font previously seen in a different pair. Thus, distinguishing between identical and recombined pairs required the retrieval of semantic associations between words in the semantic condition but the retrieval of perceptual associations between words and fonts in the perceptual condition. The processes involved in encoding and retrieving associations have been described as associative memory or relational memory, and here we use the latter expression.

Figure 1.

Participants were scanned while encoding and then retrieving associations between words (semantic condition) or between words and fonts (perceptual condition). The retrieval task was an associative recognition test that included identical and recombined pairs. In the semantic condition, recombined pairs consisted of words encoded in separate pairs, whereas in the perceptual condition, recombined pairs consisted of studied pairs written in a font previously seen in a different pair.

The TAP principle predicts that memory is a function of the overlap between encoding and retrieval operations. Behavioral studies investigating this idea typically include two encoding conditions (A and B) emphasizing different forms of processing (e.g., semantic vs perceptual) and two retrieval conditions (A′ and B′) also differing regarding these processes. In this situation, TAP predicts that memory performance should be better when study and test conditions match (A-A′ and B-B′) than when they mismatch (A-B′ and B-A′). Event-related fMRI allows a different way of investigating the TAP principle, which we used in the present study. If one assumes that different forms of processing recruit different brain regions (process-specific regions), then the TAP principle predicts that successful memory performance should be associated with the reactivation during retrieval of process-specific regions that were also activated during encoding. More concretely, we tested the prediction that differences in activity between successful and unsuccessful trials would involve encoding-retrieval overlaps (ESA-RSA) differing for semantic versus perceptual processing.

Unlike functional neuroimaging studies that compared relational and nonrelational memory conditions (Davachi and Wagner, 2002), we investigated the neural correlates of RM by comparing successful versus unsuccessful RM activity during both encoding and retrieval. As mentioned above, we defined ESA as study-phase activity for subsequently remembered versus forgotten pairs (subsequent memory effect) (for review, see Paller and Wagner, 2002) and RSA as test-phase activity for hits versus misses.

It is worth noting that we used misses as a control rather than using correct rejections. An advantage of using misses instead of correct rejections is that the latter tends to elicit MTL activity related to novelty/encoding, which may mask MTL activity related to successful retrieval (Stark and Okado, 2003). Another advantage of using misses is that they allow ESA and RSA analyses to be based on exactly the same set of items (subsequently remembered pairs, hits; subsequently forgotten pairs, misses). More generally, the advantage of using ESA and RSA to compare encoding and retrieval activity is that differences between encoding and retrieval tasks that are not directly associated with successful memory processes (e.g., task difficulty) tend to be subtracted out in the contrast between successful and unsuccessful trials. Thus, the present study allowed a more direct contrast between encoding and retrieval activity than has been possible in previous functional neuroimaging studies.

Subjects. Sixteen participants (five females), all students at Duke University, with an average age of 19.8 (SD, 1.7) years, were scanned and paid for their participation. Data from two participants were excluded because of scanner malfunction and inadequate behavioral performance. Written informed consent was obtained for each participant, and the study met all criteria for approval from the Duke University Institutional Review Board.

Stimulus materials. The stimuli consisted of 368 word pairs and 184 unique fonts. The word pairs were constructed from a pool of abstract nouns selected from the MRC Psycholinguistic database (http://www.psy.uwa.edu.au/MRCDataBase/mrc2.html). The words were 4-11 letters in length (mean, 7.8; SD, 1.9) and of moderate frequency (Kucera-Francis written frequency mean, 45). All word pairs were balanced for number of letters (±2 letters) so that perceptual cues, such as word length, would not influence memory performance. The fonts were selected from a popular website (http://www.1001fonts.com/index.html) based on legibility, as rated by the first author. Each font was scaled to match a template size (Times New Roman, 36 point). Recombined pairings for retrieval were created pseudorandomly to meet the constraints of number of letters.

Procedures. The fMRI study consisted of four semantic runs and four perceptual runs, with the order alternating and counterbalanced across subjects. Each fMRI run contained one encoding block and one retrieval block separated by a 30 s delay. Each block consisted of 46 trials, each consisting of a word pair displayed for 3.4 s. Participants were encouraged to respond within this period, and late responses were not included in the analyses. The number of study trials remaining identical per block was 29, yielding a potential total of 116 encoding trials and 116 retrieval trials per condition. The intertrial interval (a fixation cross) varied between 0 and 5.4 s and was used as baseline in the fMRI analysis (see below).

Encoding and retrieval trials in the semantic and perceptual conditions are illustrated in Figure 1. In each semantic encoding trial, a pair of words in plain font was displayed, and subjects rated semantic relatedness (1, high; 4, low) while learning the pair (intentional encoding). In each semantic retrieval trial, a pair of words in plain font was displayed, which was either identical to a studied pair or a recombined pair made by pairing words studied in different pairs. Participants made an identical/recombined decision and indicated their confidence (1, definitely identical; 2, probably identical; 3, probably recombined; 4, definitely recombined). During perceptual encoding trials, the font in which both were presented varied across trials. A different font was used for each pair, thereby avoiding interference effects. Subjects rated how well the font fitted the words (1, high; 4, low) while learning the words-font association. In perceptual retrieval trials, identical pairs were as in the semantic condition, and recombined pairs were made of studied pairs presented in the font previously seen in a different pair (Fig. 1).

fMRI scanning. Images were collected using a 4T GE scanner. Scanner noise was reduced with earplugs, and head motion was reduced with foam pads and headbands. Stimuli were presented with liquid-crystal display goggles (Resonance Technology, Northridge, CA), and behavioral responses were recorded with a four-key fiber-optic response box (Resonance Technology). Anatomical scanning started with a T1-weighted sagittal localizer series. The anterior commissure (AC) and posterior commissure (PC) were identified in the midsagittal slice, and 34 contiguous oblique slices were prescribed parallel to the AC-PC plane. High-resolution T1-weighted structural images were acquired with a 450 ms repetition time (TR), a 9 ms echo time (TE), a 24 cm field of view (FOV), a 256² matrix, and a slice thickness of 1.9 mm. Functional scanning used an inverse spiral sequence with a 1700 ms TR, a 6 ms TE, a 24 cm FOV, a 64² image matrix, and a 60° flip angle. Thirty-four contiguous slices were acquired with the same slice prescription as the anatomical images. Slice thickness was 3.75 mm, resulting in cubic 3.75 mm³ isotropic voxels.

fMRI analyses. Data were analyzed using SPM99 (Statistical Parametric Mapping; Wellcome Department of Cognitive Neurology, http://www.fil.ion.ucl.ac.uk/spm). After discarding the first four volumes, time series were corrected for differences in slice acquisition times and realigned. Anatomical images were coregistered with the functional images. Next, both anatomical and functional images were spatially normalized to a standard stereotactic space, using the Montreal Neurological Institute templates implemented in SPM99 and resliced to a resolution of 3 × 3 × 3 mm. The coordinates were later converted to Talairach and Tournoux space (Talairach and Tournoux, 1988). Finally, the volumes were spatially smoothed using an 8 mm isotropic Gaussian kernel and proportionally scaled to the whole-brain signal.

For each subject, trial-related activity was assessed by convolving a vector of the onset times of the stimuli with a synthetic hemodynamic response function. The general linear model, as implemented in SPM99, was used to model the effects of interest and other confounding effects (e.g., head movement and magnetic field drift). Statistical parametric maps were identified for each participant by applying linear contrasts to the parameter estimates (β weights) for the events of interest, resulting in a t statistic for every voxel. The parameter estimates for the different events will be subsequently referred to as effect sizes (Pastor et al., 2004). In both semantic and perceptual conditions, we coded six trial types: encoding fixation baseline, subsequently remembered, subsequently forgotten, retrieval fixation baseline, retrieval hits, and retrieval misses. The mean number of trials contributing to each trial type in the design was 39 (SD, 15). An analysis excluding the three subjects with sparse trials (<15) yielded a similar pattern of results as the analysis including all subjects (see supplemental material, available at www.jneurosci.org). Subsequently remembered trials were determined by matching the high-confidence retrieval hit responses at test to the relevant trials at study. Similar to other subsequent memory studies (Sperling et al., 2003), only the high-confidence retrieval hits were considered for analysis (low-confidence retrieval hits were modeled as a separate trial type, not included in any of the relevant analyses), because we were interested in recollection rather than familiarity or guessing (to maximize the comparison between successful and failed retrieval). Low-discriminability (d′) scores (0.06 in the semantic task and 0.12 in the perceptual task) provided additional justification for excluding low-confidence retrieval hits.

Three analyses were performed to address the three questions described in the Introduction. First, to identify areas showing differences as a function of phase, we isolated regions showing successful activity (hit > miss; p < 0.001) in either the encoding or the retrieval phase. Within these regions of interest (ROIs), masks were used to include areas showing a reliable memory × phase interaction (p < 0.05) and to exclude areas showing a memory × content interaction (p < 0.05). Second, to identify areas showing differences as a function of content, we isolated regions showing successful memory activity (hit > miss; p < 0.001) in either the semantic or the perceptual condition. Within these ROIs, masks were used to include areas showing a reliable memory × content interaction (p < 0.05) and to exclude areas showing a memory × phase interaction (p < 0.05). For purpose of illustration, regions identified in these comparisons are shown in Figures 2 and 3 at thresholds between p < 0.05 and 0.005. Finally, to identify regions generally involved in successful RM regardless of memory phase and stimulus content, we performed a quadruple conjunction isolating regions showing significant effects in each and all four conditions (semantic ESA, perceptual ESA, semantic RSA, and perceptual RSA). Because the different conditions were presented independently in separate functional runs, and each showed a hit-miss difference at p < 0.05, the significance threshold for the quadruple conduction can be estimated at p < 0.00001 (i.e., 0.05⁴). This estimate assumes that a region may be activated in one condition without being activated in other conditions, which is a reasonable assumption given the numerous activation differences across conditions reported below. Furthermore, to ensure that regions found were truly general, we excluded regions showing interactions (p < 0.05) with memory phase or stimulus content.

Figure 2.

Brain regions showing differences between successful RM activity during encoding (ESA) and during retrieval (RSA). The bar graphs display differences in the effect size of activations for remembered versus forgotten items during encoding (i.e., subsequently remembered vs forgotten) and during retrieval (i.e., hits vs misses). Diff, Difference; Dorsolat, dorsolateral; ENC, encoding; Forg, forgotten; P, perceptual; Post. Parahipp./Hipp., posterior parahippocampal cortex/hippocampus; Rem, remembered; RET, retrieval; S, semantic; Ventrolat, ventrolateral. Enclosed box displays the MTL from a sagittal slice at x = -27.

Figure 3.

Brain regions that showed ESA/RSA overlaps that differed between semantic (S) and perceptual (P) RM conditions (teal, semantic; purple, perceptual). The left ventrolateral (L Ventrolat) PFC region (BA 45; x, y, z: -46, 26, 2) was slightly more posterior/dorsal than the left ventrolateral PFC region that showed greater ESA than RSA for both semantic and perceptual RM (red area, BA 47; x, y, z: -49, 33, -8). B Parietal, Bilateral parietal cortex; Diff, difference; ENC, encoding; Forg, forgotten; L Occiptemp, left occipitotemporal cortex; R Post Parahipp G, right posterior parahippocampal cortex/hippocampus; Rem, remembered; RET, retrieval.

Results

Behavioral data

Table 1 lists the proportion of correct and incorrect responses as a function of confidence and task. The proportion of hits was similar for semantic and perceptual conditions overall (p > 0.20) but was greater for high-confidence responses in the semantic condition (p < 0.0001). The t tests comparing the proportion of hits (correctly identifying identical pairs) to the proportion of false alarms (incorrectly endorsing recombined pairs as identical) revealed significant differences (p < 0.001) for both the semantic and perceptual tasks. The proportion of hits was also significantly greater than chance for both the semantic (p < 0.001) and perceptual (p < 0.005) tasks. Finally, the proportion of high-confidence hits was significantly greater than the proportion of high-confidence false alarms for both tasks (p < 0.001). To control for the difference in confidence proportions and to reduce the role of guessing overall, only high-confidence hits were included in fMRI analyses. To investigate the possible contribution of item memory to differences between hits and misses, we conducted a behavioral follow-up study to test the level of item memory under encoding conditions equivalent to the ones used in the scanner. A group of nine participants studied the same number of pairs with the same instructions and presentation rate as in the fMRI study but, then, instead of the associative recognition used in the scanner, they performed a forced-choice old/new recognition test. The results of this test show that item recognition performance for the short blocks we used was high (words: mean, 93%, SD, 6%; fonts: mean, 79%, SD, 7%). Thus, differences in activation between hits and misses in our study mainly reflected differences in relational memory, with differences in item memory playing a relatively small role. Moreover, given that when item memory fails, relational memory also fails; the common process separating hits from misses was primarily relational memory.

Table 1.

Behavioral results: mean proportion of responses (SD)

	HC	LC	Total
Intact pairs
Hits
Semantic	0.49 (0.16)	0.18 (0.13)	0.67 (0.10)
Perceptual	0.34 (0.19)	0.30 (0.21)	0.64 (0.16)
Misses
Semantic	0.09 (0.08)	0.23 (0.10)	0.33 (0.10)
Perceptual	0.10 (0.10)	0.27 (0.12)	0.36 (0.16)
Recombined pairs
Correct rejections
Semantic	0.32 (0.16)	0.39 (0.11)	0.71 (0.12)
Perceptual	0.23 (0.17)	0.36 (0.14)	0.59 (0.19)
False alarms
Semantic	0.12 (0.11)	0.17 (0.13)	0.29 (0.12)
Perceptual	0.15 (0.11)	0.26 (0.18)	0.41 (0.19)

HC, High confidence; LC, low confidence.

Reaction times (RTs) during retrieval and encoding were analyzed with separate 2 (performance: remembered, forgotten) × 2 (content: semantic, perceptual) ANOVAs. The retrieval ANOVA yielded a significant main effect of performance (p < 0.01), which reflected faster RTs for hits (1736 ms; 195 SD) than for misses (1983 ms; 289 SD). This is a typical finding in recognition memory tasks, presumably because of greater monitoring demands (an extended search process). The encoding ANOVA yielded a significant effect of content (p < 0.05), which reflected faster RTs in the perceptual (1883 ms; 359 SD) than in the semantic condition (2116 ms; 279 SD). Separate analyses were done with the RT differences entered as nuisance covariates, and the results of these are reported in the supplemental material (available at www.jneurosci.org). Finally, to test whether differences in encoding duration (because of the differential intertrial interval) influenced memory performance, an ANOVA was performed, which revealed no significant effect (p > 0.2) on the proportion of hits across the different intertrial intervals.

fMRI data

Brain regions showing differences between ESA and RSA are listed in Table 2 and displayed in Figure 2 (for individual condition data vs baseline, see Table 4 and supplemental material, available at www.jneurosci.org). In the PFC, ESA and RSA were not associated with differences in lateralization but with differences between left PFC subregions, whereas ventrolateral regions [Brodmann area (BA) 47 and BA 44] showed greater ESA than RSA (Fig. 2A), and anterior (BA 10) and dorsolateral (BA 46) (Fig. 2B) regions showed greater RSA than ESA. In the MTL, an anterior region (left hippocampus head) showed greater ESA than RSA (Fig. 2C), whereas a posterior MTL region (left posterior parahippocampal/hippocampal) showed the converse pattern (Fig. 2D). ESA > RSA differences were also found in anterior cingulate and cerebellar regions, and RSA > ESA differences were found in posterior parietal and precuneus regions.

Table 2.

Brain regions showing significant differences between ESA and RSA

			Coordinates (T&T)
	H	BA	x	y	z	T
ESA > RSA
PFC ventrolateral	L	47	-49	33	-8	5.73
opercular	L	44	-56	15	7	3.37
MTL anterior hippocampus	L		-34	-11	-12	2.36
Anterior cingulate	M	32/8	-4	28	37	2.74
Parietal cortex	L	7	-23	-64	48	2.43
Putamen	L		-19	10	-7	2.04
Cerebellum	L		-38	-61	-29	2.54
RSA > ESA
PFC anterior	L	10/11	-30	51	-9	2.28
dorsolateral	L	46	-46	28	26	2.30
MTL post. parahipp./hipp.	L	30	-19	-41	6	5.19
Parietal cortex	L	19/39	-34	-72	35	2.59
	L	40	-49	-42	51	2.18
	L	39	-42	-68	31	2.89
	L	7	-34	-72	42	3.04
	R	7/39	27	-62	34	2.10
Parieto-occipital cortex	L	39/18	-42	-73	14	2.81
Occipital cortex	L	18	-27	-70	-6	2.30

H, Hemisphere; L, left; M, middle; R, right; T&T, Talairach and Tournoux (1988); T, statistical value; post. parahipp./hipp., posterior parahippocampal cortex/hippocampus.

Table 4.

Effect sizes of individual trial types versus baseline and difference scores for activations displayed in Figures 2, 3, 4

		Encoding											Retrieval
		Semantic					Perceptual					Semantic					Perceptual
Comparison/region	Figure and panel	R	F	R-F	R	F	R-F	H	M	H-M	H	M	H-M
ESA > RSA (both Sem and Per)
Left ventrolateral PFC (BA 47)	2A	0.81	0.24	0.57	0.80	0.16	0.64	0.54	0.48	0.06	0.11	0.16	-0.05
Left anterior hippocampus	2C	0.23	0.09	0.14	0.47	-0.14	0.61	-0.08	-0.06	-0.02	-0.25	-0.34	0.08
RSA > ESA (both Sem and Per)
Left dorsolateral PFC (BA 46)	2B	-0.03	0.06	-0.09	-0.01	0.16	-0.17	0.45	0.18	0.26	0.34	-0.02	0.35
Left post. parahipp/hipp.	2D	-0.04	0.19	-0.23	0.05	0.05	0.00	0.21	-0.25	0.45	0.27	-0.14	0.41
Sem > Per (both ESA and RSA)
Left ventrolateral PFC (BA 45)	3A	0.82	0.21	0.60	0.46	0.39	0.06	0.67	0.41	0.25	0.38	0.44	-0.06
Per > Sem (both ESA and RSA)
Left occipitotemporal	3B	0.88	0.93	-0.05	1.47	0.90	0.57	0.77	0.77	0.00	1.28	1.13	0.15
Right parietal (BA 40)	3C	0.12	0.16	-0.05	0.42	0.09	0.33	-0.04	-0.02	-0.02	-0.18	0.03	0.16
Right parahipp.	3D	0.06	0.21	-0.15	0.49	0.26	0.22	0.08	0.13	-0.05	0.34	0.05	0.29
Quadruple conjunction
Left mid-posterior hippocampus	4	0.40	0.16	0.24	0.53	0.32	0.21	0.35	0.13	0.22	0.42	0.14	0.28

Sem, Semantic; Per, perceptual; R, subsequently remembered; F, subsequently forgotten; H, hits; M, misses; post. parahipp/hipp., posterior parahippocampal cortex/hippocampus.

Brain regions showing overlaps between ESA and RSA that differed for semantic versus perceptual conditions are listed in Table 3 and displayed in Figure 3 (for individual condition data vs baseline, see Table 4 and supplemental material, available at www.jneurosci.org). ESA-RSA overlaps specific to semantic RM were found in a left lateralized cortical and subcortical network, including ventrolateral PFC (Fig. 3A), insular, striatal, and cerebellar regions. As illustrated by Figure 3A, the left ventrolateral PFC region associated with semantic RM during both encoding and retrieval (yellow/red area) was more posterior than the left ventrolateral PFC region associated with encoding of both semantic and perceptual RM (blue area). Whereas the semantic-specific activation occurred primarily in BA 45, the encoding-specific activation occurred primarily in BA 47. There was also another semantic-specific activation in the left ventromedial PFC (BA 11). ESA-RSA overlaps specific to perceptual RM were found in posterior brain regions, including left occipitotemporal cortex (BA 37/19) (Fig. 3B), bilateral posterior parietal cortex (BA 7/40) (Fig. 3C), and right parahippocampal cortex (Fig. 3D).

Table 3.

Brain regions showing significant differences between semantic and perceptual ESA/RSA

			Coordinates (T&T)
	H	BA	x	y	z	T
Semantic ESA/RSA > Perceptual ESA/RSA
PFC ventrolateral	L	45	-46	26	2	3.84
ventromedial	L	11	-23	28	-17	2.36
Auditory cortex	L	41	-34	-22	5	3.48
Caudate	L		-15	26	-1	4.24
Putamen	L		-23	1	10	2.41
Subcallosal gyrus	R	25	23	10	-13	2.09
Amygdala	L		-19	-1	-13	1.81
Cerebellum	L		-19	-56	-22	1.93
Perceptual ESA/RSA > Semantic ESA/RSA
Occipitotemporal cortex	L	19/37	-42	-71	-16	1.97
Parietal cortex	R	40	30	-52	59	3.24
	L	7	-23	-72	42	2.96
MTL parahipp. gyrus	R	35	15	-26	-4	3.22*
Anterior cingulate	M	42	0	45	5	3.14

See Table 2 for abbreviations.

^*p < 0.05.

The quadruple conjunction of semantic ESA, semantic RSA, perceptual ESA, and perceptual RSA yielded only one region in the entire brain, the left hippocampus (x, y, z: -27, -26, -5). As indicated by the bar graph in Figure 4, this region showed successful memory activity during both encoding (remembered - forgotten) and retrieval (hits - misses) and for both semantic and perceptual associations (for individual condition data vs baseline, see Table 4 and supplemental material, available at www.jneurosci.org). This finding confirms our prediction that the hippocampus plays a general role in successful RM. The MTL region associated with successful RM in general (Fig. 2, green MTL area) was located between the anterior MTL region specialized in successful RM encoding (red area) and the posterior MTL region specialized in successful RM retrieval (blue area). Thus, the existence of a common encoding/retrieval region in MTL is compatible with the existence of MTL regions differentially involved in ESA versus RSA.

Figure 4.

Results of a quadruple conjunction analysis that identified regions showing successful relational memory activity [remembered (Rem) vs forgotten (Forg)] during semantic (Sem) encoding, semantic retrieval, perceptual (Per) encoding, and perceptual retrieval. Only one region was identified in this analysis: the left hippocampus (x, y, z: -27, -26, -5). Diff, Difference; ENC, encoding; P, perceptual; RET, retrieval; S, semantic.

Discussion

In summary, the study yielded three main sets of findings. First, ESA-RSA differences were found within both the MTL and the PFC. Second, several regions activated during encoding were reactivated during retrieval in a content-specific manner. Finally, only one region in the entire brain was associated with successful RM regardless of phase (encoding vs retrieval) and content (semantic vs perceptual), the left hippocampus. These three sets of findings are discussed in separate sections below.

Encoding and retrieval

Within the MTL, a more anterior region (left hippocampus head) showed greater ESA than RSA (Fig. 2C), whereas a more posterior region (left parahippocampal/hippocampal region) showed greater RSA than ESA (Fig. 2D). This double dissociation is consistent with a meta-analysis of functional neuroimaging studies (Lepage et al., 1998), which concluded that encoding tends to activate the anterior, and retrieval tends to activate the posterior MTL (HIPER model). However, Schacter and Wagner (1999) noted that encoding activations in anterior MTL were sometimes found by RM studies, whereas encoding activations in posterior MTL were sometimes observed in non-RM studies. Thus, they suggested that, rather than encoding-retrieval differences, the anteroposterior MTL gradient might reflect relational-nonrelational differences. In the present study, RM demands were high during both phases, and hence, this alternative hypothesis cannot easily account for the present dissociation. As for the left parahippocampal/hippocampal activation, it is consistent with the notion that the hippocampus is involved in RM (Eichenbaum et al., 1994) and additionally suggests a role of the parahippocampal cortex in RM (Davachi et al., 2003, Duzel et al., 2003; Ranganath et al., 2004).

Although consistent with the HIPER model, three caveats should be noted about the anterior MTL activation (Fig. 2C). First, this activation could reflect novelty rather than encoding per se. Encoding and novelty interpretations are difficult to distinguish, because novelty promotes encoding (Tulving and Kroll, 1995; Ranganath and Rainer, 2003), and both tend to engage similar brain regions (Kirchhoff et al., 2000; Yamaguchi et al., 2004). Second, although the anterior MTL activation did not show a significant phase × content interaction, it tended to be larger for perceptual than for semantic RM. Greater hippocampal activity in the perceptual condition may reflect the use of unusual (novel) fonts or cross-domain binding between words and fonts (Mayes et al., 2001). Third, several fMRI studies of episodic encoding have found activations in posterior parahippocampal regions (Stern et al., 1996; Gabrieli et al., 1997) (but see Daselaar et al., 2004). The present results are not inconsistent with these studies, because we are not claiming that only anterior MTL regions are involved in encoding processes. What we found is that, in a direct within-subject comparison, anterior MTL regions showed greater ESA than RSA.

Within the PFC, the ESA-RSA contrast yielded a double dissociation within the left PFC, whereas left ventrolateral regions showed greater ESA (Fig. 2A), and left dorsolateral and anterior regions showed greater RSA (Fig. 2B). The finding of ESA in the left ventrolateral PFC is consistent with fMRI studies using the subsequent memory paradigm (for review, see Paller and Wagner, 2002). The finding of RSA in the left PFC is generally consistent with functional neuroimaging evidence linking left PFC to source memory (Nolde et al., 1998; Ranganath et al., 2000) but not with studies that did not find a difference in this region between successful and unsuccessful source memory trials (Dobbins et al., 2002; Kahn et al., 2004). The present study provides the first clear evidence that different left PFC subregions are differentially involved in successful encoding versus successful retrieval.

A speculative explanation for the dissociation between ventrolateral PFC (ESA > RSA) and dorsolateral PFC (RSA > ESA) is that these two areas are differentially involved in working memory maintenance versus manipulation (Petrides, 1994; Owen, 1997; D'Esposito et al., 1999). Successful RM encoding may be more dependent on continuous maintenance of incoming information, whereas successful RM retrieval may be more dependent on the reorganization of the retrieved output within working memory. The involvement of anterior PFC in RSA is consistent with evidence that this region is typically activated during retrieval (Cabeza and Nyberg, 2000). The role of anterior PFC during retrieval has been attributed to retrieval mode (Lepage et al., 2000a; Cabeza et al., 2002), manipulation of self-generated information (Christoff et al., 2001), and subgoaling/branching operations (Braver and Bongiolatti, 2002; Koechlin et al., 2003). All of these ideas could account for greater RSA than ESA.

Semantic and perceptual

As noted above, the TAP principle predicts that successful memory performance should involve the reactivation during retrieval of brain regions originally involved in encoding a particular kind of information (e.g., semantic vs perceptual). The present study confirmed this prediction: several regions showed ESA-RSA overlaps that differed between semantic and perceptual RM conditions (Fig. 3). These encoding-retrieval overlaps cannot be attributed to the overall similarity of encoding and retrieval tasks in the semantic versus perceptual conditions, because activity related to overall similarity was subtracted out by remember minus forgotten subtractions.

Successful semantic RM was associated with encoding and retrieval activity in the left ventrolateral PFC (Fig. 3A). This finding is consistent with functional neuroimaging evidence linking this area to semantic RM encoding (Kapur et al., 1996; Lepage et al., 2000b) and retrieval (Badgaiyan et al., 2002) and to semantic processing in general (for review, see Thompson-Schill, 2003). As illustrated in Figure 3A, the left ventrolateral PFC region showing an ESA-RSA overlap for semantic RM (yellow/red area, BA 45; x, y, z: -46, 26, 2) was more posterior/dorsal than the left ventrolateral PFC region showing greater ESA than RSA for both semantic and perceptual RM (blue area, BA 47; x, y, z: -49, 33, -8). Although semantic processing and episodic encoding both have been strongly associated with ventrolateral PFC (Cabeza and Nyberg, 2000), the present study suggests that they may involve different subregions within this general area.

Turning to perceptual RM, ESA-RSA overlaps were found in the left occipitotemporal (Fig. 3B), bilateral parietal (Fig. 3C), and right parahippocampal (Fig. 3D) regions. Activations in occipitotemporal and posterior parietal regions are frequently found during visual memory and imagery tasks (Cabeza and Nyberg, 2000). The left occipitotemporal activation occurred in a region known as visual word form area, which includes patches specialized in letter processing (for review, see McCandliss et al., 2003). Parietal regions have been strongly associated with attentional processes (Cabeza and Nyberg, 2000) and may reflect visual attention to font features. Finally, the right parahippocampal activation is consistent with evidence that the right MTL is involved in encoding and retrieval of nonverbal stimuli (Kelley et al., 1998; Simons et al., 2001).

General RM processes

In the whole brain, only one brain region was associated with successful RM regardless of memory phase (encoding vs retrieval) and stimulus content (e.g., semantic vs perceptual), the left hippocampus. This region was located between the anterior MTL region that was differentially involved in encoding and the posterior MTL region that was differentially involved in retrieval. Thus, the distinction between anterior and posterior MTL regions can be seen as an encoding/retrieval gradient with a middle convergence region shared by both stages (Small et al., 2001). This finding is consistent with evidence from separate functional neuroimaging studies that the left hippocampus is activated during both RM encoding (Henke et al., 1997, 1999; Lepage et al., 2000b) and retrieval (Giovanello et al., 2004; Preston et al., 2004) and during both semantic (Henke et al., 1999; Lepage et al., 2000b; Davachi and Wagner, 2002; Jackson and Schacter, 2004) and perceptual RM (Henke et al., 1997; Montaldi et al., 1998; Gonzalo et al., 2000; Killgore et al., 2000; Preston et al., 2004). However, results from separate studies cannot prove that a common hippocampal region is activated across all of these different conditions. The present study is the first to do so.

This finding fits very well with dominant theories about hippocampal function. First, it is consistent with the hypothesis that the hippocampus holds indexes of stored memory traces (Teyler and DiScenna, 1985; Squire, 1992). This hypothesis predicts that the hippocampus should be activated both when a new index is formed (encoding) and when the index is accessed (retrieval). This idea is not incompatible with the notion of hippocampal regions differentially involved in encoding versus retrieval, and the two ideas can be harmonized by the hypothesis of an encoding-retrieval gradient with a middle region shared by both processes. Second, the present finding is consistent with the view that the hippocampus plays a general and fundamental role in RM (Eichenbaum et al., 1992). The finding of a hippocampal region associated with successful RM during both encoding and retrieval and for both semantic and perceptual associations provides strong support to this view.

Conclusions

The study yielded three main sets of findings. First, encoding-retrieval (ESA-RSA) differences were found within both the MTL and the PFC. Within the MTL, successful encoding engaged a more anterior region, and successful retrieval engaged a more posterior region. Within the PFC, successful encoding recruited a left ventrolateral area, and successful retrieval recruited left dorsolateral and anterior areas. Second, several regions activated during encoding were reactivated during retrieval in a content-specific manner. For semantic RM, encoding-retrieval overlaps were found in left ventrolateral PFC, and for perceptual RM, in occipital, parietal, and right parahippocampal areas. Finally, only one region in the entire brain was associated with successful RM regardless of phase (encoding vs retrieval) and content (semantic vs perceptual): the left hippocampus. These results shed new light on the neural correlates of RM and how they change as a function of memory phase and stimuli.