A within-subject ERP and fMRI investigation of orientation-specific recognition memory for pictures

Abstract

Despite a large body of recognition memory research, its temporal, measured with ERPs, and spatial, measured with fMRI, substrates have never been investigated in the same subjects. In the present study, we obtained this information in parallel sessions, in which subjects studied and recognized images of visual objects and their orientation. The results showed that ERP-familiarity processes between 240 and 440 ms temporally preceded recollection processes and were structurally associated with prefrontal brain regions. Recollection processes were most prominent from 440 to 600 ms and correlated with activation in temporal, parietal, and occipital brain regions. Post-retrieval monitoring, which occurred in the ERP between 600 and 1000 ms as a long-lasting slow-wave over frontal channel groups, showed correlations with activation in the prefrontal and parietal cortex. These ERP/fMRI relationships showed some correspondences to source localizations of the investigated ERP memory effects.

Introduction

Recognition memory has been a focus of research interest for many years. Behavioral studies have been supplemented with ERPs and fMRI recording (see Kim, 2010; Rugg & Curran, 2007; Skinner & Fernandes, 2007; Spaniol, Patrick, Davidson, Kim, Moscovitch, & Grady, 2009, for reviews) to elucidate the cognitive and neural substrates underlying episodic retrieval. Despite the large body of research, no study has so far compared the temporal, measured with ERPs, and spatial, measured with fMRI, dynamics of recognition memory in the same subjects. In the present study, we obtained this information and present relationships between the temporal and spatial substrates of familiarity and recollection processes that underlie episodic retrieval of pictures and their orientation.

Dual-process theories of recognition memory hold that the retrieval of episodic information is supported by two independent processes: recollection and familiarity (Jacoby, 1991; Mandler, 1980; see Yonelinas, 2002 for review). Recollection corresponds to the retrieval of specific, meaningful information about a studied item and its learning context. In this case, the subject remembers not just the item (e.g., a picture), but also such information as which direction it was facing when first studied or where on the screen it was presented. Familiarity lacks the retrieval of such episodic details and arises instead from identifying a global similarity between a test item and information stored in memory. In this instance, the subject knows that he or she has seen the item, but cannot recall any additional contextual information. Multiple behavioral paradigms exist to measure the component processes of recognition memory. For the present study, objective estimates of familiarity and recollection were gathered through the successful or unsuccessful recollection of specific details from the study episode. Subjects were provided with explicit, item-related information during study (i.e., the orientation of a picture). Recollection was defined as the correct recognition of the item plus its orientation; and familiarity was specified as correct item recognition but incorrect orientation recognition. This conceptualization of familiarity and recollection has some similarities to source or associative recognition tasks.

In ERP research, familiarity and recollection have been associated with two distinct ERP components: the FN400 and parietal old/new effects. The FN400 is a negative going wave over frontal brain regions and is associated with a more positive magnitude for old as compared to new items between 300 and 500 ms. It is thought to reflect processes of familiarity. It distinguishes hits from correct rejections without being influenced by the recollection of details from the study episode (e.g., Curran, 2000; Rugg & Curran, 2007, for review). An alternate hypothesis sees the FN400 as reflecting conceptional, implicit memory (e.g., Voss & Paller, 2009), but there is counter-evidence to this interpretation (Rugg & Curran, 2007; Stenberg, Hellman, Johansson, & Rosén, 2009). The parietal old/new effect is a parietal positivity between 500 and 800 ms that is considered an index of recollection because it is modulated according to whether the eliciting items are associated with correct or incorrect retrieval of specific details from the study episode, and whether items are judged as “remembered” or “known” in subjective recognition memory tasks (Curran, 2000; see Rugg & Curran, 2007, for review).

In addition to FN400 and parietal old/new effects, late-frontal old/new effects have been observed. These memory effects are characterized by a frontal positivity and onset times usually later than 800 ms (Cruse & Wilding, 2009; Friedman & Johnson, 2000; Hayama, Johnson, & Rugg, 2008; Ranganath & Paller, 2000). Wilding and Rugg (1996) have proposed that these old/new effects reflect the engagement of post-retrieval processes that are activated whenever the outcome of the retrieval search is ambiguous or causes uncertainty (Rugg, Otten, & Henson, 2002).

fMRI research has identified brain regions underlying episodic memory functions. The role of the medial temporal lobe (MTL, including the hippocampus, perirhinal and parahippocampal cortex) in recognition memory is well documented (Diana, Yonelinas, & Ranganath, 2007; Ranganath, 2010; Squire, Stark, & Clark, 2004), although there is still a debate about the functional differentiation of the MTL with regard to recollection and familiarity (Montaldi & Mayes, 2010; Wixted & Squire, 2011). The prefrontal cortex has also been shown to play a role in recognition memory (Gallo, McDonough, & Scimeca, 2010; Shimamura, 1995; Spaniol et al., 2009). The parietal-cortex activations that have been consistently found during episodic retrieval have recently been connected to cognitive functions like attention to internal memory representations, accumulation of information retrieved from memory, or buffering of mnemonic information (Cabeza, Ciaramelli, Olson, & Moscovitch, 2008; Ciaramelli, Grady, & Moscovitch, 2008; Hutchinson, Uncapher, & Wagner, 2010; Wagner, Shannon, Kahn, & Buckner, 2005; Vilberg & Rugg, 2008).

Considering the whole brain, as done in the present study, distinctions between regions that are associated with either familiarity or recollection or both have been found. The hippocampus and perirhinal cortex have been associated with recollection and familiarity, respectively (Aggleton & Brown, 1999; Diana et al., 2007; Norman & O'Reilly, 2003)¹. Familiarity showed relatively stronger activation in superior parietal lobe, angular gyrus, insula, and cerebellum (Cansino, Maquet, Dolan, & Rugg, 2002; Dobbins, Rice, Wagner, & Schacter, 2003; Ragland, Valdez, Loughead, Gur, & Gur, 2006; Skinner & Fernandes, 2007; Slotnick, Moo, Segal, & Hart, 2003; Vilberg & Rugg, 2007, 2009; Wheeler & Buckner, 2004; Yonelinas et al., 2005). Recollection has been found to activate the intraparietal sulcus, postcentral gyrus, lingual gyrus, and inferior temporal gyrus, as well as amygdala and thalamus (reviewed in Spaniol et al., 2009). These studies have also identified brain areas that are activated for both familiarity and recollection like areas in the prefrontal cortex, precuneus, caudate nucleus, and parietal cortex. These regions appear to be connected to distinct neural networks of familiarity and recollection and represent brain areas that might serve perceptual or decision processes, which are common to both retrieval processes (Dörfel, Werner, Schaefer, Kummer, & Karl, 2009; Wheeler & Buckner, 2004).

Previous attempts to associate ERP old/new effects with their underlying neural substrates have so far relied on deductions about the ERP’s spatial distributions and between-experiment comparisons that focused on functionally parallel findings in ERP and fMRI studies. These studies have assumed that the FN400 and parietal old/new effect are generated in the lateral prefrontal and lateral parietal cortex because of their spatial distribution on the scalp (i.e., over frontal and parietal channel groups, respectively) and neural locations that have been associated with similar cognitive processes by fMRI studies (Yonelinas, Otten, Shaw, & Rugg, 2005), between-subject ERP/fMRI (Vilberg & Rugg, 2007, 2009), and monkey studies (Xiang & Brown, 2004). Late-frontal old/new effects are thought to be generated in the prefrontal cortex (Cruse & Wilding, 2009). This assumption received support from fMRI studies that showed activation related to post-retrieval monitoring in the dorsolateral prefrontal cortex (Achim & Lepage, 2005; Hayama & Rugg, 2009). So far, no study has established relationships between ERP and fMRI measures of episodic memory in the same subjects.

In the present study, we assessed the temporal, measured with ERPs, and spatial, measured with fMRI, dynamics of recognition memory in the same subjects. We were thus able to determine ERP/fMRI relationships for the processes of episodic retrieval for the first time. We recorded ERPs and fMRI in the same subjects in different sessions. In both sessions, subjects studied pictures of objects and their orientation by making subjective orientation judgments (Fig. 1). During the test phases the EEG or fMRI data was recorded while subjects made recognition judgments for old and new items. Old pictures could either be presented in the original orientation or the mirror-reversed orientation. Recognition judgments combined item and orientation decisions, thus allowing objective measurement of familiarity and recollection.

Figure 1.

Schematic of the procedure for the ERP and fMRI sessions illustrating study list, test list, memory judgment, and experimental condition.

Methods

Participants

Thirty-seven undergraduates participated in the ERP session, and 30 of these were included in the ERP analyses (60 % female, mean age M = 21.3, SD = 2.7). Of these 30 participants, 16 subjects (63 % female, mean age M = 22.3, SD = 3.0) also provided artifact-free data for the fMRI session. Subject exclusion details are below. Participants received partial course credit or payment for their participation. All subjects were right-handed, had normal or corrected-to-normal visual acuity, and gave informed consent for each session. The study was approved by the Institutional Review Board of the University of Colorado.

Stimuli and apparatus

Stimuli consisted of 804 colored pictures of objects, half of which were used in the ERP and fMRI sessions, respectively. No picture was used both in the ERP and fMRI session. Twenty additional pictures were used for practice trials.

The apparatus and testing room was the same for the study phases of the ERP and fMRI sessions. All pictures were presented on an LCD computer monitor on a black background. Pictures were 7.82 cm wide by 8.15 cm high, with a viewing distance of approximately 100 cm. During the test phases of the ERP session, pictures were presented on an LCD computer monitor, were 7.9 cm wide by 8.4 cm high, with a viewing distance of 100 cm. During the test phases of the fMRI session, pictures were presented on a standard back projection screen (Avotc, Inc.). Pictures were about 22 cm wide by 16 cm high, with a viewing distance approximately 40 cm.

Procedure

General procedure

The procedures for the ERP and fMRI sessions were the same except when noted otherwise. ERP and fMRI sessions were separated by a mean of 9.18 days (SD = 4.53, Range = 6–24). Participants always completed the ERP session first to limit subject-testing expenses. If a subject’s data was unacceptable for the less expensive ERP session due to low trial counts (less than 25 trials per condition as was the case in N = 4 subjects) or excessive eye-movement artifacts (N = 2; see section Event-related potential recording and measurement for description of eye-movement artifacts), or an experimenter error that resulted in a subject being pre-exposed to stimuli in the wrong conditions (N = 1); he or she was not invited for the more expensive fMRI session. In addition, 11 subjects were excluded for the MRI session due to problems including failure to return for the MRI session (N = 4), excessive head movements (maximum motion that was larger than 1.0 mm in x, y, or z or 1.0 degree rotation about x,y, or z; N = 3), equipment failure (N = 3), or partying all night without sleep before the session (N = 1). An additional 3 subjects were not invited back for the fMRI session because of high blink rates during the ERP session, but we were subsequently able to salvage their ERP data with blink correction (described below). We did not observe any training effects from the ERP to the fMRI session despite the fixed sequence in procedure (see section Behavioral performance—Results and Tables 1 and 2). Furthermore, counterbalancing the order of ERP and fMRI session could have undermined the ERP/fMRI correlations that are the focus of the present analyses.

Table 1.

Accuracy data (proportion) for hits and false alarms from the fMRI and ERP sessions (M: mean; SD: standard deviation).

	fMRI		ERP fMRI subgroup		ERP all subjects
Condition/Response	M	SD	M	SD	M	SD
Original    Pictures
Orientation correct/“Same”	0.76	0.12	0.78	0.09	0.76	0.09
Orientation incorrect/“Different”*	0.10	0.06	0.11	0.06	0.11	0.06
Mirror    Pictures
Orientation correct/“Different”	0.60	0.10	0.60	0.10	0.57	0.12
Orientation incorrect/“Same”*	0.23	.08	0.26	0.07	0.28	0.09
New    Pictures
Same-false alarm/“Same”	0.04	0.05	0.04	0.4	0.05	0.06
Different-false alarm/“Different”	0.03	0.04	0.03	0.03	0.04	0.05

Note. Statistical analysis of accuracy data did not take into account the conditions with incorrect orientation judgments marked with (*).

Table 2.

Reaction time data (in ms) for the fMRI and ERP sessions (M: mean; SD: standard deviation).

	fMRI		ERP fMRI subgroup		ERP all subjects
Condition	M	SD	M	SD	M	SD
Original    Pictures
Orientation correct/“Same”	1372	195	1369	209	1397	190
Orientation incorrect/“Different”	1740	252	1817	235	1811	241
Misses/“New”	1419	206	1385	228	1424	235
Mirror    Pictures
Orientation correct/“Different”	1527	203	1538	175	1560	198
Orientation incorrect/“Same”	1575	177	1605	164	1663	177
Misses/“New”	1454	243	1374	242	1420	254
New    Pictures
Correct rejections/“New”	1306	232	1171	241	1164	236
Same-false alarm/“Same”	1824	353	1922	298	1901	334
Different-false alarm/“Different”	1777	380	1808	300	1810	263

Study phase

Each study phase was conducted one day before the test phases with either ERP recording or fMRI scanning. Before the first study phase, participants completed a short practice of the study and test phase to get familiar with the task and procedure. Then, participants studied a total of 268 pictures of asymmetric common objects (Fig. 1). Pictures were presented in randomized order. Additional buffer pictures, one at the beginning and one at the end of each block, were used as practice test pictures (see test phase description). Participants indicated by button presses whether they subjectively thought the picture was oriented to left (left response key) or right (right response key). After all the pictures were shown once, they were repeated twice in different orders and separate blocks. Subjects received auditory feedback to indicate whether their orientation response was the same (bell-tone) or different (buzzer-tone) from their response to the first presentation of the picture. Pictures were presented for 2 seconds followed by a 500 ms blank/fixation period. A feedback tone prompted subjects to respond when they failed to do so within 2 seconds. Subjects were allowed a self-paced break after each study block.

Test phase

One day later, subjects were tested on their memory for studied pictures. Each test session began with a practice test-block, in which subjects practiced the respective response mapping for the ERP and fMRI session with buffer items from the study lists being used as previously studied (i.e., “old”) practice items. The actual test phases contained all 268 studied pictures randomly intermixed with 134 new pictures. Previously studied pictures were divided such that half of the pictures appeared in the original study-orientation and half appeared in the mirror-reversed orientation. Subjects were given a self-paced break after every 1.5 minutes to rest and relax their eyes in an effort to reduce blinking during EEG recording. Each test picture was presented for 3 seconds. Test trials in the ERP session included a variable duration (500–1500 ms) fixation cross, followed by a test picture. In the fMRI session, test pictures were presented without a fixation period between them. In both sessions, participants used their index and middle finger of one hand and their index finger of the other hand to respectively press a key for “same,” “different,” or “new” (Fig. 1), with “same” (index finger) and “different” (middle finger) always on the same hand and “new” on the opposite hand index finger . Assignment of left vs. right hand to “same”/”different” vs. “new” was counterbalanced across subjects.

Stimuli were randomly assigned to conditions for each subject separately. For the ERP session, the assignment of stimuli to test lists was randomized. For the fMRI session, a novel extension of a genetic algorithm (Wager & Nichols, 2003) was used that incorporated probabilistic estimates of response accuracy (from pilot results) to optimize contrast detection power (Cordes, Herzmann, Nandy, & Curran, in press).

Event-related potential recording and measurement

The EEG was recorded in the study and test blocks with a 256-channel Hydrocel Geodesic Sensor Net^TM (HGSN 256 v. 1.0, Tucker, 1993; Fig. 2) connected to an AC-coupled high-input impedance amplifier (200 MΩ, Net Amps^TM, Electrical Geodesics Inc. (EGI), Eugene, OR). Amplified analog voltages (0.1–100 Hz bandpass) were digitized at 250 Hz. The recording reference was the vertex channel (Cz). Individual sensors were adjusted until impedances were less than 50 kΩ. This is a standard impedance criterion for recording with EGI’s high-input impedance amplifiers.

Figure 2.

Geodesic sensor net layout. Electrode sites are numbered. Red clusters are regions of interest included in analyses. FPM =frontal polar medial, LAS = left anterior superior, RAS = right anterior superior, CM = central medial, LPS = left posterior superior, RPS = right posterior superior.

The EEG was digitally low-pass filtered at 40 Hz prior to segmenting into epochs of 1100 ms, starting 100 ms before target onset. Individual channels were replaced on a trial-by-trial basis with a spherical spline algorithm (Srinivasan, Nunez, Tucker, Silberstein, & Cadusch, 1996). Blinks were corrected using automated Independent Components Analysis (ICA) as implemented by the ERP PCA Toolkit (Dien, 2010), which calls on functions of the EEGLAB (Delorme & Makeig, 2004) and FieldTrip (Oostenveld, Fries, Maris, & Schoffelen, 2011) Matlab toolboxes. Trials were discarded from analysis if they contained uncorrected blinks (vertical electrooculogram differences greater than 70 µV), horizontal eye movements (horizontal electrooculogram > 70 µV), or more than 20 % of the channels were bad (average amplitude over 100 µV or over 50 µV between adjacent samples). ERPs were aligned to a 100-ms baseline before target onset, averaged separately for each channel and condition, and recalculated to the average reference. For each subject, it was ensured that a minimum of 25 trials per condition were available for analysis.

Time segments and regions of interest (ROIs) were defined according to previous research (Herzmann, Willenbockel, Tanaka, & Curran, 2011; Nyhus & Curran, 2012). Mean amplitudes for each time segment were computed by averaging the channels within each ROI for each condition and subject. ROIs for the FN400 (260–440 ms) were left anterior superior (LAS), frontal polar medial (FPM), and right anterior superior (RAS) regions. ROIs for the parietal old/new effects (440–600 ms) were left posterior superior (LPS), central medial (CM), and right posterior superior regions (RPS). ROIs for the late frontal old/new effects (600–1000 ms) were the same as for the FN400. For locations of ROIs see Figure 2.

Source localization of the ERP current source density (µV/cm²) was carried out for all statistically significant, grand mean ERP old/new effects (i.e., difference waves) using all 256 channels in the specified time windows. Activation patterns were derived by standardized Low Resolution Electromagnetic Tomography Analysis (sLORETA; Pascual-Marqui, 2002) in GeoSource, Version 2.0 (EGI, Eugene, OR). sLORETA solutions assumed standard electrode locations provided by EGI and default settings for all user-modifiable parameters within GeoSource: an isotropic Sun-Stok 4-shell spherical head model, 2447 dipoles distributed across the cortical surface with 2 mm resolution, and Tikhonov regularlization (1 × 10⁻⁴). Regularization of the inverse model refers to adding a small regularization parameter (1 × 10⁻⁴) to all singular values. The resulting voxel intensities were displayed on MRI slice views of a single-subject, typical, MNI-(Montreal Neurological Institute)-transformed brain.

fMRI acquisition and measurement

fMRI was performed in a 3.0T GE HDxMRI scanner equipped with an 8-channel head coil and parallel imaging acquisition using echo-planar imaging (EPI) with imaging parameters: Array Spatial Sensitivity Encoding Technique (ASSET) = 2, rampsampling (a GE option that allows sampling of the signal under the ramp of the readout gradient to allow faster data acquisition), Repetition Time (TR)/Echo Time (TE) = 1.5 sec/30 ms, Flip Angle (FA)=70 deg, Field of View (FOV) = 22 cm × 22 cm, thickness/gap = 3.5 mm/0.5 mm, 30 slices, resolution 64 × 64, axial acquisitions (aligned to the AC-PC line). A standard 2D co-planar T1-weighted image and a standard 3D high resolution T1-weighted SPoiled Gradient Recalled (SPGR) image (1 mm³ resolution) were also collected.

Image processing and data analysis were performed using the FMRIB Software Library (FSL) package (Analysis group, FMRIB, Oxford, UK, http://www.fmrib.ox.ac.uk/fsl/). Standard pre-processing was applied: MCFLIRT – slice time correction/motion correction, BET – brain extraction, time-series prewhitening, registration, and spatial normalization to the MNI space (2 mm resolution) with spatial smoothing using a Gaussian FWHM = 8 mm. Each normalized image sequence was temporally filtered to remove low frequency artifacts < 1/120 sec). FMRIB’s improved linear model (FILM) was then applied from which statistical inferences were based on the theory of Gaussian random fields. Regressors included all possible combinations of condition (original image, mirror image, new image) and response (“same,” “different,” or “new”) and were modeled by convolution of single trial epochs with the Double-Gamma Haemodynamic Response Function. Although all conditions were modeled, only conditions with at least 20 occurrences per subject were included in the inference of contrasts. Conditions with less than 20 occurrences per subject were model but not analyzed further (see below). Individual reaction times were also included as a regressor to control for brain activation associated with the individual variance of response speed. Four contrasts of fMRI memory activation were computed (see Fig. 1 for event type details), according to past fMRI research (Diana et al., 2007, Spaniol et al., 2009). These contrasts assume that recollection is indexed by the ability to correctly remember item and orientation, whereas familiarity is indexed by item recognition without the recollection of orientation:

1. familiarity (old[incorrect] > correct rejection),

2. recollection of original images (original[correct] > old[incorrect]),

3. recollection of mirror images (mirror[correct] > old[incorrect]),

4. recollection contrast between original and mirror images (original[correct] > mirror[correct]).

Conditions original[incorrect] and mirror[incorrect] (Fig. 1) were combined into the condition old[incorrect] because of low trial numbers. Analyses also did not consider false alarms because of low trial counts. When interpreting the present contrasts, two limitations have to be kept in mind. First, recollection and familiarity contrasts are likely to also differ with regard to confidence (Wixted & Squire, 2011). Second, the familiarity contrast may include activity related to the recollection of attributes other than orientation, so-called “non-criterial” recollection (Parks, 2007; Yonelinas & Jacoby, 1996).

Group statistical maps were computed for all contrasts using FLAME 1 (FMRIB’s Local Analysis of Mixed Effects) in FMRI Expert Analysis Tool, a part of the FSL package. Clusters were considered statistically significant if they surpassed a voxelwise threshold of Z > 3.29 and a cluster size of 50 voxels, as determined by Monte-Carlo simulations (10,000 iterations) in AFNI (Cox, 1996) to achieve a family-wise error rate (FWE) of α < 0.05.

ERP/fMRI relationships were specified for all ERP old/new effects that yielded significant results. For these old/new effects (defined as difference waves between two conditions), mean activity was calculated for each individual subject by averaging across all channels in the ROIs used for any given component in the ERP analysis (see section Event-related potential recording and measurement). A regressor containing these individual ERP values was then added, one at a time, in the general linear model of the respective fMRI contrast. For example, the regressor for the ERP difference between correct recollection of original and mirror images (i.e., original minus mirror) was added to fMRI contrast 4 (original[correct] > mirror[correct], see above). Clusters in these statistical maps were considered significant if they surpassed a voxelwise threshold of Z > 2.58 and a cluster size of 103 voxels (FWE α < 0.05).

Results and Discussion

Behavioral performance—Results

Table 1 shows accuracy data and Table 2 response times (RTs) for the fMRI session and for the ERP session. For the latter data is presented for all subjects (N = 30) and for the subset of these subjects who also completed the fMRI session (N = 16).

Analysis of behavioral data had two aims. First, we wanted to test whether subjects accurately recognized old items and thus showed a significant memory effect. We also tested whether this effect was different for original than mirror images. For accuracy data from the fMRI session and from the full sample (N = 30) of the ERP session, we conducted analyses of variance (ANOVA) with repeated measures on the factors response (“same,” “different”) and memory type (hits defined as old items with correct item recognition and orientation judgments; false alarms defined as new items with incorrect memory judgments). Thus, the analysis did only consider the four conditions: original pictures with “same” responses, mirror pictures with “different” responses, new pictures with “same” responses, and new pictures with “different” responses (see Table 1). Results were similar for the ERP and fMRI session. Hit rates were significantly higher than false alarm rates, F(1,29) = 708.1, MSE = 0.016, p < .001 and F(1,15) = 413.8, MSE = 0.016, p < .001, for the ERP and fMRI session respectively. Only hit rates were different across the two response options, F(1,29) = 116.6, MSE = 0.002, p < .001 and F(1,15) = 79.8, MSE = 0.001, p < .001, for the ERP and fMRI session respectively. Old original images were more accurately recognized than old mirror images, t(29) = 19.4, p < .001 and t(15) = 18.1, p < .001, for the ERP and fMRI session respectively, whereas false alarms did not differ, ps = .24 and .35.

Second, we wanted to rule out that there were any differences in performance between the ERP and fMRI session. For accuracy data of the subset of participants (N = 16) who had completed both the ERP and fMRI session, we conducted the same ANOVA as reported above but added the factor session (ERP, fMRI). No main effect or interactions of the factor session were found, all ps > .63. We also conducted pairwise t-tests of RT measures between the ERP and fMRI session for each condition reported in Table 2. No significant differences in RTs were observed, all ps > .12. Thus, there were no practice effects from the ERP to the fMRI session.

Behavioral performance—Discussion

The behavioral data showed that accuracy and RTs of memory judgments were similar for the ERP and fMRI sessions. Thus, no practice effects were observed even though the fMRI session always followed the ERP session. In general, participants were more accurate in remembering the orientation of original than mirror images. This does not appear to reflect a general bias to respond “same” because no such bias was found for false alarms.

ERP memory effects—Results

ERP memory effects are reported for all subjects (N = 30) from the ERP session. The same pattern of effects, with lower statistical power, was obtained for the subset of 16 participants, who also completed the fMRI session. Figure 3 shows average ERP waveforms and Figure 4 topographies of old/new effects.

Figure 3.

Average ERP waveforms for original[correct], mirror[correct], old[incorrect], and correct rejection for the FN400 (FPM, LAS, RAS, 260–440 ms), parietal old/new effect (CM, LPS, RPS, 440–600 ms), and late frontal old/new effect (FPM, LAS, RAS, 600–1000 ms).Vertical lines indicate time segments used for statistical analysis.

Figure 4.

Voltage maps of ERP old/new effects showing the FN400 260–440 ms, the parietal old/new effect at 440–600 ms, and the late frontal old/new effect at 600–1000 ms. Spherical spline interpolation was used. Asterisks denote maps that were not associated with significant differences between conditions.

ANOVAs with repeated measures on the factors of memory judgment (original[correct], mirror[correct], old[incorrect], correct rejection) and region of interest (LAS, FPM, and RAS for FN400 and late frontal old/new effect; LPS, CM, and RPS for parietal old/new effects) were conducted. Huynh-Feldt (Huynh & Feldt, 1976) and Bonferroni corrections were applied.

The main effect of memory judgment for old/new effects between 260–440 ms, F(3, 87) = 10.6, MSE = 0.91, p < .001, was due to significant differences between correct rejections and original[correct], F(1, 29) = 23.2, MSE = 0.98, p < .001, mirror[correct], F(1, 29) = 8.0, MSE = 1.03, p < .05, and old[incorrect], F(1, 29) = 21.6, MSE = 0.96, p < .001, which all showed more positive amplitudes over frontal areas than correct rejections. There was also a trend for a significant difference between original[correct] and mirror[correct], F(1, 29) = 4.6, MSE = 0.78, p = .08 (p = .04 before Bonferroni correction).

The main effect of memory judgment for old/new effects between 440–600 ms, F(3, 87) = 7.9, MSE = 1.02, p < .001, indicated that original[correct] showed significantly more positive amplitudes over parietal areas than old[incorrect], F(1, 29) = 6.3, MSE = 0.47, p < .05, mirror[correct], F(1, 29) = 14.4, MSE = 0.69, p < .001, and correct rejections, F(1, 29) = 18.4, MSE = 1.06, p < .001. Old[incorrect] also differed from correct rejections, F(1, 29) = 5.8, MSE = 1.24, p < .05. For mirror[correct], the significant memory judgment × ROI interaction, F(2, 58) = 9.9, MSE = 0.23, p < .001, indicated that this condition differed from correct rejections only over central medial regions, t(29) = 3.6, p < .001.

The main effect of memory judgment for old/new effects between 600–1000 ms, F(3, 87) = 8.0, MSE = 1.67, p < .001, was due to significant differences between correct rejections and original[correct], F(1, 29) = 16.6, MSE = 1.81, p < .001, mirror[correct], F(1, 29) = 15.9, MSE = 1.48, p < .001, and old[incorrect], F(1, 29) = 7.2, MSE = 2.4, p < .05, which all showed more positive amplitudes over frontal areas than correct rejections.

ERP memory effects—Discussion

ERP results showed old/new effects, which had slightly earlier timelines but similar patterns of amplitude differences (Fig. 3) and similar topographical distributions (Fig. 4) as ERP correlates of familiarity, recollection, and post-retrieval monitoring reported in previous studies. Old/new effects between 260 and 440 ms resembled the FN400 and were significant over frontal regions for all correctly recognized old pictures, irrespective of the accuracy of the orientation judgment. These old/new effects are therefore taken to reflect familiarity processes. A recent experiment from our laboratory used a very similar experimental design, but manipulated whether or not subjects received the amnestic drug midazolam during study (Nyhus & Curran, 2012). Following midazolam administration, FN400 old/new effects were not affected by the accuracy of the orientation judgments, as in the present results. Surprisingly, following saline administration, FN400 old/new effects were only observed when orientation judgments were correct. Familiarity effects might have been weaker in Nyhus and Curran’s study because pictures were each only studied twice (rather than 3 times as in the present study), leading to lower accuracy than in the present study.

Old/new effects between 440 and 600 ms resembled the parietal old/new effects and showed more positive amplitudes for correctly recognized original images than any other memory condition. As in previous studies, this parietal old/new effect is taken to reflect recollection processes (Rugg & Curran, 2007; Vilberg & Rugg, 2007, 2009). Interestingly, correctly recognized mirror images—a condition also associated with detailed recollections from episodic memory—showed a significantly smaller parietal old/new effect than correctly recognized original images. When considering recall-to-reject and recall-to-accept strategies of memory retrieval (Curran & Cleary, 2003; Hintzman & Curran, 1994; Rotello & Heit, 2000; Yonelinas, 1997), this pattern suggests that the present parietal effects are more closely associated with recollecting that a test picture is exactly the same as a previously studied one (recall-to-accept for original images) than with rejecting (calling “different”) a test picture based on the recollection that its test orientation is different from the studied orientation (recall-to-reject for mirror images). A similar pattern of results was found by Curran & Cleary (2003), who also observed differences in the parietal old/new effects between two memory conditions that combined item and orientation recognition and thus closely resembled the present original[correct] and mirror[correct] conditions. Not only were the present parietal amplitudes greater for the original[correct] than mirror[correct], but mirror[correct] did not differ from old[incorrect]. This suggests that recall-to-reject processes had no influence on the parietal old/new effects, which may also be consistent with previous claims that recall-to-reject processes may primarily come into play when subjects are explicitly instructed about the usefulness of this strategy (Rotello & Heit, 2000), because the present subjects received no such instructions.

Another way to conceptualize the greater parietal old/new effects for original than mirror images is that this ERP memory effect captures bottom-up recollection processes to a greater extent than top-down recollection (Cabeza et al., 2008; Ciaramelli et al., 2008). Bottom-up processes are assumed to be triggered by the similarity of the retrieval cue with the stored memory representation. Original images thus profit more from bottom-up recollection whereas mirror images require additional, top-down processes like noticing that the retrieval cue is an old item, but that its study orientation is different. Furthermore, differences between mirror and original images could also have been expected during post-retrieval monitoring indicated by late frontal old/new effect, when participants might have mentally rotated the retrieval cue in the mirror condition to compare it to the stored memory representation. Surprisingly, we did not find evidence for this top-down processing of mirror images in the ERP data because mirror images did not elicit more positive amplitudes than original images. It is possible that ERPs and/or the present ERP paradigm might not be best suited to observe effects that are indicative of top-down processing of mirror images, especially in the later time course of memory retrieval.

Pictures with incorrect orientation judgments did also elicit a small but significant parietal old/new effect. This might suggest that recollection of details from the study episode (e.g., a thought that came to mind), but not the picture orientation, accompanied the retrieval of these items (“non-criterial recollection”, Yonelinas & Jacoby, 1996). However, the topography of the old/new effects for old[incorrect] items in the FN400 (260–400 ms) and parietal old/new effect (440–600 ms) time window (Fig. 4, middle column) are very similar. Thus, it appears more likely that familiarity processes are prolonged into the time window of the parietal old/new effects. Further support for this assumption can be found in similar ERP/fMRI relationships of the old(incorrect) > new condition for the time windows of the FN400 and parietal old/new effect (see below).

Old/new effects between 600 and 1000 ms resembled the late frontal old/new effects and showed more positive amplitudes for all correctly recognized old pictures, irrespective of the accuracy of the orientation judgment. These old/new effects are taken to represent post-retrieval monitoring processes that should be more engaged for old than new conditions because the former are more likely to be associated with memory retrieval (Cruse & Wilding, 2009; Rugg et al., 2002).

fMRI memory effects—Results

Tables 3 to 6 and Figures 5 to 8 show all significant group activations (positive contrasts) and deactivations (negative contrasts)² for the contrasts of familiarity, recollection of original images, recollection of mirror images, and the recollection-contrast between original and mirror images.

Table 3.

Significant regions (corrected Z > 3.29, FWE α < 0.05, cluster size > 50) for the familiarity (old[incorrect] > correct rejection) contrast. Negative contrasts are shaded. Clusters that represent unique effects for the familiarity contrast but not the recollection contrasts (Tab. 4 and 5) are shown in red.

MNI region (L;R)	BA	cluster size (# of voxels)	Peak Z-value	MNI coordinates
Angular gyrus (L;R)	39	573;344	4.33;4.65	−40,−60,32;38,−52,32
Anterior cingulate gyrus (L;R)	32	505;454	−4.52;−4.58	0,44,4;4,42,2
Calcarine fissure (L;R)	23	442;161	−4.98;−4.96	−2,−62,20;2,−60,20
Cuneus (L;R)	17	91;212	3.66;5.60	−8,−80,42;14,−82,48
Cuneus (L;R)	17	164;180	−4.77;−4.60	−8,−60,22;4,−62,22
Heschl gyrus (L)	41/42	82	−4.19	−34,−22,12
Hippocampus (L;R)		146;83	−4.35;−3.73	−36,−24,−8;40,−18,−12
Inferior frontal gyrus, opercular part (L)	44	157	4.58	−50,20,34
Inferior frontal gyrus, triangular part (L)	45	135	4.56	−50,20,32
Inferior temporal gyrus (L;R)	20	292;894	−4.65;−4.66	−58,−4,−26;54,−14,−26
Insula (L)	13	94	4.36	−32,18,−6
Intraparietal sulcus (L;R)	39/40	218;185	3.97;4.32	−46,−58,46;44,−44,38
Lingual gyrus (L)	18/19	71	−4.03	−4,−66,10
Median cingulate gyrus (L;R)	23	651;610	−4.77;−4.65	−2,−26,34;0,−26,32
Middle frontal gyrus (L;R)	46	101;62	4.06;3.79	−32,0,54;30,2,56
Middle occipital gyrus (L)	19	212	−4.51	−46,−70,12
Middle occipital gyrus (L;R)	19	313;270	4.36;4.38	−30,−68,26;30,−64,30
Middle temporal gyrus (L;R)	21	1325;1455	−4.75;−4.63	−58,−4,−24;64,−46,16
Parahippocampal gyrus (L;R)	27	169;177	−4.07;−4.39	−26,−16,−28;24,−10,−30
Posterior cingulate gyrus (L;R)	23	191;103	−5.06;−4.38	0,−52,22;2,−48,24
Precentral gyrus (L;R)	4	150;117	4.00;4.35	−32,0,56;54,8,28
Precuneus (L;R)	7	718;652	−5.30;−5.05	−2,−58,22;2,−56,22
Precuneus (R)	7	160	5.53	12,−80,48
Putamen (R)		65	−4.32	34,4,4
Superior occipital gyrus (L;R)	19;7	129;352	4.11;4.88	−10,−82,46;30,−74,46
Superior occipital gyrus (L;R)	17/18	207;82	−4.75;−4.48	−10,−96,14;14,−96,18
Superior parietal gyrus (R)	19	163	5.41	14,−82,50
Superior temporal gyrus (L;R)	41/42	290;788	−3.99;−4.68	−54,−34,12;60,−42,16
Supramarginal Gyrus (R)	40	56	4.41	44,−42,38
Temporal pole, middle temporal gyrus (L;R)	38	158;143	−4.41;−4.04	−54,8,−34;40,24,−34
Temporal pole, superior temporal gyrus (L;R)	28	106;65	−3.96;−3.91	−26,6,−26;38,24,−30

Table 6.

Significant regions (corrected Z > 3.29, FWE α < 0.05, cluster size > 50) for the recollection-contrast between original and mirror images (original[correct] > mirror[correct]). Negative contrasts are shaded.

MNI region (L;R)	BA	cluster size (# of voxels)	Peak Z-value	MNI coordinates
Angular gyrus (L;R)	39	199;310	−4.30;−4.47	−36,−48,34;40,−48,30
Cerebelum (L;R)		204;168	−4.24;−3.78	−48,−68,−20;44,−58,−22
Cuneus (L;R)	17	122;203	−4.12;−4.27	−8,−78,42;14,−82,44
Fusiform gyrus (L)	37	90	−3.98	−44,−68,−16
Inferior frontal gyrus, opercular part (R)	44	137	−4.05	52,22,30
Inferior frontal gyrus, orbital part (R)	10	65	−3.65	44,40,−12
Inferior frontal gyrus, triangular part (L;R)	45	53;357	−3.75;−4.27	−46,22,30;48,32,18
Inferior occipital gyrus (L)	17	163	−4.18	−48,−68,−18
Intraparietal sulcus (L;R)	39/40	264;178	−3.94;−4.17	−40,−60,54;38,−50,38
Middle frontal gyrus (R)	46	148	−4.01	30,−2,56
Middle occipital gyrus (L;R)	19	258;969	−4.09;−4.62	−32,−80,20;36,−82,16
Precentral gyrus (L;R)	4	172;109	−3.80;−3.91	−28,−6,54;32,0,52
Superior frontal gyrus (R)	47	65	−3.93	30,−4,58
Superior occipital gyrus (L;R)	17/18	154;366	−4.15;−4.25	−10,−80,44;30,−76,40
Superior parietal gyrus (R)	19	59	−3.89	12,−80,52

Figure 5.

Positive (red) and negative (blue) contrasts of group activation maps (Z > 3.29, FWE α < 0.05, cluster size > 50) for the familiarity contrast (old[incorrect] > correct rejection).

Figure 8.

Positive (red) and negative (blue) contrasts of group activation maps (Z > 3.29, FWE α < 0.05, cluster size > 50) for the recollection-contrast between original and mirror images (original[correct] > mirror[correct]).

The results indicate that most effects were bilateral. Considering familiarity and recollection contrasts (Tab. 3–5, Fig. 5–7), many overlapping regions were found when looking at both positive and negative contrasts, which might be due to the similar term that contributes to the contrasts (old[incorrect]). Regions that showed both positive recollection activations and negative familiarity contrasts might reflect confidence differences between conditions insofar as accurate responses to original, mirror, and new images should be associated with higher confidence. From this perspective, focusing on effects that are unique to either familiarity or recollection irrespective of the sign might better reflect true differences between familiarity and recollection. In addition, it is possible that regions that show negative familiarity contrasts are more sensitive to novelty. Major clusters (positive or negative) of the familiarity contrast (but not for the recollection contrasts) occur in the bilateral angular gyri, bilateral prefrontal gyri, left Heschl gyrus, left insula, bilateral middle and superior occipital gyri, bilateral intraparietal sulcus, bilateral precentral gyri, bilateral middle temporal gyri, and in the bilateral temporal pole of the superior temporal gyri. For the recollection contrasts (but not the familiarity contrast), major clusters are in the right amygdala, cerebellum, right putamen, and right supramarginal gyrus.

Table 5.

Significant regions (corrected Z > 3.29, FWE α < 0.05, cluster size > 50) for the contrast indicating the recollection of mirror images (mirror[correct] > old[incorrect]) contrast. Clusters that represent unique effects for the recollection but not the familiarity contrast (Tab. 3) are shown in red.

MNI region (L;R)	BA	cluster size (# of voxels)	Peak Z-value	MNI coordinates
Amygdala (R)		103	4.05	30,0,−20
Calcarine fissure (L;R)	23	97;50	3.87;3.74	−6,−48,8;2,−56,14
Hippocampus (L;R)		249;116	4.33;3.99	−32,−12,−14;32,−2,−22
Inferior frontal gyrus, orbital part (R)	10	64	3.67	54,32,−8
Inferior temporal gyrus (R)	20	393	4.11	58,−12,−26
Intraparietal sulcus (L)	39/40	59	3.71	−28,−42,48
Median cingulate gyrus (L;R)	23	313;205	4.23;3.71	−10,−46,38;12,−42,36
Middle frontal gyrus, orbital part (L)	10	134	3.82	−2,66,−2
Middle occipital gyrus (L)	19	125	3.71	−44,−78,26
Middle temporal gyrus (L;R)	21	190;194	3.73;3.79	−58,0,−26;60,−10,−24
Paracentral lobule (R)	4	81	3.74	6,−28,72
Parahippocampal gyrus (R)	27	96	4.09	24,−6,−32
Posterior cingulate gyrus (L;R)	23	350;122	4.38;3.81	−6,−44,20;4,−44,22
Precuneus (L;R)	7	1078;913	4.25;4.02	−6,−48,18;6,−56,52
Putamen (L;R)		232;196	4.23;4.00	−24,4,−4;30,−12,4
Superior frontal gyrus, medial orbital (L)	47	223	3.9	−8,66,4
Superior parietal gyrus (R)	19	162	3.89	16,−56,52
Superior temporal gyrus (R)	41/42	143	3.69	68,−14,4
Supramarginal gyrus (R)	40	113	3.71	56,−30,26

Figure 7.

Positive (red) and negative (blue) contrasts of group activation maps (Z > 3.29, FWE α < 0.05, cluster size > 50) for the contrast indicating the recollection of mirror images (mirror[correct] > old[incorrect]).

Table 6 and Figure 8 show clusters for the recollection contrast between original and mirror images. Only clusters showing stronger brain activation for mirror images than original images as indicated by negative Z-values were found. These clusters were wide spread across the brain and found in the bilateral angular, right frontal, bilateral precentral, and bilateral occipital and parietal gyri, bilateral cerebellum, and bilateral cuneus.

fMRI memory effects—Discussion

The present finding of stronger effects for the familiarity contrast in the angular and prefrontal gyri, insula, and parietal cortex corresponds well with previous findings (Cansino et al., 2002; Dobbins et al., 2003; Ragland et al., 2006; Skinner & Fernandes, 2007; Slotnick et al., 2003; Vilberg & Rugg, 2007, 2009; Wheeler & Buckner, 2004; Yonelinas et al., 2005). Activity of the prefrontal gyrus might not be familiarity-specific but rather be explained by a larger working memory demand for “familiar” memory judgments (Wheeler & Buckner, 2004; Yonelinas et al., 2005), which are associated with less confidence and are more difficult for the participant as can be seen by increased response times when orientation judgments were incorrect (see Table 2). We also found familiarity-related effects in bilateral hippocampus. Significance of this negative contrast can be expected due to a response to novel stimuli involving the anterior portion of the medial temporal lobes. We did not find any indication that the perirhinal cortex is involved in familiarity, contrary to previous reports (Diana et al., 2007, Haskins, Yonelinas, Quamme, & Ranganath, 2008). A reason for this discrepancy may be signal drop-out due to susceptibility effects of the sphenoid sinus affecting the anterior part of the parahippocampal gyrus (entorhinal cortex, perirhinal cortex). Signal drop-out in these regions is especially strong for axial acquisitions of echoplanar data, as used here (Jin, Pelak, & Cordes, 2011).

The present recollection clusters are in line with previous studies that reported activation in the amygdala and the parietal cortex for the recollection contrast (Henson, Rugg, Shallice, Josephs, & Dolan, 1999; Spaniol et al., 2009; Yonelinas et al., 2005). Recollection-specific clusters were found in lateral parietal regions (supramarginal gyrus) whereas medial parietal regions (e.g., precuneus) were active for both familiarity and recollection contrasts. The supramarginal gyrus has been linked to bottom-up attention-to-memory (Cabeza et al., 2008; Ciaramelli et al., 2008) and has been hypothesized to be more strongly involved in recollection than familiarity. The present results support this view. Our results of overlapping and non-overlapping clusters for familiarity and recollection indicate that familiarity and recollection rely on different brain networks that share particular brain regions, which might serve perceptual or decision processes that are common to both familiarity and recollection (Dörfel et al., 2009; Wheeler & Buckner, 2004).

Recollection of mirror images, relative to recollection of original images, was associated with stronger activation in a network of brain regions previously related to retrieval and recollection (Henson et al., 1999; Spaniol et al., 2009; Wheeler & Buckner, 2004; Yonelinas et al., 2005). Increased activation in frontal, parietal, and occipital regions might be recruited for spatial transformations associated with the processing of mirror-images like mental rotation (Dong et al., 2000; Tagaris, Kim, Strupp, Anderson, Ugurbil, & Georgopoulos, 1997; Weiss et al., 2009) or visual imagery (Mellet, Tzourio-Mazoyer, Bricogne, Mazoyer, Kosslyn, & Denis, 2000; Newman, Klatzky, Lederman, & Just, 2005), which can be expected to be recruited more for mirror than original images especially during post-retrieval monitoring. Activation in parietal regions might be associated with a stronger requirement on top-down attention to memory (Cabeza et al., 2008; Ciaramelli et al., 2008). In addition, this contrast showed more activation in right than left frontal regions, which have previously been connected to diagnostic retrieval monitoring (Gallo et al., 2010). Interestingly, the recollection difference of mirror and original images showed activation clusters in similar regions as the positive familiarity contrast (see Tab. 3 and 6 and Fig. 5 and 8), specifically in the cuneus, precuneus, and angular gyrus, precentral gyrus, superior occipital gyrus, and intraparietal sulcus. Both conditions, correctly recognized mirror images and images with incorrect orientation judgments, showed the slowest reaction times and low memory accuracy. This overlap in fMRI patterns might thus reflect increased task difficulty.

Relationships between ERP and fMRI memory effects—Results

Relationships between all statistically significant ERP-memory effects and respective fMRI contrasts are shown in Tables 7 to 9, which report significant clusters for each ERP-memory effect, apart from the old/new effect between mirror and new images between 260 and 440 ms, for which no significant clusters were found. Figures 9 to 13 highlight the most relevant ERP/fMRI relationships and, in one case, the comparison to the ERP source localization (Fig. 12). Most of the relationships were negative, and some of them showed bilateral clusters but there were also unilateral clusters.

Table 7.

Significant regions (corrected Z > 2.58, FWE α < 0.05, cluster size > 103) for the ERP/fMRI relationships of ERP familiarity-effects between 260 and 440 ms. Negative contrasts are shaded.

MNI region (L;R)		BA	cluster size (# of voxels)	Peak Z−value	MNI coordinates
ERP/fMRI contrast original images > new images
Angular gyrus(L;R)		39	104;71	−3.39;−3.56	−42,−52,28;44,−46,36
Heschl gyrus (R)		41/42	125	−3.80	46,−18,12
Inferior frontal gyrus, opercular part (R)		44	39	−3.01	34,6,30
Inferior frontal gyrus, triangular part (L)		45	142	−3.20	−34,32,22
Intraparietal sulcus (R)		39/40	51	−3.44	50,−46,38
Middle frontal gyrus (R)		46	53	−3.22	26,8,52
Postcentral gyrus (R)		3	114	−3.04	50,−18,40
Superior frontal gyrus (R)		47	53	−3.09	24,8,52
Superior temporal gyrus (R)		41/42	85	−3.43	48,−18,6
Supramarginal gyrus (R)		40	40	−3.27	50,−44,38
E RP/fMRI contrast images with incorrect orientation judgment > new images
Supramarginal gyrus (R)		40	220	−3.47	60,−38,36

Note. Cluster-level threshold was 103 voxels. Some of the significant clusters extended into adjacent brain regions. This led to the reported clusters, which seemingly have less than 103 voxels but are, in fact, part of larger clusters with more than 103 voxels.

Table 9.

Significant regions (corrected Z > 2.58, FWE α < 0.05, cluster size > 103) for the ERP/fMRI relationships of ERP post-retrieval monitoring effects between 600 and 1000 ms. Negative contrasts are shaded.

MNI region (L;R)		BA	cluster size (# of voxels)	Peak Z−value	MNI coordinates
ERP/fMRI contrast original images > new images
Angular gyrus (L;R)		39	79;112	−3.00;−3.63	−42,−48,30;42,−50,36
Inferior frontal gyrus, opercular part (L)		44	93	−3.21	−52,8,24
Inferior frontal gyrus, triangular part (L)		45	235	−3.36	−48,28,18
Intraparietal sulcus (L;R)		39/40	16;76	−2.82;−3.35	−46,−56,48;46,−52,40
Middle frontal gyrus (L)		46	49	−3.49	−34,2,52
Precentral gyrus (L)		4	155	−3.62	−36,2,52
ERP/fMRI contrast mirror images > new images
Supramarginal gyrus (L)		40	188	−3.49	−60,−28,32
ERP/fMRI contrast images with incorrect orientation judgment > new images
Intraparietal sulcus (R)		39/40	80	−3.39	60,−48,46
Supramarginal gyrus (R)		40	72	−3.19	62,−46,44

Note. Cluster-level threshold was 103 voxels. Some of the significant clusters extended into adjacent brain regions. This led to the here reported clusters, which seemingly have less than 103 voxels but are, in fact, part of larger clusters with more than 103 voxels.

Figure 9.

ERP/fMRI relationships for the FN400 (260–440 ms) of original minus new images and the fMRI contrast of original > new images (Z > 2.58, FWE α < 0.05, cluster size > 50). For more information on significant clusters see Table 7. Blue indicates negative contrasts.

Figure 13.

ERP/fMRI relationships for the late frontal old/new effect (600–1000 ms) of original minus new images and the fMRI contrast of original > new images (Z > 2.58, FWE α < 0.05, cluster size > 50). For more information on significant clusters see Table 9. Blue indicates negative contrasts.

Figure 12.

Comparison of activation patterns derived by sLORETA in GeoSource using the sLORETA option (top) and ERP/fMRI relationships (bottom) at similar axial slices (z-value in MNI space) for the recollection difference (440–600 ms) between original and mirror images, highlighting possible source generators in the occipital cortex. Source solutions were calculated for the grand mean ERP difference between original minus mirror images. ERP/fMRI relationships are red for original > mirror and blue for mirror > original (see also Table 8).

Old/new ERP-differences between 260 and 440 ms, thought to indicate familiarity processes, were associated with relatively more activation for new than old images in bilateral prefrontal and right postcentral gyri (Table 7, Fig. 9).

Parietal old/new effects (i.e., differences between old and new images) between 440 and 600 ms, taken as ERP correlates of recollection processes, showed relationships with right prefrontal and right temporal gyri as well as left posterior regions in cuneus and supramarginal gyrus, which all showed stronger activation for old than new images (Table 8, Fig. 10 and 11). ERP-differences between original[correct] and old[incorrect] were associated with activation clusters in the left precentral gyrus and intraparietal sulcus showing stronger activation for images with incorrect orientation judgments than original images (Table 8). The ERP-recollection difference between original and mirror images showed relationships with bilateral activation clusters in the middle and superior occipital gyri, intraparietal sulci, superior parietal gyri, and middle temporal gyri, which were more strongly activated for mirror than original images (Table 8, Fig. 12).

Table 8.

Significant regions (corrected Z > 2.58, FWE α < 0.05, cluster size > 103) for the ERP/fMRI relationships of ERP recollection-effects between 440 and 600 ms. Negative contrasts are shaded.

MNI region (L;R)		BA	cluster size (# of voxels)	Peak Z−value	MNI coordinates
ERP/fMRI contrast original images > new images
Anterior cingulate gyrus (R)		32	130	3.55	10,36,24
Cuneus (L;R)		17	218;88	3.40;3.36	2,−76,30;4,−76,30
Inferior frontal gyrus, orbital part (R)		10	100	3.63	46,22,−14
Inferior temporal gyrus (R)		20	65	3.65	58,−20,−16
Middle frontal gyrus (R)		46	294	3.20	24,42,30
Middle temporal gyrus (L;R)		21	66;862	3.27;4.12	−48,−36,10;66,−36,−2
Precuneus (R)		7	25	2.97	8,−48,26
Putamen (L)		106	3.23	−26,14,4
Superior frontal gyrus (R)		47	81	2.98	22,52,22
Superior temporal gyrus (L;R)		41/42	83;29	3.53;3.42	−46,−34,10;56,−22,−4
Temporal pole, superior temporal gyrus (R)		28	95	3.45	48,22,−14
ERP/fMRI contrast mirror images > new images
Middle frontal gyrus (R)		46	46	3.16	44,−2,54
Precentral gyrus (R)		4	86	3.21	42,−6,44
Superior temporal gyrus (L)		41/42	97	3.27	−64,−32,24
Supplementary motor area (R)		6	128	3.46	2,2,56
Supramarginal gyrus (L)		40	49	3.39	−62,−32,26
ERP/fMRI contrast images with incorrect orientation judgment > new images
Middle frontal gyrus (R)		46	78	3.13	28,24,54
Postcentral gyrus (R)		3	107	−3.67	52,−12,36
Superior frontal gyrus (R)		47	142	3.54	26,28,52
ERP/fMRI contrast original images > images with incorrect orientation judgment
Intraparietal sulcus (L)		39/40	188	−3.10	−48,−44,42
Precentral gyrus (L)		4	107	−3.15	−42,−2,56
ERP/fMRI contrast original images > mirror images
Angular gyrus (R)		39	66	−3.31	32,−54,40
Cerebelum (R)			120	−3.24	34,−58,−24
Fusiform gyrus (R)		37	24	−3.06	34,−58,−18
Inferior occipital gyrus (R)		17	76	−3.84	36,−84,0
Intraparietal sulcus (L;R)		39/40	234;40	−3.92;−3.34	−40,−42,48;34,−48,56
Middle occipital gyrus (L;R)		19	357;441	−3.97;−4.68	−36,−82,14;36,−84,4
Middle temporal gyrus (L;R)		21	250;78	3.73;3.38	−58,−44,6;48,−34,2
Precuneus (L;R)		7	44;32	−3.72;−3.35	−12,−76,46;18,−66,48
Superior occipital gyrus (L;R)		17/18	99;35	−3.65;−3.22	−18,−64,40;20,−66,48
Superior parietal gyrus (L;R)		19	364;233	−4.04;−3.83	−18,−66,48;22,−64,56

Note. Cluster-level threshold was 103 voxels. Some of the significant clusters extended into adjacent brain regions. This led to the reported clusters, which seemingly have less than 103 voxels but are, in fact, part of larger clusters with more than 103 voxels.

Figure 10.

ERP/fMRI relationships for the parietal old/new effect (440–600 ms) of original minus new images and the fMRI contrast of original > new images (Z > 2.58, FWE α < 0.05, cluster size > 50). For more information on significant clusters see Table 8. Red indicates positive contrasts.

Figure 11.

ERP/fMRI relationships for the parietal old/new effect (440–600 ms) of mirror minus new images and the fMRI contrast of mirror > new images (Z > 2.58, FWE α < 0.05, cluster size > 50). For more information on significant clusters see Table 8. Red indicates positive contrasts.

Late frontal old/new effects between 600 and 1000 ms, which might represent post-retrieval monitoring processes, were associated with clusters of relatively more activation for new than old images in the left prefrontal and bilateral parietal regions (Table 9, Fig. 13).

Source localization with sLORETA was carried out for the same ERP memory contrasts for which ERP/fMRI correlations had been found. This was done to test for correspondences between ERP/fMRI correlations and possible source generators of the ERP memory effects. We found notable similarities for only one contrast: the ERP difference between original and mirror images in the time window of the parietal old/new effect (440–600 ms), which showed correspondences to the ERP/fMRI correlations in the occipital cortex (Fig. 12).

Relationships between ERP and fMRI memory effects—Discussion

Activations of different structural regions in the fMRI were correlated with temporally and functionally different ERP components. The FN400 (260–440 ms) was related to activation in prefrontal and right postcentral regions—previously indicated in familiarity processes (e.g., Skinner & Fernandes, 2007; Vilberg & Rugg, 2007, 2009; Wheeler & Buckner, 2004; Yonelinas et al., 2005), whereas the parietal old/new effect for original images (original minus new images between 440–600 ms) showed relationships to temporal and occipital regions—suggested to serve recollection processes (Henson et al., 1999; Spaniol et al., 2009; Yonelinas et al., 2005). The parietal ERP old/new effect for mirror images (i.e., mirror minus new) showed an additional association with the supplementary motor area, which has been previously connected to mental rotation (Dong et al., 2000; Tagaris et al., 1997; Weiss et al., 2009). The recollection difference between original and mirror images between 440 and 600 ms was related to activation in occipital, parietal, and temporal brain areas, possibly indicating the retrieval of memory representations, top-down processes during retrieval, and mental rotation during post-retrieval monitoring. Interestingly, the ERP old/new difference between images with incorrect orientation judgments and new images in the 440-to-600 ms time window, showed associations to right prefrontal and postcentral gyri. These regions were also found for the ERP/fMRI relationships of the FN400. This correspondence between ERP/fMRI correlations of old/new effects for old[incorrect] minus new images in the 260–440 ms and 440–600 ms time window might indicate that this old/new effect in the 440–600 ms time window is associated with familiarity-based rather than recollection-based recognition processes (Rugg & Curran, 2007). This suggestion receives support from the very similar topography of the old/new effects for old[incorrect] items in the time windows of the FN400 (260–400 ms) and parietal old/new effect (440–600 ms) (Fig. 4).

Late frontal old/new effects were prominent over left frontal channel groups in the ERP (Fig. 3) and associated with activation clusters in the left prefrontal gyrus, confirming previous postulations of the role of the prefrontal cortex in post-retrieval monitoring (Cruse & Wilding, 2009; Rugg et al., 2002). In addition, late frontal old/new effects were also related to activation in the angular and supramarginal gyri. Together with previous studies (Nelson et al., 2010; Sestieri, Corbetta, Romani, & Shulman, 2011), this finding suggests an involvement of the parietal cortex during post-retrieval monitoring.

Many of the observed relationships were negative, for example, more positive amplitudes for original than mirror images in the ERP between 440 and 600 ms were related to higher BOLD activation of mirror than original images. Such patterns of ERP and BOLD amplitudes are not uncommon and have been observed before in memory and other complex tasks (e.g., Bledowski et al., 2006; Lawson, Liu, Joseph, Vagnini, Kelly, & Jiang, in press). The exact reason for these findings is still not entirely clear. The polarity of ERP effects could be influenced from two different sources. More positive amplitudes for original than mirror images between 440 and 600 ms (Fig. 3) could either reflect stronger activation of a positive-going source for original pictures or it could reflect stronger activation of a negative-going source for mirror images. It is also possible that the temporal integration of information in EEG and fMRI led to these results because ERPs represent brain activation in circumscribed time windows, whereas fMRI models activation over a longer period of time. It is conceivable that ERPs are thus more sensitive to rapid, transient activity, whereas the fMRI response may be dominated by reentrant and more sustained activity introduced through task demands and top-down processes (Brem, Halder, Bucher, Summers, Martin, & Brandeis, 2009). This specificity of measuring brain activation could lead to the observation of negative ERP/fMRI correlations when the same brain area shows transient (measured by ERPs) as well as sustained activity (measured in fMRI). In addition, it could lead to the observation of dissimilarities between ERP source localization and ERP/fMRI correlations (further discussed in the next paragraph) when the same experimental pattern occurs in spatially different brain regions that are activated in either a transient or sustained manner.

For the recollection difference between original and mirror images between 440 and 600 ms, we obtained source generators with sLORETA that showed some correspondence to the ERP/fMRI correlations (see Fig. 12). This suggests that for this effect ERP/fMRI correlations are possibly due to common generators for activations in ERP and fMRI. Although this contrast showed the best correspondence, the compatibilities were rather general such that both fMRI and sLORETA implicated predominantly posterior regions, but appeared more medial and inferior in sLORETA than in fMRI. In this contrast, as in the other contrasts that did not show any correspondences, differences between source localization and ERP/fMRI correlations were found. This might be due to different sensitivities of ERP and fMRI to certain aspects of the measured brain activation like an open versus closed spatial layout of the ERP source, canceled versus intact phase-resetting contributions to the ERP, or different effects of the activity of inhibitory interneurons on EEG and fMRI (see Bledowski et al., 2006, for a detailed discussion). In addition, LORETA models the current source density (CSD) transformation, which represents only electrical fields generated near the scalp surface. LORETA thus considers only the cortical surface and is insensitive to deep sources as found in the fMRI. Measuring different physiological processes thought to be related to neural activity, LORETA and fMRI could lead to dissimilar finding which do not necessarily represent erroneous calculations by either technique (Corrigan et al., 2009).

General Discussion

The present study investigated the temporal and spatial brain processes of long-term memory. Familiarity and recollection of pictures together with their studied orientation information were objectively and sequentially measured with ERPs and fMRI in the same subjects to acquire the spatial and temporal dynamics of the neural processes mediating long-term memory.

Familiarity and recollection showed distinct patterns of brain activation in both ERPs and fMRI, as expected by dual-process models of recognition memory (Jacoby, 1991; Mandler, 1980; see Yonelinas, 2002). The study also shows that although familiarity and recollection are associated with activation in separate brain regions, they also share a number of brain areas, which may reflect common processes or may suggest independent processes in nearby networks (Dörfel et al., 2009; Wheeler & Buckner, 2004). The present study provides novel evidence by showing that familiarity and recollection did not only yield distinct patterns of activation in ERPs and fMRI, but that these activation patterns showed distinct ERP/fMRI relationships, which are in line with previously reported, between-experiment comparisons (Rugg & Curran, 2007; Vilberg & Rugg, 2007, 2009).

Recording brain activation with ERPs and fMRI in the same participants made it possible to further explain the temporal patterns of recognition memory processes and their possible underlying neural sources. The present results showed that ERP familiarity processes between 240 and 440 ms temporally preceded recollection processes and were structurally associated with prefrontal brain regions. Recollection processes were most prominent between 440 and 600 ms and correlated with activation in temporal, parietal, and occipital regions. Post-retrieval monitoring, which occurred in the ERP between 600 and 1000 ms as a long-lasting slow-wave over frontal channel groups, showed correlations with activation in the prefrontal and parietal cortex.

Figure 6.

Positive (red) and negative (blue) contrasts of group activation maps (Z > 3.29, FWE α < 0.05, cluster size > 50) for the contrast indicating the recollection of original images (original[correct] > old[incorrect]).

Table 4.

Significant regions (corrected Z > 3.29, FWE α < 0.05, cluster size > 50) for the contrast indicating the recollection of original images (original[correct] > old[incorrect]). Negative contrasts are shaded. Clusters that represent unique effects for the recollection but not the familiarity contrast (Tab. 3) are shown in red.

MNI region (L;R)	BA	cluster size (# of voxels)	Peak Z-value	MNI coordinates
Anterior cingulate gyrus (L)	32	67	3.81	0,6,30
Calcarine fissure (L;R)	23	277;165	3.99;3.81	−2,−62,18;2,−62,18
Cerebellar vermis		91	3.85	6,−58,−4
Cerebelum (R)		62	3.84	8,−58,−6
Cerebelum (R)		76	3.77	10,−60,−12
Cuneus (L;R)	17	162;128	4.20;3.69	−12,−56,24;16,−94,16
Gyrus rectus (R)	11	126	3.71	4,56,−14
Hippocampus (L;R)		106;66	3.84;3.59	−36,−14,−12;26,−4,−26
Inferior frontal gyrus, opercular part (R)	44	59	−4	54,22,28
Inferior frontal gyrus, triangular part (R)	45	114	−3.95	54,22,26
Inferior temporal gyrus (L;R)	20	93;335	3.76;4.20	−56,−2,−28;54,−10,−24
Insula (L)	13	63	−3.96	−32,20,−8
Insula (L;R)	13	16;172	3.67;4.01	−36,−8,−8;38,2,6
Lingual gyrus (L;R)	18/19	84;217	3.64;3.77	0,−62,6;10,−60,−6
Median cingulate gyrus (L;R)	23	622;728	4.31;4.01	−12,−52,32;12,−34,42
Middle frontal gyrus, orbital part (L;R)	10	303;165	3.93;3.69	0,66,−6;4,56,−12
Middle occipital gyrus (L)	19	74	4.15	−46,−70,14
Middle temporal gyrus (L;R)	21	484;729	4.26;4.37	−46,−68,12;64,−48,18
Parahippocampal gyrus (R)	27	103	3.93	22,−6,−30
Posterior cingulate gyrus (L;R)	23	264;28	4.45;3.78	−8,−52,32;12,−38,30
Precuneus (L;R)	7	930;333	4.44;3.87	−6,−52,30;12,−38,42
Putamen (R)		111	4.05	34,4,6
Rolandic operculum (R)	43	125	3.92	66,−4,12
Superior frontal gyrus, medial orbital (L)	10	100	3.66	−2,68,2
Superior occipital gyrus (L)	17/18	63	3.93	−14,−92,16
Superior parietal gyrus (R)	19	105	3.66	24,−52,68
Superior temporal gyrus (L;R)	41/42	111;1112	3.80;4.37	−52,−32,12;62,−46,18
Supplementary motor area (R)	6	128	3.74	6,−14,62
Supramarginal gyrus (R)	40	194	4.01	56,−30,24
Temporal pole, superior temporal gyrus (R)	28	54	3.68	58,4,−4