Event-related Potential Signatures of Relational Memory

Abstract

Various lines of evidence suggest that memory for the relations among arbitrarily paired items acquired prior to testing can influence early processing of a probe stimulus. The event-related potential experiment reported here was designed to explore how early in time memory for a previously established face-scene relationship begins to influence processing of faces, under sequential presentation conditions in which a preview of the scene can promote expectancies about the to-be-presented face. Prior to the current work, the earliest component documented to be sensitive to memory for the relations among arbitrarily paired items was the late positive complex (LPC), but here relational memory effects were evident as early as 270-350 msec after face onset. The latency of these relational memory effects suggests that they may be the precursor to similar effects observed in eye movement behavior. As expected, LPC amplitude was also affected by memory for face-scene relationships, and N400 amplitude reflected some combination of memory for items and memory for the relations among items.

INTRODUCTION

Contextual cues contribute to our ability to identify familiar people. Imagine, for instance, that you spot a flyer on campus that is advertising the upcoming women’s gymnastic season. The girl on the balance beam looks familiar (she is a student in one of your classes), but you are unable to determine where you might have seen her in the past. If the flyer had instead been promoting academic enrichment at the University and her photo had been superimposed on the lecture hall where you teach your class, recognition would likely have been facilitated. Intuitions like these have been borne out in behavioral research, in which the reinstatement of a scenic background context has been found to influence the speed and accuracy with which face recognition decisions are made (Rainis, 2001; Beales & Parkin, 1984; Memon & Bruce, 1983; Davies & Milne, 1982; Klee, Leseaux, Malai, & Tiberghien, 1982). The work reported here attempts to take this further by examining the time course of neural activity associated with face processing in context, using event-related brain potentials (ERPs). The current research takes advantage of the millisecond-level temporal resolution of ERPs to determine when—and, by inference, at what level of processing—reinstatement of a studied facescene relationship is first appreciated by the brain.

The time course of memory for relationships among arbitrarily associated items has been explored previously using response-signal behavioral paradigms (e.g., Dosher & Rosedale, 1989; Gronlund & Ratcliff, 1989), eye movement behavior (Hannula, Ryan, & Cohen, submitted), and ERPs (Tsivilis, Otten, & Rugg, 2001; Donaldson & Rugg, 1998, 1999; Weyerts, Tendolkar, Smid, & Heinze, 1997). Most of this work has been conducted with arbitrary pairs of items (e.g., words, pictures), in which, during the test block, pairings are either maintained with respect to previous study exposure (i.e., repeated), rearranged (i.e., re-paired), or novel. Memory for the relations among items is assessed via differences, on some measure of performance or activity, between repeated and re-paired test stimuli. Behavioral research indicates that performance dependent upon memory for the pairings requires as much as 200 msec more processing time than performance dependent upon memory for the items themselves (Gronlund & Ratcliff, 1989).

ERP studies have shown that differential brain activity to repeated pairs, as compared with activity elicited by re-pairs and novel pairs, begins approximately 600 msec after stimulus onset. This “parietal old/new effect” is maximal over centro-posterior scalp sites and takes the form of a positive shift to repeated pairs (Donaldson & Rugg, 1998, 1999; Weyerts et al., 1997, but see Tsivilis et al., 2001). It has been proposed, and intracranial recordings have provided data to support (e.g., Dietl et al., 2005; Trautner et al., 2004), that this late positive complex, hereafter referred to as the LPC, reflects processing in medial temporal lobe structures (i.e., hippocampus).¹ To date, no experiment has provided evidence for ERP effects earlier than the LPC that distinguish between maintained and rearranged pairs of items.

Modulation of earlier ERP components has been reported only with respect to memory for the items themselves. For instance, in a study in which objects were superimposed on scenes, an early (∼100 msec) frontopolar effect distinguished trials containing both a new object and a new scene from those with an old object, an old scene, or both (Tsivilis et al., 2001). In addition, modulation of a midfrontal component, significant 300-350 msec after stimulus onset (sometimes called the FN400), distinguished repeated and re-paired object-scene pairings (i.e., both the object and the scene were studied, whether or not they were studied in the same pairing) from pairs containing any novel information. Similar midfrontal “old/new effects” have been observed in several other studies, and this component has been argued by some to reflect familiarity- or item-based recognition (see Friedman & Johnson, 2000). Such effects are believed by others (e.g., Yovel & Paller, 2004) to be modulations of the well-known N400 component (discussed below), and thus, to reflect a form of “conceptual priming,” which seems to be intact in amnesic patients (Levy, Stark, & Squire, 2004; Olichney et al., 2000).

Eye movement studies (Hannula et al., submitted) also reveal effects associated with memory for relations among items and suggest that relational memory phenomena can occur quite early in processing. In that work, we showed that memory for face-scene relations is evident in patterns of viewing elicited by three-face displays when one of the faces “matches” (had previously been studied with) the scene on which the faces are superimposed; this effect takes the form of preferential viewing of the matching face, from among three faces that were all previously viewed equally often. This preferential viewing effect was evident 1-1.5 sec prior to explicit identification of the match, just 500-750 msec after the faces were presented, and occurred whether or not memory for the face-scene pairs was tested directly. Thus, memory for the face-scene relationships was expressed spontaneously in the pattern of eye movements elicited by the test display. In a variant of this paradigm, we have found that the ability to distinguish the matching face behaviorally from among other equally familiar faces is impaired in patients with amnesia following hippocampal damage (Hannula, Tranel, & Cohen, 2006), reflecting that what we have argued is the central role of the hippocampus in relational memory (Hannula et al., submitted; Cohen, 2003; Ryan & Cohen, 2003; Eichenbaum & Cohen, 2001; Ryan, Althoff, Whitlow, & Cohen, 2000; Cohen, Poldrack, & Eichenbaum, 1997; Cohen & Eichenbaum, 1993).

The time course of the eye movement effect is of particular importance for the work to be presented here. The emergence of disproportionate viewing, reaching significance within the period 500-750 msec after face presentation, is very rapid considering that subjects cannot predict where in the three-face display the matching face will be located, and that, in a typical trial, the 500-750 msec period is only enough time to permit two or three fixations. Accordingly, this provides a very conservative estimate of the time course of retrieval, identification, and use of memory for face-scene relations. This eye movement effect (which takes some time to emerge) is evident right about the time—or even before—similar effects are just beginning in the ERP waveform (the LPC, described above).

That (for neurologically intact participants) disproportionate viewing of the matching face emerged so early in time was no doubt influenced by the fact that each test trial presented a previously viewed scene for 3 sec prior to the presentation of the superimposed three-face display. Presentation of the scenic context with which a face has been bound in memory likely promoted pattern completion, in which exposure to a subset of items from an earlier study event causes reactivation of memory for the entire original event (Eichenbaum & Cohen, 2001; O’Reilly & Rudy, 2001; Cohen & Eichenbaum, 1993; Halgren, 1984; Squire, Cohen, & Nadel, 1984). The reactivation of the relevant face-scene representation upon presentation of the scene would be expected to exert a top-down influence on the processing of the matching face in the three-face test display.

Top-down influences have been suggested as part of well-known models of face processing (e.g., Bruce & Young, 1986). For example, Bruce and Young (1986) propose that recognition of a familiar face is accomplished by an interactive system of functional components. At the lowest level of the system, structural codes provide a description of a face consistent with its perceptual attributes. This information is then compared with stored structural descriptions of known faces at the level of face recognition units. Finally, when a match is obtained, specific information about that individual is made available via person identity nodes. Under normal circumstances, information proceeds through the system serially, but it is suggested that the face recognition units may be activated in advance of person identity nodes by other information with which the face shares some past relationship (explaining, perhaps, why a picture of your student in the lecture hall might prompt better recognition than a picture of the same student on the balance beam). This model of face processing, and the proposed influence of context reinstatement on face identification, was developed to account for the recognition of well-known faces. However, it is suggested that “for less familiar, and less public figures, their identity is even more bound up in the contexts with which they are associated” (p. 322), a proposal that seems consistent with the results of several context reinstatement experiments (Rainis, 2001; Beales & Parkin, 1984; Memon & Bruce, 1983; Davies & Milne, 1982; Klee et al., 1982). It follows then that the presentation of a scenic context may influence early aspects of face processing, even when the faces are pre-experimentally unfamiliar, as in our eye movement studies (Hannula et al., submitted). The preview of a scene previously associated with a face prior to presentation of the three-face test display seems to lead obligatorily to accessing perceptual information about the “matching” face, via pattern completion, and, in the course of using that retrieved information, driving the eyes to the matching face soon after it is presented.

The Current Study

The current study follows up the work of Hannula et al. (submitted), further examining the processing of facescene relationships. As in our previous work, and unlike other ERP studies of memory for relations among items, the paradigm involves sequential rather than simultaneous presentation. Here, the scene is presented, followed by a matching, a familiar but mismatching, or a novel face.² The results of interest concern the pattern of activity elicited by the face as a function of whether it matches the preceding scene. Based on the previous ERP studies, we expect changes in the LPC distinguishing between faces that match versus faces that do not match the preceding scene. However, of particular interest here is activity earlier in time after face presentation that may be indicative of top-down effects exerted by the 3-sec scene preview on subsequent processing of the superimposed face.

The earliest documented component that shows specific sensitivity to faces and that may conceivably be modulated by the scene context is the N170. But several reports have indicated that although this component distinguishes faces from other complex objects (Eimer, 2000a, 2000b; but see Schweinberger, Huddy, & Burton, 2004), it is insensitive to repetition or fame (Bentin & Deouell, 2000; Eimer, 2000a, 2000b), and thus, seems unlikely to be affected by the presence (or absence) of a match in the current experiment. Another early component that might possibly be modulated by face-scene matches is the N250r, a component sensitive to face repetition and thought to reflect the activation of perceptual representations that can support face recognition (Joyce & Kutas, 2005; Schweinberger et al., 2004; Trenner, Schweinberger, Jentzsch, & Sommer, 2004; Pfutze, Sommer, & Schweinberger, 2002; Schweinberger, Pickering, Jentzsch, Burton, & Kaufmann, 2002; Begleiter, Porjesz, & Wang, 1995; Schweinberger, Pfutze, & Sommer, 1995). However, this component is much earlier than any previously found to show sensitivity to contextual or relational effects.

A more likely candidate, perhaps, is the N400, a negative-going ERP component peaking approximately 400 msec after stimulus onset that can be elicited by faces and that is quite sensitive to context effects. In the current paradigm, expectations about the face that is to follow the scene, on the basis of pattern completion and reactivation of memory for face-scene relations, may be expected to show the same kind of context effects to which the N400 has long been shown to be sensitive in the realm of language processing. In a hallmark experiment, Kutas and Hillyard (1980) demonstrated the influence of sentential context on the N400 elicited by words: words rendered predictable by being placed in a congruent sentence (e.g., “He was stung by a bee”) elicited N400 responses of smaller amplitude than words that were anomalous in their context (e.g., “He spread the warm bread with socks”). Since then, N400 modulations have been documented in a number of experiments that employ a host of stimulus materials and task conditions (for a review, see Kutas & Federmeier, 2000). Perhaps closest to the materials and paradigm to be used here, an N400-like component has been shown to be sensitive to manipulations of object-scene congruity (Ganis & Kutas, 2003). In that experiment, designed to investigate at what level a scene exerts its effect on object processing, participants were exposed to items that were either semantically consistent with a scene (e.g., a toaster in the kitchen) or not (e.g., a toaster in the bathroom). The magnitude of a negative-going waveform that peaked around 390 msec after an object was introduced into the scene was found to vary as a function of congruity, and was interpreted as a reflection of the activation of previously established semantic associations.

The N400, or a component like it (i.e., the FN400 described earlier), has been shown in memory experiments to be sensitive to a variety of manipulations of item familiarity (see Friedman & Johnson, 2000), even for amnesic patients (Duzel, Vargha-Khadem, Heinze, & Mishkin, 2001; Olichney et al., 2000); faces have been shown to elicit such item familiarity effects, especially if the items are processed for meaning (Huddy, Schweinberger, Jentzsch, & Burton, 2003; Mnatsakanian & Tarkka, 2003). In language and object recognition experiments, the N400 has also proven sensitive to well-established, meaning-based relationships between a presented item and a prior linguistic (see Kutas & Federmeier, 2000) or scenic context (Ganis & Kutas, 2003). It is not yet clear, however, whether this component might also be sensitive to relations among meaningful items (e.g., faces, scenes) that are pre-experimentally unfamiliar.

Taken all together, various lines of evidence suggest that relational information acquired prior to probe testing can influence early processing of the probe stimulus. In the present work, we examine memory for relations between arbitrarily paired, previously unfamiliar faces and scenes, in a paradigm involving sequential, rather than simultaneous, presentation of the paired (or mis-paired) items. We ask: when in time—and how—does memory for a previously established facescene relationship begin to influence processing of a face presented several seconds after the matching (or nonmatching) scene?

METHODS

Participants

Sixteen college-aged students (6 women, 10 men) from the University of Illinois participated in exchange for course credit or payment. All were right-handed (as assessed with the Edinburgh inventory; Oldfield, 1971) with no history of neurological/psychiatric disorders; none had immediate family members who were left-handed.

Apparatus

The electroencephalogram (EEG) was recorded from 26 geodesically arranged electrodes embedded in an elastic cap, referenced on-line to the left mastoid. The 26 sites include midline prefrontal (MiPf), left and right medial prefrontal (LMPf and RMPf), left and right lateral prefrontal (LLPf and RLPf), left and right medial frontal (LMFr and RMFr), left and right mediolateral frontal (LDFr and RDFr), left and right lateral frontal (LLFr and RLFr), midline central (MiCe), left and right medial central (LMCe and RMCe), left and right mediolateral central (LDCe and RDCe), midline parietal (MiPa), left and right mediolateral parietal (LDPa and RDPa), left and right lateral temporal (LLTe and RLTe), midline occipital (MiOc), left and right medial occipital (LMOc and RMOc), and left and right lateral occipital (LLOc and RLOc); the arrangement can be seen in the head schematic at the top right of Figure 2. Blinks and eye movements were monitored via electrodes positioned on the outer canthus of each eye and the infraorbital ridge of the left eye. Electrode impedances were kept below 10 kΩ. EEG was amplified with a Sensorium polygraph set at a bandpass of 0.02-100 Hz, and was continuously digitized at 250 Hz.

Stimuli and Design

The stimuli were full-color face images of 96 men and 96 women selected from our faces database (see Althoff & Cohen, 1999), and 144 full-color images of thematically and perceptually distinct real-world scenes (e.g., a mountain scene, a cityscape, a bridge) obtained from Brand X Pictures. Each face was sized to 240 × 240 pixels; scenes was measured at 700 × 525 pixels.

Faces were superimposed on scenes to create 144 face-scene pairs (see Figure 1, top). Each pair was presented in one of four study blocks; thus 36 unique pairs were presented in random order in each block. In each of four corresponding test blocks, 36 displays were shown, each consisting of a single face superimposed on a previously viewed scene. Three different display types were used in each test block: there were 12 repeat (or match), 12 re-pair, and 12 novel displays. Collapsed across all four test blocks, there were 48 trials in each experimental condition. In each match display, the face had been studied with (and thus matched) the scene with which it was paired. In each re-pair display, the face had been studied in the prior block, but had not been previously presented with the scene with which it was now paired. In each novel display, the face had not been seen before and was presented with a previously viewed scene (see Figure 1, bottom).

Figure 1.

Top: Examples of face-scene pairs presented in the study blocks. Bottom: Examples of face-scene pairs presented in the test blocks. In each match display, the face was studied previously and had been studied with (and thus matched) that scene. In each re-pair display, the face was studied previously but had not been studied with that scene. In each novel display, the face had not been seen before.

Faces and scenes were randomly assigned to lists and were counterbalanced such that each list of faces was seen equally often with each list of scenes; lists were rotated across experimental blocks and experimental conditions.

Procedure

After providing informed consent, participants were seated in a quiet room, 40 inches from the computer monitor. Prior to the first study block, participants were told that a set of face-scene pairs would be presented on the monitor and that they should attempt to commit each pair to memory so that in a subsequent test block they could distinguish between intact pairs (i.e., match displays) and new pairs (i.e., re-pair and novel displays). It was emphasized that blinks, eye movements, and muscle tension were to be minimized while the stimuli were being presented. Written instructions were provided on the computer monitor prior to each study block to remind the subjects that they were to study the faces and scenes. Participants were asked to press a button when they were ready to begin the experiment.

Each study trial began with the presentation a scenic image for 3000 msec; thereafter, a single face was superimposed on top of the scene and the resulting facescene pair remained on the screen for 5000 msec. A 2500-msec delay was interposed between subsequent trials. Thirty-six unique face-scene pairs were presented once each in the study block; the corresponding test block followed directly afterwards. Prior to the test block, subjects were told that face-scene pairs would be presented on the monitor and that they were to use a button-press response to indicate whether each pair was intact (i.e., match displays) or new (i.e., re-pair and novel displays). It was emphasized that “new” responses were to be made on trials in which the face and the scene were both old, but had not been seen together previously, and on trials in which the face was novel, and thus, had not been presented with the scene at study. Subjects were to respond “yes” if the pair was intact and “no” if the pair was new; the hand used for each response was counterbalanced across participants. When they were ready, participants initiated the test block by pressing a button.

Each test trial began with the presentation of a previously viewed scene for 3000 msec. A face was then superimposed on the scene and the pair remained on the screen until a response was made by the participant. As in the study blocks, a 2500-msec delay was interposed between subsequent trials. All together, four study-test block sequences were completed. Subjects took breaks between blocks as necessary and were fully debriefed after the fourth, and final, test block.

ERP Data Analysis

ERP data were re-referenced off-line to the algebraic mean of the signal at the left and right mastoids. A blink correction procedure (Dale, 1994) was applied where appropriate to minimize the number of trials rejected on the basis of blink artifact. Trials that could not be blink corrected and those contaminated by eye movements, excessive muscle activity, or amplifier blocking were eliminated from the data before averaging. In total, 6.57% of the test trials were eliminated from further analysis. For each test trial to which the participant had responded with a correct button press, ERPs were computed for epochs extending from 100 msec prior to face onset until 920 msec after face onset. Artifact-free average ERPs were then created for each of the three experimental conditions, after subtracting the 100-msec preface baseline.

Based on prior reports of the components of interest and on the observed peak amplitudes, ERP data were binned into five time windows that allowed us to examine the effects of face-scene match on the processing of faces superimposed upon previously viewed scenes. Mean amplitudes were measured in each of the following time windows and condition effects were assessed by way of repeated measures omnibus analysis of variance (ANOVA) across all electrode sites: 140-190 msec (N170); 240-270 msec (N250r); 270-350 msec (“P320”); 350-500 msec (N400); and 600-900 (LPC). Where indicated, the distribution of effects was analyzed across 16 representative scalp sites using a repeated measures omnibus ANOVA with three levels of condition, two levels of hemisphere (right hemisphere and left hemisphere scalp sites), two levels of laterality (lateral and medial scalp sites), and four levels of anteriority (prefrontal, frontal, central, and occipital scalp sites). For each ANOVA, the Huynh-Feldt adjustment to the degrees of freedom (df) was applied to correct for violations of sphericity associated with repeated measures. Accordingly, for all F tests with more than 1 df in the numerator, both the corrected p value and the Huynh-Feldt epsilon value $\left(\acute{\epsilon }\right)$ will be reported.

RESULTS

Behavioral Responses

There were significant differences in response accuracy across the three display types, F(2,30) = 7.99, p = .002, ε = .92, with poorer accuracy for re-pair displays (M = 77.66% correct; SD = 9.06; range = 52.08-100% correct) than for match [M = 85.14% correct; SD = 10.02; range = 64.58-95.83% correct; t(15) = 2.72, p = .05] and novel displays [M = 88.41% correct; SD = 12.17; range = 62.5-97.92% correct; t(15) = 4.87, p < .05]; accuracy was not significantly different for match and novel displays, t(15) = 1.01, p > .05 There was also a significant difference in mean response time across the three display types, F(2, 30) = 9.78, p < .001, $\acute{\epsilon }$ = .92, with correct responses to re-pair displays (M = 1922.83; SD = 521.62; range = 1031.62-2967.20 msec) requiring more time than correct responses to match [M = 1599.34; SD = 370.11; range = 1077.74-2345.52 msec; t(15) = 3.51, p = .01] and novel displays [M = 1529.19; SD = 431.33; range = 912.61-2778.74 msec; t(15) = 5.19, p < .001], respectively; response times for match and novel displays were not statistically different, t(15) = .62, p > .05.

Event-related Potentials

Figure 2 shows, at all electrode sites, the grand-average ERP waveforms to faces presented in the context of match, re-pair, and novel displays (when responded to with a correct button press). Figure 3 shows a close-up of these ERPs at a representative sample of three electrode sites, with critical effects labeled.

Figure 2.

Grand-average (n = 16) ERP waveforms for the three conditions shown at all 26 electrode sites. Negative is plotted up. The layout of the electrode sites approximates their position on the head, as illustrated in the schematic shown at top right.

Figure 3.

Effects of condition, shown at left mediolateral central, medial central, and right mediolateral central electrode sites. The earliest significant effect of condition was seen on a positivity immediately following the N250r; this “early relational” (“P320”) effect differentiated match displays (eliciting a more positive response) from re-pair and novel displays. Effects were also found on the N400 (which was graded in amplitude with condition, with novel eliciting the most negative and match the least negative response) and on the LPC (which was enhanced for match, relative to re-pair or novel displays).

140-190 msec (N170) and 240-270 msec (N250r)

Visual inspection of the waveforms (Figure 2) reveals a peak approximately 170 msec after face onset (i.e., N170), which appears to be maximal over right occipital scalp sites; the amplitude of this component was not influenced by memory for faces or face-scene pairings, F(2,30) = 1.06, p = .36.³

Similarly, visual inspection of the waveforms (Figures 2 and 3) reveals a peak approximately 250 msec after face onset, and although there is a hint of separation between displays containing novel faces (“new”) and those containing repeated, or previously viewed, faces (“match” and “re-pair”), these differences were small and not significant, F(2,30) = 0.63, p = .54.

270-350 msec (“P320”)

Visual inspection of the waveforms suggested an effect of condition onsetting immediately after the N250r, that is, on the following positivity (which we label here the “P320” on the basis of its polarity and peak latency, or the “early relational component,” on the basis of its sensitivity to manipulations of relations among items). Analyses revealed a main effect of condition from 270-350 msec after face presentation, F(2,30) = 6.94, p = .003, $\acute{\epsilon }$ = .95. The main effect was due to faces presented in the context of match displays showing more positivegoing activity in this time window than faces presented in the context of re-pair, F(1,15) = 5.70, p = .03, or novel displays, F(1,15) = 12.89, p = .003; activity elicited by faces in re-pair and novel displays was not significantly different, F(1,15) = 1.04, p > .05. The selectivity of the effect for match trials reflects memory for the face-scene relationship established during the study block. The effect has a fairly broad, bilateral distribution, with the differences between experimental conditions largest over medial fronto-central electrode sites [interactions of Condition × Laterality and Condition × Anteriority: F(2,30) 7.56, p = .002, $\acute{\epsilon }$ = 1.0, and F(6,90) = 3.06, p = .04, $\acute{\epsilon }$ = .45, respectively].

350-500 msec (N400)

Consistent with our expectations, there was a main effect of condition from 350-500 msec after face presentation, F(2,30) = 9.26, p < .001, $\acute{\epsilon }$ = 1.00. Planned comparisons revealed that this effect was graded with respect to the experimental conditions: match displays elicited the smallest amplitude N400, re-pair displays were intermediate, and novel displays elicited the largest amplitude N400 [match vs. re-pair: F(1, 15) = 4.92, p < .05; re-pair vs. novel: F(1,15) = 5.24, p = .04]. This effect was also widely distributed but was most prominent over medial electrode sites [Condition × Laterality interaction: F(2,30) = 5.70, p = .0082, $\acute{\epsilon }$ = .98].

600-900 msec (LPC)

Consistent with previous studies, there was significant modulation of the ERP waveform from 600-900 msec as a function of experimental condition, F(2,30) = 5.26, p = .01, $\acute{\epsilon }$ = .93. Planned comparisons revealed that the LPC amplitude was greatest for match displays [match vs. re-pair: F(1, 15), 4.67, p < .05; match vs. novel: F(1, 15) = 7.63, p = .01], with no significant difference between re-pair and novel displays, F(1,15) = 1.12, p > .05, a pattern similar to that seen for the P320. This late positive effect was bilaterally distributed and most prominent over medio-central scalp sites [Condition Laterality × Anteriority interaction: F(6,90) = 3.22, p = .01, $\acute{\epsilon }$ = .91].

Summary

Sensitivity to relational information was seen in three time windows of the ERP: 270-350 msec (“P320”), 350-500 msec (N400), and 600-900 msec (LPC). In all time windows, match displays elicited more positive responses than did either novel or re-pair displays. In the N400 time window, an additional difference was observed between re-pair and novel displays (more positive responses to re-pair displays); in both of the other two time windows, the effect was selective to relational matches, with responses to re-pair and novel displays not statistically different. The three effects were distinguishable not only by their functional sensitivity (i.e., relational-only vs. graded) and time course but also by their scalp distributions. A comparison of the distribution of the match effect (difference between match and novel displays) across the three time windows revealed an interaction of time window with anteriority [F(6,90) = 5.11, p = .01, $\acute{\epsilon }$ = .32], as well as an interaction of time window, laterality, and anteriority [F(6,90) = 4.39, p = .0036, $\acute{\epsilon }$ = .66]. Pairwise comparisons revealed that, consistent with visual impressions, the “P320” effect is more frontally distributed than the N400 effect [Time window × Anteriority: F(3,45) = 4.74, p = .01, $\acute{\epsilon }$ = .83], and the LPC effect is more posterior than the N400 effect [a skew that is most apparent over medial electrode sites: Time window × Anteriority, F(3,45) = 3.78, p = .04, $\acute{\epsilon }$ = .50; Time window × Laterality × Anteriority, F(3,45) = 8.75, p = .001, $\acute{\epsilon }$ = .67].

DISCUSSION

ERPs elicited by novel faces and by repeated faces that either matched (had previously been studied with) or did not match the scene that just preceded it were examined in five time windows and were analyzed for effects of item (face) memory and of relational (facescene) memory. Effects of memory for face-scene relationships were evident, either selectively or in combination with effects of memory for faces, in three time windows, starting with the period 270-350 msec after face presentation.⁵ Mean amplitude of the ERP wave-forms elicited by match displays were distinct from those elicited by both re-pair and novel displays (which did not differ), thus showing relational memory specificity, both 270-350 msec (P320) and 600-900 msec (LPC) after face onset. In one other time window, from 350-500 msec (N400) after face onset, mean amplitude was graded with respect to previous viewing history, with match and novel displays eliciting the smallest and largest N400s, respectively. Neither of the components specifically associated with face processing (the posterior N170 and the N250r) showed sensitivity to the manipulations here. Each of these findings will be discussed in turn.

Relational Memory Effects

Effects of memory for face-scene relations were seen in two different ERP components in which the ERP to match displays was significantly different from re-pair and novel displays, which, in turn, did not differ from one another.

270-350 msec (P320)

Memory for face-scene relationships was first expressed in the modulation of a broad, medio-fronto-centrally maximal component with a peak amplitude approximately 320 msec after face onset. From 270-350 msec, ERPs to faces seen in a reinstated context (match displays) were more positive-going than ERPs to either studied faces paired with a different (though familiar) context (re-pair displays) or faces that were new (novel displays). This early positive component is thus a relational memory effect. Emerging well in advance of the LPC, which had previously been shown to be sensitive to memory for the relations among arbitrarily paired items (Donaldson & Rugg, 1998, 1999; Weyerts et al., 1997), the effect we document here is, to our knowledge, the earliest reported ERP modulation distinguishing matching pairs from re-pairings of studied items. Positivity in a similar time window has been reported for faces (presented in isolation) that had been studied along with a brief biographical description, as compared with equally familiar faces that had been studied alone (Paller, Bozic, Ranganath, Grabowecky, & Yamada, 1999); it is possible that this difference, too, reflects memory for the previously learned relationship between the face and the biographical information.

The P320 effect reported here occurs early enough after the presentation of the face to potentially be the precursor to the eye movement effect described earlier (Hannula, et al., submitted), in which preferential viewing of the matching face, from among three faces viewed equally often, was evident by 500-750 msec after the faces were presented.⁶ In that experiment, rapid expression of relational memory in eye movement behavior appears to have been a direct consequence of the opportunity for pattern completion. In a different condition in that experiment, when the three-face test display and the scene on which it was superimposed appeared simultaneously, preferential viewing of the matching face developed more slowly (1500-1750 msec after the faces were presented). Similarly, the use in the current experiment of a 3-sec preview of the scene prior to presentation of the face, rather than the simultaneous presentation conditions used in previous work on the LPC (Donaldson & Rugg, 1998, 1999; Weyerts et al., 1997), clearly provides the opportunity for pattern completion in advance of presentation of the face display, and is likely the reason that we see the effects of relational memory so early in the ERP waveform, emerging well in advance of the LPC.

600-900 msec (LPC)

Relational memory effects were also seen here on the LPC, consistent with the results of several past experiments (Donaldson & Rugg, 1998, 1999; Weyerts et al., 1997). Modulations of LPC amplitude in this study reflected memory for the relationships between arbitrarily paired items (face-scene pairings), and were not simple effects of memory for the items (faces) themselves.

The LPC has been argued by some to be a neural correlate of conscious recollection (e.g., Allan, Wilding, & Rugg, 1998; Paller & Kutas, 1992), involving memory for an item in relation to a particular study experience, and depends critically upon the integrity of medial temporal lobe structures (e.g., Duzel et al, 2001; Olichney et al., 2000). However, in the current study, the LPC effect, together with the P320 effect, may be less related to conscious recollection per se than to memory for face-scene pairings. It is important to note that the ERPs recorded here were those elicited to the appearance of the faces, occurring 3 sec after the onset of the (matching or nonmatching) scene. That the LPC to the face was largest when it matched the scene is difficult to explain by appealing only to conscious recollection. Presentation of scenes presumably elicits conscious recollection of the relevant study experiences at that point in time, seconds in advance of presentation of the face, and indeed, that recollective process is likely part of the pattern completion and reactivation process, discussed earlier, that shapes processing of the subsequently presented face. However, the ERP effects recorded here do not derive from that recollective epoch. The question then is why the onset of the matching face would elicit more recollective activity than the subsequent appearance of a nonmatching face. Intuitively, it seems that additional recollective activity would be much more likely in response to the appearance of a nonmatching face, as that would elicit recollection of a different study experience. But that is not what we observed. Accordingly, rather than appealing to conscious recollection for an explanation of the LPC effect here, we suggest that the increased LPC amplitude for match trials, like the increased P320, reflects activity related to processing the relationship between the face and the already reactivated face-scene representation. Taken together with our other findings of impairment on a variant of this task in patients with hippocampal amnesia (Hannula et al., 2006), the explanation of the relational memory effects seen here is consistent with the relational memory account of hippocampal function (Eichenbaum & Cohen, 2001; Cohen & Eichenbaum, 1993).

In attributing “relational” effects on the LPC and the P320 to aspects of the retrieval, identification, and use of memory for face-scene relations, we do not mean to distinguish relational memory from episodic memory. To the extent that episodic memory refers to the temporal-spatial context of the remembered information, that is, to some event that includes the remembered item, it is an example of relational memory. Often, when an individual participating in a recognition memory task remembers that a given test item (e.g., a specific face) was previously studied, it is an example both of episodic memory and relational memory—s/he remembers the relationship between the item and the study episode. When all the study items are pre-experimentally unfamiliar (novel), however, as in this study, correct responding does not require relational memory for the item-episode relationship; the subject need not recollect or gain explicit access to memory for the study episode. However, correct responding to questions about which face was associated with which scene does require relational memory, whether of face-scene relations or of the study episode more generally (i.e., the relations among the scene, the face, and the study episode). Accordingly, although work with the LPC has typically addressed episodic memory more generally, the use here of pre-experimentally novel materials permits us to discern effects on both the LPC and the P320 of memory for the relations between items (face-scene relations) distinct from any effects of memory for the items themselves.

Mixed Relational and Item Memory Effects

A different, graded pattern of effects was also observed in this experiment, in which re-pair trials elicited responses intermediate between match and novel displays. This pattern of results could be due to some combination of relational memory and item memory effects.

350-500 msec (N400)

The N400 is known to be sensitive to a variety of manipulations of item familiarity; systematic reductions in N400 amplitude as a function of repeated exposure to previously viewed items have been reported in several experiments (see Friedman & Johnson, 2000). Closest to the materials used here, repetition of both famous and pre-experimentally unfamiliar faces is associated with decreased N400 amplitude over frontal scalp sites (Joyce & Kutas, 2005; Eimer, 2000a). Because the magnitude of the N400 is larger for famous than for unfamiliar faces (Bentin & Deouell, 2000; Eimer, 2000a), it has been argued that this component reflects processing which supports the activation of semantic representationsbound to well-known faces, perhaps at the level of “person identity nodes” in Bruce and Young’s (1986) model of face processing (Bentin & Deouell, 2000). Further, it has been argued that repetition-related reductions in N400 amplitude are a consequence of facilitated access to these semantic representations for faces viewed earlier in the experiment (Eimer, 2000a). This interpretation is compelling when the faces are bound to rich semantic representations (i.e., famous faces), but is less convincing when the faces are pre-experimentally unfamiliar, and when study exposures are few in number and of limited duration. In this case, reliance on what are necessarily self-generated semantic labels, in service of later memory, is less likely (Joyce & Kutas, 2005). It is possible therefore that N400 amplitude is sensitive both to memory for, or the repetition of, single items as well as for the semantic or name information that might be associated with those items.

In the current experiment, displays that contained novel faces were associated with the largest amplitude N400, with repeated faces viewed out of context (“repair”) and those viewed in context (“match”) eliciting N400s of increasingly reduced amplitude. A simple effect of repetition (or item memory) would have been expected to elicit N400 reductions of equal magnitude for match and re-pair displays, irrespective of the relational manipulation, as each of these displays consist of a repeated scene along with a repeated face. The absence of such an effect in this experiment might reflect the fact that memory for face-scene relationships was emphasized both during encoding and at test, or might reflect the opportunity for pattern completion.

Across studies, several different patterns of sensitivity to item and relational information have been observed on the N400. Besson and Kutas (1993) compared responses to words repeated within the same sentence context, repeated within a different sentence, or viewed for the first time in the experiment. Words repeated within the same context elicited reduced N400s relative to words presented for the first time, whereas repetition of words within different sentences did not (even when the meaning of the word was preserved across repetition). Thus, the N400 seemed sensitive only to relational information. However, none of the words were preexperimentally unfamiliar to the participants, whereas the novel faces in the current experiment were, in fact, being encountered for the first time and even the repeated faces had only ever been seen once. It is possible, therefore, that the differences between the patterns of results in these two studies reflect a greater sensitivity of the N400 to item familiarity as such, as opposed to item recency.

Still another pattern of effects was found in work exploring the neural correlates of memory for arbitrarily paired objects and scenes. N400 amplitudes were reduced to repeated object-scene combinations regardless of whether the original pairing was maintained. Critically though, in that experiment subjects were to distinguish between old objects and new objects; the object-scene relationship was incidental to performance on the memory test (Tsivilis et al., 2001). A possible interpretation is that relational memory effects are not obligatorily (or automatically) expressed in N400 amplitude; alternatively, it is possible that relational memory effects would be evident in N400 amplitude, even in the absence of explicit instructions to identify intact pairs, if expectancies were established on the basis of pattern completion, as was the case in our experiment using preview conditions. Indeed, relational effects described previously for the N400 in language and object recognition experiments (Ganis & Kutas, 2003; Kutas & Federmeier, 2000) always involved expectancies established on the basis of a currently active semantic context (sentence or scene), which influenced the processing of a subsequently presented target item. In the current study, expectancies, on the basis of successful pattern completion, may have made processing less effortful for match displays, resulting in reduced N400 amplitude, reflecting memory for face-scene relationships acquired in a single study exposure.

Item Memory Effects

Effects of item memory alone would be inferred from differences in the ERP to novel faces versus repeated faces, regardless of whether the repeated faces matched or did not match the scene. There were no significant effects of this type on any of the components measured here, including the N250r.

240-270 msec (N250r)

Based on past work, it seemed reasonable to expect that we might see modulation of the N250r, which is the earliest, consistently observed component sensitive to the repetition of (and presumably, then, some kind of memory for) faces (Joyce & Kutas, 2005; Schweinberger et al., 1995, 2002, 2004; Trenner et al., 2004; Pfutze et al., 2002; Begleiter et al., 1995). The N250r tends to be maximal over right lateral inferior-temporal scalp sites, perhaps originating in the fusiform gyrus, and is associated with more negative-going waveforms to repeated faces than to faces being presented for the first time (see Schweinberger & Burton, 2003). Some reports have suggested that N250r modulation is relatively transient— not seen at repetition latencies that exceed 15 min (Schweinberger et al., 2002) and reduced in magnitude when two to four faces intervene between subsequent face presentations (Pfutze et al., 2002). However, recent work provides evidence for robust modulation of this component at delays as long as one hour, with a single study exposure to pre-experimentally unfamiliar faces (Joyce & Kutas, 2005).

Visual inspection of the waveforms from the current study (Figures 2 and 3) reveals a peak approximately 250 msec after face onset, and there is a hint of separation between displays containing novel faces (“new”) and those containing repeated, or previously viewed, faces (“match” and “re-pair”); however, these differences were small and not significant. The absence of a statistically significant N250r modulation in our experiment may reflect the fact that we emphasized memory for face-scene relationships, rather than memory for the faces themselves, as there is some suggestion in the literature that the nature of the task affects the magnitude of the observed differences between repeated and novel faces (Trenner et al., 2005).

Relational Memory, Amnesia, and the Brain

The current paradigm comes out of our previous work, in which we have argued that the hippocampus is critically involved in long-term memory for the relations among items (Hannula et al., submitted; Cohen, 2003; Ryan & Cohen, 2003; Eichenbaum & Cohen, 2001; Ryan et al., 2000; Cohen et al., 1997; Cohen & Eichenbaum, 1993). Neuropsychological research conducted with amnesic patients has shown that the hippocampus is critical for remembering the relationships among items in a scene (Hannula et al., 2006; Ryan et al., 2000) as well as relations among arbitrarily paired items (Hannula et al., 2006; Turriziani, Fadda, Caltagirone, & Carlesimo 2004; Giovanello, Verfaellie, & Keane, 2003; Gabrieli, Cohen, & Corkin, 1988). Even when memory is tested indirectly, via eye movement monitoring, in tasks that do not call upon or depend upon explicit remembering or conscious recollection, patients with hippocampal amnesia fail to show effects of relational memory that are expressed robustly in the patterns of viewing of neurologically intact participants (Ryan & Cohen, 2004; Ryan et al., 2000). In light of other findings from our lab using a variant of the current paradigm in which patients with hippocampal amnesia are impaired at distinguishing the matching face from nonmatching faces, it seems likely that the P320 and LPC modulations seen here as a function of memory for face-scene relationships are related to intact hippocampal functioning and would be absent in amnesia.

Less clear is how amnesia might affect the expression of memory in modulations of the N400 component, which was graded here across the different conditions. Previous work has shown that sensitivity of the N400 to manipulations of item repetition is intact in amnesia (Duzel et al., 2001; Olinchey et al, 2000). It has been proposed that medial temporal lobe regions adjacent to the hippocampus, the parahippocampal region, including the perirhinal and entorhinal cortex, may support memory for single items (Eichenbaum & Cohen, 2001; Eichenbaum, Otto, & Cohen, 1994), and a rapidly growing body of research has implicated the perirhinal cortex in this form of memory (e.g., Pihlajamaki et al., 2004; Ranganath et al., 2003; Davachi & Wagner, 2002; Wan, Aggleton, & Brown, 1999). It has also been shown that N400 effects sensitive to semantic congruity, involving derivable linguistic relationships and thought to reflect conceptual priming, are intact in amnesia (Olichney et al., 2000). This result is consistent with the relational memory theory, which proposes that the hippocampus is critically involved representing arbitrary (nonderivable) relationships among perceptually distinct items (e.g., Cohen & Eichenbaum, 1993). Accordingly, in the current paradigm, we might expect some modulation of the N400 component in patients with damage limited to the hippocampus, but not the full graded pattern seen in neurologically intact subjects. Thus, we might expect no difference in N400 amplitude for repeated faces as a function of whether they are presented in the context of match displays versus re-pair displays, as this contrast depends critically upon memory for arbitrary interitem relationships; but all repeated faces might show reduced N400 amplitude relative to displays containing novel faces, reflecting memory for (or repetition of) faces and scenes. Such an outcome would help to explain the graded pattern of N400 results observed in the current work.

General Conclusions

In this study, our goal was to use the temporal resolution of ERPs to examine how early in time relational information can affect stimulus processing, under conditions that permit pattern completion and memory reactivation. We found that under these conditions, using pre-experimentally novel materials, relational memory effects can be seen within the first 300 msec after stimulus presentation—some 200-300 msec before such effects have been found to emerge when paired items are presented simultaneously. In addition, we found that sensitivity to memory for relations emerged at several different time points (and, by inference, processing stages), and both with and without associated sensitivity to memory for the items themselves. We hypothesize that such relational memory effects require an intact hippocampal system, which supports not only encoding in memory of the relations among items but also the ability to retrieve or reactivate that memory in conditions that permit pattern completion, and to use it to guide subsequent processing and performance.