Prestimulus subsequent memory effects for auditory and visual events

Abstract

It has been assumed that the effective encoding of information into memory primarily depends on neural activity elicited when an event is initially encountered. Recently, it has been shown that memory formation also relies on neural activity just before an event. The precise role of such activity in memory is currently unknown. Here, we address whether pre-stimulus activity affects the encoding of auditory and visual events, is set up on a trial-by-trial basis, and varies as a function of the type of recognition judgment an item later receives. Electrical brain activity was recorded from the scalps of 24 healthy young adults while they made semantic judgments on randomly intermixed series of visual and auditory words. Each word was preceded by a cue signalling the modality of the upcoming word. Auditory words were preceded by auditory cues and visual words by visual cues. A recognition memory test with remember/know judgments followed after a delay of about 45 min. As observed previously, a negative-going, frontally-distributed modulation just before visual word onset predicted later recollection of the word. Crucially, the same effect was found for auditory words and observed on stay as well as switch trials. These findings emphasize the flexibility and general role of pre-stimulus activity in memory formation, and support a functional interpretation of the activity in terms of semantic preparation. At least with an unpredictable trial sequence, the activity is set up anew on each trial.

INTRODUCTION

We continuously encounter new experiences in life. Some of these will be remembered, but others will be forgotten. Since the late 1970s, research has been carried out to investigate the brain mechanisms that underlie effective memory formation. Using different measures of brain activity in both healthy and diseased humans, studies have shown that activity elicited by an event when it is initially encountered can predict whether the event will be remembered or forgotten in a later memory test (for reviews see Paller & Wagner, 2002; Rugg, Otten, & Henson, 2002; Friedman & Johnson, 2000; Wagner, Koutstaal, & Schacter, 1999). This work has generated important insights into the way in which the brain supports the creation of a new memory. It does, however, make the assumption that, in addition to retrieval-related processes, the neural and cognitive processes set in train when an event is initially encountered are the main reasons why some events are remembered but others are forgotten.

This assumption has been challenged by the recent demonstration of brain activity that can predict whether an event will later be remembered before the event has even occurred. Using a technique with high temporal resolution (scalp-recorded electrical brain activity), Otten and colleagues (2006) showed that human brain activity elicited by a cue presented just before word onset can predict recollection of the word up to at least 45 min later. The existence of such brain activity indicates that for learning to be effective, the brain needs to be in the right state when new information is encountered. Thus, memory formation not only relies on brain activity elicited by an event, but also on anticipatory processes preceding it. This finding has significant implications. First, it requires the updating of theoretical accounts of encoding, which will need to consider activity before as well as after an event. Second, it opens up the possibility that some memory problems arise because pre-stimulus activity is not conducive to effective encoding. This may suggest ways in which people can be taught to learn better by preparing themselves appropriately before approaching new information. Third, it calls for a re-evaluation of findings obtained with imaging techniques that have relatively poor temporal resolution (e.g. functional magnetic resonance imaging, fMRI). These techniques do not easily allow the differentiation of neural processes that happen within a few hundred milliseconds of each other. Thus, effects seen with these methods may reflect processes before an event, processes thereafter, or some combination of the two.

The functional role of pre-stimulus activity in memory formation is currently unknown. Otten et al. (2006) investigated pre-stimulus activity across two independent experiments that both incorporated a condition in which a semantic judgment had to be made about a visually-presented word. In Experiment 1, visual cues were presented 2.7 s before the onset of visual words. The cues indicated whether a semantic (living/non-living) or non-semantic (alphabetic/non-alphabetic) decision had to be made about the upcoming word. In Experiment 2, visual cues were presented 1.5 s before either visual or auditory words. All words required a judgment about the relative size of the words’ referents, with the cue signalling the input modality of the upcoming word. Electrical brain activity elicited in the cue-word interval predicted later recollection of words requiring living/non-living judgments in Experiment 1, and of visually-presented words in Experiment 2. In both cases, this activity was largest over frontal scalp sites shortly before word onset. No pre-stimulus effects were observed for alphabetic decision in Experiment 1 and auditory words in Experiment 2.

Pre-stimulus influences on memory formation presumably reflect the degree to which encoding-related processes are prepared ahead of stimulus presentation. An alternative interpretation is that the activity signals overall levels of attention or alertness, which vary across trials. It is well-known that attention can affect both pre-stimulus activity (Thut, Nietzel, Brandt, & Pascual-Leone, 2006; Linkenkaer-Hansen et al., 2004; Supèr, van der Togt, Spekreijse, & Lamme, 2003; Driver & Frith, 2000) and encoding success (e.g. Craik, Govoni, Naveh-Benjamin, & Anderson, 1996). However, if some words are remembered better only because people were generally more alert on those trials during encoding, it would be expected that response times on those trials are reduced and that those words give rise to enhanced early modality-specific brain activity (which is attention-sensitive, e.g. Mangun & Hillyard, 1995). This was not observed in either of the two previous experiments (Otten et al., 2006). In addition, overall levels of attention would be expected to vary randomly across trials. In contrast, the encoding-related pre-stimulus effect differs systematically across conditions. This argues against a simple attentional interpretation of the effect.

Pre-stimulus effects can also not reflect the mere anticipation of an upcoming event, as knowing when an event occurs is not sufficient to elicit the effects (cf. the alphabetic condition in Experiment 1 and auditory condition in Experiment 2 of Otten et al., 2006). Instead, pre-stimulus activity may reflect the degree to which semantic processes can be prepared ahead of an event. This notion is based on the observation that the two conditions that showed pre-stimulus influences on encoding both involved, at least in part, a decision about a word’s meaning. Adopting an appropriate semantic task set may allow the creation of a representation that is semantically more elaborate and therefore easier to retrieve (Craik & Lockhart, 1972). It may not be a coincidence that encoding-related pre-stimulus ERP activity resembles the slow frontal negativities associated with working memory control processes (e.g. Ruchkin, Grafman, Cameron, & Berndt, 2003).

A functional interpretation in terms of semantic preparation is limited by the fact that pre-stimulus activity has thus far only been found to affect the encoding of visual events (Adcock et al., 2006; Mackiewiecz, Sarinopoulos, Cleven, & Nitschke, 2006; Otten et al., 2006). If pre-stimulus effects are restricted to visual events, they may reflect a relatively low, perceptual level of processing. The lack of a reliable effect preceding semantic decisions on auditory words (Otten et al., 2006, Experiment 2) is especially puzzling in this respect. One explanation for the absence of an effect for auditory words is that the cues preceding these words were always presented visually. The requirement to switch modalities prior to auditory words may have prevented the preparation of processes relevant for effective encoding. Upon encountering a visual cue, attention may first have to be redirected from the visual to the auditory modality to enable optimal registration of the imminent auditory event (see Supplementary Fig. 5 in Otten et al., 2006, for supporting evidence). This process presumably takes some time, compromising the ability to mobilize processes that affect memory formation.

The primary aim of the current experiment was to establish whether pre-stimulus activity can affect the encoding of auditory as well as visual events. In addition, we wished to determine the time course of the activity. The demonstration that pre-stimulus activity varies rapidly enough to influence encoding on individual trials suggests that the activity can be set up within one trial (i.e. within the duration of the cue-word interval). However, pre-stimulus influences have been assessed in sequences in which semantic and non-semantic tasks, or visual and auditory information, were randomly intermixed (Otten et al., 2006). If a trial involves the same task/modality as the preceding trial (‘stay’ trials), encoding-related activity may be easier to engage because all processes can be maintained across two successive trials (cf. Herron & Wilding, 2004; Pashler, Johnston, & Ruthruff, 2001). If, on the other hand, the current trial involves a different task/modality than the preceding trial (‘switch’ trials), preparation of the upcoming decision may be more difficult as at least some of the processes have to be set up anew. Previous studies on pre-stimulus activity could not separate effects across stay and switch trials because of insufficient trial numbers.

Healthy young adults were asked to perform an incidental semantic encoding task on randomly intermixed visual and auditory words. 1.5 s before each word, a cue signalled the input modality of the upcoming event. In contrast to our previous study (Otten et al., 2006, Experiment 2), input modality was held constant between cue and word to eliminate any influence of modality switching. Auditory words were preceded by auditory cues and visual words by visual cues. A recognition memory test incorporating remember/know judgments (Tulving, 1985) followed after a delay of 45 min. Although remember/know judgments are not process pure (see e.g. Parks & Yonelinas, 2007), the remember/know paradigm has proved useful in many previous behavioural and neuroimaging studies to dissociate recognition accompanied by recollection versus a general sense of familiarity (e.g. Rugg & Yonelinas, 2003).

If pre-stimulus activity has a general modulatory role in encoding, electrical activity leading up to both auditory and visual events should predict later recognition. The type of pre-stimulus activity predicting memory formation in each case will reveal any influence of perceptual attributes and will, therefore, elucidate its functional role. Comparing effects across stay and switch trials allowed us to determine whether a task set relevant for encoding can be set up within one trial.

METHODS

The experimental procedures were kept as similar as possible to those employed in Experiment 2 of Otten et al. (2006), with one crucial exception. The cue preceding auditory words was not visual, but auditory, in nature.

Participants

The experiment was approved by the joint University College London and University College London Hospitals ethics committee. Twenty-four healthy adults, none of whom had participated in the experiments reported in Otten et al. (2006), were paid to take part. Mean of age was twenty-two years, and fourteen were women. Each volunteer gave written informed consent prior to participating, and reported to be right handed, native English speaking, and to be without neurological or psychiatric history. Color vision was checked with the abridged Ishihara plates. The data from a further six volunteers were not considered as they either forgot too few words to compute subsequent memory effects (see EEG Acquisition and Analysis below), or had eye movements that could not be corrected.

Stimulus Materials

Study and test sequences were constructed from a pool of 398 concrete nouns with a length of 3-12 letters and a written frequency of 0-500 occurrences per million (Kučera & Francis, 1967). Each word was available in written form (white uppercase Helvetica script, 500 ms duration, about 0.7° vertically and 1 – 4.5° horizontally) and spoken form (adult male voice, mean duration 650 ms, range 310 – 1130 ms). Visual cues consisted of a small image of a red square (250 ms duration, 0.7° by 1°), and auditory cues of a 500 Hz pure tone (250 ms duration).

Six sets of 60 words each were selected at random from the stimulus pool with the constraint that the distribution of word lengths was comparable across sets. The sets were rotated across subjects to create different study and test lists so that each word appeared as either old or new, and as either auditory or visual. For each subject, four sets were used to create a study list of 240 words (120 visual and 120 auditory), with the remaining two sets added to create a test list of 360 words (240 old and 120 new, half of each visual and half auditory). The words were randomly allocated to each list, and the lists were divided into blocks of 60 trials. Two filler words were added to the beginning of each block, and short rest breaks were given in between blocks. The remaining 18 words were used to create practice lists.

Tasks and Procedure

The experiment involved an incidental study phase, followed by a surprise recognition memory test. Volunteers were first prepared for the recording of electrical brain activity (see EEG Recording and Analysis below), after which the study task was explained to them. They were asked to make judgments about a series of words, half of which would be seen on a computer monitor and half heard via headphones. The later memory test was not mentioned. Visual and auditory stimuli were randomly intermixed, and a cue presented before each word indicated the upcoming stimulus modality. Visual stimuli were always preceded by a visual cue, and auditory stimuli by an auditory cue. Volunteers were instructed to create a mental image of the object denoted by each word, and decide whether the height of the object, as viewed by them, was greater than its width. The experimenter encouraged using the cue to get ready for the upcoming decision. Responses were given via a key press with the index fingers of the left and right hands (counterbalanced across subjects). Both speed and accuracy were stressed. A short practice block familiarized volunteers with the task. They then performed four experimental task blocks, which lasted about 25 min.

Instructions for the test phase were given approximately 40 minutes after the end of the last study block. Volunteers were informed that they would again be making decisions about words delivered visually and preceded by a red square, or auditorily and preceded by a tone. As before, visual and auditory words were randomly intermixed. All studied words were presented again, along with words not experienced during the study phase. Stimulus modality was held constant between study and test phases. Thus, words presented visually at study were presented visually at test, and likewise for auditory words. For each word, volunteers had to make a ‘remember’, ‘know’, or ‘new’ judgment (Tulving, 1985). They were asked to press one key if they remembered seeing or hearing the word during the study phase and recollected something specific about their initial encounter with the item, another key if they simply had a feeling that the word had been presented, and a third if they thought that the word was new. Responses were given with the index and middle fingers of one hand, and the index finger of the other hand. New responses were always assigned to the single response finger (responding hand counterbalanced across subjects), with the other index and middle fingers assigned to remember and know judgments (responding finger counterbalanced across subjects). A short practice block was followed by six experimental blocks. The test phase lasted about 35 min. Mean delay between the end of the study phase and the start of the first experimental test block was 45 min. At the end of the experiment, electrodes were removed and volunteers debriefed.

At both study and test, visual words were displayed one at a time in the centre of a computer monitor against a black background. Auditory stimuli were presented binaurally at a comfortable hearing level via headphones. Cues were presented 1.5 s before word onset. A fixation point (a plus sign) was present in the centre of the screen throughout the task except when a visual cue or visual word was presented. The inter-trial interval (time between successive cue onsets) varied between 3.5 and 6 s (mean 4.5) during the study phase, and between 4 and 5.5 s (mean 4.75) during the test phase.

EEG Acquisition and Analysis

EEG was recorded from 31 scalp sites using sintered silver/silver-chloride electrodes embedded in an elasticated cap according to an equidistant electrode montage (montage 10; www.easycap.de/easycap/e/electrodes/13_M10.htm). Vertical and horizontal eye movements were recorded bipolarly from electrodes placed at the supra- and infraorbital ridges of the right eye and at the outer canthus of each eye. Site 8 (equivalent to Fz in the 10/20 system) was used as the online reference, and an electrode placed just anterior to Fz served as ground. Signals were amplified and band-pass filtered between 0.01 and 35 Hz (Contact Precision amplifier; 3dB roll-off), and digitized at a rate of 100 Hz (12-bit resolution).

The data were digitally filtered between 0.05 and 15 Hz (96 dB roll-off, zero phase shift filter) to remove low- and high-frequency noise, and algebraically re-referenced to linked mastoids. Epochs of 2560 ms duration surrounding events of interest were extracted from the continuous EEG record obtained during the study phase. These epochs were used to create event-related potentials (ERPs), separately for activity elicited by the cues and activity elicited by the words. Cue-related ERPs were referred to a 100 ms period before cue onset, and word-related ERPs to a 100 ms period before word onset. ERP waveforms were created for each electrode site and stimulus category by averaging epochs separately for study items given ‘remember’, ‘know’, and ‘new’ judgments in the subsequent recognition test. Blink artefacts were minimized by estimating and correcting their contribution to the ERP waveforms via a standard regression technique (Rugg, Mark, Gilchrist, & Roberts, 1997). Trials on which horizontal or non-blink vertical movements occurred were excluded from the averaging process, as were trials containing EEG drifts (± 50 μV) or analog-to-digital saturation. Trials with study responses faster than 200 ms or no response were also excluded from analysis. Accuracy of study judgments was not considered as the imagery task does not have right or wrong answers. ERPs were based on a minimum of 15 artifact-free trials.

Waveforms were quantified by measuring mean amplitudes across selected latency regions. These values were submitted to repeated measures analyses of variance (ANOVAs), incorporating the Greenhouse-Geisser correction for violations of sphericity when appropriate (Keselman & Rogan, 1980; see Results for further details). Greenhouse-Geisser corrections were also employed for the analyses of the performance data.

RESULTS

Task Performance

Study

The imagery task was performed with an average response time of 1431 (s.d. 281) ms when words were presented visually. Not surprisingly given the more gradual perceptual input, judgments took longer for auditorily-presented words (average of 1656 ms, s.d. 266; Student’s t₂₃ = 10.59, P < 0.001). To evaluate whether the time to respond to an item at study was related to later memory performance, response times were compared across words given a ‘remember’, ‘know’, or ‘new’ judgment in the subsequent recognition memory test. These response times did not differ for either visual (F_{1.8, 40.5} = 0.95, P = 0.386) or auditory (F_{1.4, 31.4} = 0.77, P = 0.427) words. When ‘know’ and ‘new’ judgments were collapsed (mimicking the EEG analyses below), there was a tendency for response times to be longer when items were later given a ‘remember’ judgment. However, these differences did again not reach statistical significance (1440 vs. 1405 ms for visual words, t₂₃ = 1.65, P = 0.113; 1676 vs. 1639 ms for auditory words, t₂₃ = 2.04, P = 0.053). Accuracy of study performance was not considered as the taller/wider judgment does not have an objective correct answer.

Test

Memory performance is shown in Table 1. Other than overall slower response times for auditory words, performance did not differ between input modalities. Performance in the recognition test was characterized with measures of recognition accuracy (Pr) and response bias (Br), using the two-high threshold model (Snodgrass & Corwin, 1988). Pr refers to the difference between the proportions of hits and false alarms, and Br to the proportion of false alarms divided by (1-Pr). Pr and Br were computed separately for ‘remember’ and ‘know’ judgments. For ‘remember’ responses, recognition accuracy was 0.57 for visual and auditory words (paired-samples t₂₃ = −0.02, P = 0.983). Corresponding response bias was 0.11 and 0.13, respectively (t₂₃ = −1.00, P = 0.288). For ‘know’ responses, the ability to discriminate between old and new items was 0.15 for visual and 0.14 for auditory words (t₂₃ = 0.45, P = 0.658). Response bias was 0.13 in both cases (t₂₃ = −0.39, P = 0.698). Recognition accuracy was reliably above zero for all recognition judgments (one-sample t₂₃ > 3.51, P < 0.002).

Table 1.

Recognition memory performance.

	Recognition judgment
Word type	Remember	Know	New
Proportion of responses
Old
Visual modality	0.62 (0.17)	0.26 (0.17)	0.13 (0.07)
Auditory modality	0.62 (0.16)	0.25 (0.17)	0.12 (0.07)
New
Visual modality	0.04 (0.05)	0.11 (0.07)	0.85 (0.10)
Auditory modality	0.05 (0.04)	0.12 (0.08)	0.83 (0.11)
Mean response time (ms)
Old
Visual modality	1197 (179)	1711 (450)	1461 (275)
Auditory modality	1421 (176)	1955 (425)	1775 (323)
New
Visual modality	1551* (704)	1824* (448)	1289 (205)
Auditory modality	1670* (528)	2040* (448)	1562 (201)

Note. Values are across-subject means (s.d.). n = 24.

^*Some volunteers did not make any remember or know judgments to new words. The response times shown here are based on 15, 20, and 23 volunteers for remember judgments on new visual words, remember judgments on new auditory words, and know judgments on new words respectively.

EEG Data

Encoding-Related Activity Preceding Visual and Auditory Words

Neural correlates of memory formation were assessed with the subsequent memory approach (Sanquist, Rohrbaugh, Syndulko, & Lindsley, 1980; for reviews see Paller & Wagner, 2002; Rugg et al., 2002; Friedman & Johnson, 2000; Wagner et al., 1999). This procedure contrasts neural activity elicited by study items according to later memory performance to the items, thus allowing the separation of activity relevant for successful encoding from other ongoing processes. In keeping with our previous studies on pre-stimulus activity (Otten et al., 2006), the primary interest was in activity supporting the encoding of information into episodic memory. Accordingly, study words were classified as recollected if they were later given a ‘remember’ recognition judgment. To maximize the signal-to-noise ratio of neural activity associated with failed episodic encoding, words were classified as not recollected if they were given a ‘know’ recognition judgment or were misclassified as new.

The group averaged ERP waveforms elicited by cues preceding visual and auditory words that were later recollected or not recollected are shown for a representative frontal electrode site in Fig. 1 (whole-head data are available from the authors upon request). For visual words, the data show a nearly identical pattern to that observed in Experiment 2 of Otten et al. (2006). As found previously, waveforms diverged prior to word onset depending on later memory performance. This difference took the form of a more negative-going waveform preceding words that were later recollected, an effect that was largest over the front of the head and reversed polarity at more posterior scalp sites. The difference emerged halfway into the cue-word interval, lasted for about 500 ms, and dissipated just before word onset. This time course deviates from that observed earlier, where the effect built up gradually during the cue-word interval, reaching its maximum just before word onset.

Figure 1.

Neural activity that precedes memory formation of visual and auditory events. Group averaged ERP waveforms elicited by pre-stimulus cues at a representative frontal electrode site (site 50 from montage 10, www.easycap.de/easycap/e/electrodes/13_M10.htm; equivalent to site Fp1 of the 10/20 system). Positive values are plotted upwards. (a) ERPs elicited by visual cues signalling a semantic decision about an upcoming visually-presented word, overlaid according to whether the word was recollected (given a ‘remember’ judgment) or not recollected (given a ‘know’ or ‘new’ judgment) in the subsequent recognition memory test. The ERPs differ reliably before word onset according to later memory performance. (b) Two-dimensional voltage spline map showing the distribution of the subsequent memory effect (difference between recollected and not recollected words in the 750-1250 ms interval after cue onset) across the scalp for visual events. The map is range-scaled. The pre-stimulus effect exhibits a negative-going, frontally-maximum distribution. (c-d) As (a-b), but data for auditory cues signalling a semantic decision about an auditorily-presented word. Auditory events exhibit the same pre-stimulus effect as visual events.

Crucially, auditory words also exhibited a subsequent memory effect prior to stimulus onset. Although the overall waveforms elicited by visual and auditory inputs of course differed, in both cases a significant difference related to encoding success occurred before word onset. As for visual words, waveforms preceding auditory words that were later recollected were more negative-going than those preceding auditory words not later recollected. This effect had the same frontal scalp distribution as observed for visual words, and also showed a similar mid-interval time course.

These pre-stimulus subsequent memory effects were quantified by measuring mean amplitudes at all electrode sites during the period when the effects were largest, 750-1250 ms interval after cue onset. For visual as well as auditory words, an ANOVA confirmed a significant interaction between subsequent memory and electrode site (F_3.3,75.6 = 4.29, P = 0.006 and F_4.2,96.7 = 3.11, P = 0.017 respectively). An across-modality analysis also showed this interaction effect (F_3.4,79.1 = 5.70, P = 0.001), and indicated that it did not differ in either amplitude (F_1,23 = 0.22, P = 0.646) or scalp distribution (F_4.5,102.9 = 1.10, P = 0.364) across visual and auditory words. Follow-up analyses contrasting the amplitudes of the pre-stimulus waveforms at the left frontal site illustrated in Fig. 1 showed a significant subsequent memory main effect for visual as well as auditory words (F_1,23 = 8.24, P = 0.009 and F_1,23 = 5.90, P = 0.023 respectively), which again did not differ reliably (F_1,23 = 0.10, P = 0.757).

We also analyzed consecutive 100-ms regions to elucidate the development of the subsequent memory effect over time. Table 2 presents the results from across-modality ANOVAs on the mean amplitudes in the ten consecutive 100-ms intervals leading up to word onset. The six intervals encompassing 700-1300 ms after cue onset showed a significant subsequent memory by electrode site interaction, which for four intervals remained significant after Bonferroni correction. The same six intervals showed a significant subsequent memory main effect when the analyses were restricted to the frontal electrode site where the pre-stimulus effect was largest (shown in Fig. 1, site 50 from montage 10, www.easycap.de/easycap/e/electrodes/13_M10.htm; equivalent to Fp1 in the 10/20 system). One of these intervals survived Bonferroni correction. None of the analyses demonstrated a significant interaction between subsequent memory and input modality (Fs between 0.01 and 1.97, Ps between 0.175 and 0.906). These analyses corroborate the results from the 750-1250 ms interval, and demonstrate that the apparently longer-lasting effect in the auditory condition was not statistically significant.

Table 2.

Greenhouse-Geisser corrected statistics for across-modality analyses of variance on consecutive 100-ms intervals covering the 500-1500 ms period after cue onset.

	Subsequent Memory * Site interaction			Subsequent Memory main effect (site 50)
Interval after cue onset (ms)	df	F	P	df	F	P
500-600	2.5,56.5	0.97	0.400	1,23	1.63	0.216
600-700	3.2,74.2	1.72	0.168	1,23	2.08	0.163
700-800	3.6,83.0	5.92	< 0.001*	1,23	5.68	0.026
800-900	3.2,73.0	6.07	0.001*	1,23	8.85	0.007
900-1000	3.0,69.6	3.35	0.024	1,23	8.51	0.008
1000-1100	2.8,65.4	3.73	0.017	1,23	8.37	0.008
1100-1200	4.1,94.1	5.31	0.001*	1,23	7.60	0.011
1200-1300	4.7,107.9	6.72	< 0.001*	1,23	11.10	0.003*
1300-1400	3.9,89.8	1.65	0.170	1,23	2.89	0.103
1400-1500	5.4,125.1	1.63	0.151	1,23	4.18	0.053

^*Note. = Significant after Bonferroni correction.

Pre-stimulus Activity and Stay/Switch Trials

As the pre-stimulus effects were indistinguishable across visual and auditory words, the opportunity arises to clarify the time course of pre-stimulus activity related to encoding. To that end, trials were collapsed across stimulus modality and separated according to whether the preceding trial contained a word in the same or a different input modality. This comparison has thus far not been possible, as in previous studies too few trials were available to compute separate subsequent memory effects for stay and switch trials (Adcock et al., 2006; Otten et al., 2006).

Fig. 2 illustrates the ERP waveforms elicited by cues on stay and switch trials preceding words that were later recollected and not recollected. Subsequent memory effects are visible before stimulus onset for either trial type. In both cases, words that are later recollected are preceded by a more negative-going waveform over frontal scalp sites. An ANOVA comparing effects on stay and switch trials on the mean amplitudes in the 750-1250 ms post-cue interval revealed a significant subsequent memory by electrode site interaction (F_3.3,76.5 = 5.96, P = 0.001). This effect did not differ in either amplitude (F_1,23 = 0.26, P = 0.611) or scalp distribution (F_5.2,118.8 = 0.86, P = 0.510) across stay and switch trials. Indeed, when tested separately on each trial type, significant subsequent memory by electrode sites interactions were found for stay (F_5.1,116.3 = 4.10, P = 0.002) as well as switch (F_3.2,74.2 = 3.83, P = 0.011) trials. Restricting the analyses to the left frontal electrode site where the effects were largest gave the same results. Significant subsequent memory effects were found for both trial types (F_1,23 = 10.19, P = .004 and F_1,23 = 4.82, P = .039 for stay and switch trials, respectively), with no reliable interaction between the two (F_1,23 = 0.00, P = 0.976).

Figure 2.

Neural activity that precedes memory formation on ‘stay’ and ‘switch’ trials. Group averaged ERP waveforms elicited by pre-stimulus cues at a representative frontal electrode site (site 50 from montage 10, www.easycap.de/easycap/e/electrodes/13_M10.htm; equivalent to site Fp1 of the 10/20 system). Positive values are plotted upwards. (a) ERPs elicited by cues on ‘stay’ trials, where the preceding trial contained events in the same input modality. ERPs are collapsed across visual and auditory events, and overlaid according to whether the word was recollected or not recollected in the subsequent recognition memory test. (b) Two-dimensional voltage spline map showing the distribution of the subsequent memory effect (difference between recollected and not recollected words in the 750-1250 ms interval after cue onset) across the scalp for events on ‘stay’ trials. The map is range-scaled. (c-d) As (a-b), but data elicited by cues on ‘switch’ trials, where the preceding trial contained events in the other input modality. Negative-going, frontally-distributed pre-stimulus subsequent memory effects were observed for ‘stay’ as well as ‘switch’ trials.

Pre-stimulus Activity and Types of Recognition Judgments

The identical subsequent memory effects preceding visual and auditory words also allow an initial investigation into the type(s) of memory processes supported by encoding-related activity before an event. Although the existence, separation, and measurement of different memory processes are highly controversial issues in the memory literature (e.g. Parks & Yonelinas, 2007), the opportunity to collapse across visual and auditory study trials enables a direct comparison between ERP waveforms elicited by words later given ‘remember’, ‘know’, or ‘new’ judgments. Fig. 3 shows the group averaged waveforms for those 18 of the 24 participants who gave sufficient numbers of ‘know’ and ‘new’ responses to consider these categories in isolation.

Figure 3.

Encoding-related pre-stimulus activity, separated as a function of the type of recognition judgment items later received. Group averaged ERP waveforms at a representative frontal electrode site (site 50 from montage 10, www.easycap.de/easycap/e/electrodes/13_M10.htm; equivalent to site Fp1 of the 10/20 system). Positive values are plotted upwards. The figure displays ERPs elicited by cues signalling a semantic decision about an upcoming word, collapsed across visual and auditory events. ERPs are overlaid according to whether words received a ‘remember’, ‘know’, or ‘new’ judgment in the subsequent recognition memory test. Only words that were given ‘remember’ judgments demonstrated statistically reliable ERP differences before word onset relative words later judged as ‘new’.

Pre-stimulus subsequent memory effects appear largest for items later given ‘remember’ judgments. A clear frontally-distributed negative-going modulation precedes these words, whereas only a small difference is visible prior to words later judged as ‘known’. These observations were confirmed by the statistical analyses. An across-item (remember/know/new) ANOVA on the mean amplitudes in the 750-1250 ms post-cue interval showed a significant recognition judgment by electrode site interaction (F_6.7,114.1 = 2.35, P = 0.030). Follow-up analyses comparing ‘remember’ and ‘new’ judgments demonstrated a significant subsequent memory by site interaction (F_4.6,77.7 = 3.86, P = 0.005). The comparison between ‘know’ and ‘new’ judgments did not reveal this interaction (F_5.4,91.0 = 1.33, P = 0.259) or a significant subsequent memory main effect (F_1,17 = 0.53, P = 0.473). Restricting the analyses to a more limited temporal interval (600-1000 ms, where the effect for know judgments appears largest) again did not bring out a statistically reliable difference (F_1,17 = 1.26, P = 0.278 for the main effect of subsequent memory and F_5.5,93.6 = 1.74, P = 0.126 for the interaction with electrode site). Only items that were later judged as remembered were preceded by a significant negative-going waveform at the left frontal scalp site shown in Fig. 3 (F_1,17 = 4.93, P = 0.040).

Importantly, the direct comparison between ‘remember’ and ‘know’ judgments also did not reveal a statistically reliable difference (all P > 0.085). Thus, although the waveforms preceding items later judged as ‘remembered’ versus ‘new’ differed reliably, items judged as ‘known’ were preceded by waveforms that were intermediate between the two. In a direct test of this idea, a linear trend analysis on the data from site 50 was significant (F_1,17 = 4.93, P = 0.040).

Encoding-Related Activity Elicited by Study Items

Finally, neural activity after word onset also differed according to subsequent memory performance (Fig. 4). Visual words that were later recollected elicited a more positive-going waveform over frontal scalp sites and a more negative-going waveform over posterior sites. Auditory words showed a more modest difference in that subsequently recollected words elicited a waveform that was more negative-going over some posterior scalp sites. The statistical reliability of these effects was evaluated in the 1200-2200 ms interval, chosen on the basis of the approximate onset of the effects visible in the group average and the continuous and long-lasting nature of subsequent memory effects in the literature (Paller & Wagner, 2002; Rugg et al., 2002; Friedman & Johnson, 2000; Wagner et al., 1999). The across-modality analysis showed a significant interaction between subsequent memory and electrode site (F_5.9,134.9 = 2.55, P = 0.024), not modulated by input modality (F_5.4,124.5 = 0.73, P = 0.608). However, separate analyses in each modality showed a reliable subsequent memory by site interaction only for visual words (F_7.5,173.5 = 2.32, P = 0.024). Auditory words did not show this interaction (F_4.1,94.4 = 1.10, P = 0.364), or a reliable subsequent memory main effect (F_1,23 = 2.73, P = 0.112).

Figure 4.

Encoding-related neural activity after word onset. Group averaged ERP waveforms elicited by the presentation of visual and auditory words at two representative frontal and occipital electrode sites (sites 36, 50, 42, and 44 from montage 10, www.easycap.de/easycap/e/electrodes/13_M10.htm). Positive values are plotted upwards. (a) ERPs elicited by visual words that were recollected or not recollected in the subsequent recognition memory task. (b) ERPs elicited by auditory words that were later recollected or not recollected.

Possible attention-related effects on the amplitudes of the exogenous components elicited by visual and auditory study words were evaluated by measuring their mean amplitudes in the 40-ms regions surrounding the time of maximum amplitude. For visual words, the P1 and N1 were measured between 70-110 and 130-170 ms respectively at the two occipital sites where these components are typically largest (sites 44 and 42 of montage 10; www.easycap.de/easycap/e/electrodes/13_M10.htm). For auditory words, the N1 and P2 were measured between 100-140 and 200-240 ms respectively at central site 1. None of these components showed reliable subsequent memory effects (F_1,23 = 0.17, P = 0.679 and F_1,23 = 2.31, P = 0.143 for visual P1 and N1; F_1,23 = 0.17, P = 0.683 and F_1,23 = 0.06, P = 0.815 for auditory N1 and P2).

DISCUSSION

The data replicate our earlier demonstration that neural activity before an event can influence memory formation (Otten et al., 2006). More importantly, the data provide four new findings that significantly enhance our understanding of anticipatory processes in encoding. First, the data show that pre-stimulus activity can influence the encoding of auditory as well as visual events. Second, pre-stimulus activity was found to affect encoding on switch as well as stay trials. Third, encoding-related pre-stimulus activity seems to depend on the degree of overlap between a cue and stimulus event. And, finally, pre-stimulus influences on encoding were more apparent for items later given ‘remember’ than ‘know’ judgments. Below, we discuss each finding in turn.

As found previously, electrical brain activity elicited by a cue presented just before a visual word predicted later recollection of the word (cf. Otten et al., 2006). Crucially, the same effect was observed here for auditory words. In both cases, a negative-going ERP modulation largest over frontal scalp sites predicted encoding success. This modulation started around 750 ms after cue onset and dissipated shortly before the word appeared. The lack of any discernible differences in time of occurrence, size, and scalp distribution across input modalities suggests that the physical differences between auditory and visual events do not affect the modulation. Thus, the modulation is unlikely to reflect the differential processing of early perceptual attributes. This adds confidence to the conclusion outlined in the Introduction that the modulation does not signal variations across trials in the degree of general alertness or readiness to attend to the upcoming information (cf. Otten et al., 2006). Also in the present experiment, study trials that were later recollected were not associated with faster response times or enhanced early modality-specific brain activity (Mangun & Hillyard, 1995).

The generalizability of the frontal negative pre-stimulus ERP effect across visual and auditory information suggests that anticipatory processes may play a wide role in memory formation. One possibility is that this type of pre-stimulus activity reflects the preparation of higher-level processes conducive to creating a durable representation in memory. These processes may involve semantic attributes, given that the pre-stimulus effect has so far only been found in tasks that involve the processing of a word’s meaning (Otten et al., 2006, Experiment 1). If the cue is used to mobilize semantic processing resources ahead of word presentation, the ensuing representation may be richer and easier to retrieve (Craik & Tulving, 1975). Alternatively, pre-stimulus activity may index motivation-, reward-, or emotion-related processes (Adcock et al., 2006; Mackiewicz et al., 2006).

Otten et al. (2006, Experiment 2) previously failed to find a pre-stimulus subsequent memory effect for auditory information. The task used in that experiment closely resembled that used here, with one crucial exception. For visual as well as auditory words, a visual pre-stimulus cue was used. A small image of an eye or ear signalled the input modality of the upcoming word. As explained earlier, the requirement to switch modalities prior to auditory words may have prevented the preparation of processes relevant for effective encoding because attention first needs to be redirected from the visual to the auditory modality to enable optimal registration of the imminent auditory event. This process presumably takes time, compromising the ability to mobilize processes that affect memory formation. When the cue and word are presented in the same input modality, there is no need to engage switching mechanisms, maximizing the opportunity to prepare encoding-related processes for both input modalities. Accordingly, neural activity preceding auditory as well as visual events predicted memory formation in the present experiment. In combination, the present and previous findings indicate that pre-stimulus subsequent memory effects depend on the degree of overlap between a pre-stimulus cue and stimulus event.

A remarkable feature of pre-stimulus activity is that it can influence encoding on a trial-by-trial basis. It can be engaged within 1.5 s (the cue-word interval used in the present experiment) and released quickly enough to be available again for the next trial. A caveat to this conclusion comes from the initial need to evaluate pre-stimulus activity by collapsing across two types of trial. In the first, the preceding trial contained events in the same input modality as those on the current trial (stay trials, e.g. visual-visual pairs). In the other, the preceding events were in the different input modality (switch trials, e.g. auditory-visual pairs). It could be argued that the observed effects only arose because of stay trials, where the same processes could be maintained across two consecutive trials. The more time there is to set up and maintain pre-stimulus activity, the stronger its potential influence on encoding. A direct comparison across stay and switch trials revealed that the effect is present on, and indistinguishable across, these two trial types. Thus, the frontal negative pre-stimulus activity indeed seems to be set up anew on each trial. The similarity of the effects on stay and switch trials also argues against the idea that the frontal negative ERP activity reflects task switching processes. In that case, larger activity should have been evident on switch trials. This does not preclude the possibility that encoding-related pre-stimulus effects only occur in situations where there is a requirement to engage in some kind of switching across trials. Without this requirement, it may be possible to maintain a task set or preparatory state across series of trials (cf. Otten, Henson, & Rugg, 2002).

The transient nature of pre-stimulus activity is surprising given that only one semantic task had to be performed during the study phase. If pre-stimulus activity reflects the mobilization of semantic resources, why would the activity not be engaged at the beginning of the study phase and maintained throughout? A likely explanation lies in the use of an intermixed, unpredictable trial sequence. Semantic decisions were made about visual and auditory events that were randomly allocated to each trial. Thus, until a cue appeared, it could never be predicted whether a decision would have to be made about a visual or an auditory event. In that situation, it may be more efficient to set up pre-stimulus activity on each trial. Perhaps some of the processes associated with maintaining such activity across trials are shared with those associated with switching between unpredictable events, or perhaps maintaining the activity over extended periods of time is effortful in the face of regular switches. If engaging pre-stimulus activity indeed depends on the particulars of the experimental sequence, it may be under voluntary control rather than a naturally occurring state (Meeter et al., 2004).

The idea that pre-stimulus influences on encoding are flexible is boosted by considering the timing of the effects across experiments. In the previous study (Otten et al., 2006, Experiment 2), the effect built up gradually in the course of the cue-word interval, reaching its maximum just before word onset. Here, the effect started earlier and had mostly dissipated by the time the word was encountered. These temporal differences suggest that the time at which encoding-related processes are initiated depends on the experimental parameters. The crucial parameter in this case is the type of cue used in the two studies. The cues used previously consisted of images of an eye or an ear; in the present experiment they were a tone or an image of a square. The discrimination between a small picture of an eye or an ear requires careful attention to perceptual detail. Discriminating between a square and a tone, on the other hand, can be done on the basis of minimal perceptual input. Indeed, the P200 (a component related to ease of perceptual discrimination; e.g. Lindholm & Koriath, 1985) was much larger in the current experiment (compare Fig. 1 with Fig. 1 in Otten et al., 2006, Experiment 2). When the cue is easier to identify, it is presumably possible to prepare task-related processes (including those related to encoding) earlier. This complements our earlier conclusion that pre-stimulus influences on encoding depend on the degree of overlap between the type of pre-stimulus cue and the type of stimulus event. In general, pre-stimulus effects on encoding are expected to be influenced by any variable that influences ease of preparation, such as the time in between cue and event, the type of upcoming decision, and individual differences.

Reliable subsequent memory effects also occurred after word onset. As typically observed, events that were later recollected elicited a more positive-going waveform over frontal sites at study (Paller & Wagner, 2002; Rugg et al., 2002; Friedman & Johnson, 2000; Wagner et al., 1999). Notably, this effect was only statistically significant for visual words, although there was a hint of a more negative-going posterior modulation for both visual and auditory words. The apparent variations in neural activity following visual and auditory events point to a qualitative difference in the way in which durable memories are formed of events with different input modalities. More importantly in the present context, however, is the observation that auditory events elicited a reliable subsequent memory effect before, but not after, stimulus onset. This supports earlier suggestions that pre- and post-stimulus effects do not necessarily co-occur, and are therefore at least partially dissociable (Otten et al., 2006). The same conclusion arises from the opposite polarities of pre- and post-stimulus effects. Memory formation is thus associated with qualitatively different functional and neural processes before and after an event. A full understanding of human long-term memory therefore requires the consideration of both.

A final, more tentative, aspect of the present data concerns the pattern of responses across study items later given ‘remember’, ‘know’, or ‘new’ judgments. Pre-stimulus influences on encoding were most pronounced for items later given ‘remember’ judgments. At first glance, this seems to support the conclusion that pre-stimulus influences on encoding are especially important for episodic memory (recollection). However, the single-step remember/know procedure that was used in the experiment is not process pure, and ‘remember’ judgments may involve recollection as well as high levels of familiarity (e.g. Parks & Yonelinas, 2007). Inspection of the retrieval-related ERPs (not shown here) provides some assurance that the majority of ‘remember’ judgments involved recollection. These trials elicited the left parietal old/new ERP effect typically held to index recollection (Rugg et al., 2002). Nonetheless, a firm interpretation of the pattern of results across the different types of recognition judgments awaits experiments in which complimentary approaches, such as source judgments or combined remember/know and confidence judgments (Yonelinas, Otten, Shaw, & Rugg, 2005), are used. The gradual nature of the effects across ‘remember’, ‘know’, and ‘new’ judgments may suggest that pre-stimulus activity leading to later recollection and familiarity differs in kind rather than type (cf. Ranganath et al., 2004; Davachi, Mitchell, & Wagner, 2003; Rugg & Yonelinas, 2003). Alternatively, the memory processes underlying these recognition judgments may have differed in strength rather than type in the present circumstances.

In conclusion, the present data establish the general role of pre-stimulus activity in memory formation, irrespective of the input modality of the to-be-encoded event. The data also suggest that pre-stimulus activity is sufficiently flexible to affect encoding on a trial-by-trial basis. Together, the findings help delineate the functional role and time course of neural activity preceding memory formation. A question of particular interest for future research is how the pre-stimulus ERP activity observed here relates to (i) encoding-related pre-stimulus effects in different brain regions identified with fMRI (Adcock et al., 2006; Mackiewicz et al., 2006) and (ii) pre-stimulus effects in other cognitive domains (e.g. Driver and Frith, 2000; Supèr et al., 2003; Linkenkaer-Hansen et al., 2004; Düzel et al., 2005; Kounios et al., 2006; Thut et al., 2006; Haynes et al., 2007).