Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2008 Jul 1.
Published in final edited form as: Neuroimage. 2007 Apr 18;36(3):933–942. doi: 10.1016/j.neuroimage.2007.04.014

Interactions between working memory and visual perception: An ERP/EEG study

Yigal Agam 1, Robert Sekuler 1
PMCID: PMC1974846  NIHMSID: NIHMS26266  PMID: 17512216

Abstract

How do working memory and perception interact with each other? Recent theories of working memory suggest that they are closely linked, and in fact share certain brain mechanisms. We used a sequential motion imitation task in combination with EEG and ERP techniques for a direct, online examination of memory load’s influence on the processing of visual stimuli. Using a paradigm in which subjects tried to reproduce random motion sequences from memory, we found a systematic decrease in ERP amplitude with each additional motion segment that was viewed and memorized for later imitation. High-frequency (>20 Hz) oscillatory activity exhibited a similar position-dependent decrease. When trials were sorted according to the accuracy of subsequent imitation, the amplitude of the ERPs during stimulus presentation correlated with behavioral performance: The larger the amplitude, the more accurate the subsequent imitation. These findings imply that visual processing of sequential stimuli is not uniform. Rather, earlier information elicits stronger neural activity. We discuss possible explanations for this observation, among them competition for attention between memory and perception and encoding of serial order by means of differential activation strengths.

Introduction

The interplay between working memory and visual perception has been the subject of considerable debate. Even though working memory has received much interest from researchers in psychology and cognitive neuroscience, remarkably little is known about the mechanisms on which it depends, or about the factors that set its limits. At the theoretical level, two general classes of models posit different interactions between memory and perception. One approach conceptualizes working memory as a set of specialized buffers for the storage of information (Baddeley, 2003), which are controlled by an attention-based structure, the central executive. The multi-component, modular model emphasizes limited amounts of activation and temporal decay in the buffers as the sources of capacity limitations in working memory (Baddeley & Logie, 1999). Physiological demonstrations of activity in the prefrontal cortex (PFC) during the memory retention period have lead to the identification of the PFC as the neural substrate for the proposed storage buffers (Goldman-Rakic, 1987; Postle, 2006). An alternative view treats working memory not as a separate module, but as an emergent property, which harnesses existing neural mechanisms specialized for sensory perception and long-term memory representations (Cowan, 1999, 2000; Jonides, Lacey, & Nee, 2005; Pasternak & Greenlee, 2005; Postle, 2006). By this account, after the visual stimulus has disappeared from sight, visuospatial working memory is achieved by prolonged activations, via attention, of the same occipital and parietal regions that are thought to mediate visual perception (Druzgal & D’Esposito, 2001; Postle, Druzgal, & D’Esposito, 2003; Todd & Marois, 2004; Vogel & Machizawa, 2004). In the “embedded-process”, or “emergent property” framework, the prefrontal cortex does not provide the actual substrate for memory storage, but rather mediates attentional control of the sensory reactivation process (Curtis & D’Esposito, 2003; Lebedev, Messinger, Kralik, & Wise, 2004; Postle, 2005). Consequently, this approach emphasizes attentional control as a limiting factor in working memory capacity (Cowan, 1999, 2000).

Attempts to choose between these theoretical accounts have produced mixed results. The key issues that distinguish the two accounts are (i) the degree to which short-term storage of visual information overlaps with early stages of visual processing, and (ii) whether memory’s capacity limit is dictated by some limit on attentional selection of visual input. Several groups used visual search paradigms in conjunction with a working memory task, measuring the degree to which the content of working memory affects search effciency. Woodman, Vogel, and Luck (2001) found no detrimental effect on visual search when subjects had to concurrently remember a visual object. More recently, however, they found that when some location in space had to be remembered, the slope of the function relating reaction time to search array size increased, indicating reduced search efficiency (Woodman & Luck, 2004). A similar result was reported by Oh and Kim (2004). Olivers, Meijer, and Theeuwes (2006) explored the relation between the content of working memory and the items in the search array, and reported that loading working memory with information increased interference from singleton distractors (see also Lavie & de Fockert, 2005), especially when those distractors were identical to or shared features with the remembered material. This is inconsistent with studies by Downing and Dodds (2004) and Woodman and Luck (In press), which found no such interference effects. A second line of inquiry examined how perceptual processing is affected by a concurrent working memory task. Awh and colleagues, using functional imaging (Awh et al., 1999), ERPs (Awh, Anllo-Vento, & Hillyard, 2000), and behavioral measures (Awh, Jonides, & Reuter-Lorenz, 1998), showed that remembering spatial locations has the same perceptual consequences as spatial attention, i.e., increased sensitivity to visual stimuli at those locations, as measured by BOLD signal changes, ERP amplitude and reaction times. Theeuwes, Olivers, and Chizk (2005) demonstrated the same principle using eye movements: Remembering a location caused the subjects’ gaze to deviate, just as if they were directing their attention towards that location. Downing (2000) showed that when subjects remembered a face at one of two locations, perceptual judgements about stimuli subsequently shown in that location were faster than in the other location. In the verbal domain, Shulman and Greenberg (1971) observed a perceptual deficit, as measured by response time, in recognizing a digit while remembering a list of consonants, a deficit that depended on list length. Fougnie and Marois (2006) used a dual-task study that joined multiple object tracking and working memory. Somewhat in contradiction with other studies, they found that although the two tasks interfered with each other, working memory capacity cannot be explained solely by limits on attention, and working memory likely involves its own distinct, capacity-limited subprocesses.

We believe one useful approach, which has not been used so far, is to directly measure neural respnses to visual stimuli at the same time they are being loaded into working memory, rather than use indirect measures, such as visual search, or task-irrelevant stimuli, like those used in the spatial memory studies. We also sought to use a single-task paradigm, which would eliminate the complexities and added cost of handling two tasks at the same time (Olivers et al., 2006; Lavie, Hirst, de Fockert, & Viding, 2004). We reasoned this could be achieved by means of a procedure in which items of visual information are presented sequentially, so that subjects have to keep early items in working memory while continuing to encode each subsequently-presented item. Under these conditions, perceptual responses can be gauged in the presence of a steadily increasing load on working memory. By recording neural responses to the presentation of each additional item, we could measure the consequences of the growth in the amount of stored information. Scalp EEG recordings provide a good basis for such an analysis: First, they provide excellent temporal resolution, which is essential for evaluating differences between responses to closely-spaced stimuli. Second, ERP and EEG markers provide valuable information about visual perception: ERP amplitude is known to correlate with attentive visual processing (Hillyard & Münte, 1984; Hillyard, Vogel, & Luck, 1998; Luck, Woodman, & Vogel, 2000; Awh et al., 2000), as does activity in the high-frequency (beta and gamma) bands of the EEG (Gruber, Müller, Keil, & Elbert, 1999; Müller, Gruber, & Keil, 2000; Tallon-Baudry, Bertrand, Hena3, Isnard, & Fischer, 2005). Thus, we used those electrophysiological markers to track changes in subjects’ processing of incoming visual motion information while they were attempting to hold previously seen motion in working memory.

We recorded scalp EEG from human adults who performed a sequential imitation task (Agam, Bullock, & Sekuler, 2005; Agam, Galperin, Gold, & Sekuler, In press). Fig. 1a shows a schematic diagram of the experimental paradigm. On each trial, subjects viewed a moving disc whose trajectory comprised five randomly oriented, connected linear segments. Then, several seconds later, subjects used a stylus and a graphic tablet to reproduce the trajectory from memory (see also supplementary video clips). We focused on the period during which subjects were viewing the moving disc, the idea being that as the disc progresses, there is more that the subject has to hold in memory of what he or she has already seen, so we would be measuring responses to the disc’s motion under conditions of varying load in working memory.

Figure 1.

Figure 1

Open in a new tab

Stimuli and results. A: Example of a memory trial. Note that while viewing and reproducing the stimulus, the disc did not leave a trace, so subjects only saw its instantaneous position; the dashed lines are for illustration purposes. B: Behavioral results. Segment orientation error is plotted against segment serial position. Error bars are within-subject SEM (Loftus & Masson, 1994). C: ERPs at the midline electrode locations for the memory condition, time-locked to the onset of the disc’s appearance on the display. Vertical dashed lines indicate the onset of individual motion segments. The 4.75 seconds for which ERPs are displayed correspond to the period during which subjects viewed the stimulus models.

Methods

Subjects and Procedure

Seventeen right-handed subjects (8 male, 9 female, age range 18–26) participated after providing written informed consent. Each of the observers performed between 200 and 240 trials of the imitation task (memory condition). Each motion stimulus was generated by the steady movement of a yellow disc (1 deg visual angle in diameter) against a black background on a computer screen, which subjects viewed from a distance of 57 cm. Each model comprised a novel set of five directed motion segments, each 1.5 deg visual angle long, whose orientations varied quasi-randomly. To ensure proper perceptual processing of the individual motion segments, the angular difference between the orientations of each two adjacent segments was between 30 and 150 degrees. The moving disc took 525 ms to traverse each segment; Successive segments were separated by a 225 ms pause, in which the disc remained stationary. The inter-segment pause was meant to prevent perceptual smoothing of the trajectory due to abrupt changes in motion direction (Sekuler, Siddiqui, Goyal, & Rajan, 2003; Brown & Voth, 1937). Subjects had to knit together the directed components in their mind’s eye, and hold the trajectory in memory for 3.75 seconds. They then tried to reproduce it with a stylus on a graphic tablet (Wacom, Vancouver, WA). The accuracy of the imitation was assessed by an automatic segmentation algorithm (Agam et al., 2005), which used temporal and spatial criteria to decompose the imitation into individual segments. After segmentation, the orientation of each reproduced segment was compared to the orientation of the corresponding segment in the stimulus model. The absolute difference between orientations provided the principal measure of error in imitation (see Agam et al., 2005, In press, for discussions of alternative measures).

All seventeen subjects also performed a control task, meant to help us distinguish between effects of memory load and vision-related effects, such as adaptation. In the control condition, stimuli were visually identical to those in the imitation task. For nine subjects, on one third of 240 trials (which were subsequently excluded from the analysis), the speed of the disc through one of the five motion segments changed relative to the normal speed by up to 25%. Subjects had to indicate whether such a speed change happened on a given trial. The eight remaining subjects performed a different control task: At the onset of every one of 100 trials, an arrow pointed in a random direction. Subjects were asked to count on how many, out of 15 segments, the direction of motion matched the direction of the arrow. Only the first five segments, which were visually identical to the stimuli in the memory condition, were included in the analysis. In both control tasks, subjects did not have to remember and imitate the seen motion sequence, but only to make a perceptual judgement.

Since there were more trials in the memory condition than in the control condition, memory trials were randomly excluded from analysis, until the number of trials was equal in both conditions for each subject. Although this procedure decreased the signal-to-noise ratio in the memory data, it ensured that any effects seen in the memory, but not in the control condition were not due to a difference in noise levels. For the analysis in Fig. 4, which pertains to the memory condition only, no trials were excluded.

Figure 4.

Figure 4

Open in a new tab

Correlation between ERP amplitude and behavioral performance. A: Imitation accuracy. Left panel shows orientation error as a function of segment serial position for each quarter of the trials in the memory condition. Right panel shows mean error across all five segments plotted against quartile. B: Mean ERP energy across all electrode locations. Left panel shows the energy of the ERPs time-locked to the onset of segments 2 to 5 in each group. Circles denote energy in segment 1. Right panel shows the energy of the averaged ERPs of all five segments as a function of imitation accuracy. Note that the energy values in the right panel are smaller, since the averaging of all segments reduces noise levels. All error bars are within-subjects SEM for each curve independently.

Electrophysiological Recordings and Analysis

We recorded from 129 electrode sites at 250 Hz using an Electrical Geodesics (Eugene, OR) system. Data were cleaned of bad channels, re-referenced to the grand average and reduced to a montage comprising 27 standard electrode locations, two vertical, bipolar channels above and below each eye, and one horizonal, bipolar eye channel, using BESA (MEGIS Software GmbH, Munich). Data were then averaged and analyzed using MATLAB (The Mathworks, Natick, MA). Blink and eye movement artifacts were eliminated by rejecting epochs in which the difference between the maximum and minimum voltage at any channel exceeded 70 μV. Data were notch-filtered at 60 Hz and high-pass filtered at 1 Hz. For ERP analysis, an additional low-pass filter was applied at 20 Hz. To calculate the energy of the ERP signal, the ERP trace at each electrode was squared, summed across the entire 750 ms duration of the segment and divided by the sample rate, then log-transformed (natural log) to correct for non-normality due to lower bounding at zero. For EEG analysis, energy spectra were computed using a 512-point (Hann-windowed) Fourier transform on each segment. Oscillatory energy was summed across evenly-spaced points within each frequency band (0.1 octave frequency steps), divided by the number of FFT points and by the sample rate, and log-transformed.

Results

As explained earlier, to score the accuracy of subjects’ reproductions in the memory condition, we defined the error for each segment as the absolute difference in orientation between the reproduced segment and the corresponding segment in the stimulus. Behaviorally, the results demonstrated a pronounced primacy effect and a modest, one-item recency effect (Fig. 1b), confirming previous findings with this paradigm (Agam et al., 2005).

Fig. 1c shows event-related potentials (ERPs) at five midline electrode locations, time-locked to stimulus disc’s appearance, and encompassing the entire 4.75-second period in which subjects viewed the stimulus disc during the memory task. As is clear from the figure, every motion segment elicits a distinct electrical signature at a fixed delay from its onset. To examine changes across successive segments, we superimposed traces corresponding to individual segments, and tested the averaged ERPs and energy spectra for differences between segments in the memory and control conditions. As can be appreciated from Fig. 1c, the ERP associated with the first segment is quite different from the other segments’ ERPs, making a direct comparison difficult, particularly when selecting specific components of the ERP for comparison. This may be due to the onset of the disc’s motion, which not only entails an abrupt change in the visual scene, but also a change in context and in the subject’s arousal level. Such attentional transients are also known to be accompanied by a strong decrease in oscillatory activity, particularly in the alpha band (Klimesch, 1996). Another reason for the difference between the first segment and others could be that each ERP lasts longer than its respective segment, and extends into the next one. The first segment, during which working memory is “empty”, would not be contaminated by such residual effects. In any case, when all five segments were included in the analysis, both conditions produced highly significant differences between segments in both time and frequency, in which the first segment was clearly an outlier. To obtain more meaningful observations related to the effects of memory load per se, we excluded the first segment from further analysis, and focused on segments 2 through 5.

Analysis of segment-by-segment ERPs focused on epochs of interest identified by visual inspection of the data: We identified the notable peaks in the data and focused our analysis on short intervals encompassing those peaks. Such subjective choices of time bins may seem arbitrary, but note how different the ERPs to motion segments (Fig. 1C) are from the “standard” ERPs that are typically elicited by a brief flash of a static stimulus (As Fig. 1C shows, the ERP to the appearance of the static disc at time 0 actually resembles a typical visual ERP.) The non-standard form of the ERPs to the moving stimulus means that we could not rely on well-characterized components such as N1, P1, etc., found with more common visual stimuli (see Luck, 2005, for a review of ERP components), and had to resort to our own choice of time epochs. The large frontal peak may be likened to the frontal P3a component. However, the P3a is known to be elicited by unexpected, task irrelevant stimuli (Squires, Squires, & Hillyard, 1975; Polich & Comerchero, 2003), which makes us think it represents something else here. We shall therefore avoid using “standard” notations for the ERP peaks.

The ERPs around identified peaks were averaged and subjected to a repeated measures ANOVA with factor segment. Fig. 2 demonstrates the segment-by-segment differences between ERPs in the memory and control conditions at five midline electrode locations. The memory condition produced widely-distributed differences between successive segments of the trajectory. Specifically, as each successive segment was seen, ERP amplitude decreased. These effects were widely distributed, but largest at frontal and central locations. We found very little difference between segments in the perceptual control condition, indicating that differences among segments in the memory condition are not caused by visual processes alone, but arise from the requirement to encode and maintain successive components of the seen trajectory in working memory. Note that our failure to reliably reject the null hypothesis in the control condition does not, by any means, imply that nothing is happening in that condition (Loftus, 1996); In fact, the data show a certain downward trend in ERP amplitude. But the weak control result, coupled with the clear significance in the memory condition, does show that memory load has a much stronger effect on the ERPs than merely the repeated motion or the passage of time.

Figure 2.

Figure 2

Open in a new tab

Modulation of ERP amplitude. A: Left and middle panels show ERPs time-locked to the onset of segments 2 to 5 in the memory and control conditions, respectively. Each color corresponds to a different serial position, and the horizontal black lines represent averaging epoch boundaries. Asterisks denote significance levels (ANOVA with factor segment) for the mean potential at each tested time epoch (p values above 0.05 are unmarked). The right panels show the energy of the ERPs from the left and middle panels; The solid and dashed lines represent the energy in the memory and control conditions, respectively. B: Mean energy of the ERPs across all 27 electrode locations, as a function of segment serial position. In all energy plots, Error bars are within-subject SEM for each condition, and the solid and dashed circles indicate the energy for the first segment, which is only given for reference; It was not included in any statistical analysis.

When observing the ERP traces in Fig. 2A, one gets a sense that the ERPs in the memory condition decrease in a multiplicative fashion. Consider, for example, electrode Fz: It looks as if each segment, throughout the entire 750 ms, is a reduced version of its predecessor. This phenomenon has been termed “gain control”, and described by Hillyard et al. (1998) — in the context of attentional modulation — as “an amplitude change without any modification of waveform”. With our data, these changes are difficult to assess by simply choosing visible peaks, as some parts of the ERPs, particularly the early and late parts of segments, have moderate slopes. To provide, along with the ERP traces, a measure that reflects changes in the ERP amplitude but does not depend on a specific, subjective choice of time bins, we calculated the sum of the squared samples in each segment’s ERP for the entire duration of each segment. This is a measure that is akin to the energy of the ERP trace, and reflects how much the ERP deviates from zero, or from a “resting” state. In using this measure, we sacrifice some temporal resolution for a more compact, component-independent representation of the ERP. We emphasize that this measure has not been used before as an index of visual processing, and is not meant to replace the analysis of peaks in the raw waveforms, but only to complement it as a more compact indicator, which seems appropriate given the multiplicative nature of the differences between segments in our data.

The right column in Fig. 2A shows the energy plots for each electrode, which clearly show significant decreases in amplitude in the memory, but not the control, condition. Of the midline electrodes, Oz showed an interaction between condition and serial position (p < 0.005). In addition, we calculated the mean energy of the averaged ERP signal across the entire scalp, measured at 27 electrode locations (Fig. 2B). This whole-head measure provides a useful, compact index of the amplitude effects. Even though position was a significant factor in the control condition (p < 0.03, one-way ANOVA), energy did not decrease consistently, and in fact increased at the fifth segment. Conversely, in the memory condition, the ERPs decreased monotonically and highly significantly (p < 0.0001) in energy. A two-way ANOVA confirmed an interaction between factors condition and serial position (p < 0.05).

Although more speculative than ERP measures, there is growing evidence that high-frequency (>20 Hz) oscillatory activity is correlated with visual processing. To complement the ERP analysis and obtain more information about the differential processing of motion segments in our imitation task, we examined changes in oscillatory activity across time, while subjects were viewing the stimulus models. To that end, we divided the frequency spectrum into four standard frequency bands: theta, 4–8 Hz; alpha, 8–13 Hz; beta, 13–30 Hz; and gamma, 30–58 Hz (Hwang et al., 2005; Sederberg et al., 2006), and calculated the total energy within each of the frequency bands (see Methods). Changes in oscillatory energy during the presentation of segments 2–5 were assessed at each electrode location using a repeated-measures ANOVA with factor segment for each frequency band.

High-frequency oscillations in the EEG demonstrated serial-position dependent effects analogous to those seen with ERPs. Significant differences between segments were widespread, and were observed considerably more in the memory condition. As Fig. 3A shows, segmentwise differences in the EEG reveal a loss of energy in the high frequencies (beta and gamma bands) as a function of serial position. In addition to comparing oscillatory activity in individual electrodes, we calculated the mean energy across the entire scalp (27 locations) in each of the frequency bands. Fig. 3B shows the mean energy in each band as a function of segment serial position. A one-way ANOVA with factor segment yielded a significant result only in the beta and gamma bands (p < 0.0001 for both) in the memory (but not the control) condition. A two-way ANOVA indicated an significant interaction between condition and serial position in the beta and gamma bands (p < 0.015, p < 0.01, respectively).

Figure 3.

Figure 3

Open in a new tab

Modulation of high-frequency oscillations. A: Energy in four standard frequency bands in the memory and control conditions at midline electrode locations. Each group of bars corresponds to a different frequency band, and shows the energy for segments 3, 4 and 5 (left, middle and right bars, respectively) relative to the energy for segment 2. B: Mean energy across all electrodes in each band as a function of serial position, in the memory (solid line) and control (dashed line) conditions. The solid and dashed circles indicate the energy in the first segment. All significance markers are similar to Fig. 2. Error bars are within-subject SEM for each condition.

Do our findings have any bearing on subjects’ actual performance in the memory task? To find out, we sorted trials in the memory condition according to the accuracy of the imitation on each trial. Orientation errors were averaged across all five segments, and trials were binned into four quartiles according to the size of the mean error. Therefore, for each subject we had four equally populous groups of trials, representing four different levels of performance (Fig. 4A, p < 0.01, one-way ANOVA with factor mean segment error). To examine whether the ERP generated during encoding can predict the quality of subsequent imitation, we compared the ERPs in each of the four groups. Fig. 4B shows the energy of the ERPs across the entire scalp, for individual segments (left) and averaged across segments 1 to 5 (right) in each group. The amplitude of the ERPs associated with viewing stimulus segments was positively correlated with subsequent performance: The larger the ERP amplitude, the more accurately subjects then reproduce the motion stimulus. When considering the dynamics of the ERPs across serial positions within each group (Fig. 4B, left), an interesting pattern can be seen: In all four datasets, ERP amplitude decreased significantly (p < 0.005 in all sets, ANOVA with factor segment); Later segments were less effective in eliciting ERPs than earlier ones. We did not observe a similar correspondence with behavioral performance in the frequency data, possibly because of the higher noise levels in each group of trials.

An issue that must be addressed with our results is the possible contamination of the data by eye movements. Subjects were required to track the moving disc, and were more likely to move their eyes during the later part of the trajectory, as the disc moved further away from its starting point. Eye movements have a very strong effect on EEG signals, and must be considered in any study using that technique. Several factors convince us that eye movements cannot account for our results: 1) Effects are robustly seen in both anterior and posterior electrodes, whereas eye artifacts are mostly restricted to anterior sites; 2) We do not know of any reported impact of eye movements on high-frequency oscillations; 3) It is unlikely that eye movement patterns differ between the memory and control conditions; and 4) The correlation between ERP amplitude and success in imitation is difficult to explain by means of differences in eye movement patterns. To provide an estimate of how much eye movements may have affected our data, we include result from an eye-tracking study carried out in our lab (Maryott and Sekuler, unpublished observations, see appendix) using a task identical to our memory condition. Data from six subjects show that on average, the eyes move about the length of a segment while viewing each stimulus segment. The small difference between segments (maximum 3 pixels), and the fact that there was no statistically significant difference (p > 0.5) suggest that eye movements did not have a strong influence on our current results.

Discussion

The results reported here provide evidence that working memory load modulates neural responses to visual motion stimuli. As more segments had to be held in memory, ERP amplitude decreased. The same pattern was observed for high-frequency oscillations. Using a perceptual control task with identical stimuli, we demonstrated that the observed drop in amplitude is indeed due to the requirement to remember the motion sequences.

One important question is what the ERPs represent: Do they really index perception? One could argue that the differences between the ERPs within the memory condition reflect purely memory-related processes such as storage and rehearsal, and are not perception-related. If that were the case, then we would expect a qualitative difference between the waveforms for the memory and control conditions, as the control does not involve working memory. What we see, however, is that the major peaks happen exactly at the same time in the control as they do in the memory condition, therefore our results are unlikely to be exclusively related to memory processes. That is not to say that working memory does not modulate the ERPs: It does, but seems to affect slow-wave activity, which was not analyzed here. Numerous studies have found slow-wave ERP markers that are modulated by memory load (Vogel & Machizawa, 2004; Löw et al., 1999; Ruchkin, Johnson, Canoune, & Ritter, 1990) or by the type of information being remembered (Bosch, Mecklinger, & Friederici, 2001; Rolke, Heil, Hennighbausen, Häussler, & Rösler, 2000; Mecklinger & Pfeifer, 1996; Mecklinger, 1998). We intentionally filtered out that aspect of the ERPs (using a 1 Hz high-pass filter) in order to examine interactions with perception-related processes.

We shall now outline some possible explanations for our results. First, the decrease in ERP amplitude with successive segments suggests that processing of the disc’s motion becomes less and less effective as more of the trajectory’s segments have to be held in working memory; It seems that early motion segments benefit from increased attention compared to later ones, whose processing is shared with the rehearsal of early segments. Selective visual attention has been robustly linked to changes in ERP amplitude. Many studies (Hillyard & Münte, 1984; Hillyard et al., 1998; Luck et al., 2000; Awh et al., 2000) found that attended stimuli elicit larger ERPs than ignored ones. Such a correlation between attention and ERP amplitude may indicate that working memory “competes” for attention with the perceptual processing of successive visual stimuli. This is further supported by a drop in high-frequency oscillatory activity, which has also been shown to correlate with attention, as shown by EEG (Gruber et al., 1999; Müller et al., 2000), intracranial EEG (Tallon-Baudry et al., 2005), MEG (Sokolov et al., 1999) and monkey single-unit (Womelsdorf, Fries, Mitra, & Desimone, 2006) experiments. All those studies found increases high-frequency activity when stimuli within the receptive field of a neuron, or in the case of EEG and MEG, in the contralateral visual field, were attended compared to when they were ignored.

This proposed competition for processing resources has some interesting implications: Without resorting to temporal decay or storage buffer size, competition provides a simple explanation of why working memory is so severely limited in capacity. Working memory capacity limitations, at least in the nonverbal domain, could arise from a tradeo3 between the processing of incoming visual information and the need to keep remembered visual material active. Furthermore, competition for attention can account for the primacy effect, i.e., the better recall of early items (Fig. 1b), as an outcome of the uneven distribution of attentional resources available to encode successive motion segments, so that earlier segments benefit from more robust encoding (Sederberg et al., 2006). Our findings do not correlate with the recency effect (Fig. 1B): The electrophysiological effects did not flatten out at the fifth segment. We have suggested previously (Agam et al., 2005) that the primacy effect results from interference during retrieval, which might explain the lack of correlated data during encoding. Finally, we showed that ERP amplitude during presentation of a stimulus model is a good predictor of the accuracy of subsequent imitation. If larger ERP amplitude indeed represents elevated attention, then this predictive relationship could indicate that if a subject is able to maintain a sufficiently high attention level early on in the sequence, as seen for the two high-performance bins, the drop in attention would not be as consequential and the probability of good recall would increase.

The idea that working memory capacity is a derivative of a more general limit on attention has been proposed previously in various forms (Cowan, 1999, 2000; Kane, Conway, Bleckley, & Engle, 2001; Postle, 2006; Jonides et al., 2005). Although the results of the present study agree with such a hypothesis, they are not dispositive. The rival multi-component model could, in principle, accommodate our data, by assuming that working memory does indeed comprise of “stand-alone”, specialized buffers, but controlling the content of those buffers pulls attention away from perceptual processing. However, if that is the case, then the multi-component model could benefit from the inclusion of an explicit connection between the so-called “central executive” — the main attentional construct — and the “crystallized systems” — sensory representations and long-term memory stores (Baddeley, 2003). In other words, the “central executive” should also describe what is called “selective attention” in other contexts. That way, the multi-component model could more successfully account for interactions between attention-driven sensory processing and working memory maintenance.

An alternative explanation is one that postulates an encoding mechanism at play, rather than a consequence of limited resources. For example, some models for the representation of serial order in memory (Grossberg, 1978; Page & Norris, 1998; Farrell & Lewandowsky, 2002; Bullock & Rhodes, 2003; Botvinick & Plaut, 2006) assume an activation strength-based code: In the most simple implementation, the internal representations of all the items in a sequence are activated, prior to the beginning of recall, at differential strengths which are ordered by the serial position of each item: The strongest item is most activated, and activation strength decreases monotonically, with the last item most weakly activated, thus providing an implicit code for serial order. How such a “primacy gradient” could be constructed, however, has been unclear. Our results, although too coarse in spatial resolution to provide a detailed account of such a mechanism, may reflect the construction of an activation strength gradient while the sequence is being seen and encoded. Arbib, Bonaiuto, and Rosta (2006) proposed an attention-based mechanism that arranges the representations of action in a sequence in descending order of activity strength. On the other, strength-based encoding may in fact not be a mechanism per se but an implicit consequence of limitations on processing serial information with equivalent reliability across successive items, as proposed above.

We used a control condition to rule out low-level visual effects, such as sensory adaptation. One could still argue, though, that what we observed is indeed adaptation: A reasonable argument would be that since in the memory condition ERP amplitude is larger early on in the motion sequence relative to the control condition (Fig. 2B), the visual system adapts more strongly. While this is by all means an interaction between memory and perception, it is different from the attention-based explanation. Some points, however, argue against adaptation: As seen in Fig. 2B, the total ERP energy for segment 5 is smaller in the memory condition than in the control condition (p < 0.055, t-test), arguing against convergence to the same physiological state. This is also mostly true when examining each individual electrode (Fig. 2A). Similarly, EEG activity in the beta and gamma bands (Fig. 3B) begins at the same level and drops precipitously in the memory condition to a much lower final level than the control condition. Furthermore, Fig. 4B shows that for the top three quartiles of performance, amplitude is similar for the second segment, but does not fall o3 to the same final level.

In conclusion, we report a unique observation related to the encoding of sequential stimuli. More research needs to be carried out to fully understand why the amplitude of neural responses to stimuli drops throughout the sequence and what kind of process is reflected by this phenomenon. One direction we are currently taking is repeating each stimulus model multiple times, so that subjects become more familiar with the model after each presentation (see also Agam et al., In press). Studying how serial position amplitude effects are modulated by familiarity may provide useful clues as to what causes these effects. If the amplitude effects are due to decreased attention, then the competition for attention between memory and perception might be resolved through model repetition. On the other hand, if the amplitude effects reflect a more elaborate encoding mechanism, then they should persist following learning.

Supplementary Material

01
02
Download video file (81.7KB, mov)
03
Download video file (81.8KB, mov)
04
Download video file (83.7KB, mov)
05
Download video file (77.9KB, mov)

Figure A0.

Figure A0

Open in a new tab

Interactions between Memory and Perception

Acknowledgments

We thank Jessica Maryott for collecting eye-tracking data. Supported by NSF grant SBE-0354378 and NIH grant R01MH068404. E-mail: yigal@brandeis.edu

Appendix

Eye-tracking Data

These data were taken from an eye-tracking study, in which six subjects performed a task identical to the memory task in the present study. For each viewed motion segment, we calculated the distance (in pixels) between the point on the display where the subjects were looking at the onset of the segment and at the end of the segment. This distance is plotted against segment serial position. As can be seen, the maximal difference between segments is small (about three pixels, equivalent to 1/13 degrees visual angle), and insignificant (p > 0.5, ANOVA).

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Agam Y, Bullock D, Sekuler R. Imitating unfamiliar sequences of connected linear motions. Journal of Neurophysiology. 2005;94(4):2832–2843. doi: 10.1152/jn.00366.2005. [DOI] [PubMed] [Google Scholar]
  2. Agam Y, Galperin H, Gold BJ, Sekuler R. Learning to imitate novel motion sequences. Journal of Vision. doi: 10.1167/7.5.1. In press. [DOI] [PubMed] [Google Scholar]
  3. Arbib MA, Bonaiuto J, Rosta E. The mirror system hypothesis: From a macaque-like mirror system to imitation. In: Cangelosi A, Smith ADM, Smith K, editors. Proceedings of the 6th international conference on the evolution of language (EVOLANG6); World Scientific Publishing Company. 2006. pp. 3–10. [Google Scholar]
  4. Awh E, Anllo-Vento L, Hillyard SA. The role of spatial selective attention in working memory for locations: evidence from event-related potentials. Journal of Cognitive Neuroscience. 2000;12(5):840–847. doi: 10.1162/089892900562444. [DOI] [PubMed] [Google Scholar]
  5. Awh E, Jonides J, Reuter-Lorenz PA. Rehearsal in spatial working memory. Journal of Experimental Psychology: Human Perception & Performance. 1998;24(3):780–790. doi: 10.1037//0096-1523.24.3.780. [DOI] [PubMed] [Google Scholar]
  6. Awh E, Jonides J, Smith EE, Buxton RB, Frank LR, Love T, Wong EC, Gmeindl L. Rehearsal in spatial working memory: evidence from neuroimaging. Psychological Science. 1999;10(5):433–437. [Google Scholar]
  7. Baddeley A. Working memory: looking back and looking forward. Nature Reviews Neuroscience. 2003;4(10):829–839. doi: 10.1038/nrn1201. [DOI] [PubMed] [Google Scholar]
  8. Baddeley AD, Logie RH. Working memory: The multiple component model. In: Miyake A, Shah P, editors. Models of working memory: Mechanisms of active maintenance and executive control. Cambridge: Cambridge University Press; 1999. pp. 28–61. [Google Scholar]
  9. Bosch V, Mecklinger A, Friederici AD. Slow cortical potentials during retention of object, spatial, and verbal information. Cognitive Brain Research. 2001;10(3):219–237. doi: 10.1016/s0926-6410(00)00040-9. [DOI] [PubMed] [Google Scholar]
  10. Botvinick MM, Plaut DC. Short-term memory for serial order: A recurrent neural network model. Psychological Review. 2006;113(2):201–233. doi: 10.1037/0033-295X.113.2.201. [DOI] [PubMed] [Google Scholar]
  11. Brown JF, Voth AC. The path of seen movement as a function of the vector-field. American Journal of Psychology. 1937;49A:543–563. [Google Scholar]
  12. Bullock D, Rhodes B. Competitive queuing for serial planning and performance. In: Arbib M, editor. Handbook of brain theory and neural networks. Cambridge, MA: MIT Press; 2003. pp. 241–244. [Google Scholar]
  13. Cowan N. An embedded-process model of working memory. In: Miyake A, Shah P, editors. Models of working memory: Mechanisms of active maintenance and executive control. Cambridge: Cambridge University Press; 1999. pp. 62–101. [Google Scholar]
  14. Cowan N. The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral & Brain Sciences. 2000;24:87–185. doi: 10.1017/s0140525x01003922. [DOI] [PubMed] [Google Scholar]
  15. Curtis CE, D’Esposito M. Persistent activity in the prefrontal cortex during working memory. Trends in Cognitive Sciences. 2003;7(9):415–423. doi: 10.1016/s1364-6613(03)00197-9. [DOI] [PubMed] [Google Scholar]
  16. Downing PE. Interactions between visual working memory and selective attention. Psychological Science. 2000;11(6):467–473. doi: 10.1111/1467-9280.00290. [DOI] [PubMed] [Google Scholar]
  17. Downing PE, Dodds CM. Competition in visual working memory for control of search. Visual Cognition. 2004;11(6):689–703. [Google Scholar]
  18. Druzgal TJ, D’Esposito M. Activity in fusiform face area modulated as a function of working memory load. Cognitive Brain Research. 2001;10(3):355–364. doi: 10.1016/s0926-6410(00)00056-2. [DOI] [PubMed] [Google Scholar]
  19. Farrell S, Lewandowsky S. An endogenous distributed model of ordering in serial recall. Psychonomic Bulletin & Review. 2002;9(1):59–79. doi: 10.3758/bf03196257. [DOI] [PubMed] [Google Scholar]
  20. Fougnie D, Marois R. Distinct capacity limits for attention and working memory: evidence from attentive tracking and visual working memory paradigms. Psychological Science. 2006;17(6):526–534. doi: 10.1111/j.1467-9280.2006.01739.x. [DOI] [PubMed] [Google Scholar]
  21. Goldman-Rakic PS. Cellular and circuit basis of working memory in prefrontal cortex of nonhuman primates. In: Uylings HBM, Van Eden CG, De Bruin JPC, Corner MA, Feenstra MGP, editors. Progress in brain research. Amsterdam: Elsevier; 1987. pp. 325–336. [DOI] [PubMed] [Google Scholar]
  22. Grossberg S. A theory of human memory: Self-organization and performance of sensory-motor codes, maps, and plans. In: Rosen R, Snell F, editors. Progress in theoretical biology. Vol. 5. New York: Academic Press; 1978. pp. 233–374. [Google Scholar]
  23. Gruber T, Müller MM, Keil A, Elbert T. Selective visual-spatial attention alters induced gamma band responses in the human EEG. Clinical Neurophysiology. 1999;110(12):2074–2085. doi: 10.1016/s1388-2457(99)00176-5. [DOI] [PubMed] [Google Scholar]
  24. Hillyard SA, Münte TF. Selective attention to color and locational cues: an analysis with event-related brain potentials. Perception & Psychophysics. 1984;36:185–198. doi: 10.3758/bf03202679. [DOI] [PubMed] [Google Scholar]
  25. Hillyard SA, Vogel EK, Luck SJ. Sensory gain control (amplification) as a mechanism of selective attention: electrophysiological and neuroimaging evidence. Philosophical Transactions of the Royal Society, London, Series B. 1998;353(1373):1257–1270. doi: 10.1098/rstb.1998.0281. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Hwang G, Jacobs J, Geller A, Danker J, Sekuler R, Kahana MJ. EEG correlates of verbal and nonverbal working memory. Behavioral & Brain Functions. 2005;1(20) doi: 10.1186/1744-9081-1-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Jonides J, Lacey SC, Nee DE. Processes of working memory in mind and brain. Current Directions in Psychological Science. 2005;14(1):2–5. [Google Scholar]
  28. Kane MJ, Conway ARA, Bleckley MK, Engle RW. A controlled-attention view of working memory. Journal of Experimental Psychology: General. 2001;130(2):169–183. doi: 10.1037//0096-3445.130.2.169. [DOI] [PubMed] [Google Scholar]
  29. Klimesch W. Memory processes, brain oscillations and EEG synchronization. International Journal of Psychophysiology. 1996;24(1–2):61–100. doi: 10.1016/s0167-8760(96)00057-8. [DOI] [PubMed] [Google Scholar]
  30. Lavie N, de Fockert J. The role of working memory in attentional capture. Psychonomic Bulletin & Review. 2005;12(4):669–674. doi: 10.3758/bf03196756. [DOI] [PubMed] [Google Scholar]
  31. Lavie N, Hirst A, de Fockert J, Viding E. Load theory of selective attention and cognitive control. Journal of Experimental Psychology: General. 2004;133(3):339–354. doi: 10.1037/0096-3445.133.3.339. [DOI] [PubMed] [Google Scholar]
  32. Lebedev MA, Messinger A, Kralik J, Wise SP. Representation of attended versus remembered locations in prefrontal cortex. PLOS Biology. 2004;2(11):e365. doi: 10.1371/journal.pbio.0020365. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Loftus GR. Psychology will be a much better science when we change the way we analyze data. Current Directions in Psychological Science. 1996;5(6):161–171. [Google Scholar]
  34. Loftus GR, Masson MEJ. Using confidence intervals in within-subject designs. Psychonomic Bulletin & Review. 1994;1(4):476–90. doi: 10.3758/BF03210951. [DOI] [PubMed] [Google Scholar]
  35. Löw A, Rockstroh B, Cohen R, Hauk O, Berg P, Maier W. Determining working memory from ERP topography. Brain Topography. 1999;12(1):39–47. doi: 10.1023/a:1022229623355. [DOI] [PubMed] [Google Scholar]
  36. Luck SJ. An introduction to the event-related potential technique. Cambridge, MA: MIT Press; 2005. [Google Scholar]
  37. Luck SJ, Woodman GF, Vogel EK. Event-related potential studies of attention. Trends in Cognitive Sciences. 2000;4(11):432–440. doi: 10.1016/s1364-6613(00)01545-x. [DOI] [PubMed] [Google Scholar]
  38. Mecklinger A. On the modularity of recognition memory for object form and spatial location: a topographic ERP analysis. Neuropsychologia. 1998;36(5):441–460. doi: 10.1016/s0028-3932(97)00128-0. [DOI] [PubMed] [Google Scholar]
  39. Mecklinger A, Pfeifer E. Event-related potentials reveal topographical and temporal distinct neuronal activation patterns for spatial and object working memory. Cognitive Brain Research. 1996;4(3):211–224. doi: 10.1016/s0926-6410(96)00034-1. [DOI] [PubMed] [Google Scholar]
  40. Müller MM, Gruber T, Keil A. Modulation of induced gamma band activity in the human EEG by attention and visual information processing. International Journal of Psychophysiology. 2000;38(3):283–299. doi: 10.1016/s0167-8760(00)00171-9. [DOI] [PubMed] [Google Scholar]
  41. Oh SH, Kim MS. The role of spatial working memory in visual search efficiency. Psychonomic Bulletin & Review. 2004;11(2):275–281. doi: 10.3758/bf03196570. [DOI] [PubMed] [Google Scholar]
  42. Olivers NL, Meijer F, Theeuwes J. Feature-based memory-driven attentional capture: Visual working memory content affects visual attention. Journal of Experimental Psychology: Human Perception & Performance. 2006;32(5):1243–1265. doi: 10.1037/0096-1523.32.5.1243. [DOI] [PubMed] [Google Scholar]
  43. Page MPA, Norris DG. The primacy model: a new model of immediate serial recall. Psychological Review. 1998;105(4):761–781. doi: 10.1037/0033-295x.105.4.761-781. [DOI] [PubMed] [Google Scholar]
  44. Pasternak T, Greenlee MW. Working memory in primate sensory systems. Nature Reviews Neuroscience. 2005;6(2):97–107. doi: 10.1038/nrn1603. [DOI] [PubMed] [Google Scholar]
  45. Polich J, Comerchero MD. P3a from visual stimuli: Typicality, task and topography. Brain Topography. 2003;15(3):141–152. doi: 10.1023/a:1022637732495. [DOI] [PubMed] [Google Scholar]
  46. Postle BR. Delay-period activity in the prefrontal cortex: One function is sensory gating. Journal of Cognitive Neuroscience. 2005;17(11):1679–1690. doi: 10.1162/089892905774589208. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Postle BR. Working memory as an emergent property of the mind and brain. Neuroscience. 2006;139(1):23–38. doi: 10.1016/j.neuroscience.2005.06.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Postle BR, Druzgal TJ, D’Esposito M. Seeking the neural substrates of visual working memory storage. Cortex. 2003;39(4–5):927–946. doi: 10.1016/s0010-9452(08)70871-2. [DOI] [PubMed] [Google Scholar]
  49. Rolke B, Heil M, Hennighbausen E, Häussler C, Rösler F. Topography of brain electrical activity dissociates the sequential order transformation of verbal versus spatial information in humans. Neuroscience Letters. 2000;282(1–2):81–84. doi: 10.1016/s0304-3940(00)00891-0. [DOI] [PubMed] [Google Scholar]
  50. Ruchkin DS, Johnson R, Canoune H, Ritter W. Short-term memory storage and retention: An event-related brain potential study. Electroencephalography & Clinical Neurophysiology. 1990;76(5):419–439. doi: 10.1016/0013-4694(90)90096-3. [DOI] [PubMed] [Google Scholar]
  51. Sederberg PB, Gauthier LV, Terushkin V, Miller JF, Barnathan JA, Kahana MJ. Oscillatory correlates of the primacy effect in episodic memory. Neuroimage. 2006;32(3):1422–1431. doi: 10.1016/j.neuroimage.2006.04.223. [DOI] [PubMed] [Google Scholar]
  52. Sekuler R, Siddiqui A, Goyal N, Rajan R. Reproduction of seen actions: Stimulus-selective learning. Perception. 2003;32(7):839–54. doi: 10.1068/p5064. [DOI] [PubMed] [Google Scholar]
  53. Shulman HG, Greenberg SN. Perceptual deficit due to division of attention between memory and perception. Journal of Experimental Psychology. 1971;88(2):171–176. doi: 10.1037/h0030879. [DOI] [PubMed] [Google Scholar]
  54. Sokolov A, Lutzenberger W, Pavlova M, Preissl G, Braun C, Birbaumer N. Gamma-band MEG activity to coherent motion depends on task-driven attention. Neuroreport. 1999;10(10):1997–2000. doi: 10.1097/00001756-199907130-00001. [DOI] [PubMed] [Google Scholar]
  55. Squires NK, Squires KC, Hillyard SA. Two varieties of long-latency positive waves evoked by unpredictable auditory stimuli. Electroencephalography & Clinical Neurophysiology. 1975;38:387–401. doi: 10.1016/0013-4694(75)90263-1. [DOI] [PubMed] [Google Scholar]
  56. Tallon-Baudry C, Bertrand O, Hena3 MA, Isnard J, Fischer C. Attention modulates gamma-band oscillations differently in the human lateral occipital cortex and fusiform gyrus. Cerebral Cortex. 2005;15(5):654–662. doi: 10.1093/cercor/bhh167. [DOI] [PubMed] [Google Scholar]
  57. Theeuwes J, Olivers CNL, Chizk CL. Remembering a location makes the eyes curve away. Psychological Science. 2005;16:196–199. doi: 10.1111/j.0956-7976.2005.00803.x. [DOI] [PubMed] [Google Scholar]
  58. Todd JJ, Marois R. Capacity limit of visual short-term memory in human posterior parietal cortex. Nature. 2004;428:751–754. doi: 10.1038/nature02466. [DOI] [PubMed] [Google Scholar]
  59. Vogel EK, Machizawa MG. Neural activity predicts individual differences in visual working memory capacity. Nature. 2004;428(6984):748–751. doi: 10.1038/nature02447. [DOI] [PubMed] [Google Scholar]
  60. Womelsdorf T, Fries P, Mitra PP, Desimone R. Gamma-band synchronization in visual cortex predicts speed of change detection. Nature. 2006;439(7077):733–736. doi: 10.1038/nature04258. [DOI] [PubMed] [Google Scholar]
  61. Woodman GF, Luck SJ. Visual search is slowed when visuospatial working memory is occupied. Psychonomic Bulletin & Review. 2004;11(2):269–274. doi: 10.3758/bf03196569. [DOI] [PubMed] [Google Scholar]
  62. Woodman GF, Luck SJ. Do the contents of visual working memory automatically influence attentional selection during visual search? Journal of Experimental Psychology: Human Perception & Performance. doi: 10.1037/0096-1523.33.2.363. In press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  63. Woodman GF, Vogel EK, Luck SJ. Visual search remains efficient when visual working memory is full. Psychological Science. 2001;12(3):219–224. doi: 10.1111/1467-9280.00339. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

01
NIHMS26266-supplement-01.doc (19KB, doc)
02
Download video file (81.7KB, mov)
03
Download video file (81.8KB, mov)
04
Download video file (83.7KB, mov)
05
Download video file (77.9KB, mov)

RESOURCES