Skip to main content
NCBI home page
The Journal of Neuroscience logoLink to The Journal of Neuroscience
. 2009 Jun 24;29(25):8032–8038. doi: 10.1523/JNEUROSCI.0952-09.2009

Searching for Targets within the Spatial Layout of Visual Short-Term Memory

Bo-Cheng Kuo 1,2, Anling Rao 1, Jöran Lepsien 3, Anna Christina Nobre 1,✉
PMCID: PMC6666054  PMID: 19553443

Abstract

Recent studies have revealed that the internal representations that we construct from the environment and maintain in visual short-term memory (VSTM) to guide behavior are highly flexible and can be selectively modulated according to our task goals and expectations. In the current study, we conducted two experiments to compare and contrast neural mechanisms of selective attention related to searching for target items within perceptual versus VSTM representations. We used event-related potentials to investigate whether searching for relevant target items from within VSTM representations involves spatially specific biasing of neural activity in a manner analogous to that which occurs during visual search for target items in perceptual arrays. The results, replicated across the two experiments, revealed that selection of a target object within a search array maintained in VSTM proceeds through a similar mechanism as that in the perceptual domain. In line with previous results, N2pc potentials were obtained when targets were identified within a perceptual visual-search array. Interestingly, equivalent N2pcs, with similar time courses and scalp distributions, were also elicited when target items were identified within a VSTM representation. The findings reinforce the notion of highly flexible VSTM representations that can be modulated according to task goals and suggest a large degree of overlap in the spatially specific neural mechanisms of target selection across the perceptual and VSTM domains.

Introduction

Recent studies have revealed that the internal representations that we construct from the environment and maintain in visual short-term memory (VSTM) to guide behavior are highly flexible and can be selectively modulated and accessed by attentional mechanisms according to our task goals and expectations (Griffin and Nobre, 2003; Landman et al., 2003). However, how attentional mechanisms operate on VSTM representations remains largely unknown. Here, we investigated whether searching for relevant target items from within VSTM representations involves spatially specific biasing of neural activity in a manner analogous to that which occurs during visual search for target items in perceptual arrays.

The nature of VSTM representations is not fully understood, but growing evidence suggests that some of the intrinsic spatial configuration of the original perceptual array is preserved along with the remembered items (Gratton, 1998; Jiang et al., 2000). Partially shared neural mechanisms for coding perceptual and VSTM representations may support these functional parallels. One prevailing account of the neural organization of VSTM proposes that representations of relevant items are supported by the concerted action of multimodal cortical regions involved in executive control, especially prefrontal cortex, and posterior brain areas that participate in the perception of these same items (D'Esposito et al., 2000; Fuster, 2000; Petrides, 2000; Passingham and Sakai, 2004; Harrison and Tong, 2009). According to this view, top-down signals from prefrontal areas bias activity in posterior regions to maintain, monitor, and/or manipulate information in VSTM (Awh and Jonides, 2001; Curtis and D'Esposito, 2003). The participation of retinotopically organized and functionally specialized visual areas in VSTM representations opens the possibility that signals about predictions or task goals that involve VSTM representations may bias activity in visual areas analogously to how top-down signals bias activity in visual areas during perception (Desimone and Duncan, 1995; Treue, 2003). For instance, spatial selection of task-relevant items from VSTM could be achieved by “top-down” biasing of topographically organized neuronal activity in the posterior areas.

We exploited the well established event-related potential (ERP) marker of attentional selection within visual search, the N2pc, to probe whether the selection of targets within VSTM similarly involves spatially specific biasing of neural activity. Although the sources of the N2pc are not fully characterized, the differential potential across contralateral versus ipsilateral sites of the posterior scalp is thought to reflect spatial biasing of activity in posterior parietal and occipital–temporal areas (Hopf et al., 2000), correlated with the selection of target items and/or the suppression of distracting items (Kiss et al., 2008; Hickey et al., 2009). Across two experiments, we tested for the presence of an N2pc when participants searched for a target within a stimulus array held in VSTM, based on shape or color. Both experiments also contained visual search conditions, so that the effects of attentional selection within VSTM and perception could be compared directly. In addition, we tested for the presence of a recently described ERP marker of VSTM search (N3RS) (Nobre et al., 2008).

Materials and Methods

Across both experiments, participants viewed a sample shape (experiment 1) or color (experiment 2) and a search array of different loads (two or four shapes or colored items). Their task was to state whether the sample shape (or color) was present in the search array. In the visual search condition, the sample shape (or color) appeared before the search array, and participants searched for the target item within the perceptual search array. In the VSTM condition, the sample shape (or color) appeared after the search array, and participants searched for the target item within the internalized VSTM representation of the search array. In line with previous studies, attentional selection of a target during visual search was expected to generate an N2pc effect (Luck and Hillyard, 1994; Eimer, 1996). The main experimental question was whether an N2pc effect would also occur when spatial selection occurred within a VSTM representation, indicating spatially specific biasing of mnemonically held information. We also sought to replicate a recently described load-dependent N3RS effect related to searching within VSTM (Nobre et al., 2008). We predicted that the amplitude of the N3RS should increase monotonically with the increasing load of retro-search within VSTM.

Experiment 1

Participants.

Nineteen healthy right-handed (Oldfield, 1971) participants took part in this experiment. All were student members of the Department of Experimental Psychology at the University of Oxford. The behavioral and ERP analyses included data from a subset of 14 participants (nine females and five males; age range, 20–25 years; mean age, 21.21 years) who all showed above-chance performance and had a sufficient number of artifact-free ERP trials (>40; see below) across all task conditions. Twelve subjects were undergraduate students who participated for course credit. Two subjects were graduate students who participated voluntarily. Visual acuity was normal or corrected-to-normal. Informed written consent was obtained from all of the participants before taking part in the study, the nature of which was fully explained. The experimental methods were non-invasive and had ethical approval from the Central University Research Ethics Committee of the University of Oxford.

Task design.

The task is illustrated in Figure 1a (top row). Participants viewed two stimulus arrays and searched for a target amid an array of two or four items presented perceptually in a visual search task (V2 and V4) or maintained in short-term memory in a VSTM search task (M2 and M4). The search type (visual, VSTM), search load (two-item, four-item), and the correct search response (target-present, target-absent) were manipulated factorially within participants, in a three-way repeated-measures design.

Figure 1.

Figure 1.

Open in a new tab

Schematic of experimental trials: visual search and VSTM search (top) used in experiment 1 (a) and experiment 2 (b). In visual search trials, a single item was initially presented in the center of the screen, followed by an array of two or four peripherally located items. Participants were asked to search for the initial item within the second array and indicate whether the item was present (50%) or absent (50%). In VSTM search trials, an array of two or four peripheral items was initially presented, followed by a single centrally presented item. Participants were asked to search for this item within the previously presented array now maintained in VSTM. In both types of search, the delay between presentation of the first and second items varied between 800 and 1300 ms. Sensitivity measures (d′) and mean response times (milliseconds) in target-present trials across the two types of search tasks (visual and VSTM) and two types of search load (two-item and four-item) were measured (bottom). Error bars represent SEM.

A trial began with a centrally displayed asterisk (500–1000 ms duration), which signaled the onset of each trial. In visual search trials, one item was initially presented in the first array in the center of the monitor (200 ms), followed, after a randomized variable interval (800–1300 ms), by an array of two or four peripherally located items (200 ms). Participants searched for the initial item within the second array and indicated whether the item was present or absent. In VSTM search trials, an array of two or four peripheral items was initially presented (200 ms), followed, after a randomized variable interval (800–1300 ms), by a single centrally presented item (200 ms). Participants searched for the single target item within the previous array maintained in VSTM. Participants were instructed to indicate whether the item was present (50%), by pressing the left mouse button, or absent (50%), by pressing the right mouse button, when the second visual array was displayed. All trial types were intermixed in a randomized and unpredictable order. The interval between trials, which included a 2000 ms response period, varied randomly between 2500 and 3500 ms. There were 768 trials in total (96 target-present and 96 target-absent trials in each search-type and load condition). To avoid fatigue and provide participants with time to rest their eyes, the experiment was completed in 48 mini blocks of 16 trials, which participants could self-initiate. One additional practice block was completed at the beginning, to familiarize participants with the task.

Stimuli.

A black background was used throughout the experiment. The stimuli used for search arrays were drawn from the set of novel meaningless closed shape contours previously developed by Endo et al. (2003), which are difficult to verbalize. All stimulus contours were white. Stimulus arrays were composed of two (50%) or four (50%) shapes randomly selected from a subset of eight items. Each stimulus was 1.72° visual angle in size and was positioned randomly in one of the four possible peripheral locations of an invisible 2 × 2 matrix. Stimulus locations were centered at 2.58° eccentricity and 2° elevation. Two-item arrays contained stimuli presented along a diagonal (top left and bottom right; or top right and bottom left). Short durations were chosen for the presentations of the arrays and probes (200 ms; see above) to discourage participants from making eye movements and because these durations were shown to be effective for measuring the N2pc (Brisson and Jolicoeur, 2007), our ERP marker of interest.

Experimental procedure.

Participants were comfortably seated in a dimly illuminated, electrically shielded room, facing a computer monitor placed 100 cm in front of them. Each participant was given written as well as verbal instructions about the task requirements, and they completed 16 practice trials before the actual experiment to ensure they understood the task. They were also instructed to maintain fixation on a small fixation marker at the center of the monitor during the active parts of the experiment and to respond as accurately and quickly as possible. Participants responded using their right hand. Participants were also asked to minimize blinking and moving their eyes throughout the experiment.

Behavioral analyses.

Perceptual sensitivity to detect target stimuli (d′ score) (Green and Swets, 1966) and reaction times (RTs) in target-present trials were analyzed by a 2 (search type: visual and VSTM) × 2 (search load: two-item and four-item) repeated-measures ANOVA. In the d′ measure, hit rate was defined as the conditional probability that the participants responded “target-present” given that the target was presented, and the false-alarm rate was defined as the conditional probability that the participants responded “target-present” given that the target was absent. The d′ scores were calculated by hits and false alarms in each condition using the following equation (Engelmann and Pessoa, 2007):

graphic file with name zns02509-6548-m01.jpg

In the RT analysis, only correct responses were included. Post hoc contrasts were performed to guide interpretation when appropriate.

EEG recording

The EEG was recorded continuously using NuAmp amplifiers (Neuroscan) from 40 silver/silver-chloride electrodes placed on the scalp with an elasticized cap, positioned according to the 10–20 international system (AEEGS, 1991). The montage included six midline sites (FZ, FCZ, CZ, CPZ, PZ, and OZ) and 14 sites over each hemisphere (FP1/FP2, F3/F4, F7/F8, FC3/FC4, FT7/FT8, C3/C4, T7/T8, CP3/CP4, TP7/TP8, P3/P4, P7/P8, PO3/PO4, PO7/PO8, and O1/O2). Electrodes were placed around the eyes to monitor for blinks and eye movements. Additional electrodes were used as ground and reference sites. Electrodes were referenced to the right mastoid site (A2). The electrode between FPZ and FZ (AFZ) on the midline served as the ground electrode. Electrode impedances were kept below 5 kΩ. The ongoing brain activity at each electrode site was sampled every 1 ms (1000 Hz analog-to-digital sampling rate). Activity was filtered with a low-pass filter of 300 Hz. The EEG was recorded continuously during the entire duration of each experimental run. The computer used for stimulus presentation sent digital codes (via the parallel port) to the EEG-acquisition computer, indicating exactly when each stimulus was presented.

EEG processing

Offline, the EEG was re-referenced to the algebraic average of the right and left mastoids. Bipolar electro-oculogram (EOG) signals were derived by computing the difference between the voltages at electrodes placed to the side of each eye [horizontal EOG (HEOG)] and above and below the left eye [vertical EOG (VEOG)]. The re-referenced and transformed continuous data were further low-pass filtered (40 Hz) to exclude high-frequency noise.

The continuous EEG was then segmented into epochs starting 100 ms before and ending 500 ms after each stimulus array. The EEG epochs were normalized to the baseline period before stimulus presentation. Data quality was inspected using automated algorithms. Epochs containing excessive noise or drift (±100 μV) at any electrode between −100 and +500 ms were excluded. In addition, epochs with eye-movement artifacts (blinks or saccades) were rejected. Blinks were identified as large deflections (±50 μV) in the HEOG or VEOG electrodes. Saccades were identified by visually inspecting HEOG traces individually. Finally, trials with incorrect behavioral responses were also discarded.

Epochs were averaged according to search type, search load, target side, and response type. In the visual condition, ERPs were derived from the onset of the search arrays. In the VSTM condition, ERPs were derived from the onset of the centrally presented probe item, which triggered search in the array maintained in VSTM. ERPs from trials containing targets located on the right side and from trials containing targets located on the left side were combined by an averaging procedure that preserved the electrode location relative to the target side (ipsilateral or contralateral). The ERPs of main interest were those in target-present trials, across the two search types and two search loads, resulting in four conditions: visual search with two items (V2), visual search with four items (V4), VSTM search with two items (M2), and VSTM search with four items (M4). To maintain an acceptable signal-to-noise ratio, a lower limit of 40 artifact-free trials per subject per condition was set.

EEG analysis

Statistical analyses were performed using the mean voltage value (mean amplitude) over successive time bins across different scalp regions. The first aim of this experiment was to determine whether search in VSTM relied on spatially specific neural processes similar to those observed during visual search. We therefore focused on identifying the predicted N2pc potential during the visual search task and testing for the presence of an N2pc potential in the VSTM search task. The mean amplitudes of the N2pc were measured between 240 and 300 ms at the parietal–occipital electrodes (PO7/PO8, P7/P8, P3/P4, and TP7/TP8) contralateral and ipsilateral to the side of the target. A four-way repeated-measures ANOVA was computed on the mean amplitudes of the N2pc, testing the effects of search type (visual and VSTM), search load (two-item and four-item), electrode side (ipsilateral and contralateral to target), and electrode pair (PO7/PO8, P7/P8, P3/P4, and TP7/TP8). Only effects including the factor of electrode side, reflecting differential activity observed in contralateral versus ipsilateral electrodes, were of interest.

To compare the scalp distributions of the N2pc in the visual and VSTM search conditions, we calculated the voltage difference between contralateral and ipsilateral electrodes at all lateral parietal–occipital electrode pairs (CP3/CP4, TP7/TP8, P3/P4, P7/P8, O1/O2, and PO7/PO8). The differential voltage was then normalized over these electrode pairs in each condition for each participant (McCarthy and Wood, 1985). A three-way repeated-measures ANOVA (search type, search load, electrode pair) was used to test for the differential distribution of the normalized N2pc.

A subsidiary aim of this experiment was to identify load-dependent neural activity related to VSTM search, by comparing the processes of searching retroactively for a task-relevant target from four items versus two items in the VSTM representation. Successive pointwise t tests were used to compare the effects of load during VSTM search (M4 vs M2) to identify reliable sustained effects (Guthrie and Buchwald, 1991; Murray et al., 2002). This exploratory analysis guided the selection of the time window with which to test for the presence of an N3RS effect (Nobre et al., 2008). The N3RS was quantified by measuring mean amplitudes at the midline electrode and flanking electrodes from frontal to parietal sites (F3/Cz/4, FC3/Z/4, C3/Z/4, CP3/Z/4, and P3/Z/4) during the time window between 340 and 400 ms. Mean amplitudes were analyzed in three-way repeated-measures ANOVA, testing the effects of search load (two-item and four-item), electrode position along the anteroposterior axis (five levels of region), and electrode position along the lateral axis (three levels of electrode sites). The Greenhouse-Geisser ε correction for nonsphericity was applied to all analyses when appropriate (Jennings and Wood, 1976).

Experiment 2

The goal of this experiment was to replicate the results of experiment 1 using different search stimuli and search based on a different feature: color. Consequently, the stimuli used in the search and probe arrays differed from those in the previous experiment, but the methods were the same in all other aspects, including task design, experimental procedure, behavioral analyses, and EEG recording and analyses.

Participants.

Thirteen healthy right-handed participants were paid to participate in this experiment. All participants were undergraduate and graduate student members of the Department of Experimental Psychology at the University of Oxford. The behavioral and ERP analyses included data from a subset of 12 participants (eight females and four males; age range, 21–27 years; mean age, 22.33 years) who all showed above-chance performance and had a sufficient number of artifact-free ERP trials (>40; see above) across all task conditions. Once again, visual acuity was normal or corrected-to-normal. Informed written consent was obtained from all of the participants before taking part in the study, the nature of which was fully explained. The experimental methods were non-invasive and had ethical approval from the Central University Research Ethics Committee of the University of Oxford.

Stimuli.

Search arrays were composed of two or four colored squares randomly selected from a set of eight possible colors (red, green, yellow, blue, gray, pink, purple, and brown). Each stimulus subtended ∼1 × 1° and was positioned randomly in one of the four possible peripheral locations of an invisible 2 × 2 matrix, whose positions were centered at ∼2.6° eccentricity and 2° elevation. The task is illustrated in Figure 1b (top row).

EEG analysis.

The only difference in the analysis of the EEG data was the temporal window used for the analysis of the N3RS (320–380 ms), which the exploratory analysis indicated occurred earlier in the second, color-search experiment.

Results

Behavioral results

Figure 1 (bottom rows) and Table 1 show the mean d′ scores and RTs in each condition for both shape-based search [experiment 1 (Fig. 1a)] and color-based search [experiment 2 (Fig. 1b)]. The pattern of d′ modulation was equivalent across experiments. In both experiments, the d′ scores showed a significant effect of search type (shape, F(1,13) = 528.60, p < 0.001; color, F(1,11) = 9.96, p < 0.01) and a significant effect of load (shape, F(1,13) = 16.74, p < 0.05; color, F(1,11) = 192.20, p < 0.001). Higher mean d′ scores occurred in the visual search task and the two-item condition than in the VSTM search task and the four-item condition. A significant interaction between search type and load was also observed in both cases (shape, F(1,13) = 16.63, p < 0.05; color, F(1,11) = 19.56, p < 0.05). This interaction arose because the difference of detection sensitivity between the M2 condition and the M4 condition was larger than the difference between the V2 condition and the V4 condition (shape, 1.11 ± 0.87 d′ difference between M2 and M4 and 0.67 ± 0.80 d′ difference between V2 and V4, t(13) = 4.08, p = 0.001; color, 0.85 ± 0.24 d′ difference between M2 and M4 and 0.32 ± 0.27 d ' difference between V2 and V4, t(11) = 4.44, p = 0.001).

Table 1.

Mean d′ scores and reaction times (in milliseconds) of correct responses in each condition

Visual
 VSTM
Two items Four items Two items Four items
Shape
    d′ score 2.37 ± 0.99 1.70 ± 0.27 1.68 ± 0.96 0.57 ± 0.25
    RT 495.32 ± 105.79 564.59 ± 109.95 504.67 ± 124.61 596.53 ± 139.58
Color
    d′ score 2.81 ± 0.97 2.49 ± 0.98 2.36 ± 0.45 1.52 ± 0.43
    RT 354.46 ± 102.39 366.07 ± 104.46 346.95 ± 98.13 450.25 ± 142.59
Open in a new tab

In experiment 1, the RT data showed only a significant main effect of search load (F(1,13) = 54.08, p < 0.001), with faster RTs in the two-item condition than in the four-item condition. No other effects were significant (p > 0.1). In experiment 2, RTs were also significantly modulated by search load (F(1,11) = 25.85, p < 0.001), with faster RTs in the two-item condition compared with the four-item condition. In addition, the main effect of search type (F(1,11) = 5.99, p < 0.05) and interaction between search type and load (F(1,11) = 20.42, p = 0.001) were also significant. RTs were faster in the visual search task compared with the VSTM search task. The interaction indicated that the load effect was larger in the VSTM condition (103.30 ± 68.09 ms difference between M2 and M4) than in the visual condition (11.61 ± 20.91 ms difference between V2 and V4) (t(11) = 4.52, p = 0.001).

Electrophysiological results

N2pc effect

Figure 2 demonstrates that the N2pc was elicited by the target-present arrays at parietal–occipital electrode sites contralateral to the side of the target in both visual and VSTM tasks in both experiments.

Figure 2.

Figure 2.

Open in a new tab

The N2pc effects in the visual and VSTM domains showed equivalent time courses (grand-averaged waveforms) and topographies in experiment 1 (a) and experiment 2 (b). ERP waveforms averaged across all participants are shown for the four-item search load over contralateral and ipsilateral locations of a representative lateral posterior electrode pair. The N2pc was elicited between 240 and 300 ms at the parietal–occipital electrodes (e.g., P7/P8) contralateral (waveforms, solid lines; topographies, blue/right side) and ipsilateral (waveforms, dashed lines; topographies, red/left side) to the side of the target. The topographic maps isolated the lateralized differences in voltage between contralateral and ipsilateral sites. These maps show the symmetrical relative differences in voltage, which is more negative over the contralateral scalp and more positive over the ipsilateral scalp. The voltage distributions are shown from posterior perspective. The color scale shows the range of possible voltage values, normalized for the topographies. Blue indicates negative voltage, and red indicates positive voltage.

In experiment 1, analysis of N2pc amplitudes showed a significant main effect of electrode side (F(1,13) = 75.01, p < 0.001), replicating the pattern of enhanced negative voltage over contralateral electrodes. The interaction of search type with electrode side was also significant (F(1,13) = 5.88, p < 0.05). Follow-up subsidiary ANOVAs testing for the N2pc in each task condition separately revealed the presence of the N2pc effect in both the VSTM (F(1,13) = 5.97, p < 0.05) and visual (F(1,13) = 137.10, p < 0.001) conditions. Direct comparison of the voltage difference between contralateral versus ipsilateral electrodes indicated that the N2pc was larger during the visual search than during VSTM search (t(13) = 2.43, p < 0.05). A significant three-way interaction (search type, electrode side, and electrode pair) (F(1.81,13) = 4.62, p < 0.05) indicated that the N2pc was largest in the visual search condition at P7/P8. There were no modulations of the N2pc by search load.

In experiment 2, the presence of the N2pc was also confirmed by a significant main effect of electrode side (F(1,11) = 7.40, p < 0.05). During color search, the N2pc effect was also significantly modulated by search type (F(1,11) = 6.68, p < 0.05). In this case, the N2pc effect was larger during VSTM search than in visual search (t(11) = 2.58, p < 0.05). A significant three-way interaction was observed involving search type, search load, and electrode side (F(1,11) = 11.90, p < 0.05). Follow-up analyses for each type of search were conducted to clarify the pattern of effects. Analysis of the visual search condition revealed a significant interaction between electrode side and load (F(1,11) = 9.70, p = 0.01). During visual search, the N2pc effect was significant only when the array contained four items (F(1,11) = 18.82, p = 0.001). The effect of electrode side was not significant for two-item arrays (p > 0.1). Analysis of the VSTM search condition revealed a main effect of electrode side (F(1,11) = 13.84, p < 0.05), which did not interact with load (p > 0.1). Overall, the results indicated that the difference in the N2pc effect between two and four items was larger during visual search (1.03 ± 1.15 μV) than during VSTM search (0.29 ± 1.19 μV) (t(1,11) = 3.45, p < 0.05).

Analyses using normalized differences between voltage over the contralateral versus ipsilateral posterior scalp showed no significant differences in the topographical distribution of the N2pc effect in the visual versus VSTM conditions of either experiment (search type × electrode pair in experiment 1, F(3.64,47.37) = 1.27, p = 0.29; experiment 2, F(2.78,30.61) = 1.13, p = 0.35).

N3RS effect

Figure 3 shows the replication of the load-dependent VSTM search N3RS effect in experiment 1 (a) and experiment 2 (b). Grand-averaged ERP waveforms are shown from representative midline electrodes (e.g., FCz, Cz, and CPz) in the two load conditions of the VSTM search tasks.

Figure 3.

Figure 3.

Open in a new tab

The grand-averaged waveforms at representative midline electrodes (FCz, Cz, and CPz) showing the N3RS during the VSTM search and its modulation by VSTM load (two-item, dashed lines; four-item, solid lines) in experiment 1 (a) and experiment 2 (b). The temporal window for N3RS analysis is indicated by shading, and arrows point to the effect.

Search load exerted a significant main effect in both experiment 1 (F(1,13) = 8.24, p < 0.05) and experiment 2 (F(1,11) = 26.73, p < 0.001). Search load did not interact with electrode position during shape-based search in experiment 1 (p > 0.1). During the color-search task in experiment 2, however, the load effect was larger over more frontal electrodes, as revealed by a significant interaction between search load and electrode position along the anteroposterior axis (F(1.258,13.83) = 11.29, p < 0.05).

Discussion

In this study, two ERP experiments revealed that selection of a target object within a search array maintained in VSTM proceeds through a mechanism that is qualitatively similar to target selection in the perceptual domain. In line with previous results, the N2pc was obtained during visual search within a perceptual array (Luck and Hillyard, 1994; Eimer, 1996). The main novel finding was that a similar N2pc was also elicited when target items were identified within a VSTM representation. The N2pc in the visual and VSTM domains had equivalent time courses and topographies, suggesting overlap in the spatially specific neural mechanisms of target selection in the two cases. Finally, we also replicated the finding of a later potential (N3RS) associated with VSTM search in a load-dependent manner (Nobre et al., 2008).

Behavioral search performance showed broadly similar patterns in the perceptual and VSTM domains. In both experiments, the number of items (load) through which the search had to be performed significantly affected sensitivity measures (d′) and RTs. Replicating well established findings, search was less accurate and slower as load increased (Sternberg, 1966; Treisman and Gelade, 1980). As expected, VSTM search performance was relatively poorer, indicating some degradation in the quality or number of items maintained in memory. Nevertheless, participants could perform all task conditions above chance levels and below ceiling.

Accurate target identification was accompanied by a similar lateralized ERP signature over the posterior scalp. A clear N2pc was elicited for identifying targets based on shape (experiment 1) or color (experiment 2), independently of whether search was performed on a visual or VSTM array. In both experiments, the N2pc showed similar onset times for the different search types and load conditions. The similar onset latency across search types suggests that the spatial selection mechanisms within VSTM triggered by the central probe can access an internalized representation that is being actively maintained and argues against the need for a time-consuming step of forming a new mental image of the array when the probe appears. The similar onset latency across search loads suggests that the neural mechanisms triggering the N2pc reflect parallel operations during search that are not contingent on target identification.

N2pc amplitude was also insensitive to search load, except for in one case. When searching for a color amid a perceptual array containing only two items (visual search condition of experiment 2), the potentials across the two sides of the posterior scalp (i.e., the N2pc amplitude) were not significantly different. This replicates the observation that the N2pc effect may be absent in very easy search conditions (Luck and Hillyard, 1994). Across all other conditions in the two experiments, the amplitude of the N2pc did not vary significantly as a function of search load (Jolicoeur et al., 2008). Our findings suggest that the parallel mechanism in question may be relatively insensitive to the amount of competition among stimuli within the search array. However, we cannot rule out that N2pc modulations would become apparent with more extensive and systematic testing of parameters related to competition among stimuli (e.g., crowding, item proximity, and similarity).

We speculate that the N2pc may reflect the spatial layout of a top-down biasing signal prioritizing feature-based or object-based information about the target identity (Desimone and Duncan, 1995). In the case of experiment 1, these biases would be based on the shape of the target object, whereas in experiment 2, they would be based on the color of the target object. The top-down biases would lead to enhanced processing at locations in which the object attributes matched the task-relevant, target attributes and/or suppression of processing at locations in which the object attributes do not match the task-relevant ones (Astle et al., 2009). The timing of the N2pc is compatible with such an effect. It precedes RTs that signal target identification by at least 100 ms and is consistent with the buildup of attentional biases of neural activity in extracellular cortex observed during object-based visual search with single-unit recordings in nonhuman primates (Chelazzi et al., 1993, 1998), especially taking into account the slower neuronal response times in human versus primate visual areas (Schroeder et al., 2001).

Identification of an equivalent N2pc effect in VSTM search as observed during visual search strongly suggests the operation of biasing mechanisms on some common or similar neuronal populations. In the first instance, it provides converging evidence for the suggestion that at least some aspect of the spatial layout of the original visual array is preserved in VSTM (Gratton, 1998; Jiang et al., 2000). However, because we did not manipulate the location of gaze or other aspects of the spatial layout, our data cannot explain the precise nature of the spatial layout that is maintained in VSTM, although this would be an interesting topic to explore in future studies.

Our findings would be most well explained by the participation of functionally specialized posterior visual and parietal areas in maintaining VSTM, possibly under the coordination of executive brain areas, such as the prefrontal cortex (D'Esposito et al., 2000; Fuster, 2000; Petrides, 2000; Passingham and Sakai, 2004; Postle, 2006). Furthermore, they suggest that VSTM-related activity in these areas can be flexibly modulated according to task goals (Ranganath et al., 2004; Gazzaley et al., 2005, 2007; Lepsien and Nobre, 2007). Under this interpretation, it is possible to envisage the sharing of some selection mechanisms across perceptual and VSTM representations. In both cases, the biasing of activity in visual or parietal areas according to target attributes would influence subsequent processes of target identification and decision making.

In the case of VSTM search, it was also possible to investigate nonlateralized brain activity that varies as a function of search load. In the two current experiments, one probe stimulus, presented at the end of the trial, triggered search through a VSTM array containing two or four items. When comparing search through four versus two items, a load-dependent negative potential distributed over midline electrodes followed the N2pc. This potential was described previously in a study using a VSTM color search (Nobre et al., 2008) and was labeled N3RS for its latency and functional implication in “retroactively searching” through a VSTM array. A similar load modulation of the N3RS was obtained in both experiments of the current study. The latency of the N3RS also varied according to the difficulty of target selection, being delayed for the slower shape-based search compared with the faster color-based search. The N3RS, therefore, is a possible marker of time-consuming evaluation of putative targets or devaluation of competing targets. These search mechanisms are presumed to operate on neural activity that is biased according to the task goals and could proceed either serially (Treisman and Gelade, 1980) or through parallel computations that take longer to resolve with increasing numbers of distractor stimuli (Duncan and Humphreys, 1989). If this interpretation is correct, one might expect to observe similar load-dependent modulations during search through visually presented, perceptual arrays. Unfortunately, in this case, it may prove difficult to compare brain activity without contamination from physical differences in the arrays (related to the number of stimuli). Although elucidating the functional underpinnings of the N2pc and N3RS is beyond the scope of the present study, our findings suggest that these potentials index distinct stages during search. Contrary to its most common interpretation, the N2pc may reflect a parallel biasing mechanism, whereas the N3RS may be more closely linked to target selection processes hitherto attributed to the N2pc.

It is intuitive to consider VSTM search to be mediated by mental imagery. Although often treated under a separate “literature,” imagery may be integral to the maintenance and retrieval of VSTM representations, as well as to the formation of anticipatory attentional biases. The spatial specificity and preservation of depictive analog qualities have been central themes in mental imagery research (Shepard and Metzler, 1971; Kosslyn et al., 1978). Furthermore, similarly for what we propose for VSTM, imagery representations are thought to rely on activity in some of the same neural populations that are activated during perception (Ganis et al., 2004; Stokes et al., 2009). The degree to which the spatiotopic attentional biases we observed during VSTM search depend on a mediating role of mental imagery remains to be tested. To the extent that our task engaged mental imagery, our findings lend additional support to the preservation of an analog spatial representation and to a sharing of common neural mechanisms with perception. We are currently testing whether spatiotopic selection biases in VSTM can also occur in the absence of mental imagery.

In summary, we identified similar spatially specific neural markers during search for target items within perceptual and VSTM arrays. Our findings suggest that shared spatially organized activity participating in perceptual and VSTM representations is modulated by similar attentional biasing mechanisms that lead to target selection. Our findings bolster the growing notion that VSTM representations are not merely the end product of perception but are themselves highly flexible and under the control of changing task goals and expectations (Lepsien et al., 2005; Lepsien and Nobre, 2006; Makovski et al., 2008; Nobre et al., 2008; Sligte et al., 2008).

Footnotes

The research was supported by an award from the James S. McDonnell Foundation and a project grant from the Wellcome Trust (A.C.N.). B.-C.K. was supported by the Scholarship of Graduate Students Study Abroad Program by the National Science Council in Taiwan. We thank Dr. Duncan Astle for his comments and suggestions on a previous version of this manuscript.

References

  1. AEEGS. American Electroencephalographic Society guidelines for standard electrode position nomenclature. J Clin Neurophysiol. 1991;8:200–202. [PubMed] [Google Scholar]
  2. Astle DE, Scerif G, Kuo B, Nobre AC. Spatial selection of features within perceived and remembered objects. Front Hum Neurosci. 2009;3:6. doi: 10.3389/neuro.09.006.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Awh E, Jonides J. Overlapping mechanisms of attention and spatial working memory. Trends Cogn Sci. 2001;5:119–126. doi: 10.1016/s1364-6613(00)01593-x. [DOI] [PubMed] [Google Scholar]
  4. Brisson B, Jolicoeur P. The N2pc component and stimulus duration. Neuroreport. 2007;18:1163–1166. doi: 10.1097/WNR.0b013e3281e72d1b. [DOI] [PubMed] [Google Scholar]
  5. Chelazzi L, Miller EK, Duncan J, Desimone R. A neural basis for visual search in inferior temporal cortex. Nature. 1993;363:345–347. doi: 10.1038/363345a0. [DOI] [PubMed] [Google Scholar]
  6. Chelazzi L, Duncan J, Miller EK, Desimone R. Responses of neurons in inferior temporal cortex during memory-guided visual search. J Neurophysiol. 1998;80:2918–2940. doi: 10.1152/jn.1998.80.6.2918. [DOI] [PubMed] [Google Scholar]
  7. Curtis CE, D'Esposito M. Persistent activity in the prefrontal cortex during working memory. Trends Cogn Sci. 2003;7:415–423. doi: 10.1016/s1364-6613(03)00197-9. [DOI] [PubMed] [Google Scholar]
  8. Desimone R, Duncan J. Neural mechanisms of selective visual attention. Annu Rev Neurosci. 1995;18:193–222. doi: 10.1146/annurev.ne.18.030195.001205. [DOI] [PubMed] [Google Scholar]
  9. D'Esposito M, Postle BR, Rypma B. Prefrontal cortical contributions to working memory: evidence from event-related fMRI studies. Exp Brain Res. 2000;133:3–11. doi: 10.1007/s002210000395. [DOI] [PubMed] [Google Scholar]
  10. Duncan J, Humphreys GW. Visual search and stimulus similarity. Psychol Rev. 1989;96:433–458. doi: 10.1037/0033-295x.96.3.433. [DOI] [PubMed] [Google Scholar]
  11. Eimer M. The N2pc component as an indicator of attentional selectivity. Electroencephalogr Clin Neurophysiol. 1996;99:225–234. doi: 10.1016/0013-4694(96)95711-9. [DOI] [PubMed] [Google Scholar]
  12. Endo N, Saiki J, Nakao Y, Saito H. Perceptual judgments of novel contour shapes and hierarchical descriptions of geometrical properties (in Japanese) Shinrigaku Kenkyu. 2003;74:346–353. doi: 10.4992/jjpsy.74.346. [DOI] [PubMed] [Google Scholar]
  13. Engelmann JB, Pessoa L. Motivation sharpens exogenous spatial attention. Emotion. 2007;7:668–674. doi: 10.1037/1528-3542.7.3.668. [DOI] [PubMed] [Google Scholar]
  14. Fuster JM. Executive frontal functions. Exp Brain Res. 2000;133:66–70. doi: 10.1007/s002210000401. [DOI] [PubMed] [Google Scholar]
  15. Ganis G, Thompson WL, Kosslyn SM. Brain areas underlying visual mental imagery and visual perception: an fMRI study. Brain Res Cogn Brain Res. 2004;20:226–241. doi: 10.1016/j.cogbrainres.2004.02.012. [DOI] [PubMed] [Google Scholar]
  16. Gazzaley A, Cooney JW, McEvoy K, Knight RT, D'Esposito M. Top-down enhancement and suppression of the magnitude and speed of neural activity. J Cogn Neurosci. 2005;17:507–517. doi: 10.1162/0898929053279522. [DOI] [PubMed] [Google Scholar]
  17. Gazzaley A, Rissman J, Cooney J, Rutman A, Seibert T, Clapp W, D'Esposito M. Functional interactions between prefrontal and visual association cortex contribute to top-down modulation of visual processing. Cereb Cortex. 2007;17:i125–i135. doi: 10.1093/cercor/bhm113. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Gratton G. The contralateral organization of visual memory: a theoretical concept and a research tool. Psychophysiology. 1998;35:638–647. [PubMed] [Google Scholar]
  19. Green DM, Swets JA. Huntington, NY: Krieger; 1966. Signal detection theory and psychophysics. [Google Scholar]
  20. Griffin IC, Nobre AC. Orienting attention to locations in internal representations. J Cogn Neurosci. 2003;15:1176–1194. doi: 10.1162/089892903322598139. [DOI] [PubMed] [Google Scholar]
  21. Guthrie D, Buchwald JS. Significance testing of difference potentials. Psychophysiology. 1991;28:240–244. doi: 10.1111/j.1469-8986.1991.tb00417.x. [DOI] [PubMed] [Google Scholar]
  22. Harrison SA, Tong F. Decoding reveals the contents of visual working memory in early visual areas. Nature. 2009;458:632–635. doi: 10.1038/nature07832. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Hickey C, Di Lollo V, McDonald JJ. Electrophysiological indices of target and distractor processing in visual search. J Cogn Neurosci. 2009;21:760–775. doi: 10.1162/jocn.2009.21039. [DOI] [PubMed] [Google Scholar]
  24. Hopf JM, Luck SJ, Girelli M, Hagner T, Mangun GR, Scheich H, Heinze HJ. Neural sources of focused attention in visual search. Cereb Cortex. 2000;10:1233–1241. doi: 10.1093/cercor/10.12.1233. [DOI] [PubMed] [Google Scholar]
  25. Jennings JR, Wood CC. Letter: the epsilon-adjustment procedure for repeated-measures analyses of variance. Psychophysiology. 1976;13:277–278. doi: 10.1111/j.1469-8986.1976.tb00116.x. [DOI] [PubMed] [Google Scholar]
  26. Jiang Y, Olson IR, Chun MM. Organization of visual-short term memory. J Exp Psychol Learn Mem Cogn. 2000;26:683–702. doi: 10.1037//0278-7393.26.3.683. [DOI] [PubMed] [Google Scholar]
  27. Jolicoeur P, Brisson B, Robitaille N. Dissociation of the N2pc and sustained posterior contralateral negativity in a choice response task. Brain Res. 2008;1215:160–172. doi: 10.1016/j.brainres.2008.03.059. [DOI] [PubMed] [Google Scholar]
  28. Kiss M, Van Velzen J, Eimer M. The N2pc component and its links to attention shifts and spatially selective visual processing. Psychophysiology. 2008;45:240–249. doi: 10.1111/j.1469-8986.2007.00611.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Kosslyn SM, Ball TM, Reiser BJ. Visual images preserve metric spatial information: evidence from studies of image scanning. J Exp Psychol Hum Percept Perform. 1978;4:47–60. doi: 10.1037//0096-1523.4.1.47. [DOI] [PubMed] [Google Scholar]
  30. Landman R, Spekreijse H, Lamme VAF. Large capacity storage of integrated objects before change blindness. Vision Res. 2003;43:149–164. doi: 10.1016/s0042-6989(02)00402-9. [DOI] [PubMed] [Google Scholar]
  31. Lepsien J, Nobre AC. Cognitive control of attention in the human brain: insights from orienting attention to mental representations. Brain Res. 2006;1105:20–31. doi: 10.1016/j.brainres.2006.03.033. [DOI] [PubMed] [Google Scholar]
  32. Lepsien J, Nobre AC. Attentional modulation of object representations in working memory. Cereb Cortex. 2007;17:2072–2083. doi: 10.1093/cercor/bhl116. [DOI] [PubMed] [Google Scholar]
  33. Lepsien J, Griffin IC, Devlin JT, Nobre AC. Directing spatial attention in mental representations: interactions between attentional orienting and working-memory load. Neuroimage. 2005;26:733–743. doi: 10.1016/j.neuroimage.2005.02.026. [DOI] [PubMed] [Google Scholar]
  34. Luck SJ, Hillyard SA. Spatial filtering during visual search: evidence from human electrophysiology. J Exp Psychol Hum Percept Perform. 1994;20:1000–1014. doi: 10.1037//0096-1523.20.5.1000. [DOI] [PubMed] [Google Scholar]
  35. Makovski T, Sussman R, Jiang YV. Orienting attention in visual working memory reduces interference from memory probes. J Exp Psychol Learn Mem Cogn. 2008;34:369–380. doi: 10.1037/0278-7393.34.2.369. [DOI] [PubMed] [Google Scholar]
  36. McCarthy G, Wood CC. Scalp distributions of event-related potentials: an ambiguity associated with analysis of variance models. Electroencephalogr Clin Neurophysiol. 1985;62:203–208. doi: 10.1016/0168-5597(85)90015-2. [DOI] [PubMed] [Google Scholar]
  37. Murray MM, Wylie GR, Higgins BA, Javitt DC, Schroeder CE, Foxe JJ. The spatiotemporal dynamics of illusory contour processing: combined high-density electrical mapping, source analysis, and functional magnetic resonance imaging. J Neurosci. 2002;22:5055–5073. doi: 10.1523/JNEUROSCI.22-12-05055.2002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Nobre AC, Griffin IC, Rao A. Spatial attention can bias search in visual short-term memory. Front Hum Neurosci. 2008;1:1–9. doi: 10.3389/neuro.09.004.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Oldfield RC. The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia. 1971;9:97–113. doi: 10.1016/0028-3932(71)90067-4. [DOI] [PubMed] [Google Scholar]
  40. Passingham D, Sakai K. The prefrontal cortex and working memory: physiology and brain imaging. Curr Opin Neurobiol. 2004;14:163–168. doi: 10.1016/j.conb.2004.03.003. [DOI] [PubMed] [Google Scholar]
  41. Petrides M. The role of the mid-dorsolateral prefrontal cortex in working memory. Exp Brain Res. 2000;133:44–54. doi: 10.1007/s002210000399. [DOI] [PubMed] [Google Scholar]
  42. Postle BR. Working memory as an emergent property of the mind and brain. Neuroscience. 2006;139:23–38. doi: 10.1016/j.neuroscience.2005.06.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Ranganath C, Cohen MX, Dam C, D'Esposito M. Inferior temporal, prefrontal, and hippocampal contributions to visual working memory maintenance and associative memory retrieval. J Neurosci. 2004;24:3917–3925. doi: 10.1523/JNEUROSCI.5053-03.2004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Schroeder CE, Mehta AD, Foxe JJ. Determinants and mechanisms of attentional modulation of neural processing. Front Biosci. 2001;6:D672–D684. doi: 10.2741/schroed. [DOI] [PubMed] [Google Scholar]
  45. Shepard RN, Metzler J. Mental rotation of three-dimensional objects. Science. 1971;171:701–703. doi: 10.1126/science.171.3972.701. [DOI] [PubMed] [Google Scholar]
  46. Sligte IG, Scholte HS, Lamme VAF. Are there multiple visual short-term memory stores? PLoS ONE. 2008;3:e1699. doi: 10.1371/journal.pone.0001699. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Sternberg S. High-speed scanning in human memory. Science. 1966;153:652–654. doi: 10.1126/science.153.3736.652. [DOI] [PubMed] [Google Scholar]
  48. Stokes M, Thompson R, Cusack R, Duncan J. Top-down activation of shape-specific population codes in visual cortex during mental imagery. J Neurosci. 2009;29:1565–1572. doi: 10.1523/JNEUROSCI.4657-08.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Treisman AM, Gelade G. A feature-integration theory of attention. Cognit Psychol. 1980;12:97–136. doi: 10.1016/0010-0285(80)90005-5. [DOI] [PubMed] [Google Scholar]
  50. Treue S. Visual attention: the where, what, how and why of saliency. Curr Opin Neurobiol. 2003;13:428–432. doi: 10.1016/s0959-4388(03)00105-3. [DOI] [PubMed] [Google Scholar]

Articles from The Journal of Neuroscience are provided here courtesy of Society for Neuroscience

RESOURCES