Skip to main content
PMC home page
The Journal of Neuroscience logoLink to The Journal of Neuroscience
. 2011 Apr 20;31(16):6199–6207. doi: 10.1523/JNEUROSCI.6453-10.2011

Visual Search Demands Dictate Reliance on Working Memory Storage

Roy Luria 1,✉, Edward K Vogel 1
PMCID: PMC3758569  PMID: 21508243

Abstract

Previous research suggested that working memory (WM) does not play any significant role in visual search. In three experiments, we investigated the search difficulty and individual differences in WM capacity as determinants of WM involvement during visual search tasks, using both behavioral and electrophysiological markers [i.e., the contralateral delay activity (CDA), which is a marker for WM capacity allocation]. Human participants performed a visual search task that contained a target, neutral distractors, and a flanker distractor. Overall, we found that, as the search difficultly increased (as indicated by longer reaction times), so did the role of WM in performing the search task (as indicated by larger CDA amplitudes). Moreover, the results pinpoint a dissociation between the two types of factors that determined the WM involvement in the search process. Namely, individual differences in WM capacity and search difficulty independently affected the degree to which the search process relied on WM. Instead of showing a progressive role, individual differences in WM capacity correlated with the search efficiency in all search conditions (i.e., easy, medium, and difficult). Counterintuitively, individuals with high WM capacity generally relied less on WM during the search task.

Introduction

Models of visual search argue that a range of cognitive mechanisms must work in concert to help us find targets among distractors in the visual environment. Aside from the obvious perceptual and attentional processing demands, several models have also proposed that visual working memory (WM) is critical for a number of important operations during visual search, such as representing the target template, comparing the target template to possible candidate items, and categorizing the items in the search array (Duncan and Humphreys, 1989; Bundesen, 1990; Desimone and Duncan, 1995). Consistent with this view, evidence from several primate electrophysiological studies has indicated that, during visual search, neurons that are selective for the search target often remain active during a delay period before the onset of the search array, presumably indicating a neural correlate of a target template being actively maintained in memory. Moreover, cells in inferotemporal cortex also show enhanced firing rates during search itself, just before a saccadic eye movement toward that target (Chelazzi et al., 1993). Interestingly, the same brain areas show template-related activity during the delay period, followed by an enhanced response to a matching target during visual WM tasks (Miller and Desimone, 1993, 1994), which led some to argue that visual search is just a variant of a working memory task (Desimone and Duncan, 1995).

These proposals have received limited empirical support from human research. For example, Woodman et al. (2001) used a dual-task approach to investigate the reliance on WM during visual search, by having participants perform both a visual search task and a change detection task (intended to fill visual WM) on each trial. Surprisingly, they found that visual search efficiency was not impaired while participants were concurrently engaged in a demanding WM task, suggesting that WM plays a nonsignificant role during visual search. Other dual-task studies have found some evidence for the involvement of WM during visual search but only when the target varied from trial to trial (Woodman et al., 2007) or when spatial WM was filled (Oh and Kim, 2004; Woodman and Luck, 2004) rather than the object WM task used by Woodman et al. (2001b). However, from these dual-task approaches alone, it is difficult to assess the extent to which WM was used during search because dual-task costs might also reflect processes necessary to coordinate the two tasks (to reduce interference) or even the reliance of offline long-term memory to help shoulder some of the WM demands.

The current work aimed to assess and quantify the reliance on WM storage during visual search by using a single visual search task while measuring an event-related potential (ERP) component that reflects the amount of information that is currently active in WM (Vogel and Machizawa, 2004). Thus, the focus of the current study was to measure how much information (e.g., targets and distractors) from a visual search display is represented in WM during the search process. Moreover, we examined this relationship across various levels of search difficulty: when the search was easy (“pop-out”), medium, and difficult, and when the target position was cued in advance. In addition, to further characterize this link between WM and visual search, we examined how individuals differed in their WM and search abilities.

Materials and Methods

Overview.

The current study used a perceptual search task with easy, medium, and difficult search conditions (Lavie and Cox, 1997; Johnson et al., 2002), in which subjects had to indicate which one of the two possible target letters was present in a circular display (Fig. 1A,B).

Figure 1.

Figure 1.

Open in a new tab

A, The experiment procedure used in experiment 1. In the actual experiment, we used green and red colors, which are represented in the figure using shades of gray. B, The various search conditions used across experiments. Subjects had to identify whether the letter Z or X was present in the imaginary circle array and ignore anything outside the circle. Only the red or the green side was attended.

To quantify the online role of WM during the visual search task, we used an electrophysiological measure of WM capacity, the contralateral delay activity (CDA) (Vogel and Machizawa, 2004). The CDA has been shown to be an excellent marker for visual WM capacity allocation during change detection tasks (Vogel et al., 2005; McCollough et al., 2007). Importantly, the CDA has also been observed in tasks that required a mental manipulation of visual information even without an explicit delay period (Jolicoeur et al., 2006, 2008; Brisson and Jolicoeur, 2007; Drew and Vogel, 2008; Prime and Jolicoeur, 2010). For example, Drew and Vogel (2008) measured the CDA in a multiple object tracking task, while Prime and Jolicoeur (2010) used the CDA as an index of WM involvement during a mental rotation task. Consequently, the CDA can be a useful and sensitive measure of the reliance on WM storage even in situations without explicit memory requirements, and, in the present study, we will take advantage of this property in the context of a visual search task.

The goal of experiment 1 was to verify that WM plays a role during the current visual search paradigm. We used a difficult search task in which the subject must find a predefined letter among five unique letter distractors. In one condition, the position of the target was cued in advance, eliminating any ambiguity as to where the target would appear and hence greatly reducing the need to search for the target among the distractors (Fig. 1A). By reducing the search difficulty in this way, we expect that the demands for representing information in WM should be greatly decreased, which would result in a reduction in the CDA amplitude that is accompanied by faster response times. Moreover, such a relationship would predict that the individuals who could better use this spatial cue for the target position would rely less on WM during the search task. Consequently, the “cuing benefit” on response time should correlate with CDA amplitude. In experiments 2 and 3, we used a similar approach to examine whether WM plays an increasing role in visual search as the search becomes more difficult by manipulating the uniqueness of the distractors in the search array, which is well known to increase overall response times during search (Duncan and Humphreys, 1989). Instead of using a cue that eliminated the need to search (as in experiment 1), in experiments 2 and 3, we manipulated the search difficulty. Assuming that WM plays an important role during the search process, we should expect to find an increase in WM involvement as the search becomes more difficult.

In addition to measuring the sheer quantity of information represented in WM across these search difficulty manipulations, experiments 2 and 3 also aimed to examine whether the identity of the distractors within the array determined how much information was represented in WM during the search process. We measured this in two ways. First, as described above, we manipulated search difficulty by increasing the uniqueness of the distractors within the search array as a means to evaluating whether an increasing number of distinct distractor identities leads to an overall larger reliance on representing more information in WM. Second, on some trials, the circular search array was accompanied by a peripheral distractor (flanker) that either matched the target for that trial (compatible), matched the incorrect target (incompatible), or was not present (no flanker). By comparing the CDA amplitude for trials with and without the flanker, we can evaluate under which search conditions the peripheral flanker stimulus was represented in WM. To distinguish between the two types of distractors, in the following sections, we will refer to the peripheral distractor as the “flanker” and to the letter distractors around the target as the “distractors.”

Participants.

All participants gave informed consent following the procedures of a protocol approved by the Human Subjects Committee at the University of Oregon. All subjects were members of the University of Oregon community and were paid $10 per hour for participation. Each experiment had 18 participants in the final analysis (8 females and 10 males in experiment 1, 6 females and 12 males in experiment 2, and 12 females and 8 males in experiment 3). Subjects with >25% rejection rate attributable to eye blink or eye movement were replaced (one in experiment 1 and two in experiment 2).

Measuring visual working memory capacity.

In each experiment, participants first completed a behavior-only visual working memory task before starting the ERP experiment. The working memory task consisted of a change detection task with arrays of four, six, and eight colored squares with a 1 s retention interval (Luck and Vogel, 1997; Vogel et al., 2001). We computed each individual's visual memory capacity with a standard formula (Pashler, 1988; Cowan, 2001). The formula is K = S(H − F), where K is the memory capacity, S is the size of the array, H is the observed hit rate, and F is the false alarm rate.

Electroencephalography recording.

ERPs were recorded in each experiment using our standard recording and analysis procedures (McCollough et al., 2007), including rejection of trials contaminated by blinks or large (>1°) eye movements. We recorded from 22 standard electrode sites spanning the scalp, including international 10/20 sites F3, F4, C3, C4, P3, P4, O1, O2, PO3, PO4, T5, and T6, as well as nonstandard sites occipital left (OL) and occipital right (OR) (midway between O1/2 and T5/6). The horizontal electrooculogram (EOG) was recorded from electrodes placed 1 cm to the left and right of the external canthi to measure horizontal eye movement, and the vertical EOG was recorded from an electrode beneath the right eye referenced to the left mastoid to detect blinks and vertical eye movements. Trials containing ocular artifacts, movement artifacts, or amplifier saturation were excluded from the averaged ERP waveforms. Furthermore, participants who had >25% of trial rejections in any condition were excluded from the analysis. The electroencephalography and EOG were amplified by an SA Instrumentation amplifier with a bandpass of 0.01–80 Hz (half-power cutoff, Butterworth filters) and were digitized at 250 Hz by a personal computer-compatible microcomputer.

Experiment 1: stimuli and procedure.

Each trial started with the presentation of a fixation point (“+”) in the middle of the screen for 500 ms. Then, spatial cues (“−”) identified the stimuli positions (Fig. 1A). In the “neutral” condition, all positions were cued. In the target cued condition, only the future target position was cued (please note that, although the cue was 100% valid, we actually cued two positions, one at each side). The cues were presented for 500 ms, followed by a fixation-only display (250 ms), followed by the visual search display that flashed for 500 ms. The next trial started 800 ms after the response of the preceding trial. The visual search array consisted of a bilateral display in which each side included six letters, arranged to form a circle. One side of the display was red, and the other was green (randomly determined). Half of the subjects searched for the target on the green side, and the other half on the red side. One of the letters was either a X or a Z, and the subjects had to indicate which of these target letters were presented in each trial (Fig. 1A). The letters M, N, K, S, and V (the exact order was randomly determined) occupied the remaining five positions. The design of experiment 1 resembles a study by Johnson et al. (2002) with the exception that we did not use flanker stimuli in experiment 1.

Each participant performed 16 trials of practice in each cue condition, followed by 16 blocks of 80 trials each (eight blocks for each cue condition).

Experiments 2 and 3: stimuli and procedure.

Except as note below, all other procedures were identical to experiment 1. In experiment 2, we used the difficult visual search condition (identical to experiment 1) and an easy (pop-out) search condition. Experiment 3 involved the same easy condition as in experiment 2 and a medium visual search condition (Fig. 1B). In both the easy and medium search conditions, the visual search array flashed for 200 ms. The difficult search condition (experiment 2) was flashed for 500 ms.

In the easy search condition, an O was presented as a distractor in all the remaining positions (Fig. 1B, left). In the medium search condition, the letters M, N, and K occupied three of the remaining five positions, and the other two positions were occupied with O (Fig. 1B, left). The exact position of the target and the distractor letters was randomly determined in each trial. In addition, in 75% of the trials, a bilateral flanker letter was presented outside the circle. The identity of the flanker letter was either an X or Z (randomly determined), and subjects were instructed to ignore it (Fig. 1B, left and right). On the other 25% of trials, no flanker was presented so that the display consisted of only a circle of six letters on both sides of fixation (Fig. 1B, middle).

Each participant performed 16 trials of practice in each experimental condition, followed by 16 blocks of 80 trials each (eight blocks for each condition).

CDA analysis.

The raw EEG wave was segmented into 900 ms epochs starting 200 ms before the search display array onset. Only correct trials were included in the analysis. Separate average waveforms for each condition were then generated, and difference waves were constructed by subtracting the average activity recorded from the electrodes ipsilateral to the memorized array from the average activity recorded from electrodes contralateral to the memorized array. For all statistical purposes that were related to the CDA, we used the averaged activity between 350 and 600 ms time locked to onset of the search display.

We will only present the results from the OL/OR electrodes because that is where the CDA amplitude is most evident, but the exact same pattern of results were observed over the O1/O2, P7/P8, and PO3/PO4 pairs of electrodes.

CDA and flanker size correlations (experiments 2 and 3).

To estimate whether the flanker stimulus was represented in visual WM and whether this representation correlated with the behavioral flanker effect, we subtracted the CDA amplitude during the no-flanker condition from the compatible and incompatible conditions for each subject. Behavioral flanker size was calculated for each individual level by subtracting reaction time (RT) in the incompatible condition from RT in the compatible condition.

Results

Experiment 1

Behavioral results

Target cued trials were 124 ms faster than neutral trials (1047 and 923 ms for the neutral and the cue-valid conditions, respectively; t(17) = 6.35; p < 0.0001; the confidence interval is 28 ms), indicating that the search was considerably easier in the target cued condition.

Electrophysiological results

As can be seen in Figure 2, the CDA had a lower amplitude in the target cued condition relative to the neutral condition (t(17) = 5.67; p < 0.0001; 0.03 and −0.069 μV for the target cued and neutral conditions, respectively; the confidence interval is 0.18 μV), demonstrating that fewer items were recorded in visual WM in the target cued condition (Fig. 2A). This confirms that WM played a significant role only when a search was needed.

Figure 2.

Figure 2.

Open in a new tab

Results for experiment 1. A, The CDA for the neutral (all positions cued) and the target cued conditions, time locked to the stimulus array. The gray bar above the x-axis indicates the stimuli presentation time. B, Correlation between (CDA neutral − CDA target cued) and (RT neutral − RT target cued).

Correlations between CDA and behavioral performance

The cuing benefit in RT (RT neutral − RT cue-valid) correlated with the respective CDA difference (i.e., CDA neutral − CDA valid; r = −0.66; p < 0.005), such that the subjects who showed a greater RT benefit from the cue were more likely to show a corresponding CDA reduction during search (Fig. 2B). This relationship indicates that the subjects who were using the cue less efficiently were also relying on WM to a greater extent during search. Along similar lines, we also tested for a relationship between an individual's WM capacity (measured in the change detection procedure) and his or her reliance on WM storage during the visual search task (measured again as the reduction in CDA amplitude between the cued and neutral conditions). Interestingly, WM capacity strongly correlated with the cue-related CDA difference (r = 0.51; p < 0.05), such that high-capacity individuals showed a larger cue-related CDA reduction than low-capacity individuals (note that, because in experiments 2 and 3, the “raw” CDA amplitude correlated with WM capacity in all conditions, we used a partial correlation controlling for the initial CDA amplitude). In other words, low WM individuals were less able to use the cue to reduce the involvement of WM during the search process than their high-capacity counterparts. This observed relationship is highly consistent with previous reports demonstrating an inverse relationship between memory capacity and the efficiency of attentional filtering (Vogel et al., 2005; Fukuda and Vogel, 2009, 2011).

The results of experiment 1 demonstrated that search depends on WM primarily when active search is required to find the target. It further established a direct link between the reliance on WM storage (as indicated by the CDA) and the overall RT improvement when a cue minimized the need to search among the distractors (Emrich Al-Aidoors et al., 2009), such that faster searches rely less on WM. In experiments 2 and 3, we further test the hypothesis that WM plays an increased role in visual search as finding the target becomes more difficult, at both the task and the individual level.

Experiments 2 and 3

Experiment 2 compared easy (pop-out) and difficult (identical to experiment 1) search tasks. Experiment 3 included the same easy (pop-out) search condition as in experiment 2 and a medium-difficulty search condition (Fig. 1B). If WM is indeed progressively involved in visual search as the search becomes harder, we would expect comparable increases in search RT and CDA amplitudes. In addition, the search display (in all conditions) sometimes included a flanker stimuli [whose identity could either match or mismatch the target (Lavie and Cox, 1997)] that appeared outside the “circle” of possible targets. Participants were instructed to ignore the flanker and to only look for the target within the circular display of items.

We first analyzed the behavioral results to verify that the search difficulty manipulation resulted in longer RTs, comparing the easy search condition to the difficult (experiment 2) and medium (experiment 3) search conditions. We then examined whether the same pattern was evident in the CDA amplitude as expected if the visual search difficulty dictates the degree of WM involvement. In the third part of Results, we analyzed whether the CDA amplitude and accuracy data correlate with individual differences in WM capacity. Finally, we analyzed whether the identity of the flanker influenced search performance and CDA amplitudes.

Experiment 2

Behavioral results

As expected, RT was 237 ms faster in the easy pop-out condition relative to the difficult search condition [763 and 1001 ms in the easy and difficult search conditions, respectively; F(1,17) = 95.59; mean square error (MSE) = 18,342.68; p < 0.001; the confidence interval is 22 ms].

Electrophysiological results

Mean CDA amplitude in a time window of 350–600 post-search array was taken as a dependent variable. Similar to the RT analysis, the CDA mean amplitude was higher in the difficult search condition (−1.56 μV) relative to the easy search condition (−0.39 μV; the confidence interval is 0.27 μV; F(1,17) = 37.06; MSE = 0.99; p < 0.001) (Fig. 3), reflecting that WM was more engaged when the visual search task was difficult.

Figure 3.

Figure 3.

Open in a new tab

Results for experiment 2. The CDA time locked to the onset of the stimulus array in the easy and difficult search conditions for the OL/OR electrode. The gray bar above the x-axis indicates the stimuli presentation time for the difficult condition. The gray bar below the x-axis indicates the stimuli presentation time for the easy search condition.

The present results allowed us to further test our argument: if indeed the CDA is a reliable measure of WM involvement during the search process, then even within the difficult search condition, the trials in which the subject found the target quickly (fast RT) should show smaller CDA amplitudes relative to the trials in which they took longer to find the target (slow RT). We argue that the fast–slow RT difference should also be mirrored in the CDA amplitude because, presumably on fast trials, WM would be engaged to a lesser extent. To test this prediction, we divided the RT distribution for each participant into the fast and slow trials (25 vs 75 percentile) and analyzed the CDA amplitude separately for these conditions. The data from the difficult search are the most informative for this analysis, because this is the condition in which WM was needed to perform the search (whereas the easy search exhibited almost no CDA amplitude). RT in for the 25 percentile was 781 ms, and RT in the 75 percentile was 1207 ms. As can be seen in Figure 4, the CDA was initially comparable between the fast and slow search trials but then appeared to become considerably smaller for the fast trials as the trial progressed. We measured this time course effect by comparing the two types of trials over two CDA time windows: 350–500 and 550–700 ms. Although fast and slow trial amplitudes were not significantly different over the 350–500 ms window (t(17) = 0.27; p > 0.78), the CDA for fast search trials was significantly smaller than the slow search trials in the 550–700 ms time window (t(17) = 3.43; p < 0.01). This general pattern is consistent with the view that, on both types of trials, there is initially a heavy reliance on WM but that if the target is found quickly these WM resources are released.

Figure 4.

Figure 4.

Open in a new tab

The CDA amplitude for fast and slow trials in the difficult search conditions in experiment 2. The gray bar above the x-axis indicates the stimuli presentation time.

WM capacity and CDA/accuracy correlations

WM capacity positively correlated with the CDA amplitude (r = 0.58, p < 0.05 and r = 0.60, p < 0.05 for both the easy and difficult search conditions, respectively) (Fig. 5A,B). These relationships indicate that low WM individuals rely more on WM to perform the search task. Presumably, high WM capacity participants were able to efficiently localize the target and minimize their need for WM storage during search, whereas low WM capacity participants were less efficient in localizing the target and consequently held more (irrelevant) information from the search display in WM. The difference in search efficiency was also reflected in a correlation between WM capacity in the individual accuracy level in the difficult condition (r = 0.45; p = 0.059), indicating that the high WM individuals were more accurate in finding the target.

Figure 5.

Figure 5.

Open in a new tab

A, Correlation between the CDA in the easy search condition and working memory capacity in experiment 2. B, Correlation between the CDA in the difficult search condition and working memory capacity in experiment 2.

Flanker correlations

First, we analyzed whether the flanker affected search performance and whether this effect was dependent on the search difficulty. An ANOVA with the independent variables of search (easy vs difficult) and compatibility (compatible, incompatible and no-flanker) and RT as a dependent variable yielded significant main effects of search (F(1,17) = 95.59; MSE = 18,342.68; p < 0.001) and compatibility (F(2,34) = 7.37; MSE = 810.56; p < 0.005). Importantly, the interaction between these variables was also significant (F(2,34) = 4.22; MSE = 787.87; p < 0.05; confidence interval is 13 ms), indicating that the 21 ms flanker effect (incompatible vs compatible) was significant in the easy search condition (F(1,17) = 15.49; MSE = 243.335; p < 0.005) but was not significant in difficult search condition (F < 1, 5 ms) (Fig. 6A). Thus, the flanker only affected behavior in the easy search condition. This pattern of results nicely replicate those of Lavie and Cox (1997).

Figure 6.

Figure 6.

Open in a new tab

A, RT in the easy and difficult search conditions, for the flanker incongruent, congruent, and neutral trials in experiment 2. Error bars represent confidence intervals. B, Correlation between (CDA compatible − CDA neutral) and the flanker effect in experiment 2. diff, Difference. *p < 0.005.

In terms of the CDA, when search was easy, the amplitudes of the compatible and incompatible conditions were higher than the no-flanker condition (the CDA amplitude was −0.20 μV in the no-flanker condition, −0.41 μV in the compatible condition, and −0.56 μV in the incompatible condition), a difference that was only marginally significant (F(1,17) = 3.26, MSE = 0.13, p = 0.08 and F(1,17) = 3.62, MSE = 1.41, p = 0.07 for the compatible and incompatible condition, respectively). Thus, the CDA analysis indicated that the flanker was stored in WM during the easy search condition. We then isolated the flanker representation by subtracting the CDA amplitude of the compatible and incompatible conditions from the CDA amplitude of the no-flanker conditions (i.e., CDA compatible − CDA no-flanker and CDA incompatible − CDA no-flanker) and correlated these difference values to the RT flanker effect (RT incompatible − RT compatible). We found a significant correlation between the compatible CDA difference (i.e., compatible − no flanker) and the flanker compatibility effect on search RT (RT incompatible − RT compatible) (r = −0.51; p < 0.05) (Fig. 6B). Thus, individuals who showed an increased CDA on compatible trials relative to no-flanker trials also showed large flanker compatibility effects on search performance and vice versa. Such a relationship helps to demonstrate that the identity of information (i.e., the flanker) that was represented in WM affected visual search behavior. Surprisingly, no significant correlations were found between the flanker compatibility effect on RT and the incompatible CDA difference (r = −0.15; p = 0.56). Previous evidence has suggested that the incompatible flanker condition primarily correlates with response related activity, which is at a later processing stage than WM (Gratton et al., 1988, 1992). Furthermore, in the difficult search condition, no significant correlations were observed for either compatible (r = −0.24; p = 0.32) and incompatible (r = −0.01; p = 0.96) difference waves. The lack of correlations under the difficult search condition is to be expected because the behavioral data did not show a significant flanker compatibility effect.

Experiment 3

Behavioral results

RT was 120 ms faster in the easy pop-out condition relative to the medium search condition (658 and 758 ms for the easy and medium search conditions, respectively; F(1,17) = 89.38; MSE = 4371.06; p < 0.001; confidence interval is 32 ms), indicating that finding the target was indeed more difficult in the medium visual search condition.

Electrophysiological results

Mean CDA amplitude in a time window of 350–600 post-search array was taken as a dependent variable. WM was engaged in the medium search condition to a greater extent relative to the easy condition (F(1,17) = 12.34; MSE = 0.33; p < 0.001), as indicated by a higher CDA mean amplitude in the medium search condition (−1.11 μV) relative to the easy search condition (−0.44 μV) (Fig. 7).

Figure 7.

Figure 7.

Open in a new tab

Results for experiment 3. The CDA time locked to the onset of the stimulus array in the easy and medium search conditions for the OL/OR electrode. The gray bar above the x-axis indicates the stimuli presentation time.

WM capacity and CDA/accuracy correlations

In both the easy and medium search conditions, WM capacity significantly correlated with the CDA amplitude (r = 0.43, p = 0.07 and r = 0.55, p < 0.05 for the easy and the medium searches, respectively) (Fig. 8A,B). Thus, regardless of the search difficulty, high WM capacity participants relied less on WM during the search than did the low-capacity participants. Presumably, high WM individuals were more efficient in locating the target, whereas low-capacity individuals also encoded irrelevant distractor items, as was also reflected by a correlation between WM capacity and accuracy in the medium condition (p = 0.61; p < 0.05), indicating that high WM individuals were more accurate in finding the target.

Figure 8.

Figure 8.

Open in a new tab

A, Correlation between the CDA in the easy search condition and working memory capacity in experiment 3. B, Correlation between the CDA in the medium search condition and working memory capacity in experiment 3.

Flanker correlations

First, we analyzed whether the flanker affected the search performance and whether this effect was dependent on the search difficulty. An ANOVA with the same variables as in experiment 2 yielded significant main effects of search (F(1,17) = 89.38; MSE = 4371.06; p < 0.001) and compatibility (F(2,34) = 9.48; MSE = 451.10; p < 0.001). Importantly, the interaction between these variable was also significant (F(2,34) = 3.41; MSE = 283.68; p < 0.05; confidence interval is 8 ms), indicating that the 12 ms flanker effect (incompatible vs compatible) was significant in the easy search condition (F(1,17) = 10.24; MSE = 125.48; p < 0.01) but was not significant in the medium search condition (F < 1, −3 ms) (Fig. 9A). Thus, we only found evidence that the flanker affected behavior in the easy search condition, replicating the results of experiment 2.

Figure 9.

Figure 9.

Open in a new tab

A, RT in the easy and medium search conditions, for the flanker incongruent, congruent, and neutral trials in experiment 3. Error bars represent confidence intervals. B, Correlation between (CDA compatible − CDA neutral) and the flanker effect in experiment 3. *p < 0.01.

In terms of the CDA, when the search was easy, the amplitudes of the compatible and incompatible conditions were both higher than the no-flanker condition (the CDA amplitude was −0.35 μV in the no-flanker condition, −0.45 μV in the compatible condition, and −0.52 μV in the incompatible condition), a difference that was only marginally significant for the incompatible condition but was not significant in the compatible condition (F(1,17) = 1.82, MSE = 0.03, p = 0.19 and F(1,17) = 3.52, MSE = 0.06, p = 0.07, for the compatible and incompatible condition, respectively). We then isolated the flanker representation in WM by subtracting the CDA amplitude of the compatible and incompatible conditions from the CDA amplitude of the no-flanker conditions (e.g., CDA compatible − CDA no-flanker and CDA incompatible − CDA no-flanker) and correlated this difference to the RT flanker effect (RT incompatible − RT compatible). As observed in experiment 2, there was a significant correlation between the compatible CDA difference wave and the flanker compatibility effect on RT (r = −0.59; p < 0.05). Thus, a larger (negative) CDA difference between the compatible and no-flanker condition was associated with a larger flanker effect (Fig. 9B). Therefore, although the flanker effect did not reach a significant level in the group-level CDA analysis, the subjects that did represent the flanker in WM also showed a respective large compatibility effect on search RT. Again, no significant correlations were found between the flanker RT effect size and the CDA difference for the incompatible condition (r = −0.03; p = 0.88) or in the medium search condition for both compatible (r = −0.37; p = 0.12) and incompatible (r = −0.38; p = 0.11) difference waves.

Analyzing the N2pc

Please note that, just before the CDA, in all three experiments, we can observe an earlier component named the N2pc. The N2pc is believed to reflect spatial attention, and, because it was affected by our manipulations, we analyzed the N2pc (time range of 200–250 ms relative to the memory array) in each experiment.

In experiment 1, the N2pc was significantly different between the cued and the neutral (“all positions”) conditions (t(17) = 2.66; p < 0.05). However, unlike the CDA, the difference between the cued and the neutral conditions did not correlate with the corresponding RT difference (r = 0.001; p > 0.9), nor did this N2pc difference correlate with WM capacity (even after partialling out the initial N2pc amplitude; r = −0.02; p > 0.9).

In experiment 2, the N2pc amplitude was significantly different between the easy and the difficult search conditions (t(17) = 5.54; p < 0.005). However, the N2pc amplitude did not correlate with WM capacity in the easy condition (r = 0.007; p > 0.9) and in the difficult condition (r = 0.13; p > 0.13). To rule out the lack of reliability as a reason for these low correlations, we correlated the N2pc amplitude between the difficult the easy conditions. This time, the correlation was high and significant (r = 0.78; p < 0.0005), ruling out reliability as the reason for the low correlations with WM.

In experiment 3, the N2pc amplitude was significantly different between the easy and the medium search conditions (t(17) = 5.54; p < 0.005). However, the N2pc amplitude did not correlate with WM capacity in the easy condition (r = 0.28; p > 0.24) and in the medium condition (r = 0.36; p > 0.14). To rule out the lack of reliability as a reason for these low correlations, we correlated the N2pc amplitude between the medium and the easy conditions. This time, the correlation was high and significant (r = 0.92; p < 0.0005).

These results are in line with previous studies that have already found dissociations between the CDA and the N2pc (Jolicoeur et al., 2008; McCollough et al., 2008; Perron et al., 2009) and support the notion that the CDA and N2pc reflect different mental processes.

Establishing a general link between WM and the visual search task

In this final section of Results, we aimed at further summarizing the link between the reliance on WM and the demands of the visual search task. This is important considering that previous studies did not find any substantial link between individual differences in WM capacity and visual search performance (Kane et al., 2006; Sobel et al., 2007; Poole and Kane, 2009). To this end, we analyzed the easy, medium, and difficult search conditions (taken from experiments 2 and 3) in terms of both RT and CDA. The results are presented in Figure 10. As expected, RT increased as search became more difficult (F(2,51) = 21.52; MSE = 14,851.08; p < 0.0001). Importantly, the CDA followed the same pattern (F(2,51) = 9.02; MSE = 0.66; p < 0.0005), indicating that as the search became increasingly difficult there was an increasing involvement of WM. This pattern of results is critical in establishing a strong link between the demands of the visual search task and WM representations.

Figure 10.

Figure 10.

Open in a new tab

Combined results for experiments 2 and 3 for easy medium and difficult search conditions. Bar graph (left y-axis), RT data. Line graph (right y-axis), CDA amplitude. Error bars represent confidence intervals. *p < 0.05.

Discussion

The current pattern of results supports a tight connection between WM and visual search. First, as the search task became increasingly difficult (as indicated by slower RTs), the reliance on WM was also increased (as indicated by larger CDA amplitudes). This means that, in general, as the search became harder, participants tended to rely more on WM to complete the search process and find the target. This increased reliance on WM likely reflects the temporary storage of distractors from the search display as the subject progresses toward finding the target. Thus, this basic pattern of results is highly consistent with the predictions made by the classes of visual search models that presume a critical role for WM during the search process itself (Duncan and Humphreys, 1989; Bundesen, 1990; Desimone and Duncan, 1995).

At a more specific level, the present results identify under which conditions individual differences in visual WM capacity become an important regulator of visual search performance. In experiments 2 and 3, WM involvement in the search task (as indicated by the CDA) correlated with WM capacity over and above any search difficulty (i.e., in the easy medium and difficult search conditions). Experiment 1 showed that mainly high WM capacity individuals were likely to effectively use the cue to benefit their search performance. Thus, in a single paradigm, we provided evidence that performance in the visual search task is affected by both a task property (i.e., search difficulty) and a task-independent mechanism (individual differences in WM capacity). Moreover, it seems that, at least under the present conditions, search difficulty and WM capacity did not seem to interact, because WM capacity correlated (to approximately the same extent) with search performance in the easy, medium, and difficult search conditions.

We propose that the reason why WM is involved in visual search especially when the search becomes difficult is because WM plays an important role in filtering irrelevant information (Vogel et al., 2005; Awh and Vogel, 2008; Fukuda and Vogel, 2009). Rejecting irrelevant distractors based on their identity becomes more demanding as the search becomes more difficult. Namely, in the easy search condition, it is easier to reject distractors and lock on the target, resulting in less information that is represented in WM and thus low CDA amplitude. However, when the search becomes more difficult, filtering irrelevant information is less efficient so that more items end up in WM, resulting in a high CDA amplitude. Further corroborating this argument is the finding that fast trials show a faster drop in the CDA amplitude than slow trials. Presumably, after finding the target, there is no longer a need to hold any information from the search display in WM. The same general idea explains why high WM individuals rely less on WM to perform the search process but low WM capacity individuals tend to rely more on WM during the search. This is likely because high WM individuals are very efficient in rejecting distractors and locking on the target, whereas low-capacity individuals are less efficient at disengaging attention from distractors and consequently maintain more irrelevant information in WM during the search (Fukuda and Vogel, 2011). This argument was supported by the correlations between WM capacity and accuracy in experiments 2 and 3, demonstrating that the high-capacity individuals were more accurate in finding the target than low-capacity individuals in the medium and difficult search conditions.

The results of experiment 1 are consistent with our claim that the search difficulty dictates the extent of WM involvement. When the target position was not cued in advance, the search process that is supposed to select only the target also ended up transferring irrelevant distractors into visual WM. Minimizing the search process by cuing the position of the target in advance resulted in faster search performance and less reliance on WM, presumably because only the target was represented in WM.

Experiments 2 and 3 replicated the behavioral effects predicted by perceptual load theory (Lavie, 2000; Lavie et al., 2004). Namely, the flanker distractor was filtered out of processing only in the difficult and medium search (high perceptual load) conditions so that the flanker effect was significant only in the easy search condition. However, it seems that participants were able to filter out the distractors around the target only in the easy search conditions but not in the medium and difficult search conditions (as indicated by the low CDA amplitude during easy search and the comparably high CDA amplitudes in the medium and difficult search conditions). Thus, there seems to be a dissociation in how the search process treats the distractors that are around the target, which must be excluded based on their identity, and flanker stimuli, which may be filtered on the basis of location.

The correlation between WM capacity and visual search that were found in all of the experiments are interesting in light of previous studies that tried and failed at finding any substantial relation between individual differences in WM capacity and visual search performance. For example, Kane et al. (2006) found no correlations between search efficiency and WM capacity in both easy and difficult search conditions. Sobel et al. (2007) did find a correlation between search efficiency and WM capacity but only when a habitual response had to be overcome, so it is not clear whether this relation is a result of search efficiency per se or attributable to the fact that a habitual response had to be inhibited. Recently, Woodman and Arita (2011) have provided evidence that, in a visual search task in which the target identity changed randomly on every trial, WM stored the target template just before the search, thus bridging some of the gap between human and primate studies (Miller and Desimone, 1993, 1994). In line with this recent work, the current study consistently found a relationship between WM and visual search as predicted by several models of WM and attention (Desimone and Duncan, 1995; Cowan, 2001; Kane and Engle, 2002). Note that these previous studies that did not find a clear relationship used verbal WM measures (i.e., the operation span task) that were not specific (and perhaps less sensitive) to visual WM aspects. Moreover, those studies relied exclusively on behavioral measures of search performance, whereas we examined an online neural measure of online WM storage.

Another difference between the current and previous studies is that we only flashed the visual array for a limited time, whereas in former studies the visual array remained on the screen until the response. Historically, visual search was studied using both unlimited presentation time (and thus measuring reaction times as a dependent variable) and limited presentation time (Duncan and Humphreys, 1989) and thus concentration on accuracy. The two methods yielded comparable behavioral results, in terms of the search difficulty affecting the set-size slope. Nevertheless, using limited presentation time might change the subject's strategy so that storing items in working memory becomes more probable only when the array disappears and only when subjects cannot easily find the target (i.e., in the medium and difficult search conditions). However, please note that the CDA is apparent even when the array is displayed for an unlimited period of time (for a CDA in multiple object tracking, see Drew and Vogel, 2008; for a CDA in visual search task with unlimited presentation time, see Emrich et al., 2009), so that the relatively quick offset of the display cannot be the only reason why the CDA emerges. Moreover, a careful look at our data also argues against this offset type of interpretation. Namely, if WM is not needed when the target can be easily found (such as in the pop-out condition), the same should be true for the trials in which the target was quickly found in the difficult condition. However, our results from the difficult search condition clearly show that, even when the target was found relatively fast (average RT for the 25 percentile was 781 ms), WM was initially involved to the same degree relative to instances when the target was found relatively slow. That is, the slow and fast trials only differed in the onset of the decrease in the CDA but not in the initial involvement of WM. Moreover, the fast RT in the difficult condition was approximately similar to the RT in the easy search condition (781 vs 763 ms for the difficult and easy conditions, respectively), but, despite similar RTs, the easy search condition showed very little involvement of WM during the search process. Future research may further investigate this question by letting the stimuli stay on the screen until the response.

The current results also shed new light on the ongoing debate between early and late selection. We can identify early selection as being more efficient in information filtering, because capacity is allocated only (or mostly) to the relevant items, whereas late selection might inefficiently process task irrelevant items. Our data show that early and late selection are somewhat flexible mechanisms (Vogel et al., 2005) and that, under the same conditions, some individuals used more of a late selection approach, whereas others were better able to take advantage of early selection processes. Namely, the specific locus of filtering (early vs late) was dependent on the individual WM capacity so that, even when the search was easy and the overall performance indicated early selection (i.e., efficient filtering), some individuals (with low WM capacity) used a more late selection mode of processing. When the search task was difficult and the general performance indicated late selection (i.e., inefficient filtering), still some individuals (with high WM capacity) were able to use relatively more early selection mode of processing by efficiently blocking irrelevant items from reaching visual WM.

The neural substrates underlying visual search are beginning to be elucidated and are thought to rely primarily on interactions between prefrontal cortical structures and posterior parietal cortex (Bushman and Miller, 2007; Woodman et al., 2007a). These neural mechanisms are consistent with recent work by McNab and Klingberg (2008) who have found that filtering of irrelevant information was dependent on activity in the basal ganglia and prefrontal cortex. This in turn is consistent with the broad view that visual search mechanisms depend on working memory structures.

Footnotes

This work was supported by National Institutes of Health Grant 3 R01 MH087214-02S1 (E.K.V.).

References

  1. Awh E, Vogel EK. The bouncer in the brain. Nat Neurosci. 2008;11:5–6. doi: 10.1038/nn0108-5. [DOI] [PubMed] [Google Scholar]
  2. Brisson B, Jolicoeur P. A psychological refractory period in access to visual short-term memory and the deployment of visual-spatial attention: multitasking processing deficits revealed by event-related potentials. Psychophysiology. 2007;44:323–333. doi: 10.1111/j.1469-8986.2007.00503.x. [DOI] [PubMed] [Google Scholar]
  3. Bundesen C. A theory of visual attention. Psychol Rev. 1990;97:523–547. doi: 10.1037/0033-295x.97.4.523. [DOI] [PubMed] [Google Scholar]
  4. Buschman TJ, Miller EK. Top-down versus bottom-up control of attention in the prefrontal and posterior parietal cortices. Science. 2007;315:1860–1862. doi: 10.1126/science.1138071. [DOI] [PubMed] [Google Scholar]
  5. Chelazzi L, Miller EK, Duncan J, Desimone R. A neural basis for visual search in inferior temporal (IT) cortex. Nature. 1993;363:345–347. doi: 10.1038/363345a0. [DOI] [PubMed] [Google Scholar]
  6. Cowan N. The magical number 4 in short-term memory: a reconsideration of mental storage capacity. Behav Brain Sci. 2001;24:87–185. doi: 10.1017/s0140525x01003922. [DOI] [PubMed] [Google Scholar]
  7. Desimone R, Duncan J. Neural mechanisms of selective visual- attention. Annu Rev Neurosci. 1995;18:193–222. doi: 10.1146/annurev.ne.18.030195.001205. [DOI] [PubMed] [Google Scholar]
  8. Drew T, Vogel EK. Neural measures of individual differences in selecting and tracking multiple moving objects. J Neurosci. 2008;28:4183–4191. doi: 10.1523/JNEUROSCI.0556-08.2008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Duncan J, Humphreys GW. Visual search and stimulus similarity. Psychol Rev. 1989;96:433–458. doi: 10.1037/0033-295x.96.3.433. [DOI] [PubMed] [Google Scholar]
  10. Emrich SM, Al-Aidroos N, Pratt J, Ferber S. Visual search elicits the neural marker of visual working memory. PLoS One. 2009;4:e8042. doi: 10.1371/journal.pone.0008042. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Fukuda K, Vogel EK. Human variation in overriding attentional capture. J Neurosci. 2009;29:8726–8733. doi: 10.1523/JNEUROSCI.2145-09.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Fukuda K, Vogel EK. Individual differences in recovery time from attentional capture. Psychol Sci. 2011;22:361–368. doi: 10.1177/0956797611398493. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Gratton G, Coles MG, Sirevaag EJ, Eriksen CW, Donchin E. Pre- and poststimulus activation of response channels: a psychophysiological analysis. J Exp Psychol Human. 1988;14:331–344. doi: 10.1037//0096-1523.14.3.331. [DOI] [PubMed] [Google Scholar]
  14. Gratton G, Coles MG, Donchin E. Optimizing the use of information: Strategic control of activation of responses. J Exp Psychol Gen. 1992;121:480–506. doi: 10.1037//0096-3445.121.4.480. [DOI] [PubMed] [Google Scholar]
  15. Johnson DN, McGrath A, McNeil C. Cuing interacts with perceptual load in visual search. Psychol Sci. 2002;13:284–287. doi: 10.1111/1467-9280.00452. [DOI] [PubMed] [Google Scholar]
  16. Jolicoeur P, Sessa P, Dell'Acqua R, Robitaille N. On the control of visual spatial attention: evidence from human electrophysiology. Psychol Res. 2006;70:414–424. doi: 10.1007/s00426-005-0008-4. [DOI] [PubMed] [Google Scholar]
  17. Jolicoeur P, Brisson B, Robitaille N. Dissociation of the N2pc and sustained posterior contralateral negativity in a choice response task. Brain Res. 2008;1215:160–172. doi: 10.1016/j.brainres.2008.03.059. [DOI] [PubMed] [Google Scholar]
  18. Kane MJ, Engle RW. The role of prefrontal cortex in working-memory capacity, executive attention, and general fluid intelligence: an individual differences perspective. Psychon Bull Rev. 2002;9:637–671. doi: 10.3758/bf03196323. [DOI] [PubMed] [Google Scholar]
  19. Kane MJ, Poole BJ, Tuholski SW, Engle RW. Working memory capacity and the top-down control of visual search: exploring the boundaries of “executive attention.”. J Exp Psychol Learn Mem Cogn. 2006;32:749–777. doi: 10.1037/0278-7393.32.4.749. [DOI] [PubMed] [Google Scholar]
  20. Kane MJ, Conway ARA, Hambrick DZ, Engle RW. Variation in working memory capacity as variation in executive attention and control. In: Conway ARA, Jarrold C, Kane MJ, Miyake A, Towse JN, editors. Variation in working memory. New York: Oxford UP; 2007. pp. 21–48. [Google Scholar]
  21. Lavie N. Selective attention and cognitive control: dissociating attentional functions through different types of load. In: Monsell S, Driver J, editors. Attention and performance XVIII. Cambridge, MA: Massachusetts Institute of Technology; 2000. pp. 175–194. [Google Scholar]
  22. Lavie N, Cox S. On the efficiency of attentional selection: efficient visual search results in inefficient rejection of distraction. Psychol Sci. 1997;8:395–398. [Google Scholar]
  23. Lavie N, Hirst A, de Fockert JW, Viding E. Load theory of selective attention and cognitive control. J Exp Psychol Gen. 2004;133:339–354. doi: 10.1037/0096-3445.133.3.339. [DOI] [PubMed] [Google Scholar]
  24. Luck SJ, Vogel EK. The capacity of visual working memory for features and conjunctions. Nature. 1997;390:279–281. doi: 10.1038/36846. [DOI] [PubMed] [Google Scholar]
  25. McCollough AW, Machizawa MG, Vogel EK. Electrophysiological measures of maintaining representations in visual working memory. Cortex. 2007;43:77–94. doi: 10.1016/s0010-9452(08)70447-7. [DOI] [PubMed] [Google Scholar]
  26. McNab F, Klingberg T. Prefrontal cortex and basal ganglia control access to working memory. Nat Neurosci. 2008;11:103–107. doi: 10.1038/nn2024. [DOI] [PubMed] [Google Scholar]
  27. Miller EK, Desimone R. Scopolamine affects short-term memory but not inferior temporal neurons. NeuroReport. 1993;4:81–84. doi: 10.1097/00001756-199301000-00021. [DOI] [PubMed] [Google Scholar]
  28. Miller EK, Desimone R. Parallel neuronal mechanisms for short-term memory. Science. 1994;263:520–522. doi: 10.1126/science.8290960. [DOI] [PubMed] [Google Scholar]
  29. Oh SH, Kim MS. The role of spatial working memory in visual search efficiency. Psychon Bull Rev. 2004;11:275–281. doi: 10.3758/bf03196570. [DOI] [PubMed] [Google Scholar]
  30. Pashler H. Familiarity and visual change detection. Percept Psychophys. 1988;44:369–378. doi: 10.3758/bf03210419. [DOI] [PubMed] [Google Scholar]
  31. Perron R, Lefebvre C, Robitaille N, Brisson B, Gosselin F, Arguin M, Jolicoeur P. Attentional and anatomical considerations for the representation of simple stimuli in visual short-term memory: evidence from human electrophysiology. Psychol Res. 2009;73:222–232. doi: 10.1007/s00426-008-0214-y. [DOI] [PubMed] [Google Scholar]
  32. Poole BJ, Kane MJ. Working memory capacity predicts the executive control of visual search among distractors: the influence of sustained and selective attention. Q J Exp Psychol (Colchester) 2009;62:1430–1454. doi: 10.1080/17470210802479329. [DOI] [PubMed] [Google Scholar]
  33. Prime DJ, Jolicoeur P. Mental rotation requires visual short-term memory: evidence from human electric cortical activity. J Cogn Neurosci. 2010;22:2437–2446. doi: 10.1162/jocn.2009.21337. [DOI] [PubMed] [Google Scholar]
  34. Sobel KV, Gerrie MP, Poole BJ, Kane MJ. Individual differences in working memory capacity and visual search: the roles of top-down and bottom-up processing. Psychon Bull Rev. 2007;14:840–845. doi: 10.3758/bf03194109. [DOI] [PubMed] [Google Scholar]
  35. Vogel EK, Machizawa MG. Neural activity predicts individual differences in visual working memory capacity. Nature. 2004;428:748–751. doi: 10.1038/nature02447. [DOI] [PubMed] [Google Scholar]
  36. Vogel EK, Woodman GF, Luck SJ. Storage of features, conjunctions, and objects in visual working memory. J Exp Psychol Hum Percept Perform. 2001;27:92–114. doi: 10.1037//0096-1523.27.1.92. [DOI] [PubMed] [Google Scholar]
  37. Vogel EK, McCollough AW, Machizawa MG. Neural measures reveal individual differences in controlling access to working memory. Nature. 2005;438:500–503. doi: 10.1038/nature04171. [DOI] [PubMed] [Google Scholar]
  38. Woodman GF, Arita JT. Direct electrophysiological measurement of attentional templates in visual working memory. Psychol Sci. 2011;22:212–215. doi: 10.1177/0956797610395395. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Woodman GF, Luck SJ. Visual search is slowed when visuo-spatial working memory is occupied. Psychon Bull Rev. 2004;11:269–274. doi: 10.3758/bf03196569. [DOI] [PubMed] [Google Scholar]
  40. Woodman GF, Vogel EK, Luck SJ. Visual search remains efficient when visual working memory is full. Psychol Sci. 2001;12:219–224. doi: 10.1111/1467-9280.00339. [DOI] [PubMed] [Google Scholar]
  41. Woodman GF, Kang MS, Rossi AF, Schall JD. Nonhuman primate event-related potentials indexing covert shifts of attention. Proc Natl Acad Sci U S A. 2007;104:15111–15116. doi: 10.1073/pnas.0703477104. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Woodman GF, Luck SJ, Schall JD. The role of working memory representations in the control of attention. Cereb Cortex. 2007;17(Suppl 1):118–124. doi: 10.1093/cercor/bhm065. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from The Journal of Neuroscience are provided here courtesy of Society for Neuroscience

RESOURCES