The bandwidth of consolidation into visual short-term memory (VSTM) depends on the visual feature

Abstract

We investigated the nature of the bandwidth limit in the consolidation of visual information into visual short-term memory. In the first two experiments, we examined whether previous results showing differential consolidation bandwidth for color and orientation resulted from methodological differences by testing the consolidation of color information with methods used in prior orientation experiments. We briefly presented two color patches with masks, either sequentially or simultaneously, followed by a location cue indicating the target. Participants identified the target color via button-press (Experiment 1) or by clicking a location on a color wheel (Experiment 2). Although these methods have previously demonstrated that two orientations are consolidated in a strictly serial fashion, here we found equivalent performance in the sequential and simultaneous conditions, suggesting that two colors can be consolidated in parallel. To investigate whether this difference resulted from different consolidation mechanisms or a common mechanism with different features consuming different amounts of bandwidth, Experiment 3 presented a color patch and an oriented grating either sequentially or simultaneously. We found a lower performance in the simultaneous than the sequential condition, with orientation showing a larger impairment than color. These results suggest that consolidation of both features share common mechanisms. However, it seems that color requires less information to be encoded than orientation. As a result two colors can be consolidated in parallel without exceeding the bandwidth limit, whereas two orientations or an orientation and a color exceed the bandwidth and appear to be consolidated serially.

Introduction

Successful visual behavior requires the ability to process information from dynamic, continuously changing surroundings. Visual information processing thus entails the creation and storage of durable representations of the fleeting characteristics of a given fixation. This durable storage is commonly referred to as visual short-term memory (VSTM). It is generally accepted that VSTM can hold about 3–4 items for simple visual features (Luck & Vogel, 1997; Pashler, 1988), although whether such a capacity limit reflects limits in discrete slots or continuous resources is currently under intense debate (Bays & Husain, 2008; Wilken & Ma, 2004; Zhang & Luck, 2008). Regardless of the nature of such capacity limit, the need to process highly dynamic input has led to the suggestion that the visual system can rapidly encode and consolidate new items into VSTM, although at the expense of losing old items (Ballard, Hayhoe, & Pelz, 1995; Becker & Pashler, 2002; O’Regan, 1992; Jeremy M. Wolfe, Klempen, & Dahlen, 2000).

Previous research suggests that the consolidation process itself has a limited capacity, or bandwidth. For example, several studies varied the set-size of a briefly presented memory array and found worse performance as the set size increased, despite the fact that even the larger set-sizes were small enough that they should not have exceeded the storage limit of VSTM (Jolicœur & Dell’ Acqua, 1998; Vogel, Woodman, & Luck, 2006; West, Pun, Pratt, & Ferber, 2010). These findings are consistent with the view that the bandwidth of VSTM consolidation is limited. However, varying set-size might also introduce different amounts of decision noise or interference among items (Eckstein, Thomas, Palmer, & Shimozaki, 2000; Palmer, Verghese, & Pavel, 2000). As the number of items in the memory set increases, the number of decisions at test also increases (e.g., in change detection task, participants need to decide whether each item changed). In addition, memory representations for different items could interfere with each other (e.g., due to similarity), and the more items to be maintained, the more likely that interference will occur. Thus, the findings of worse performance with higher set-sizes could be attributable to either consolidation limits or limits in post-consolidation processes.

We recently employed a sequential/simultaneous paradigm to investigate the bandwidth limit of consolidation (Becker, Miller, & Liu, 2013; Liu & Becker, 2013; Mance, Becker, & Liu, 2012). This method allows an investigation of consolidation while holding the memory load, decision noise, and interference constant. In this paradigm, two items are briefly presented and masked, either sequentially or simultaneously (John Duncan, 1980; Hoffman, 1978; Shiffrin & Gardner, 1972). Comparing performance in the sequential and simultaneous condition allows one to infer whether or not multiple items can be consolidated in parallel (Scharff, Palmer, & Moore, 2011a, 2011b). In both conditions, the memory load is the same while the number of items that need to be concurrently consolidated differs. Better performance in the sequential condition implies either a serial or limited-capacity parallel process, whereas equivalent performance in the two conditions implies a parallel process.

Using the sequential/simultaneous paradigm, we have investigated the consolidation of orientation and color information and have obtained different results. In the color experiments, we found equivalent performance in the sequential and simultaneous condition, suggesting a parallel process up to two items (Mance et al., 2012). However, in the orientation experiments, we found better performance in the sequential than the simultaneous condition (Becker et al., 2013), suggesting a serial (or limited-capacity parallel) process. Furthermore, using a continuous measure of memory precision, we were able to demonstrate that consolidation of orientation information is strictly serial (Liu & Becker, 2013).

These results suggest that the bandwidth of consolidation depends on the visual feature and provide strong constraints on theories of VSTM consolidation. However, before accepting the notion that color and orientation have different consolidation bandwidths, it is necessary to exclude procedural differences that might have contributed to our initial observations. Specifically, most of our orientation experiments (Becker et al., 2013; Liu & Becker, 2013) required the orientations to be bound to a specific spatial location, while our color experiments did not require this binding (Mance et al., 2012). It is possible that this methodological difference accounts for the observed bandwidth difference. To investigate this possibility, Experiment 1 investigated the consolidation of colors using a method that required the colors to be bound to a specific spatial location, thereby replicating our orientation methods. In Experiment 2, we measured memory precision and used a mixture model (Liu & Becker, 2013; Zhang & Luck, 2008) to provide converging evidence regarding the nature of the consolidation process for color. Finally, in Experiment 3, we paired a color stimulus with an orientation stimulus in the sequential-simultaneous paradigm to further probe the dependence of VSTM consolidation on visual features.

Experiment 1

Our prior experiments suggesting the parallel consolidation of two colors (Mance, Becker, & Liu, 2012) involved the presentation of two test stimuli followed by a probe stimulus at fixation. Participants were required to indicate whether or not the probe color matched either of the test stimuli. By contrast, most of our previous experiments suggesting the serial consolidation of orientation (Becker, et al., 2013; Exps 1a, 1b, and 2; Liu and Becker, 2013) presented a box outline at the location of one of the test stimuli, and participants had to indicate the orientation of that probed item. Thus a key difference between these methods was that the orientation experiments required observers to bind each orientation to a specific spatial location, but the color experiments did not. While features may necessarily be bound to their spatial locations during initial encoding (Treisman & Zhang, 2006), this spatial binding may dissipate once the item is fully consolidated into working memory (Logie, Brockmole, & Jaswal, 2011; Woodman, Vogel, & Luck, 2012). Thus, it is possible that the orientation experiments found lower consolidation bandwidth because they required the orientation to be bound to spatial locations at the time of report. If the feature-location binding was lost between consolidation and report, the requirement to use the location cue may necessitate additional processing. The color experiments did not require this additional processing, which may have produced a greater consolidation bandwidth. To investigate this possibility, in Experiment 1 we examined the consolidation of color information using the same type of location probe that we have previously used in our orientation experiments.

Methods

Participants

Participants were 12 students from Michigan State University (3 male, 9 female). In all experiments, the sample size was based on our previous studies using the same experimental paradigm (Becker et al., 2013; Liu & Becker, 2013; Mance et al., 2012). All gave written informed consent and were naïve as to the purpose of the study. Participants were compensated $10 per session.

Stimuli and Display

The stimuli were circular colored patches (2°) and appeared in one of four possible locations at the corners of an imaginary square centered on fixation (eccentricity=6°). They could be one of four colors: red, green, blue, or yellow, set at the maximum saturation achievable by the monitor (e.g., red is [255 0 0]). Both the colors and locations of the stimuli were randomly selected, without replacement, from their four possible values. The masks were 2.4° circular 10×10 checkerboard patterns, with the color of each check randomly sampled from the four color values. The background was gray, and a small white circular fixation point (0.3°) was presented in the center of the screen throughout the experiment. Participants were instructed to keep their gaze on this point.

The experiment was programmed in MGL (http://gru.brain.riken.jp/doku.php?id=mgl:overview), a set of OpenGL libraries running in MATLAB (The MathWorks, Natick, MA) on an Apple iMac computer. The stimuli were displayed on a 19″ cathode ray tube (CRT) monitor with a refresh rate of 96 Hz. The monitor was positioned 57 cm away from the chinrest which was aligned with the center of the screen.

Main Task

Participants performed a color identification task in one of three conditions (Figure 1). In the Set Size 1 (SS1) condition, a single color patch was presented and followed by a mask. In the Sequential (Seq) condition, one color patch was presented (and masked), then a second color patch was presented (and masked) in a different location. In the Simultaneous (Simu) condition, two color patches were presented (and masked) at the same time in two different locations. Each trial began with a 500ms fixation period followed by the onset of the stimuli, which were presented for the appropriate exposure duration determined for each participant via the thresholding procedure (see below). All color patches were followed by a 200ms mask. At the end of the trial, the cue (a black square outline) appeared to indicate the location of the target stimulus. The cue remained on the screen until response. Participants were instructed to report the target color via button-press. Responses were made using the A, S, 4, and 5 keys (4, 5 on the number pad) to indicate red, green, blue, and yellow, respectively. The first letters of the colors’ names (‘R’, ‘G’, ‘B’, ‘Y’) were posted in that order directly above the keyboard for reference. Feedback was provided after incorrect responses via low-pitched tones.

Figure 1.

The schematics of trials in Experiment 1. The exposure duration was determined individually for each participant in the thresholding task.

The three presentation conditions (SS1, Seq, and Simu) were run in blocks of 75 trials, with a prompt at the beginning of each block informing participants of the block type. The blocks were arranged into two superblocks, each containing a random sequence of the three block types, for a total of six blocks.

Thresholding Procedure

A thresholding tasks was performed by all participants prior to participation in the main task. The thresholding task was identical to the Seq and Simu conditions described above, except that the stimulus exposure duration was varied using the method of constant stimuli to manipulate difficulty. One of the following seven durations was used for any given trial: 10.4, 20.8, 41.7, 85.3, 125, 166.7, or 333.3 ms. Participants ran two blocks of both the Seq and Simu conditions to equate any practice effects. We used the data from the Seq blocks to determine the exposure duration. The proportion correct was calculated for each duration, and the data were fit with the exponential function:

$$Pc=\delta +\gamma (1-e^{-\beta t})$$

Pc is percent correct, t is exposure duration, and δ, λ, β are free parameters that control the shape of psychometric function. Data were fit with standard maximum likelihood methods and the duration that produced ~85% correct for these sequential trials was used for the stimulus presentation duration for all conditions in the main task.

Results

The average exposure duration across participants was 67.7ms (range 41.7–114.6 ms). A one-way repeated-measures ANOVA was performed on the proportion correct (Figure 2). There was a main effect of condition (F(2,22) = 26.15, p < 10⁻⁴, η_p² = 0.28). Most relevant to our main research question, follow-up paired t-tests revealed that there was no significant difference between the Simu and Seq conditions (t(11) = 0.86, p = .41). The main effect resulted because performance in the SS1 condition was better than both the Seq and Simu conditions (both p < .05). For the Seq condition, we also compared accuracy for target stimuli that appeared first vs. target that appeared second in the sequence and found no significant difference (target first: 0.81, target second: 0.77, t(11) = 1.76, p = .10), indicating there was no order effect. This was expected given our SOA in the sequential display (>700 ms) was greater than typical estimates of attentional dwell time (200–500 ms) using similar displays (e.g., J. Duncan, Ward, & Shapiro, 1994; Kyllingsbaek & Bundesen, 2007).

Figure 2.

Results of Experiment 1. Figure 2A shows the proportion correct for each condition. Error bars display the within-subjects standard error of the mean (Cousineau, 2005). Figure 2B shows individual data plotting Seq performance on the x-axis and Simu and SS1 performance on the y-axis values. Most diamonds are above the identity line (dashed line going through the origin), indicating most participants had better performance in SS1 than Seq. Most squares are around the identity line, indicating overall equivalent performance in Seq and Simu across participants.

Comparison to previous orientation data

To further verify that there is a genuine difference between color and orientation, we directly compared the results across experiments. In Becker et al., (2013), we asked participants to remember and report the orientation of briefly presented grating stimuli in the same Sequential/Simultaneous protocol. In Experiment 1b of that paper, grating stimuli were selected from a set of ten possible orientations and participants (n=10) reported whether the target grating was tilted to the right or left of vertical. In Experiment 2, the number of possible gratings was reduced to four (horizontal, vertical, and the two diagonal 45° tilted gratings), and participants (n=10) responded by indicating whether the target grating was oblique or cardinal. Figure 3 re-plots performance for those two experiments from Becker et al., (2013) along with the current Experiment 1. It is apparent that performance in the simultaneous condition is worse than in the sequential condition for the previous orientation experiments, but there is a negligible difference for color. Separate mixed-factor ANOVAs were conducted with the presentation condition and experiments as factors. Comparing the current Experiment 1 and Experiment 1b in Becker et al., (2013), there was a significant main effect of presentation condition (F(1, 20)=29.5, p<10⁻⁴, η_p²=0. 43) and experiment (F(1, 20)=16.6, p<0.001, η_p²=0.45), as well as a significant interaction (F(1, 20)=19.4, p<0.001, η_p²=0.28). The significant interaction occurred because simultaneous presentation only reduced performance in the orientation experiment. Comparing the current Experiment 1 with Experiment 2 in Becker et al., (2013) also yielded a significant main effect of presentation condition (F(1, 20)=7.98, p<0.01, η_p²=0.25) and presentation by experiment interaction (F(1, 20)=4.47, p<0.05, η_p²=0.14), again confirming that simultaneous presentation only impaired performance for orientation stimuli. In addition, this pattern remained even when the overall task difficulty was relatively consistent across experiments, as was demonstrated by a non-significant main effect of experiment (F(1, 20)<1).

Figure 3.

Comparing color results from current Experiment 1 to orientation results from Becker et al., (2013).

Discussions

These analyses established that performance was equivalent for the color feature in the sequential and simultaneous conditions when we used the same location cue as in our previous orientation experiments. The results suggest that the different findings between earlier color and orientation experiments were due to the consolidation process and not due to differences in the methodology that had been applied. Our results demonstrated that both colors in the Simu condition were consolidated as well as in the Seq condition. Given the extreme temporal constraints placed on the exposure durations, the data suggest a parallel consolidation process even though participants had to encode both the location and color values. These results thus extend our previous findings from the probe-matching experiments where it was not necessary to bind color and location (Mance et al., 2012).

While equivalent performance for the simultaneous and sequential conditions suggests parallel consolidation of two colors, it is still possible that this parallel consolidation is rather limited. The response required in Experiment 1 was a four-alternative forced choice categorization of highly discriminable colors. Under these circumstances, an impoverished representation of both colors could be sufficient for a correct response. Hence, it is possible that parallel consolidation of color results in only imprecise representations, while forming more precise representations of the colors may require a limited-capacity or serial consolidation process. If so, we would expect a more precise memory representation in the Seq than the Simu condition, but the measurements in Experiment 1 may have been too insensitive to detect a difference in precision. By contrast, if two colors can be consolidated in parallel, the precision of the memory representation should be equivalent in the Seq and Simu conditions. Experiment 2 was designed to further investigate the nature of this parallel consolidation process by assessing the precision of the memory representation.

Experiment 2

In all of our experiments with color (Experiment 1 and our previous experiments in Mance et al., 2012), we have used highly discriminable colors with a discrete response and found evidence for parallel consolidation for two colors. As stated above, these tasks only required a low-resolution representation, such that representing the approximate feature would be sufficient for a correct response. It is possible that forming low-resolution VSTM representations requires less bandwidth such that two approximate colors can be consolidated in parallel. However, if the task requires high-resolution information, then consolidation could consume more bandwidth such that two colors would need to be consolidated in a serial manner.

To further probe the bandwidth limit in consolidation of color information, in Experiment 2, participants were asked to recall the precise hue of the color stimuli. If consolidation of high-resolution color information requires a limited-capacity process, performance should be more precise in the sequential than the simultaneous presentation condition. By contrast, if the consolidation of two colors is truly a parallel process, then the precision of the recall performance should be equivalent for both presentation conditions. A continuous measure of color memory also allowed us to perform a mixture model analysis (Zhang & Luck, 2008), which is capable of distinguishing between unlimited parallel, limited-capacity parallel, and strictly serial processes (see Liu & Becker, 2013). We previously used this method to investigate the consolidation of orientation and found strong evidence that orientations were consolidated via a strictly serial process (Liu & Becker, 2013). The application of the same type of model to color data should provide insight into the nature of any possible differences in the consolidation processes for color and orientation.

Finally, we note that a serial process can mimic a parallel process if the presentation time is so long that the serial process is allowed to complete multiple iterations. To rule out this possibility, we used two exposure durations. If consolidation was implemented via a serial process that switched between stimuli in the simultaneous condition, reducing the presentation duration should produce a more pronounced drop in performance for the simultaneous than the sequential condition. If however, consolidation was implemented via a truly parallel process, shortening the duration should affect both conditions to similar extent.

Methods

Participants

There were 14 participants in total (13 females, 1 male), five of which also participated in Experiment 1. Participants were compensated for $10 per session.

Stimuli and Display

Stimuli were color patches of the same shape, size, and eccentricity as in Experiment 1. However the colors of the patches were randomly sampled from a circle in the CIE L*a*b* color space (radius = 60, a = 20, b = 38, luminance = 70). The only constraint on color selection was that in the Seq and Simu conditions the two colors could not be within 45° of one another on the color circle. The masks were 8×8 checkerboard patterns with the color of each check randomly sampled from the same color circle.

The stimuli were displayed on a 21″ CRT monitor refreshed at 100 Hz. The monitor was calibrated with an i1Pro spectrophotometer (X-Rite, Grand Rapids, MI), to derive the transformation from the CIE L*a*b space to the monitor RGB space (Westland & Ripamonti, 2004).

Task and Procedure

Participants again were presented with a single color patch, or two color patches either sequentially or simultaneously (Figure 4). The trial structure was similar to that of Experiment 1, except at the end of the trial, a color ring (thickness: 2°, eccentricity: 11°) depicting the color circle in the CIE L*a*b space was presented. Participants were instructed to click on the ring where the color matched the target’s hue. Targets were again indicated by the same location cue used in Experiment 1.

Figure 4.

The schematics of trials in Experiment 2.

Two fixed exposure durations, 70 ms and 150 ms, were used in separate sessions for each participant. In each session, the three presentation conditions (SS1, Seq, Simu) were run in blocks of 75 trials, with block order sequenced the same as in Experiment 1. Six participants ran the short duration (70ms) session first, and the other eight ran the long duration (150ms) session first.

Data analysis

For each trial, we calculated the offset (error) in the participant’s color setting as the circular deviation between the reported and the true target color on the color wheel. For descriptive data analyses, we computed the mean and the variance of the offset using circular statistics (Berens, 2009). We used the log of the variances for statistical tests because they are more normally distributed. We also fit the offset data with a model that assumes performance results from the mixture of a proportion of “guess” trials (g) in which participants did not consolidate the target into VSTM, and a second proportion of “known” trials (1-g) in which the item was consolidated into memory (Zhang & Luck, 2008). Under this model, guess trials conform to a uniform distribution and known trials conform to a circular normal distribution with a mean (θ) and standard deviation (σ). The model was fit to the observed color offset data (for both individual and aggregate data) using standard maximum likelihood methods (Myung, 2003). These analyses mirror the analyses we performed to investigate the consolidation of orientation information in Liu and Becker (2013).

Results

Descriptive Analysis

On average, participants’ color settings were centered on the color of the cued item and did not show systematic bias (Figure 5A). A 2 (duration) × 3 (condition) repeated-measures ANOVA on the mean offsets found no significant main effect nor their interaction (duration: F(1,13)=2.97, p = .11; condition: F(2, 26) = 1.69, p = .20; interaction: F(2, 26) < 1). A second 2 (duration) × 3 (condition) repeated-measures ANOVA was performed on the log variance data (Figure 5B). There were significant main effects of both duration (F(1, 13) = 11.68, p < .01, η_p² = 0.47) and condition (F(2, 26) = 33.83, p < 10⁻⁴, η_p² = 0.72), with no interaction effects (F(2, 26)<1). The main effect of duration resulted from lower precision (higher variance) in the shorter duration. This evidence for lower precision with less encoding time suggests that reducing the time available for encoding decreased the amount of information that one could extract from the display.

Figure 5.

Results of Experiment 2. Figure 4A plots the average bias in participant’s color recall. Figure 4B plots the log of the variance of each condition. Error bars display the within-subject standard error of the mean. Figure 4C plots the log variance of the SS1 and Simu condition against that of the Seq condition, for the short duration presentation. Figure 4D shows the same scatter plot for the long duration presentation. In both cases, the diamonds tend to be below the identity line, indicating smaller variance for SS1 than Seq, but the squares cluster around the identity line, indicating equivalent variance for Simu and Seq.

The main effect of condition resulted from a higher precision (lower log variance) in the SS1 condition relative to the Simu and Seq conditions. Planned comparisons (paired t-tests) confirmed that the SS1 condition differed from both the Simu and Seq conditions for both durations (all p<.001). More importantly, there was no significant difference between the Simu and Seq conditions for either duration (short: t(13)=0.64, p = .53; long: t(13)=0.28, p = .78). The equivalent performance in the Seq and Simu conditions suggest that two colors can be consolidated in parallel, which is consistent with the results of Experiment 1 and our earlier color results (Mance et al., 2012). We also examined accuracy for target stimuli that appeared either first or second in the Seq condition, and found no significant difference in log variance for either the short duration condition (target first: 6.01, target second: 6.16, t(13)=1.23, p=0.24) or the long duration condition (target first: 5.73, target second: 5.27, t(13)=2.15, p=0.051).

Model Fitting

The three parameters of the mixture model include a measure of bias of the memory representation (θ), precision of the memory representation (σ), and guess rate (g). The purpose of fitting the mixture model to the data is to further evaluate the parallel nature of color consolidation. If two concurrent stimuli are consolidated into VSTM in parallel, then there would be no more random guessing (g) in Simu than in the Seq condition. If this parallel process was unlimited then there should be no difference in the precision of the memory representations between the Simu and Seq conditions. By contrast, if the process was a limited-capacity parallel process, then memory representations should be less precise in the Simu than in the Seq condition. A strictly serial process, such as was found using a similar model to examine the consolidation of orientation information (Liu & Becker, 2013), should produce higher guess rates in the Simu than the Seq condition but no difference in precision between the two conditions.

We fitted individual participant data with the mixture model and obtained three parameter estimates for each participant: bias (θ), standard deviation (σ), and guess rate (g) (Figure 6). We then performed two-way repeated-measures ANOVAs on these model parameters with factors of duration (short, long) and presentation condition (SS1, Seq, Simu). For the bias, we found no significant main effects nor their interaction (Duration: F(1,13)=1.05, p = .32; Condition: F(2, 26) = 1.26, p = .30; interaction (F(2, 26) = 2.88, p = .07). For the guess rate there was a main effect of condition (F(2, 26)=9.06, p < .001, η_p² = 0.86), but neither duration (F(1,13)=3.12, p = .10) nor their interaction (F(2, 26)<1) was significant. For the standard deviation, there were significant main effects for both condition (F(2, 26) = 15.06, p < 10⁻⁴, η_p² =0. 83) and duration (F(1,13)=18.82, p <.001, η_p² = 0.20), but not their interaction (F(2, 26) = 1.17, p = .33). The fact that there was a main effect of duration suggests that limiting the time for encoding made the task more difficult.

Figure 6.

Model fitting results to Experiment 2 data. A mixture model was fit to individual participant data and the mean parameter values are plotted. Figure 5A shows the bias parameter θ, Figure 5B shows the guess rate (g), Figure 5C shows the standard deviation of the circular normal distribution (σ). Error bars display the within-subjects standard error of the mean.

To isolate the source of the main effects of condition for the standard deviation and guess rate parameters, we ran paired t-tests comparing the three conditions within a given exposure duration. For the short duration, SS1 had a lower guess rate and a smaller standard deviation (higher precision) than both the Seq and Simu conditions (all p’s<.02). More importantly, there was no difference between the Seq and Simu condition in terms of either their guess rate (p=.21) or standard deviation (p = .70), consistent with the consolidation of both colors in parallel. For the long duration, we again found that SS1 had a lower guess rate and smaller standard deviation than both the Seq and Simu conditions (all p’s<.04). Again there was no difference in the guess rate (p= .47) between Seq and Simu conditions, however the standard deviation for the Seq condition was significantly smaller than the Simu condition (p<.01).

Comparison to previous orientation data

We compared the model parameters from the short duration condition to our previous experiment using orientation stimuli (Liu & Becker, 2013). In that study we asked participants (n=12) to recall the orientation of briefly presented grating stimuli in the same Sequential/Simultaneous protocol and used the mixture model to analyze their recall data. The guess rate and precision parameters for the current Experiment 2 and the orientation experiment from Liu & Becker (2013) are re-plotted in Figure 7A and 7B. Mixed-factor ANOVA showed that there were no significant effects for the precision parameter (all F <1). However, for the guess rate parameter, there were significant effects for presentation condition (F(1, 24) = 15.0, p < .001, η_p²= 0.24), experiment (F(1, 24) = 7.18, p < .05, η_p²= 0.23), as well as their interaction (F(1, 24) = 24.4, p < 10⁻⁴, η_p²= 0.39). This interaction resulted because simultaneous presentation reduced performance for orientation stimuli, but not for color stimuli.

Figure 7.

Comparing color results from current Experiment 2 to orientation results from our previous experiment in Liu & Becker (2013).

Discussions

The descriptive statistics demonstrate almost identical performance and memory precision (log variance) for the Seq and Simu conditions. There are two interpretations for this finding of equivalent performance for consolidating one or two items into VSTM during a brief period. The first is that two colors can be consolidated in parallel without taxing capacity limits. The second is that the duration of the stimulus presentation was long enough to allow for either a limited-capacity parallel process to complete processing of both items, or for a serial process to complete processing of the first item and then switch to and complete processing of the second item. However, our use of two presentation durations allows us to rule out this latter interpretation. Overall performance and memory precision (log variance) was reduced when the presentation duration was shortened. This reduction in performance provides evidence that, at least at the shorter duration, there was insufficient time to complete processing of the items. Under a limited-capacity parallel or serial scenario, the drop in performance we saw with the shorter duration should have been more severe in the simultaneous condition. However, this was not the case. Even in the short duration, performance was equivalent for simultaneous and sequential presentation. This provides strong evidence that two colors can be consolidated in parallel just as well as a single color can be consolidated.

This conclusion is further bolstered by the modeling results. First, we found no difference in guess rate between the simultaneous and sequential conditions at either duration. Second, we also found no difference in the memory precision parameter between these two conditions at the short duration. All of these patterns are consistent with the parallel consolidation of two colors into VSTM. Interestingly, we did find that the sequential condition had higher memory precision under the long presentation duration. This finding suggests that, given enough time, a second process that might be serial or limited-capacity can be used to improve memory precision. This second process might be eye movements to the stimulus, or verbal encoding. We do not believe this finding is problematic for our overall claim that the initial consolidation of two colors is performed in parallel. For this claim, the short duration is the most informative condition. However, this finding suggests that it is critical to use a duration that is adequately short to assess the initial consolidation phase.

Interestingly, our finding of increased memory precision with longer stimulus duration is at odds with results from Experiment 4 of Zhang & Luck (2008). These authors presented three colored squares for a fixed duration (100 ms) but varied the delay between stimulus and mask (either 10 ms or 240 ms). They found decreased guess rate but similar precision with longer SOA, suggesting that consolidation into VSTM was a discrete process, because additional processing time did not lead to improved precision. Our short and long durations were 70 ms and 150 ms, respectively, and we found increased precision with the longer duration (Figure 6C). Thus the consolidation process seems to be continuous rather than discrete, and it is possible that the 110 ms SOA used in the previous study was too long such that precision has already reached the asymptotic level.

Lastly, it is worth noting that performance in the SS1 condition was always better than performance in either of the presentation conditions for two items. This finding replicates our earlier work. In that work and here, we attribute this SS1 superiority to post-consolidation processes such as reduced interference and/or reduced decision noise.

Experiment 3

Experiments 1 and 2 further extended our previous finding of equivalent performance in the sequential and simultaneous conditions for color stimuli (Mance et al., 2012). This was the case both when the task required color-location binding and when the task required high-resolution memory representation. These results stand in stark contrast to our previous work on orientation (Becker et al., 2013; Liu & Becker, 2013), which demonstrates that two orientations were consolidated in a serial fashion. In sum, there is compelling evidence that two orientations are processed in a serial manner while two colors can be processed in an unlimited capacity, parallel manner.

One straightforward interpretation of these discrepancies is that there are two independent mechanisms, a serial process for the consolidation of orientations and a parallel process for consolidating colors. We have previously speculated that consolidation into VSTM requires establishing distinct neuronal assemblies for each item (Becker et al., 2013). Recent neuroimaging studies have suggested that working memory representations are maintained in sensory areas (Harrison & Tong, 2009; Riggall & Postle, 2012; Serences, Ester, Vogel, & Awh, 2009). If so, the different results for color and orientation could be due to these features being processed primarily by distinct visual areas. For example, color may rely more on V4, whereas orientation may rely more on V1. If this is the case, then consolidation of color and orientation might proceed in largely independent manner. Experiment 3 was designed to test this hypothesis by testing the bandwidth of consolidating a color stimulus and an orientation stimulus simultaneously. If the consolidation processes for these two feature dimensions are independent, we should observe similar performance in sequential and simultaneous presentation. If, however, the two processes rely on some common mechanisms, we would expect interference between the two feature dimensions such that a lower performance should be observed for the simultaneous than the sequential condition.

Methods

Participants

Sixteen students from Michigan State University participated for compensation at a rate of $10/hour. Three participants were excluded due to large thresholds (see more details below) such that results were based on thirteen participants. All participants were naive as to the purpose of the study.

Stimuli and Display

The stimulus presentation of Experiment 3 followed the same display setup as the first two experiments, except that the stimuli were different. Stimuli consisted of four color patches and four sinusoidal gratings (contrast: 0.7, spatial frequency: 2 cycles/deg). The color patches were identical as those in Experiment 1, and the grating orientations were horizontal, vertical, and the two diagonals (45° & 135°). The gratings were rendered in a circular aperture presented on a grey background. The edge of the aperture was smoothed to ensure a gradual transition in luminance at the border of the grating (see Figure 8A for an example of the grating stimulus). Due to the smooth edge of the aperture, the diameter of the gratings was set to 2.3°, 0.3° larger than the color patches, to approximately equate the perceived size of the two types of stimuli. On each trial participants were presented with both a grating and a color patch. Both stimuli were then masked, with the color masks identical as Experiments 1 and 2 and the gratings masked by circular apertures containing pixel noise pattern generated with a random uniform distribution over all possible luminance levels (see Figure 8A for an example of the mask). Again a square outline was used to indicate the target’s location after the stimulus presentation.

Figure 8.

Stimuli and results of Experiment 3. Figure 6A shows examples of orientation stimuli used in Experiment 3 (left: grating, right: mask). Figure 6B shows the average duration threshold for the orientation and color stimuli. Figure 6C shows the average proportion correct for each condition. Error bars display the within-subjects standard error of the mean (Cousineau, 2005). Figure 6D shows the scatter plot of individual participant data, plotting accuracy in the simultaneous condition against the sequential condition for both the color and orientation stimulus.

Main task

The task and procedure were identical to those of Experiment 1, with the following exceptions. Participants reported the feature of the target indicated by the location cue by pressing one of eight possible keys on a computer keyboard. The ‘A’, ‘S’, ‘D’, and ‘F’ keys were used for horizontal, 45°, vertical, and 135° orientation, respectively while ‘4’, ‘5’, ‘6’, and ‘+’ keys (on the numeric keypad) were used for red, green, blue, and yellow color, respectively. The response mapping was posted above the keyboard for reference throughout the experiment. For colors, the reminders were the first letter of each color (‘R’, ‘G’, ‘B’, ‘Y’), while for orientations, the reminders were lines drawn in the corresponding orientation. Stimuli were presented for the duration that was individually determined for each participant (see “Thresholding task” below). We did not include the SS1 condition in Experiment 3, as it did not have direct bearing on our predictions. We note here that we have consistently found a superior performance in SS1 compared to both the sequential and simultaneous conditions. Sequential and simultaneous trials were blocked (100 trials/block) for a total of four blocks (2 blocks per condition). The block order was randomized with the constraint that two of the same type could not be run consecutively.

Thresholding task

Before the main task, each participant performed a thresholding task very similar to those used in Experiments 1 and 2. On each trial, only one stimulus was presented, either a color patch or an orientated grating. Color and orientation thresholding were conducted in separate blocks (a total of 120 trials were obtained for each feature), within which the stimulus exposure duration was varied across trials. The proportion correct data were fit separately for each stimulus type with an exponential function as in Experiment 1. A 85% threshold was calculated for each stimulus type and the mean of those two thresholds was used in the main experiment as the stimulus exposure time.

Results and Discussions

We first examined data from the thresholding task. In general, duration thresholds were similar for color and orientation. However, three participants showed extremely long duration threshold for orientation (> 300 ms). We suspect they might have difficulty using the response mapping for orientation—our own experience is that it took more effort to learn the response mapping for orientation than that for color. We hence removed data from these participants from our analyses, such that the results reported here were based on 13 participants. We should note, however, that including these three subjects in our analyses produced essentially the same overall pattern of results. Across the 13 participants, the measured duration threshold was very similar for color and orientation (Figure 8B), and there was no significant difference between the two thresholds (t-test, t(12)=0.12, p=0.91). The lack of difference between color and orientation threshold also justified our approach to use the average threshold as the stimulus exposure time in the main task. The average stimulus exposure time across participants was 35 ms (range: 20–80 ms).

Average proportion correct across participants in the main task is shown in Figure 8C. We performed a 2 (stimulus type) × 2 (condition) repeated-measures ANOVA on the proportion correct data and found a significant main effect of condition (F(1,12) = 7.77, p < .05, η_p² = 0.39) as well as a significant interaction effect between stimulus type and condition (F(1, 12) = 6.36, p < .05, η_p² = 0.35). The main effect of stimulus type was not significant (F(1, 12) < 1). To further test our prediction of independent consolidation for color and orientation, we performed separate t-tests to compare sequential and simultaneous performance within each feature dimension. For color, performance was marginally (t(12)=1.99, p=.07) better for the sequential than simultaneous presentation. For orientation, performance was significantly better for sequential than simultaneous presentation (t(12)=4.97, p<0.001). Thus, simultaneous presentation impaired performance for orientation more than color. This differential effect of stimulus type on presentation condition accounted for the interaction effect in the overall data. Here, we again did not find a significant difference in performance for targets appearing in the first vs. second interval in the Seq condition (target first: 0.76, target second: 0.71, t(12)=1.84, p=0.09). No significant difference was observed when we examined color and orientation data separately.

These results showed that consolidating color and orientation simultaneously incurred a cost relative to consolidating them sequentially. We would like to note that performance for both stimulus types was numerically lower in the simultaneous than the sequential condition, although this decrement was less reliable for color than for orientation, leading to the interaction effect. Overall, these results argue against our original hypothesis that the consolidation of color and orientation rely on independent mechanisms, as that should produce equivalent performance in the sequential and simultaneous condition for both stimulus types. Thus the consolidation of color and orientation likely shares a common mechanism at some level. Furthermore, this common mechanism exhibits differential efficiency in processing color and orientation information. We will discuss possible reasons for this differential efficiency in the General Discussion. For now, we conclude that consolidation of color and orientation are non-independent processes.

General Discussions

Our data provide strong evidence that two colors can be consolidated into VSTM just as quickly and accurately as a single color. In our earlier work we also found evidence for parallel consolidation of two colors, but this ability did not extend to two orientations. In fact, when investigating orientation we found strong evidence for strictly serial processing. One potential confound was the fact that different methods were used to assess memory of color and orientation. So it was possible that the observed difference in consolidation ability between features was an artifact of methodological differences.

Here we used the same methods that we have previously used in our orientation studies to investigate the consolidation of color. Experiment 1 used a location cue that was similar to Experiments in Becker et al., (2013), and Experiment 2 used a similar type of continuous response and the same modeling techniques as Liu and Becker (2013). Yet here we come to the exact opposite conclusions as those orientation experiments. Our cross-study comparisons provide strong evidence that there is a real difference in the bandwidth to consolidate colors and orientations, with at least two colors being able to be consolidated in parallel while two orientations are consolidated in a strictly serial manner.

Why should there be this difference in the consolidation process of color and orientation? A simple hypothesis would be that consolidation of color and orientation relies on independent processes. This would predict that one color and one orientation can be consolidated as well in sequential as in simultaneous presentation. In Experiment 3, we tested this prediction by presenting a color patch and an oriented grating either sequentially or simultaneously. However, we found worse memory performance for both features in the simultaneous condition than the sequential condition, with orientation showing a larger decrement than color. This finding suggests that consolidation of color and orientation shares common processes. Thus, a single mechanism might be responsible for consolidating different features, but the bandwidth of consolidation varies for different features.

We speculate that this differential bandwidth arises due to differential informational demand when encoding different features. Specifically, color may require less information to be encoded, thus consuming less bandwidth, than orientation. For a uniform color patch, encoding of any local region is sufficient to derive the stimulus color, whereas for a circular grating, a larger region needs to be encoded to compute the stimulus orientation. In other words, any single pixel in a color patch has sufficient information about the stimulus color, whereas a single pixel in a grating does not contain information about its orientation. This argument is similar to the distinction between boundary feature and surface feature discussed by (Alvarez & Cavanagh, 2008).

From a functional point of view, the accurate perception of color requires color constancy, while simple orientations do not require constancy. Color constancy requires a light source with multiple wavelengths, and multiple objects that have different reflectance properties. Most models of color constancy assume that it is achieved by coding the relative L, M, and S cone activity across multiple independent objects or surfaces that have different surface reflectance (see Brainard, 2004 for review). This requirement to simultaneously represent multiple colored surfaces may have resulted in a system that can simultaneously consolidate color information from multiple distinct objects at once, thereby ensuring the rapid computation underlying color constancy.

Another difference between color and orientation is the fact that, while both features are based on the continuous variation of physical properties (wavelength and angle), color is perceived more categorically than orientation. It is possible that the categorical coding of color requires less information to be encoded and hence consumes less consolidation bandwidth. In addition, color categories have easy access to verbal labels and perhaps verbal encoding provides an additional channel for memory consolidation. However, we think verbal encoding alone cannot explain all of our results. First, verbal codes would be less useful in the current Experiment 2 where stimuli colors were randomly selected on the color wheel and participants need to recall the precise hue of the target. Second, in our previous study (Becker et al., 2013), we facilitated verbal encoding of orientation information by making the stimuli and judgment more categorical (e.g., left- vs. right-tilted, cardinal vs. oblique, see also Wolfe, Friedman-Hill, Stewart, & O’Connell, 1992). In all these experiments, we consistently found a lower consolidation bandwidth for orientation than color. Finally, we should note that having the possibility of verbal encoding does not necessarily lead to better performance. For example, Stevanoski and Jolicoer (2007) tested working memory consolidation of color stimuli and found that activating a verbal code for color was more, rather than less, capacity demanding in terms of the use of a central mechanism. For these reasons, we do not think verbal encoding plays a significant role in our findings.

All of these factors could contribute to a more efficient processing of color than orientation. Regardless of the exact reason for the disparity in efficiency, a possible interpretation of our results is that consolidation has a fixed bandwidth in terms of the amount of information that can be simultaneously processed, and this amount is sufficient to accommodate two colors, but only one orientation. Thus, two colors can be consolidated in parallel, whereas two orientations can only be consolidated serially. This scenario was depicted in Figure 9, where consolidation was shown as the process that connects perceptual analysis to working memory stores. The bandwidth of consolidation (depicted as the height of the rectangle) is large enough to accommodate two colors but only one orientation at a time (for more detailed explanations see figure caption). Under this scenario, one color and one orientation will also exceed the bandwidth (not depicted), leading to a lower performance in the simultaneous than the sequential condition. However, the item that requires less information to be encoded (color) is less affected by a limit in consolidation than the item that requires more information to be encoded (orientation). It is worth pointing out that our experimental protocol limits encoding time (via the thresholding procedure), which is necessary to prevent serial shift of processing among items before the mask terminates consolidation. We think the bandwidth limit we revealed is a set capacity limit of the amount of information that can be consolidated at once, and it is distinct from temporal limits due to limited processing time. If more time is available, it could allow multiple iterations of this consolidation process to occur, which could increase performance levels as more samples/instances of the items are stored.

Figure 9.

Schematic of the consolidation bandwidth limit. We envision the consolidation as the intermediate step in transferring the results of perceptual analysis to working memory stores. However, the bandwidth, or the amount of information that can be transmitted at once, is limited, which is depicted by the height of the central rectangle. The figure shows hypothetical scenarios when two colors (left panel) or two orientations (right panel) are shown simultaneously. Both stimuli can be encoded in parallel by the perceptual system, but because of the brief and masked presentation, the system cannot consolidate one item and then switch to the other item. Because color requires less information to be encoded, two colors can be consolidated at the same time such that they can be transferred to the working memory store before the mask. However, orientation requires more information to be encoded, so that only one stimulus can be consolidated at a time before the mask onset which effectively eliminates the other stimulus from entering into working memory store.

Our results seem to rule out explanations based on differences in the nature of early perceptual representations between color and orientation. We have previously suggested that a larger perceptual space for color than orientation (i.e., three-dimensional space of brightness, hue, saturation vs. one-dimensional space for orientation) may make it easier to represent multiple colors than multiple orientations simultaneously without interference among representations (Becker, Miller, Liu, 2013). At the neurological level this might mean that the consolidation of each of the two colors is supported by a distinct neural ensemble, while the consolidation of each of the two orientations may rely on overlapping neural ensembles. In light of our new results, this explanation seems less viable as one would expect that the neural ensemble for a color and an orientation should be at least as distinct as those for two colors, which would predict efficient consolidation of one color and one orientation. Instead, our results hint at a common central mechanism that sets the bottleneck of consolidation that works independently of specific sensory representations. The key factor is how much information a particular feature requires to be encoded in order to consolidate the information into VSTM. Our results also have implications for research on VSTM storage, as one needs to take into account of the limited consolidation bandwidth when studying storage limit. Our results show that if the memory array is presented very briefly and masked, only a few items can be consolidated. Under these conditions, performance may reflect a limit in consolidation, in addition to (or instead of) a limit in storage. For this reason, we recommend sufficiently long exposure time or stimulus-mask SOA (e.g., 100 ms per item) when investigating VSTM storage.

Clearly, addition research will be necessary to determine the source and functional significance of the difference in consolidation bandwidth between color and orientation. Even so, our current results provide clear evidence that this consolidation limit is not equal for all visual features. Furthermore, our results suggest that a common central mechanism is the rate-limiting factor for consolidation of features into VSTM.