Systematic Differences in Visual Working Memory Performance Are Not Caused by Differences in Working Memory Storage

Abstract

People vary in their performance on visual working memory tasks, and these individual differences covary with a wide range of higher-level cognitive processes including fluid intelligence. Performance also varies across study displays, purportedly driven by both low- and higher-level processes Understanding what causes these sources of systematic variability has been crucial for developing theories of working memory. However, here we find that all such variability in performance on a test of visual working memory can be accounted for by concurrent variability in visual iconic memory: A person with relatively high working memory capacity will have high iconic memory capacity, and a particularly easy working memory display will also be easy under iconic memory conditions. These results are supported by a nonparametric factor analysis and hierarchical Bayesian model comparison. In a second experiment the relationship between iconic and working memory holds even when they are measured with substantially different experimental paradigms, and a third experiment suggests that the relationship between tests of iconic and working memory is driven by mechanisms other than iconic or working memory storage, such as variation in perceptual or attentional processes.

Our ability to retain visual information for a matter of seconds is accomplished by visual working memory. A hallmark of this memory system is its surprisingly limited capacity: accurate knowledge about only a few visual items can be retained simultaneously (Awh et al., 2007; Luck & Vogel, 1997). For example, in a color version of the cued partial-report change-detection task (Becker et al., 2000) several colored squares are shown briefly, and following a retention interval, the location of an item is cued. An item of either the same or different color is then shown at the cued location (Figure 1A), and participants must report whether the item has changed or not. Performance on this task is remarkably poor: even with only eight studied items accuracy is typically near 50%, suggesting that accurate color information can only be retained for about four items, even over a mere 1-second retention interval. Perhaps even more surprising than this capacity limit is the finding that performance on the change-detection task correlates with a myriad of higher order cognitive functions, including general fluid intelligence (Cowan et al., 2005). These performance differences are sometimes suggested to reflect individual differences in capacity (Cowan et al., 2005; Fukuda et al., 2010), but have also been attributed to other cognitive functions such as differences in encoding strategies (Cusack et al., 2009; Vogel et al., 2005) or attentional control abilities (Unsworth et al., 2015). Understanding the cognitive mechanisms that drive individual differences in visual working memory, as well as deficits in it such as occur with aging (Jost et al., 2011) and disease (Lee et al., 2010) is a critical step toward addressing the broader question of how information is represented and maintained by the visual system.

Figure 1.

Structure of a Trial and Average Memory Decay Curve

Note. (A) The timing for each event in a trial is shown below each panel in milliseconds. (B) Average capacity estimates from Experiment 1 for each retention interval (black points), which exhibit exponential decay (dotted line).

Performance on visual working memory tasks has also been shown to vary across study displays. Some of this variability is likely to be driven by low-level perceptual effects, such as the finding that performance is superior for particular colors (Bae et al., 2014; Morey, 2011) and orientations (Pratte et al., 2017). However, other sources of systematic display variability may reflect higher level processing, such as a perceptual grouping of items (Nassar et al., 2018; Woodman et al., 2003) or by leveraging statistical properties of the display (Brady & Alvarez, 2011, 2015; Brady & Tenenbaum, 2013). Moreover, performance is better when the probed item is in the top half of the display (Quirk et al., 2018) and when the same feature is probed on two consecutive trials (e.g., Bae & Luck, 2019). Like performance differences across people, the cognitive mechanisms that underlie these display effects are not known, and could arise from any of the processes required of a visual working memory task including visual perception, encoding strategies, attentional selection, visual memory storage capacity, or decision processes.

One way to examine the processes that drive performance differences in visual working memory tasks is to modify the task such that one or more of the potential processes are no longer required. If doing so removes some or all of the person or display effects, then those effects must have arisen from the omitted process. For example, Sperling (1960) found that in visual memory tasks like change detection, shortening the retention interval between the study display and the item cue leads to substantial improvements in performance far beyond working memory capacity. For example, Figure 1B shows that memory capacity declines precipitously as the retention interval is increased from 33 to 1,000 ms. This pattern has been taken to suggest that we have a visual sensory memory that has a high capacity, but that decays rapidly. This system was later termed iconic memory as it seems to maintain a snapshot (or “icon”) of the entire visual scene (Neisser, 1967). A small subset of the information held in iconic memory is thought to be transferred to the limited-capacity working memory store before it decays, mediated by visual attention (Atkinson & Shiffrin, 1968). According to this standard model, performance in change-detection tasks with very short retention intervals will be limited by all of the processes required at longer intervals, with the exception of working memory storage capacity limits. For example, at long retention intervals a person with a working memory capacity limit of five items will perform far better than someone with a working memory capacity of two items. However, at short retention intervals both people can load the single cued item from iconic memory into working memory. Because the effective load on working memory in such iconic memory tasks is therefore only one item, iconic memory performance should be similar despite differences in working memory capacity. Therefore, if person or display effects in a working memory task with long retention intervals are absent from an iconic memory task with short intervals, then those effects must stem from differences in working memory storage capacity. Alternatively, if variability in a test of working memory is similar to variability in a test of iconic memory, then those effects cannot be due to differences in working memory storage capacity, but must instead arise from limitations in processes that are needed in both tasks, such as perceptual or attentive mechanisms.

Many characteristics of working memory task performance are also present in iconic memory tasks, suggesting that they may reflect processes other than working memory storage. For example, performance on individual trials in working memory tasks is often characterized as all-or-none, implying that the working memory storage is discrete (Rouder et al., 2008; Zhang & Luck, 2008). However, we have found that iconic memory also exhibits signatures of all-or-none storage, such that the discrete nature of working memory might be inherited from processes that precede it such as attentional selection (Pratte, 2018). Similarly, the precision of items held in working memory decreases as set size increases from one to about three items, and this effect has been at the center of debates regarding working memory storage mechanisms (Bays, 2018; van den Berg & Ma, 2014; Zhang & Luck, 2008). However, we have observed identical effects of set size on the precision of representations in iconic memory, again implying that they reflect processes other than working memory storage (Pratte, 2020).

Although a great deal is known about individual differences in working memory, individual differences in iconic memory have rarely been examined. Miller et al. (2010) did find that a measure of intelligence is correlated with iconic memory decay rates. Likewise, Lu et al. (2005) found that iconic memory decayed more rapidly in older adults with mild cognitive impairment compared with controls. However, whereas those studies focused on the rate of decay in iconic memory, other work suggests that memory impairments can occur to a similar degree in the maximum capacities of both iconic and working memory. For example, people with clinically low intelligence show similar impairments in a visual memory task with short retention intervals that measures iconic memory, and tasks with longer intervals that measure working memory (Mosley, 1978; Pennington & Luszcz, 1975). Likewise, Hahn et al. (2011) found that although the decay rate of iconic memory was similar in people with schizophrenia and controls, patients showed a similar degree of impaired memory performance at both short and long retention intervals. These findings suggest that at least some of the factors driving individual differences in working memory might also manifest in tasks that measure iconic memory, and are therefore not driven by individual differences in working memory storage capacity per se. Instead, the presence of similar effects in both iconic and working memory conditions suggests that these individual differences are driven by differences in perception, attention or other processes that would limit task performance regardless of the memory retention interval.

Here we examine the extent to which person and display variability in working memory may also be present in iconic memory. Experiment 1 is similar to standard change-detection tasks, but with a memory retention interval that varied from very short intervals thought to tap iconic memory, to longer intervals thought to rely on working memory. Systematic person or display variability at short intervals reflects differences in iconic memory (Figure 2A), whereas variability at longer intervals reflects differences in working memory (Figure 2B). If effects on iconic and working memory are unrelated then memory decay curves will cross (Figure 2C), whereas similar variability in iconic and working memory would produce decay curves that remain relatively high or low for all retention intervals (Figure 2D). It is reasonable to expect some correlation between effects on iconic and working memory performance, as several processes mediate performance on both tasks (e.g., variability in visual perception will affect performance regardless of retention interval), and our goal is to quantify the strength of this relationship. To do so, two analytic approaches were used to explore the latent processes that drive variability across several retention intervals. First, an exploratory factor analysis provides a simple, nonparametric approach for quantifying the number of latent factors needed to account for variability across iconic and working memory conditions. Second, a hierarchical Bayesian model is developed that provides estimates of latent iconic and working memory ability simultaneously for each person and study display in the experiment, allowing for an examination of how effects on working memory performance relate to effects on iconic memory. The results of both approaches converge on a surprisingly parsimonious conclusion: the factor analysis suggests that only a single underlying factor is needed to account for person and display variability across both short and long retention intervals, and the hierarchical model implies that person and display effects in working memory are nearly identical to those in iconic memory.

Figure 2.

Predicted Effects of Variability in Iconic Memory and Working Memory on Memory Decay Curves

Note. (A) Variability in iconic memory (parameterized in the Bayesian model as $I=W+A$). (B) Variability in working memory (W). (C) Variability in both iconic and working memory when those effects are unrelated. (D) Variability when effects on iconic and working memory are identical.

Experiment 1

Method

Transparency and Openness

We report all data exclusion criteria and all experimental manipulations. Most of the statistical analyses are standard or referenced below, and were performed using the R statistical language. The hierarchical Bayesian model was also implemented in R, and details of how the model was estimated are reported in the online supplemental materials. The experimental data may be obtained from the Open Science Framework (https://doi.org/10.17605/OSF.IO/CXEH6). This study was not preregistered.

Participants

Two hundred students at Mississippi State University participated in return for course credit. One participant did not complete the study and was removed from analysis. An additional 23 participants were not considered in analysis as they exhibited extremely poor performance even in the easiest condition (memory capacity less than .5 items), suggesting that they did not follow task instructions. The remaining 173 participants were included in all analyses. Participants provided informed consent prior to participation, and all studies were approved by the institutional review board at Mississippi State University.

Stimuli

Stimuli were displayed on 24 in. LCD monitors using the Psychophysical Toolbox (Kleiner et al., 2007) for Octave (Eaton et al., 2014) running on the Linux operating system. The structure of a trial is shown in Figure 1A. A fixation circle (.4° visual angle) was present throughout the trial, and participants were asked to maintain fixation. Each trial began with the presentation of a study array (50 ms), comprising eight colored squares subtending 1° visual angle. Study items were always located in the same eight spatial positions along an invisible circle (3.5° radius). Colors on each trial were chosen randomly from a set of 10 highly discriminable colors (brown, pink, red, green, blue, yellow, orange, cyan, magenta, and teal). A memory retention interval followed the study array, with duration chosen randomly on each trial from one of six values (33, 83, 167, 300, 500, or 1,000 ms). Following the retention interval a memory probe was presented (500 ms), consisting of a line starting at fixation and pointing toward the center of one of the eight study locations (2° long). This cue allowed participants to direct attention to the probed item, thereby mitigating potential masking effects from the subsequent test display. A test item was then presented in the probed study location, and was either the same color as the study item presented at that location, or was selected randomly from the two colors not included in the study array. Participants reported whether the test item was the same color as the probed study item (using the $z$ key on a computer keyboard) or that the color had changed (? key). Following this response, performance feedback was provided as an “X” or check mark at fixation for incorrect and correct reports, respectively (500 ms). Responses faster than 150 ms were followed by a 3-second presentation of “TOO FAST! Wait to continue,” as such responses indicate that participants were not following task instructions. The next trial began following a one second intertrial interval.

Procedure

The experiment began with 20 practice trials, followed by 420 experimental trials during which participants were invited to take a break every 100 trials. Study colors and the test color (for change trials) were chosen randomly for each trial, but critically, the study colors for each trial were identical for every participant (e.g., Brady & Alvarez, 2015). However, the retention interval condition and change/same condition were counterbalanced and randomized across trials for every participant. Consequently, all participants saw the same 420 study arrays in the same order, but for each display participants were randomly assigned to one of the six retention interval conditions and to whether the test on that trial was a change or same condition. This design provided both change and same trials at each retention interval for every study display, allowing for the examination of iconic memory decay curves for individual participants by averaging over trials, or for individual displays by averaging over participants.

Results

For the $i$th person studying the $j$th display in the $k$th retention interval condition, responses on change trials can be a hit $\left(H_{ijk}=1\right)$ or miss $\left(H_{ijk}=0\right)$, and responses on same trials can be a false alarm $\left(F_{ijk}=1\right)$ or a correct rejection $\left(F_{ijk}=0\right)$. These outcomes on each trial follow a Bernoulli distribution with parameters for hit rates $\left(h_{ijk}\right)$ and false alarm rates $\left(f_{ijk}\right)$. We use Cowan’s K to map these rates onto a measure of memory performance (Cowan, 2001; Rouder et al., 2011), computed by subtracting the hit rate from the false alarm rate. The result provides the probability that an item is successfully stored in memory, and multiplying it by the study set size (eight) provides an estimate of memory capacity (K). Averaging over both participants and trials produces hit and false alarm rates for each retention interval condition $\left(h_{\cdot \cdot k},f_{\cdot \cdot k}\right)$, and the results show the typical finding that performance is highest at short retention intervals and falls exponentially as the retention interval increases (Figure 1B). This pattern implies that iconic memory plays a role at short intervals, but decays rapidly such that performance at longer intervals must rely on the capacity-limited working memory system (Sperling, 1960). The critical question is how iconic and working memory performance vary systematically across particular people and displays. We take two approaches for examining these effects: a nonparametric factor analysis, and parametric hierarchical Bayesian model comparison.

Factor Analysis

The degree to which individual differences in working memory are related to those in iconic memory can be assessed by investigating correlations between measures of these two processes. Because performance at shorter retention intervals disproportionately reflects iconic memory, whereas performance at longer intervals is more sensitive to working memory, the extent to which person and display effects are correlated across retention interval conditions provides a way to examine the extent to which effects on iconic and working memory covary. For example, Figure 3A shows correlations across retention interval conditions that result when data are simulated from a model in which effects on iconic and working memory performance are independent (see online supplemental materials for details). The short retention interval conditions are correlated with one another, as they are all in part driven by variability in iconic memory, and longer retention intervals are correlated with one another due to shared variance from variability in working memory. Critically, however, the correlations across short and long retention intervals are far smaller, reflecting the fact that there is little shared variability across these conditions. Exploratory factor analysis provides two ways to assess how many latent factors are needed to explain such patterns of correlations. First, the scree plot corresponding to the simulated data (insert in Figure 3A) levels off only after adding three or more factors, implying a two-factor solution (Cattell, 1966). Second, Horn’s parallel analysis (Horn, 1965; implemented in the R package psych) quantifies the number of factors needed to explain the correlation matrix by comparing eigenvalues from the data with those generated by randomly permuting the data. This approach has been shown to be highly accurate (e.g., Humphreys & Montanelli, 1975), and here it correctly concludes that two latent factors were used to generate the simulated data: one driving variability in iconic memory, and one driving variability in working memory.

Figure 3.

Correlations in Performance Across Retention Interval Conditions

Note. Correlation matrices represent (A) data simulated from a model in which iconic and working memory are independent; (B) observed correlations across participants computed by averaging data over displays; and (C) observed correlations across displays computed by averaging over participants. Inserts show scree plots (solid line) corresponding to each correlation matrix, along with baseline values generated from randomized data (dashed line).

The factor analysis approach was used to quantify the number of factors driving person-to-person variability across retention intervals. Responses were first averaged over trials to provide hit rates $\left(h_{i\cdot k}\right)$ and false alarm rates $\left(f_{i\cdot k}\right)$ for each participant at each retention interval, and then subtracted to compute a measure of memory performance for each person and condition $\left(K_{ik}=h_{i\cdot k}-f_{i\cdot k}\right)$. Figure 3B shows the correlations in these measures for pairs of retention interval conditions, and unlike the simulation in which iconic and working memory were independent, the correlations across people are nearly identical across every pair of retention intervals. This pattern implies that the correlation between iconic and working memory conditions (e.g., 33 and 1,000 ms retention intervals) is similar to the correlation between two conditions that largely measure working memory (e.g., 500 and 1,000 ms). In line with this pattern, the scree plot (insert in Figure 3B) suggests a one-factor solution, and likewise Horn’s parallel analysis suggests that a single factor is sufficient to account for the entire correlation matrix. This result implies that a single latent factor explains variability in performance across people, regardless of whether the data came from iconic memory or working memory conditions.

The same approach was used to investigate variability across particular study displays. The data were first averaged over participants to produce hit rates $\left(h_{.jk}\right)$ and false alarm rates $\left(f_{.jk}\right)$ for each display at each retention interval, and then subtracted to produce a measure of performance for each display at each retention interval condition $\left(K_{jk}=h_{.jk}-f_{jk}\right)$. The correlations in performance across displays (Figure 3C) follow the same pattern as the correlations across people: iconic and working memory performance are just as correlated with one another as working memory performance is with itself. This pattern is again corroborated by the scree plot, which levels off after a single factor, and by Horn’s analysis which suggests that a single latent factor explains the variability across displays for all retention intervals. This result implies that a single latent factor explains variability in performance across particular study displays, regardless of whether the data came from iconic memory or working memory conditions.

Hierarchical Bayesian Analysis

The factor analysis approach has the desirable property of being nonparametric, with no assumptions regarding the shape of iconic decay curves (e.g., exponential vs. power function) or how latent effects map on to hit and false alarm rates. However, conclusions from the factor analysis warrant some care as there are potential issues with the approach. First, in order to obtain correlations in performance across people it was necessary to average the data over displays. Likewise, to compute performance for each display it was necessary to average the data over people. However, the high correlations across conditions for both participants and displays (see Figure 3) suggest that there is a large amount of systematic variability across both people and displays, and averaging data over such factors can lead to distorted parameter estimates and an overestimation of certainty in those estimates (Clark, 1973; Morey, 2011; Rouder & Lu, 2005). A second consideration is that accepting a one-factor solution, whether from the leveling off of scree plots or Horn’s parallel analysis, is qualitative rather than statistical. Finally, whereas each dependent variable (aggregate measures of capacity at each retention interval) is constructed from many trials with known statistical properties (e.g., they are binomially distributed), none of this information is leveraged by the factor analysis, which simply considers correlations among the aggregate scores. To address all of these issues a mathematical model of memory decay curves was constructed that simultaneously accounts for participant and display variability in change-detection performance. The models either allowed independent sources of latent variability in iconic and working memory, or forced effects on iconic and working memory to be related. Formal model comparison using the deviance information criterion (DIC; Spiegelhalter et al., 2002) provides a statistical approach for determining whether there is unique variability in iconic and working memory across people or displays, or if all systematic variability in iconic and working memory performance is driven by a single underlying factor, as is suggested by the factor analysis.

The goal of the hierarchical model is to account for the iconic memory decay function while simultaneously allowing for variability across people and across displays, in both iconic and working memory processes. Following a similar model developed by Morey (2011), the Cowan (2001) high-threshold model is used to link latent memory processes to the data. Hit and false alarm rates are determined by the probability that a probed item is in memory for the $i$th person, $j$th display, and $k$th retention condition $\left(p_{ijk}\right)$, and each person’s response bias to report “change” $\left(g_i\right)$,

$$h_{ijk}=p_{ijk}+\left(1-p_{ijk}\right)\times g_i$$

$$f_{ijk}=\left(1-p_{ijk}\right)\times g_i$$

According to this model a hit occurs when the probed item was in memory (with probability $p_{ijk}$) or when the item is absent from memory (with probability $1-p_{ijk}$) and the participant guesses “change” (with probability $g_i$). A false alarm occurs when the probed item is not in memory (with probability $1-p_{ijk}$) and the participant guesses “change.” Memory performance across retention intervals is characterized by an exponential decay function,

$$p_{ijk}=\Phi \left(W_{ij}+A_{ij}{e}^{-\lambda R_{ij}}\right),$$

where $W_{ij}$ is working memory performance for the $i$th person and $j$th display, and $A_{ij}$ is the increase in performance for that person and display as the retention interval $\left(R_{ij}\right)$ approaches zero. Iconic memory performance for the $i$th person and $j$th display can therefore be computed as $I_{ij}=W_{ij}+A_{ij}$. Although it is possible to parameterize the model using $I_{ij}$ directly, we found that model estimation was far more stable by first estimating $A_{ij}$ and then computing iconic memory performance $I_{ij}$ from $A_{ij}$ and $W_{ij}$. Parameter $\lambda$ determines the rate of decay, and is fixed across participants and displays to aid in model identifiability. The exponential decay function produces values that can range from zero to infinity, but the model must account for the probability that an item is in memory $\left(p_{ijk}\right)$ which ranges from zero to one. The Gaussian cumulative distribution function, or “probit” function $\left(\Phi \right)$ is therefore used to constrain the exponential function to be between zero and one.

Because each participant sees each display only once, as either a change or same condition, and in only one of the retention interval conditions, memory performance cannot be estimated freely for every person by display combination. Additive models are therefore placed on $W_{ij}$ and $A_{ij}$ to make the model identifiable,

$$W_{ij}={\mu }^W+{\alpha }_i^W+{\beta }_j^W,$$

$$A_{ij}={\mu }^A+{\alpha }_i^A+{\beta }_j^A,$$

where ${\mu }^W$ is the average working memory performance and ${\mu }^A$ is the average amount that is added to get iconic memory performance. Parameters ${\alpha }_i^W$ are zero-centered person effects on working memory performance, and ${\beta }_i^W$ are zero-centered display effects on working memory performance. Likewise, ${\alpha }_i^A$ and ${\beta }_i^A$ are person and display effects, respectively, on the increase in performance for iconic memory over working memory. These effects (and person effects on $g_i$) are treated hierarchically by assuming that they follow a normal distribution, which often aids in model identifiability and yields more accurate parameter estimates (e.g., Rouder & Lu, 2005). For this experiment the model includes 1,367 parameters, and accurate estimation is accomplished by using Bayesian model estimation techniques (see online supplemental materials), many of which have been used successfully in similar memory models (e.g., Morey, 2011; Pratte et al., 2010; Rouder & Lu, 2005).

The critical question is how person and display effects in working memory are related to effects in iconic memory. In the full model these effects are free to be completely independent; however, Figure 4A shows that the estimated person effects on working memory $\left({\alpha }_i^W\right)$ are highly correlated with person effects on iconic memory $\left({\alpha }_i^I={\alpha }_i^W+{\alpha }_i^A\right)$. Likewise, Figure 4B shows that display effects on working memory $\left({\beta }_j^W\right)$ are highly correlated with display effects on iconic memory $\left({\beta }_j^I={\beta }_j^W+{\beta }_j^A\right)$. This strong relationship between variability in iconic and working memory mirrors the factor analysis result: the extent to which a person has good working memory, or a particular display will be easily remembered in working memory, is nearly perfectly predictable by performance in iconic memory conditions.

Figure 4.

Person and Display Effect Estimates From the Full Hierarchical Model

Note. Although the model allows effects to be completely independent, person effects on working memory are highly similar to person effects on iconic memory (A). Likewise, display effects on working memory are nearly identical to display effects on iconic memory (B). The resulting (predicted) memory decay curves vary substantially across people (C) and across displays (D), but the strong relationship between effects on iconic and working memory suggest that entire decay functions simply shift up and down.

In the strictest case that effects on iconic and working memory are numerically identical, the points in Figure 4 should fall along the diagonal, and the amount that must be added to get from iconic to working memory performance should be a constant for each person $\left({\alpha }_i^A=0\right)$ and each display $\left({\beta }_j^A=0\right)$. A restricted model was fit to the data in which these effects were forced to be zero, such that iconic memory is equal to working memory plus a constant $\left(I_{ij}=W_{ij}+{\mu }^A\right)$. However, the full model provided a better account than this restricted model $(\Delta \text{DIC}=15)$, suggesting that the person and display effects on working memory are not exactly equal to those on iconic memory. But importantly, even if the true latent effects on iconic and working memory are exactly equal, that equality will only manifest in parameter estimates if the model and all of its assumptions are accurate. In particular, whereas we used the probit function to map latent memory ability onto the probability that an item is in memory, this choice is arbitrary and was made for mathematical convenience. Although different choices would not change the model substantively, they will change the extent to which high/low task performance maps on to high/low latent memory effect estimates, and will therefore change the observed relationship between these effects. Indeed, the effects on iconic and working memory in Figures 4A and B are nearly identical, but there appears to be a systematic deviation from equality whereby the effects are linearly related but with a slope that differs from unity.

To allow for the possibility that effects on iconic and working memory are related, but not necessarily numerically equivalent, a model was constructed in which effects on iconic memory are a linear function of effects on working memory,

$$W_{ij}={\mu }^W+{\alpha }_i^W+{\beta }_j^W,$$

$$A_{ij}={\mu }^A+\gamma {\alpha }_i^W+\varphi {\beta }_j^W.$$

Parameter $\gamma$ determines the slope of the relationship between person effects on iconic and working memory, and $\varphi$ the relationship between display effects. These slope parameters are fixed across people and displays, and force all effects on iconic memory to be linear functions of effects on working memory (i.e., perfectly correlated). This restricted linear model provides a better account of the data than the full model $(\Delta \text{DIC}=21)$, which places no constraint on the relationship between iconic and working memory, suggesting that person and display effects on working memory can be accurately characterized as a linear function of those effects on iconic memory.

Simulation studies were used to assess the ability of this modeling approach to differentiate between cases where the effects on iconic and working memory are independent, verses cases where they are similar. Data were simulated from situations in which the true iconic and working memory effects were either independent, or were correlated with one another to varying degrees. Fitting the full and restricted (linear) models to each simulated data set revealed that the full model typically outperformed the restricted model unless the true correlation between effects was greater than .9 (see online supplemental materials). Therefore, the superiority of the restricted model fit to the experimental data implies that effects on working memory performance across people, and across displays, are nearly perfectly correlated with the effects on iconic memory. These modeling results therefore support the factor analysis result that only a single latent construct is needed to explain differences across people, and differences across displays, in both iconic and working memory.

The Causes of Display Variability

The factor analysis and hierarchical Bayesian model suggest that there is systematic variability in memory performance across particular displays, and that it is nearly as large as the variability across people. For example, Figure 5A shows the memory decay curve for a particular display, obtained by averaging the data over participants. Average performance on this trial is nearly perfect, suggesting that most participants responded correctly in both change and same conditions, and did so regardless of the retention interval. Alternatively, performance for the trial shown below it (Figure 5D) is extremely poor, even at short retention intervals. We examined several possible sources of this systematic display variability by considering whether particular aspects of each trial are related to memory performance. For example, performance has been shown to depend on the color of the probed study item, and the color of the lure on change trials (Morey, 2011). However, the study displays corresponding to each trial in Figure 5 are shown in Figures 5B and E, and even though they have the same study color (yellow) and lure color (brown), performance is remarkably different across these trials implying that there are other sources of display variability. For example, it has also been shown that performance tends to be higher when the probed item is in the top half of the display (Quirk et al., 2018), which may account for some of the advantage in Figure 5A. In addition to particular display characteristics, it has been suggested that there is a serial dependence across trials, such that performance on the current trial can be affected by what feature was tested on the previous trial (e.g., Bae & Luck, 2019). The rightmost panels in Figure 5 show the study displays for trials that preceded those shown to the left, and when performance is good (top) the previous trial had both the same probe color and the same probe position as the current trial. Overall, it appears that there are several potential causes of systematic variability across displays, and the modeling results imply that all of them have similar effects on iconic and working memory.

Figure 5.

Memory Decay Curves for Specific Experimental Displays

Note. In the top row the observed memory decay curve for trial 247 (A) was computed by averaging data over participants, and suggests that nearly all participants answered correctly on this trial regardless of the retention interval. The probed item was yellow and located at the top of the display (B), and on change trials the probe was brown (denoted here by the fixation color). On the previous trial the probed item was also yellow and located at the top of the display (C). The bottom row shows performance for trial 137 (D), the corresponding stimulus display (E), and the display of the preceding trial (F).

Because the hierarchical models developed above account for both person and display variability simultaneously, they provide reliable estimates of memory performance for each display. We can therefore assess several potential sources of systematic display variability by examining whether they predict performance across the 420 displays as estimated by the hierarchical model. Display effects from the best-fitting restricted linear model $\left({\beta }_j^W\right)$ were used to quantify the difficulty of each display. In this single-factor model the display effects reflect memory performance for both iconic and working memory. However, the results are similar when other measures are used, including factor scores from the factor analysis, or the separate measures of iconic and working memory display effects from the full hierarchical model. These values were treated as the dependent variable in an analysis of variance (ANOVA) that included several potential sources of trial-to-trial variability as predictor variables. Because some of the factors are highly unbalanced (e.g., there are far fewer trials that have the same color probed twice than not) the effects were assumed to be additive without interactions, and the Type II sums of squares were used to compute test statistics.

Systematic differences in performance across displays were due to large effects of which study location was probed $\left({\eta }_p^2=.35\right.,F\left(\mathrm{7,387}\right)=29.69,p<.01)$, which study color was probed $\left({\eta }_p^2=.18,F\left(\mathrm{9,387}\right)=9.33,p<.01\right)$, and whether the probed location on the current trial was the same as that on the previous trial $\left({\eta }_p^2=.18,F\left(\mathrm{1,387}\right)=83.09,p<.01\right)$. Smaller but significant effects were due to the influence of whether the same color was probed on two trials in a row $\left({\eta }_p^2=.06,F\left(\mathrm{1,387}\right)=22.78,\right.p<.01)$, and the color of the lure on change trials $\left({\eta }_p^2=.05,F(9\right.,387)=2.47,p<.01)$. In line with previous studies, performance tended to be superior for probed items located in the top half of the display, and for colors that may have been visually salient (including red, blue, and teal). But because the consequences of these effects were nearly identical in iconic and working memory conditions, they must result from processes other than working memory capacity limits per se, such as perceptual or attentional effects at study.

Discussion

The results of Experiment 1 suggest that individual differences in iconic memory performance account for differences in working memory performance. Because working memory capacity limits do not act as a bottleneck on performance in the iconic memory conditions, these individual differences cannot be due to differences in working memory capacity. There are, however, many other processes that might cause this similar variability under iconic and working memory conditions, particularly in Experiment 1 where the working and iconic memory tasks were identical except for the duration of the retention interval. As an extreme example, perhaps some participants were simply better at perceiving color than others. Or perhaps some participants were better able to orient attention toward the cued color, or were better at retaining the single color in working memory until the response period. In this case the fidelity of color working memory for a single item might limit performance under both iconic and working memory conditions, even though the working memory capacity limit should not affect performance under iconic memory conditions.

If the causes of individual differences in Experiment 1 reflect something idiosyncratic to the color change-detection task, such as color perception or a task strategy, then the relationship between iconic and working memory effects should not be present if they are measured with substantially different paradigms. To test this possibility, in Experiment 2 working memory was measured with the color change-detection task, but iconic memory was assessed with a measure of iconic memory that differs substantially from color change detection. If performance on these qualitatively different tasks is highly correlated across people, then the relationship between iconic and working memory observed in Experiment 1 does not merely reflect individual differences in color change-detection ability, but must reflect variability in more task-general processes.

Experiment 2

Method

Participants

Fifty-seven students participated in Experiment 2. Three did not complete the task, and analyses were performed on the remaining 54 participants.

Stimuli

Experiment 2 consisted of two tasks. To measure working memory the change-detection task was similar to that of Experiment 1 but included only a single, long retention interval (1,000 ms). In addition, only six (rather than eight) stimuli were studied to ensure that the task was not too difficult overall. Iconic memory was measured with a letter-identification task. The stimuli for this task were adapted from Eriksen and Collins (1967, 1968) and consisted of three English letters arranged from left to right. The letter stimuli (Figure 6A) were composed of white dots on a gray background, with letters made of dots (.25° visual angle) presented within a noisy background of smaller dots (15°). When all dots are presented simultaneously the three letters are trivially easy to identify. However, in the task each letter display was divided into two separate displays, each including only half of the signal dots superimposed in noise dots (see Figure 6A). When viewing either half-display alone it is difficult to identify the letters. However, when the two half-displays are presented one after another with a very short interstimulus interval (ISI), iconic memory retains the first display while the second is presented, yielding the perception of both displays being presented simultaneously such that the letters are easily identified.

Figure 6.

Example Stimulus (A) and Trial Structure (B) of the Letter-Identification Task

Note. In this example each stimulus half combines to make the letter triplet “XOM”, and the participant reported perceiving the letters “VOW”. In Experiment 2 the interstimulus interval (ISI) was always 50 ms. In Experiment 3 the ISI was variable (16, 50, or 1,000 ms) on trials with both half-displays. On the remaining trials there was no ISI and the second half-stimulus was blank.

The positions of dots for each letter were defined on a $9\times 9$ grid, similar to a digital clock display. Each element in the grid subtended .25° visual angle, such that each letter was 2.25° in width and height. A random half of each letter’s dots were chosen for the first and second half displays. The three-letter displays were constructed by abutting three of the $9\times 9$ grids, producing a stimulus defined by a $9\times 27$ location grid, and noise dots were then added to each of the $9\times 27$ grid locations with a 20% probability. The first letter was always a $T,V$ or $X$; the second letter an $A,O$ or $U$, and the third letter an $H,M$ or $W$.

The structure of a trial is shown in Figure 6B. Each trial began with the sequential presentation of each half-display, each presented for 16.6 ms separated by a 50 ms blank interval. This brief retention interval is well within the window by which the two displays are integrated as one due to iconic memory (Eriksen & Collins, 1967), and was chosen in pilot studies to obtain performance that was well above floor and below ceiling. After a 500 ms blank period (to avoid masking of the second half-stimulus display) a response display showed each of the nine letters in a $9\times 9$ grid, and participants used the computer keyboard to report the three letters they perceived during the stimulus display. Each column of letters contained the three possible letters for that position (e.g., the leftmost letters were $T,V$ and $X$ arranged top-to-bottom), and participants were required to choose one letter in each column. Each choice was highlighted by turning the dots for that letter white, and participants confirmed their responses by pressing the space bar. Upon doing so the correct three letters were shown, thereby providing performance feedback by highlighting which letters were identified correctly and which were incorrect. Following feedback (1 second), a 1-second intertrial interval preceded the start of the next trial.

Procedure

Each participant first completed 150 trials of the letter-identification task, taking a break every 50 trials. Letters were chosen randomly on each trial from the three possibilities in each position. After completing the letter-identification task participants completed 300 trials of the change-detection task, taking a break every 50 trials. More trials were administered for change-detection than letter-identification to achieve similar measurement accuracy for both tasks, as in change detection half the trials are in the change condition and half are the same condition. Whereas in Experiment 1 the study stimuli were identical for all participants, in the change-detection task of Experiment 2 colors were chosen randomly from the ten possible colors for each of the six stimuli, and on change trials the lure was randomly chosen from the remaining four colors.

Results

Working memory capacity was computed from change-detection performance for each participant using Cowan’s K as described in Experiment 1. Average capacity was 3.3 colors, similar to capacity measured at the 1,000 ms retention interval in Experiment 1. Performance in the letter-identification task was measured as the average accuracy across the three letters, such that chance performance is one letter and ceiling performance is three letters. Participants accurately reported 2.20 letters, which is well above chance and below ceiling. Critically, the change-detection measure of working memory and the letter-identification measure of iconic memory were significantly correlated across participants $(\rho =.45,t(52)=3.63,p<.001)$. Although factor analysis cannot be used here to decompose the variance, as there are only two task measures, this correlation is similar to that observed between change-detection measures of iconic and working memory in Experiment $1\left({\rho }_{33ms,1000ms}=.57\right)$, suggesting that the relationship between measures of iconic and working memory observed in Experiment 1 is similar when these memory systems are measured with substantially different experimental paradigms.

Discussion

The results of Experiment 2 extend the main finding of Experiment 1, that individual differences in iconic and working memory task performance are similar, to a case in which these memory systems are measured with substantially different tasks. This result implies that the relationship between iconic and working memory in Experiment 1 is not confined to the change-detection task, but must reflect some more general mental processes. The letter-identification task presumably does not rely on visual working memory, as reporting the three letters is trivially easy, but only if they can be successfully identified. Letter identification in this task is aided by iconic memory, suggesting that the relationship between this task and the change-detection working memory tasks might reflect shared variance due to iconic memory processes. However, several other processes might support letter-identification accuracy, including perceptual abilities. In particular, in the original version of this task Eriksen and Collins (1967) hand-drew each half-display for each stimulus letter, such that it was very difficult to identify a letter from either of the half-displays. Nonetheless, performance even at long ISIs was above chance, suggesting that this task can be performed to some degree without strictly relying on iconic memory to integrate the two stimulus halves. In our version of this task the stimuli varied from trial to trial in the distribution of noise dots, and in how the signal dots were divided across the half-displays, and it is likely that performance in our version is also partially determined by perceptual abilities such as identifying letters hidden in noise, regardless of ones iconic memory abilities.

In Experiment 3 we measured the extent to which performance on the letter-identification task reflects iconic memory, working memory, and perceptual abilities. To do so the half-displays were separated by short intervals such that they could be integrated using iconic memory, long intervals where only working memory could support such integration, or only a single half-display was shown such that performance reflects perceptual abilities. Investigating how performance on these conditions varies across participants provides a way to measure the extent to which these visual processes determine performance on the letter-identification task.

Experiment 3

Method

Experiment 3 was identical to the letter-identification task in Experiment 2 with the following exceptions. Thirty-six students participated, but one did not complete the experiment providing data from 35 participants for analysis. Whereas in Experiment 2 the ISI was fixed at 50 ms, in Experiment 3 the ISI was manipulated across trials such that the two half-displays were separated by a 16, 50, or 1,000 ms blank interval. In a fourth condition only the first of the two half-displays was presented, such that reports had to be made based on only one half of the stimulus dots. These four experimental conditions were presented in equal proportion and randomized across trials, yielding 75 trials in each condition for each participant (300 trials total).

Results

Mean accuracy for each condition is shown in Figure 7A. When only one of the two half-displays was shown participants accurately reported about 1.8 letters, which is above the chance baseline of one letter $\left(t\right(34)=24.11,p<.001)$ suggesting that performance on this task can be in part supported by visual perception, without the need for iconic memory. At the 1,000 ms retention interval, where iconic memory cannot combine the two half-displays, performance was slightly higher than the single-display condition $\left(t\right(34)=6.23,p<.001)$. We suspect that in this condition participants are effectively given two chances to perceive the letters within each half-display, leading to a slight boost in performance over the single-display condition. Critically, performance was far better in the iconic memory conditions than the 1,000 ms ISI condition, for both the 16 ms ISI $\left(t\right(34)=16.19,p<.001)$ the 50 ms ISI $\left(t\right(34)=9.08,p<.001)$. Taken together these results suggest that performance in this task can be in part supported by visual perception of a single half-display, but in conditions with brief ISIs combining each half-display in iconic memory allows for a significant increase in performance.

Figure 7.

Mean Accuracy (A) and Correlations Across Participants (B) in Experiment 3

Note. Relationships are shown for the single-display condition (“single”), and the two-display conditions with interstimulus intervals of 16, 50, and 1,000 ms. Error bars in (A) are standard errors of the mean, insert in (B) shows the scree plot corresponding to the correlation matrix.

Performance on the letter-identification task reflects many processes, any of which may contribute to individual differences in task performance. Because different processes are more or less important in the different conditions of Experiment 3, the correlations across conditions can be used to measure the extent to which the various processes contribute to individual differences. For example, if individual differences in iconic memory abilities are important, than the short retention-interval conditions should be more correlated with one another than they are with the single-display or long retention-interval conditions. Alternatively, if perception or working memory are driving performance differences, the single-display and long retention-interval conditions should be more correlated with one another than they are with the shorter retention-interval conditions that are clearly supported by iconic memory (given the higher accuracy in these conditions). However, as can be seen in the correlation matrix (Figure 7B), the correlations across every pair of conditions are high and are all remarkably similar. A factor analysis clearly implies a one-factor solution (Horn’s parallel analysis, and the scree plot in Figure 7B), suggesting that an individual’s performance across all four conditions can be explained by a single latent factor. Therefore, although performance at the 50 ms ISI is clearly boosted by iconic memory, individual differences in this condition are not driven by differences in iconic memory ability. Instead, individual differences in the letter-identification task are independent of the ISI, much as in Experiment 1 where individual differences were similar across retention intervals. Taken together, individual differences on all of these tasks appear to reflect very general mechanisms that are common to simple perceptual tasks, rather than particular aspects of iconic or working memory storage.

General Discussion

Understanding why performance on visual working memory tasks varies across people and displays has been central to theory development. Performance on such tasks relies on many processes including perception, iconic memory, attention, working memory and decision making, and variability in any of these might cause individual differences and stimulus differences in task performance. However, the results of Experiment 1 show that all such variability in a working memory task, in which the memory retention interval was one second, is also present in an iconic memory task in which retention intervals were as brief as 33 ms. This shared variability in iconic and working memory performance cannot be explained by limitations in working memory capacity, as performance in iconic memory conditions is not limited by the working memory storage bottleneck. Instead, these results suggest that person and display variability in working memory tasks must arise from variability in mechanisms other than limitations in working memory storage capacity.

If performance differences across individuals and displays are not caused by differences in working memory capacity, then what other processes might be driving these large and systematic effects? Variability in visual perception will surely cause variability in working memory performance, as perceiving information is a necessary precursor to storing it. Moreover, if visual working memory storage is carried out using the same neural machinery as visual perception, as some findings suggest (Harrison & Tong, 2009), then any differences in perception should be accompanied by similar differences in working memory storage. For example, perceptual mechanisms cause differences in the precision of visual memory across particular colors (Bae et al., 2014) or orientations (Pratte et al., 2017). It is also possible that differences in color perception across people might cause subtle variation in how particular colors are seen (e.g., see Emery & Webster, 2019). However, in the change-detection task used here the color stimuli were clearly distinguishable, such that color perception would have to be clinically impaired for some people, and for some displays, to explain the large performance effects observed in both iconic and working memory. Moreover, in Experiment 2 the relationship between iconic and working memory remained even though the iconic memory task differed substantially from the change-detection task used to measure working memory, particularly in that it had no color component. The source of these effects must therefore be in a processes that occurs after the early stages of visual perception.

The next stage in visual processing is often thought to involve iconic memory (Atkinson & Shiffrin, 1968), such that differences in iconic memory capacity could affect performance in any task that relies on vision. At first glance the results of Experiment 2 suggest a role of iconic memory in causing individual differences in working memory performance: the letter-identification task was designed to measure iconic memory, and performance on that task was correlated with performance on the change-detection working memory task. However, the results of Experiment 3 show that individual differences in the letter-identification task are similar regardless of whether iconic memory supports task performance. In particular, performance in conditions that at least in part rely on iconic memory (i.e., short ISIs) are equally correlated with one another as they are with conditions that do not require iconic memory (such as when a single half-display is shown). Taken together, these results suggest that the relationship between iconic and working memory tasks is not driven by individual differences in iconic memory processes. Instead, there must be differences in processes common to both iconic and working memory tasks, other than capacity limits on iconic or working memory storage per se.

Variability in attentional processes across people and displays is another potential source of variability in iconic and working memory tasks. Although early theories of iconic memory implied that iconic memory was a preattentive store (Sperling, 1960), recent work suggests that attention may play a critical role in iconic memory storage (Mack et al., 2015, 2016; Persuh et al., 2012; Ruff et al., 2007). The transfer of information from iconic to working memory is also thought to depend on attention (Atkinson & Shiffrin, 1968), and working memory itself is thought to be highly intertwined with attentional processes (e.g., see Fougnie, 2008; Oberauer, 2019). There are therefore several ways that individual differences in attention might cause variability in visual memory. However, the finding that this variability is so similar across iconic and working memory tasks provides constraint on how attention might be affecting them. For example, Cowan et al. (2005) proposed that a limited pool of attentional resources is what causes the capacity limits observed in working memory. However, it is not clear how such a limit could explain the concurrent variability in iconic and working memory. Although working memory might be limited by how many items can be perceived simultaneously (e.g., Cavanagh & Alvarez, 2005; Tsubomi et al., 2013), iconic memory performance clearly circumvents this limitation as its capacity is far greater than the few items thought to define the limits of attention. In iconic memory conditions the item probe directs attention to a single item while it is still stored in iconic memory, such that any attentional capacity limit, or differences in this limit across displays and people, should not affect iconic memory performance as long as capacity is more than one item. Therefore, a limit in the amount of available attentional resources cannot account for the nearly identical individual and display effects seen in iconic and working memory tasks.

Characteristics of attention other than its capacity have been suggested to drive individual differences in working memory performance. For example, performance on visual working memory tasks is highly correlated with measures of attentional control, such as antisaccade performance (Unsworth et al., 2015). Moreover, when irrelevant distractor stimuli are included in the memory study array, neural measures of working memory storage suggest that the irrelevant items are more likely to be loaded into memory for participants with low working memory performance (Vogel et al., 2005). Similarly, low-capacity individuals are worse at suppressing automatic attentional shifts to salient but irrelevant distractor stimuli (Gaspar et al., 2016). It has therefore been suggested that individual differences in attentional control affect working memory performance due to differences in encoding strategies: individuals with poor attentional control attempt, but fail, to encode the entire display in a working memory task. Alternatively, individuals with better attentional control take a more strategic approach of selecting a subset of stimuli to encode, providing for better working memory performance (Cusack et al., 2009). However, this account cannot explain the concurrent variability in iconic memory performance observed here. If participants who performed well in working memory conditions did so by selectively encoding only a few stimuli on each trial, then they should perform worse in iconic memory conditions than participants who attempt (and should nearly succeed) to encode the entire display. But the opposite pattern is clear: individuals who performed well in working memory conditions also performed well in iconic memory conditions, suggesting that these effects do not reflect differences in how many items participants attempt to encode on each trial.

There are other ways that individual differences in attentional control might affect visual memory performance. For example, one’s ability to control attention might affect the degree to which attention fluctuates from trial to trial: a better ability to maintain attentional focus on the task at hand during an experimental trial should lead to higher estimates of memory capacity. Such an attentional focus account could explain why we observe identical individual differences at both short and long retention intervals: people exhibiting poorer performance are simply not as focused on the task. The notion that some people are just better at maintaining attention throughout boring tasks explains the more general observation that nearly all assessments of cognitive ability are correlated with one another to some extent, a pattern Spearman called the positive manifold (Spearman, 1904). Individual differences in maintaining attention might affect the rate at which participants exhibit complete lapses on some trials, especially as low working memory capacity is predictive of more frequent mind wandering (Mrazek et al., 2012). However, rather than complete lapses of attention on some trials, recent work suggests that there are graded fluctuations of attention from trial to trial, and that individuals with lower capacity simply have more frequent “partial disengagement” of attention (Adam et al., 2015). These attentional fluctuations have been shown to affect stimulus encoding (Murray et al., 2011; Myers et al., 2014), which would explain our finding of similar variability in iconic and working memory tasks. However, other work suggests that attentional fluctuations affect working memory storage in particular, rather than just the perceptual processes that precede it (Adam et al., 2015, 2018). But if that is the case, and individual differences in attentional control are what cause differences in working memory performance, then we should not have observed similar effects under iconic and working memory conditions.

Using a hierarchical Bayesian modeling approach in Experiment 1 we found that task performance differed across particular study displays to a similar degree as it did across individuals. Critically, this systematic variability across displays was nearly identical under iconic memory and working memory conditions: if performance is relatively high for a particular display under working memory conditions, it will be high under iconic memory conditions with far shorter retention intervals. Like individual differences, this equivalency across iconic and working memory constrains the possible loci of these effects. For example, iconic and working memory performance was superior for items in the top half of the display, and for items with similar features as were probed on the previous trial. However, because these effects were equally present in iconic and working memory conditions, they cannot arise from processing differences specific to working memory storage. Instead, we suspect that differences in attentional allocation—to the top of the display and to items that are similar to those on previous trials—affect both encoding of the study display and the degree to which items are maintained in the focus of attention during the subsequent memory retention period. For example, for the trial shown in Figure 5A performance is near ceiling and hardly decays at longer retention intervals. Therefore, whatever causes this trial to be particularly easy has little to do with working memory storage, or higher level processes that are thought to occur in working memory such as the formation of perceptual chunks (Nassar et al., 2018). Instead, attention appears to be biased toward the probed item on this trial during study, and remains so throughout the retention interval as would be expected from bottom-up, automatic attentional biases, which have been shown to have large effects on working memory performance (e.g., Constant & Liesefeld, 2021; Ravizza & Conn, 2022).

Many aspects of performance on a working memory task can be attributed to perceptual processes or other mechanisms that precede working memory storage. Here we found that a fundamental aspect of working memory performance—substantial individual differences and display effects—are nearly identical in an iconic memory task. Although there remain several possible sources of these effects, such as differences in attentional control across people and attentional allocation across study displays, the results rule out working memory storage as the cause of such variability. Moreover, these results imply that other individual differences that covary with visual working memory performance, including fluid intelligence, will covary to a similar degree with performance on an iconic memory task. This implication is surprising, as is it not clear why variability on a low-level perceptual task would predict performance on a high-level construct such as fluid intelligence. Further work could reveal sources of variability in intelligence that cannot be explained by variability in iconic memory, such as by using a confirmatory factor analytic approach (e.g., Unsworth & Spillers, 2010). However, that iconic memory performance can account for differences in working memory performance, across people and displays, implies that performance differences in visual working memory tasks might have little do with visual working memory capacity limits.