Working memory retrieval as a decision process

Abstract

Working memory (WM) is a core cognitive process fundamental to human behavior, yet the mechanisms underlying it remain highly controversial. Here we provide a new framework for understanding retrieval of information from WM, conceptualizing it as a decision based on the quality of internal evidence. Recent findings have demonstrated that precision of WM decreases with memory load. If WM retrieval uses a decision process that depends on memory quality, systematic changes in response time distribution should occur as a function of WM precision. We asked participants to view sample arrays and, after a delay, report the direction of change in location or orientation of a probe. As WM precision deteriorated with increasing memory load, retrieval time increased systematically. Crucially, the shape of reaction time distributions was consistent with a linear accumulator decision process. Varying either task relevance of items or maintenance duration influenced memory precision, with corresponding shifts in retrieval time. These results provide strong support for a decision-making account of WM retrieval based on noisy storage of items. Furthermore, they show that encoding, maintenance, and retrieval in WM need not be considered as separate processes, but may instead be conceptually unified as operations on the same noise-limited, neural representation.

Introduction

Working memory (WM) is a fundamental cognitive process that underpins many everyday behaviors. In the visual domain, WM selectively preserves salient information over time (Baddeley, 2003; Sperling, 1960) and across eye movements (Henderson, 2008; Irwin, 1991), to guide efficient exploration of the environment (Soto, Humphreys, & Rotshtein, 2007). However, despite the fact that we understand many aspects of visual WM, there is no consensus on underlying mechanisms (see for example Bays & Husain, 2008; Rouder et al., 2008; Zhang & Luck, 2008). In recent years, most studies designed to examine WM mechanism have focused on the pattern of errors made by participants, and these have proven to be crucial to current debates (Bays & Husain, 2008; Emrich & Ferber, 2012; Rouder et al., 2008; van den Berg, Shin, Chou, George, & Ma, 2012; Wilken & Ma, 2004; Zhang & Luck, 2008). By contrast, there has been far less emphasis on response times (RTs). Although these were of fundamental interest in the past (Sternberg, 1966), mental chronometry approaches to WM have fallen out of favor, perhaps because influential proposals based on RTs, such as Sternberg's serial scanning model, were found to be inadequate (for a review see Greene, 1992).

It is nevertheless clear that response times can provide highly informative, mechanistic perspectives on cognitive processes (Jensen, 2006; Luce, 1986; Posner, 1978). In the field of perceptual judgment, for example, critical insights have come from studying choices and their timing, treating both as the product of neural decision mechanisms (Palmer, Huk, & Shadlen, 2005). A single general decision format capable of explaining data from behavior as well as neurophysiology has emerged (Bogacz, Brown, Moehlis, Holmes, & Cohen, 2006; Gold & Shadlen, 2007; Smith & Ratcliff, 2004): a scalar decision variable accrues evidence in support of potential choices over time, and the first to reach a threshold or “bound” is chosen (Carpenter & Williams, 1995; Laming, 1968; Ratcliff, 1978; Usher & McClelland, 2001). Resulting decision models have proven powerful: They can explain response times for judgments about sensory features such as brightness and motion (Ratcliff, 2002; Roitman & Shadlen, 2002); they capture details of the neural activity in various brain regions underlying such decisions (Hanes & Schall, 1996; Shadlen & Newsome, 2001); and they can adjudicate between different hypotheses about experimental influences on behavior (de Gardelle & Summerfield, 2011; Hanks, Ditterich, & Shadlen, 2006; Yang & Shadlen, 2007). As well as simple sensory judgments, similar decision processes are thought to govern responses in value-based decisions (Hare, Schultz, Camerer, O'Doherty, & Rangel, 2011).

These findings suggest that general principles might apply across different types of cognitive process. Here we ask whether retrieval of information from visual WM can also usefully be conceptualized as a decision that leads to a choice—one that operates on internal evidence, analogous to when choices are based on external sensory evidence. To address this question we measure how RT distributions are influenced by the quality of memory. Our approach is inspired by research that has led to the proposal that there is a very limited WM resource available to hold items over short durations (Bays & Husain, 2008; Palmer, 1990; Salmela, Lähde, & Saarinen, 2012; Wilken & Ma, 2004). Experimental findings suggest that variability of WM representations around the true value increases with memory load, so precision of recall for object attributes falls with increasing number of items. Such a limited resource account of WM representations is based on the principle that memory representations are noisy, and get noisier with increasing memory load (Bays & Husain, 2008; Palmer, 1990; Salmela et al., 2012; Wilken & Ma, 2004). If retrieval from WM uses a decision process that depends upon the quality of evidence, we would expect to see systematic changes in RT distribution as a function of quality of memory, as indexed by the variability or precision of report. Moreover, if such effects apply across a range of WM paradigms, this would suggest a fundamental principle that can provide crucial insights into the mechanisms underlying WM.

The decision model we use to interpret our data is Linear Approach to Threshold with Ergodic Rate (LATER) (Carpenter & Williams, 1995; Reddi, Asrress, & Carpenter, 2003). In most respects LATER resembles the standard accumulator-type decision model, but it has the unique feature that the rate of accumulation is fixed within trials (varying only stochastically from trial to trial), making it the simplest “rise to threshold” variant (Figure 1A and B). It can adequately explain performance in brightness and motion judgments, as well as several other two-choice variants, using comparatively few parameters (Anderson & Carpenter, 2010; Carpenter, Reddi, & Anderson, 2009; Emeric et al., 2007; Halliday & Carpenter, 2010; Noorani, Gao, Pearson, & Carpenter, 2011; Sinha, Brown, & Carpenter, 2006; Story & Carpenter, 2008). Moreover, it accommodates decisions between more than two alternatives, a problem with which many other models are still grappling (Churchland, Kiani, & Shadlen, 2008; Krajbich & Rangel, 2011).

Figure 1.

The LATER model in working memory. (A) A decision variable or signal, representing commitment to a particular response, is set off by a stimulus (in this case the memory probe, a response prompt), rises linearly from an initial level (S₀) at a rate picked from a normal distribution with mean μ and SD σ, and initiates a response upon reaching threshold (S_T). The resulting RT distribution is skewed. (B) This is reflected in an asymmetric cumulative density function. However, when plotted on a reciprocal time x axis and probit y axis (a reciprobit plot) it is linear, allowing parameters such as μ and σ to be estimated easily. Increasing memory load reduces the memory resource available per item (Bays & Husain, 2008; Wilken & Ma, 2004). This could influence the LATER mechanism in two ways: reducing mean rate of rise μ (C), which would be evident in self-parallel shifts of the reciprobit RT distribution (D), or raising decision threshold (E), causing reciprobit swivel (F).

According to LATER, response time can only be influenced by a change in the quality of evidence, reflected either in the rate of rise of evidence or signal (μ) or the position of the decision threshold relative to priors (S_T-S_0, signal threshold and initial signal level, respectively). Importantly, these different mechanisms can be distinguished by their effects on RT. Thus a change in only μ translates and stretches the distribution to longer RT (Reddi et al., 2003) (Figure 1C), resulting in a pure horizontal shift on reciprobit axes (Figure 1D). Changes in the other parameters (Figure 1E), by contrast, alter the slope in reciprobit plots (Figure 1F). Existing decision studies have found that the coherence of sensory evidence primarily affects the rate of rise or mean drift rate in decision models (Ratcliff, 1978; Reddi et al., 2003) and in neuronal spike rates (Roitman & Shadlen, 2002; Shadlen & Newsome, 2001).

If memory guides choice by way of a decision mechanism acting on internal rather than sensory evidence, we would make several key predictions about memory retrieval.

Prediction 1: Response time to retrieve an item should increase with memory load

Having a coarser memory of an item should provide sparser evidence to the decision process, and thereby slow responses. One way to lower precision of memory is simply by increasing the number of items to be remembered (Bays & Husain, 2008; Palmer, 1990; Salmela et al., 2012; Wilken & Ma, 2004), so increasing memory load should lead to systematic increases in RT.

Prediction 2: Rate of rise of evidence accumulation (μ) for the decision process should vary linearly with memory precision

Importantly, the decision model parameter μ can be estimated directly from individual subjects' RT distributions, while memory precision can be estimated from performance variability, allowing us to ask whether precision is the currency drawn upon by the decision process.

Prediction 3: Alternative manipulations of memory precision should also have predictable consequences for RT

Tasks in which memory load is fixed but memory precision is manipulated by other means should nonetheless produce corresponding, predictable changes in RT. For example, if memory resources are prioritized to an item by cueing or are degraded by an extended maintenance delay period, responses should be respectively speeded and slowed in a pattern predicted by the model.

Methods

Experiments 1 and 2: Procedure

A total of 33 participants (14 female, aged 18–37) with normal or corrected-to-normal visual acuity were tested after giving informed consent. The experiments conformed to the Declaration of Helsinki and were approved by University College London's NHS Research Ethics Committee. To guarantee naïveté as to the aims of the experiment, no experimenters from the research team acted as subjects, and the aims of the experiment were only revealed to the subjects once they had finished.

Participants were seated 70 cm from a 21 in. CRT monitor with a refresh rate of 140 Hz. A depiction of a typical trial and details of events and timings are given in Figure 2A and 2B (the stimuli used and design of the tasks are similar to those reported in Bays & Husain, 2008). Experiment 1 was designed in C++ and run under Windows XP. Experiment 2 was designed and run in Matlab 7.10 using the Psychophysics Toolbox (Version 3) extensions (Brainard, 1997).

Figure 2.

Measuring precision of location and orientation memory. (A) Stimuli and sequence of events on a location-judgment trial. The array featured one, two, four, or six items; here an array size (N) of two items is shown. After the sample display is blanked, subjects' memory for location of a randomly chosen item is tested by redisplaying the item displaced horizontally through distance Δ (0.5°, 2°, or 5°). The subject presses a button to report the direction of displacement. (B) An orientation judgment trial (this time with a set size of four items). A randomly chosen item is redisplayed, rotated through an angle Δ (5°, 20°, or 45°). Red circles indicate gaze position. (C) For each memory load, performance (proportion of responses judging the displacement or rotation as away from fixation) is plotted as a function of the actual displacement or rotation magnitude Δ. Memory precision is measured as the reciprocal of the SD of the fitted cumulative Gaussian.

In Experiment 1, while subjects fixated a cross to one side of the screen, an array of one, two, four, or six items was presented for 1000 ms to the other side, centered 10° from fixation. The stimuli differed for the two tasks: for the location task they consisted of colored squares (0.8° × 0.8°); for the orientation task, randomly oriented colored arrows (radius 1.25°). For the orientation task Bays and Husain (2008) used a parameter space comprising the full 360° of possible arrow orientations, whereas we constrained our stimuli to a 180° range (the upper half of the circle). The rationale for this change was to avoid the possibility of ambiguity in responses to large amplitude rotations, which could in theory be judged either clockwise or anticlockwise in a 360° space. In both tasks all items were located in random positions within the boundaries of an invisible square (9° × 9°), at least 3° apart. Highly distinguishable colors (red, green, blue, yellow, cyan, white) were randomly selected for the items, without repetition. After a 500 ms blank delay period, a single item from the sample display—the probe—was redrawn slightly displaced (location task) from its original position, or slightly rotated (orientation task). Three levels of difficulty (Δ) were used: for the location task, 0.5°, 2°, or 5° of leftwards or rightwards displacement and for the orientation task, 5°, 20°, or 45° of clockwise or counterclockwise rotation. Subjects were instructed to report the direction of the change as quickly and accurately as possible by pressing the left or right arrow key on a number pad. Response time was recorded from the presentation of the probe.

Eye position was monitored online at 1000 Hz using a frame-mounted infrared eye tracker (Eyelink 1000, SR Research), and trials were repeated if eye position deviated more than 2° from the fixation cross. The position of the fixation cross alternated between the left and right side of the screen trial by trial. Thirty-one participants took part in Experiment 1, each performing either the location or orientation task (except for six subjects who did both tasks, counterbalanced for the order in which they were done), resulting in 19 datasets for the location and 18 for the orientation task. Participants undertook four blocks of 48 trials in total, save for four of the location task subjects, who completed a longer version (32 blocks or 1,536 trials each, with breaks) to provide larger datasets, so that response accuracy (psychophysical performance) and response times could be compared on a within-subject basis. One of these four participants was subsequently excluded from the analysis (see below).

The two participants taking part in Experiment 2 had not been involved in Experiment 1 and were again naïve to its aims. The design of Experiment 2 was similar to that of the orientation task in Experiment 1, except that the stimulus always consisted of two items, the magnitude of the rotation of the target item was always Δ = 20°, and the stimulus array was preceded by a square cue (0.8° × 0.8°) shown at fixation. The square was shown for 1 s, followed by a gap of 500 ms and then the fixation cross, after which trials proceeded as for Experiment 1. Trials were split into three different types: In 50% of trials the square was filled with the color of the target item in that trial (VALID trials), in 25% it was filled with the color of the other item in the stimulus array (INVALID trials), and in the remaining 25% of trials the square was an empty black frame (NEUTRAL trials). Both participants completed 19 blocks of 32 trials.

Experiments 1 and 2: Analysis

Individual subjects' response times were analyzed separately for each condition, meaning each combination of array size N and displacement or rotation magnitude Δ. Trials with the same displacement Δ towards and away from fixation did not differ in RT and were collapsed together. The data for one location task subject were excluded from the analysis due to anomalously low accuracy, leaving 18 datasets for this task. Because of considerable intersubject variability in response time, for the group results (Figure 3, top) response time and its associated error bars represents the mean ± SEM of the 18 individual subjects' scaled median response times (τ), calculated as

Figure 3.

Set size and memory discrimination difficulty affect response time for both location and orientation tasks. Top: group average of median correct RT for 18 subjects as a function of set size N and displacement/rotation size Δ; error bars represent SEM calculated after excluding variability associated with between-subject differences (Cousineau, 2005). Bottom: group memory precision (1/error function SD, see Figure 2) for 18 subjects as a function of set size N.

They thus represent the residual intrasubject SEM as calculated according to Cousineau (2005). However, for the purposes of statistical comparison raw response times were entered in a four-by-three way repeated-measures analysis of variance (ANOVA) on number of array items (one, two, four, six) and displacement (0.5°, 2°, 5° for the location task or 5°, 20°, 45° for the orientation task). Memory precision (Figure 3, bottom) was calculated from the group's psychophysical performance function at each array size N. A cumulative Gaussian regression model was fit to the relationship between response probability and stimulus displacement Δ for each array size, as reported previously in Bays and Husain (2008). Similar output measures were recorded: The standard deviation σ of the best-fitting Gaussian provided an estimate of response variability, and 1/σ a measure of memory precision.

For the four participants who provided larger datasets, psychophysical performance (measured as described above) and response times were analyzed individually. Results for one of these subjects could not be used (see above) because she had anomalously low accuracy and highly aberrant RT distributions that were not amenable to the following conventional analysis. To test whether distributions of reciprocal response time were normally distributed (and that we were justified modelling them in LATER terms) we created reciprobit plots (see Figure 1) of correct RT for each subject and condition: 1/RT was plotted cumulatively on a probit y axis (a skewed cumulative probability axis upon which Gaussian distributions are linear), a transform that yields straight lines with median μ and slope 1/σ, direct estimates of the corresponding LATER model parameters. We used the Kolmogorov-Smirnov one-sample test to test for deviation from the normal distribution. In conditions in which large numbers of error responses are produced, reciprobit distributions are known to deviate from linear in the (long RT) tail, making them unsuitable for normality testing (Noorani et al., 2011). We therefore censored (only from this normality analysis) conditions in which more than 10% of errors were made.

In order to determine which of the two alternative models in Figure 1C and E better captured the effect of N and Δ on RT, using a likelihood ratio test in bespoke software (SPIC, Carpenter, 1994) we tested two possible distribution transformations (Figure 1D and F): self-parallel shift of the recinormal distribution along the rate axis, and rotation (swivel) around a common intercept on the infinite time axis. We carried out seven such comparisons for each of the three subjects, one for each value of N to test the effect of only changing Δ (four), as well as one for each Δ to test the effect of only changing N (three). Out of the 21 comparisons carried out, 12 were significantly in favor of distribution shift, and only one (for Δ = 0.5 for one subject i.e., for conditions in which numerous errors distort the distributions) was in favor of swivel. Log likelihood ratios (LLRs) from individual comparisons were summed across subjects and conditions to determine which hypothesis (shift or swivel) offered a superior fit overall. We carried out a single likelihood ratio test for each subject in Experiment 2, testing for shift or swivel of the 1/RT distributions shown in Figure 5, and again summed the LLRs.

Figure 5.

Influence of working memory allocation on response time. RT distributions for two subjects in different cueing conditions, on reciprobit axes (reciprocal RT is plotted cumulatively on a probit ordinate). One of the two items was endogenously precued by color, either the item subsequently probed (valid cue) or the other, nontarget item (invalid cue); in 25% of trials the cue was neutral. X axis labels show 1/RT (response rate) values and corresponding RTs.

Experiment 3: Procedure

Ten neurologically normal subjects (five female, aged 18–29), with normal or corrected-to-normal visual acuity, took part in Experiment 3. The experiment was designed in C++ and run under Windows XP. Stimuli were presented using the same experimental setup as described above. Each memory array consisted of one or six oriented bars presented on a gray background in an imaginary circle (radius 4.4°) around fixation. In the six-item case, all bars were equally spaced and bars differed by at least 10° in orientation, which was otherwise random. The colors of the bars in each trial were randomly selected out of eight highly discriminable colors.

Each trial began with the presentation of a central white fixation cross (0.8° diameter) for 500 ms, followed by a test array of oriented bars shown for 500 ms. A memory probe, consisting of a centrally positioned bar of one of the colors in the memory array, and having random orientation, was then presented at 0.1, 1, 2, or 3 s after disappearance of the array. Using a response dial (PowerMate USB Multimedia controller, Griffin Technology, Nashville, TN) subjects rotated the randomly oriented probe bar to match the remembered orientation of the target item (the bar of the same color in the test array). Each of the participants performed between 10 and 15 sessions of 40 trials, 20 trials for each of the array sizes, five of which used each delay duration.

Experiment 3: Analysis

In each trial raw error was recorded as the angular deviation between the orientation reported by the subject and the orientation of the target item. Distributions of error were fitted using a mixture model incorporating responses of three distinct types: those directed at the target, those directed at a different (non) target and responses distributed randomly. A detailed description of the model is depicted in previous manuscripts (e.g., Bays, Catalao, & Husain, 2009). Maximum likelihood estimates (Myung, 2003) of the model parameters were obtained separately for each subject and condition using an expectation maximization algorithm (MATLAB code available at: http://bayslab.com/code/JV10/). This process yields the standard deviation of error for responses directed at the target, taken here as the reciprocal of memory precision, as well as the proportion of nontarget and random responses. Response time was recorded as the time elapsed between appearance of the probe bar and the subject pressing the dial to register the response. Both sets of raw data were entered in a two-by-four way repeated-measures ANOVA on array size (one or six) and delay (0.1, 1, 2, or 3 s). However, for the purposes of Figure 6, RTs were corrected for intersubject variability, as for Experiment 1 above (Cousineau, 2005).

Figure 6.

Influence of delay period on memory precision and response time. (A) Group mean (± SEM) of the standard deviation (precision⁻¹) for subjects' matching responses in a task testing free report from memory of stimulus orientation, as a function of the number of items presented and the delay before the cue to respond. (B) Group median response time for the same conditions, reflecting the combined duration of memory recall and rotation of the dial to the remembered orientation (± within-subject SEM, calculated after excluding variability associated with between-subject differences, as described in Cousineau, 2005).

Results

We tested the effect of memory load on retrieval time using a change discrimination task in which subjects viewed sets of colored blocks or oriented lines on a screen, and reported the direction of change in position or orientation of one of the items when it reappeared after a delay (Figure 2). The number of items to be remembered (N), and judgment difficulty (magnitude of change to target stimulus after the delay, Δ), were manipulated across trials in a full factorial design. Letters or digits have been used in most previous studies of memory retrieval dynamics, forcing researchers to use recognition tasks in which subjects indicate whether or not a new visual stimulus differs from a reference set (Hockley, 1984; Ratcliff, 1978; Sternberg, 1966, 1969). Such recognition tasks are inherently asymmetric, introducing a choice bias. We used the location and orientation of simple geometric shapes in this study because these features are symmetric and continuous (they may be varied parametrically) and their representation in memory is well-characterized (Bays et al., 2009; Bays & Husain, 2008).

Prediction 1: Retrieval time should increase with memory load

A coarser memory representation of an item should provide sparser evidence to the decision process, demanding longer evidence accumulation and thereby slowing responses. One of the ways in which precision of memory recall can be varied is by increasing memory load (Bays & Husain, 2008; Palmer, 1990; Salmela et al., 2012; Wilken & Ma, 2004). Figure 3 (top) shows how response times for judgments of both location and orientation were influenced by number of items (N) to be remembered as well as difficulty of discrimination (Δ, size of the displacement). Consistent with Prediction 1, there was a powerful effect of memory load. As subjects remembered more items, they took significantly longer to discriminate the change in one of them, for all values of Δ, main effect of N, location task: F(3, 51) = 34.3, p < 0.001; orientation task: F(3,51) = 45.2, p < 0.001. Response times were also significantly longer the smaller the displacement to be discriminated from memory, location task: F(2, 34) = 10.3, p < 0.001; orientation task: F(2, 34) = 19.1, p < 0.001. Figure 3 (bottom), which reports response accuracy for the group, replicates the finding that memory precision declines initially rapidly and then more slowly with increasing set size (Bays & Husain, 2008).

Prediction 2: Rate of rise of evidence in the decision process should vary linearly with memory precision

In principle, numerous models for the way response time depends on evidence in memory could potentially account for the observed median response times, but details of the shape or fine structure of RT distributions can distinguish between models (Noorani & Carpenter, 2011). Idiosyncrasies in individuals' response times result in this information being lost when results are combined across subjects, so to investigate Prediction 2 more thoroughly we recorded and analyzed separately larger datasets from three subjects on the location task (totaling some 4,680 trials).

The LATER model (Figure 1) predicts that not response time but its reciprocal, analogous to response rate in Hertz, will be distributed normally. Figure 4A shows example response time distributions for two subjects, for the effect of N with Δ held constant (Figure 4A, top) and vice versa (Figure 4A, bottom). In each condition the reciprocal of response time is plotted cumulatively on a probit scale (a reciprocal-probit, or reciprobit, plot), upon which a Gaussian distribution follows a straight line. For the conditions in which fewer than 10% errors were produced (for an explanation of why this is necessary, see Methods), the distributions of 1/RT for none of the three subjects deviated from normality (Kolmogorov-Smirnov one-sample test, p > 0.05). Satisfying this core requirement of LATER strengthens the argument that memory retrieval is a process that evolves over time with a rate distributed across trials in a Gaussian fashion.

Figure 4.

Influence of set size and probe displacement on reciprocal RT and the LATER model parameter μ. (A, top) Example RT distributions from two subjects for different set size conditions (all for Δ = 5°) on reciprobit axes, in which reciprocal RT is plotted cumulatively on a probit ordinate. X axis labels show 1/RT (response rate) values and corresponding RTs. (A, bottom) Example RT distributions from two subjects for different displacements (all for N = 1), reciprobit axes. Note that data points in green are identical because they represent the same condition. (B) Values for the decision model parameter μ (measured from the RT distributions in A) for three subjects as a function of array size. Individual subjects' values were scaled relative to the group mean, as described in Cousineau (2005). (C) The same values for μ as a function of subjects' memory precision (1/error function SD, see Figure 2).

The reciprobit plots in Figure 4A reveal further that distributions of 1/RT for conditions differing only in the value of N or of Δ are shifted in a self-parallel manner along the rate axis, as in Figure 1D, rather than exhibiting the change in slope shown in Figure 1F, as confirmed by comparing the two alternatives using likelihood methods (log likelihood ratio [LLR] = 122.1, equivalent to p < 0.0001; for details see Methods). In LATER model terms, this means that only the mean rate of evidence accrual (μ)—and not its variance (σ²), start point (S₀), or threshold (S_T)—changes between conditions. But does μ vary in proportion to the evidence from which it is derived (memory precision), as we would predict?

For each subject and condition we estimated the LATER parameter μ, keeping all other parameters fixed across conditions (see Methods). μ varies as a nonlinear function of array size (Figure 4B), a relationship predicted by the nonlinear trend in RT observed in the population curves (Figure 3). The individual subjects' memory precision values were estimated from their performance functions at each set size (according to procedures reported previously in Bays and Husain [2008], see Methods). Figure 4C shows that μ varies linearly with memory precision. Considering each of the three displacement values separately, μ was strongly correlated with memory precision (R²₁₀ > 0.975, p < 0.02). This key finding supports the theory that, when memory is used to guide behavior, the degree of noise in the representation of a stored visual feature determines the quality of evidence supplied to the decision mechanism.

Prediction 3: Manipulations of memory precision should have predictable consequences for RT

A decision-making account of retrieval from memory predicts analogous effects on RT when set size and stimuli are held constant, but other interventions known to influence memory precision—such as item cueing or maintenance delays—are used to manipulate memory representations (e.g., Bays & Husain, 2008). According to the decision model, the effects these manipulations have on memory precision should be accompanied by corresponding changes in the parameter μ, yielding parallel reciprobit distributions.

We tested this prediction in two new subjects performing a version of the orientation task in which only arrays of two randomly oriented bars were used. One of the oriented bars was cued by a square of its color appearing at fixation before the trial began, and either the cued bar or the other bar was subsequently tested (VALID and INVALID trials, respectively). In NEUTRAL trials an uninformative cue appeared.

Figure 5 shows that, relative to NEUTRAL trials, on trials with cueing the cued item experienced an RT advantage (VALID trials), and there was a response time cost for the other item (INVALID trials). We compared the individual subjects' latency distributions in pairs, VALID versus NEUTRAL and NEUTRAL versus INVALID. None of the three distributions of 1/RT for either of the subjects deviated from normality (Kolmogorov-Smirnov one-sample test, p > 0.05). Mean reciprocal RT differed significantly for both comparisons for both subjects (t > 2.9, p < 0.005, using an unpaired t test because the number of trials differed between conditions). All reciprocal latency distributions differed significantly from one another (Kolmogorov-Smirnov two-sample test, p < 0.001). This difference represented a parallel shift with μ increased for the cued item and reduced for the uncued one (summed LLR = 5.259, equivalent to p = 0.015). These effects are consistent with Prediction 3: since cueing increases memory precision for one item at the expense of others, we could expect the decision mechanism to accrue evidence faster (higher μ) than in the neutral condition for the cued item and slower for the item not cued, altering RT systematically across the conditions as found.

To demonstrate how widely applicable the decision model explanation is, we next manipulated the quality of the information in memory in a different way, by adjusting the length of the delay period before the probe appeared. Furthermore, to test whether our decision-making approach applies beyond two-alternative forced-choice situations, we used a task in which subjects reported remembered visual orientations freely by rotating a dial. This analog report method has seen recent success in explaining the errors people make in remembering locations and orientations (Bays et al., 2009; Bays, Wu, & Husain, 2011; Gorgoraptis, Catalao, Bays, & Husain, 2011; Pertzov, Bays, Joseph, & Husain, 2013; Wilken & Ma, 2004; Zhang & Luck, 2008), but has yet to be used in studies of the timing of memory retrieval.

Our 10 subjects were shown either one or six colored bars and were asked to remember their orientation. After a delay of 100 ms, 1, 2, or 3 s, the colored probe bar appeared and subjects responded by rotating the probe—using the dial—to the orientation in which they remembered the target item. This free report procedure yields a distribution of errors across trials, for which the standard deviation of the distribution of response errors serves as an inverse measure of memory precision (a parameter estimated instead for the fitted psychophysical error functions used in the experiments above). Though inevitably incorporating delays due to rotating the dial, the time at which the dial was pressed to register the matched orientation was taken as response time, permitting a comparison to be made between an analog scale of memory error and corresponding RT.

As shown in Figure 6, at the shortest delay, memory was more accurate for the one-item than the six-item condition, and the effect was exaggerated by the length of the delay. There were significant main effects of load, F(1, 9) = 27.8, p = 0.001, and of delay, F(3, 27) = 9.4, p < 0.001. However, although the length of the delay only mildly influenced memory error when one feature was stored, there was a profound degradation of memory with six items, leading to a significant interaction between load and delay, F(3, 27) = 5.5, p = 0.004. This loss of precision of recall over time with increasing number of items may be related to increased competition from multiple items residing simultaneously in memory (Pertzov et al., 2013). A similar pattern was also observed for RT, crucially with a significant interaction between load and delay, F(3, 27) = 3.12, p = 0.039. The effects were smaller than those for error, but RTs display high variability, and subjects were given specific instructions to respond accurately rather than being told to make speeded responses. Under these circumstances, the resemblance in the pattern of results suggests a close association between memory quality and the decision process accessing it.

Discussion

The evidence presented here is consistent with the view that judgments based on features stored in visual WM are the result of a decision process that evaluates the noisy evidence in memory before reaching a threshold and initiating a response. Response time was strongly influenced by memory quality: It increased as more items had to be stored (Figure 3), reduced when responding to an item with greater task relevance (Figure 5), and mirrored the degradations in memory accuracy that occur over maintenance delays (Figure 6). A simple decision mechanism (LATER) can explain these global patterns and the specific shape of RT distributions for individual subjects (Figure 4). In this model, commitment to a memory-based choice accumulates towards a response threshold at a rate proportional to the precision with which remembered items are stored (Figure 1).

This mechanism has broad appeal because it can explain behavior under several different conditions: with different stimulus features (location and orientation), cued and uncued items, and even different response types (forced choice by button press or free match to sample by rotation of a dial) and response priorities (speed or accuracy). The three experiments indicate that, rather than impinging separately on the choice process, all these factors influence the precision of feature representations in memory, the currency drawn upon as evidence by the decision process to evaluate potential responses.

This conceptual framework is based on the assumption that information remembered about a visual feature takes time to be expressed in a response specifically because an assessment of noisy evidence in WM evolves over time. The more noise is present, the slower the decision evolves and the longer the response time. Such a proposal challenges classical views that the delay in retrieval is due to the difficulty of accessing information from storage. Much previous work on retrieval has focused on letter or digit recognition and construed the process as a search for items in memory (Sternberg, 1966), a view that still prevails (see for example Astle, Summerfield, Griffin, & Nobre, 2012). Sternberg's serial exhaustive scanning model, which predicts a linear increase in RT with set size because each discrete item takes a fixed time to retrieve, was hugely influential but failed to accommodate inconsistencies such as probability (Theios & Walter, 1974), repetition (Baddeley & Ecob, 1973), and serial position (Corballis, 1967) effects (as reviewed in Greene, 1992).

The findings here also show that RT does not increase linearly but instead bears a decelerating relation to the number of remembered items. However, crucially, our analysis reveals that this is because the decision process rate of rise μ depends linearly on memory precision rather than memory load (Figure 4), the former varying in an inverse power manner with set size (Figure 3). Recent research in WM has provided evidence, based on analysis of the pattern of errors, that the representation for short-term storage is a highly limited resource (Bays & Husain, 2008; Elmore et al., 2011; Wilken & Ma, 2004) that can nevertheless be flexibly deployed (Bays et al., 2009; Gorgoraptis et al., 2011; Pertzov et al., 2013; Zokaei, Gorgoraptis, Bahrami, Bays, & Husain, 2011). With increasing memory load, the representation of each item becomes more variable around its true value (Bays & Husain, 2008; Palmer, 1990; Salmela et al., 2012; Wilken & Ma, 2004), providing noisier evidence for a decision process to operate upon. Our findings here, based on RT distributions across several different WM paradigms, demonstrate that there is a clear relationship between the precision of memory report and time taken to reach a decision. It is perhaps worth noting that we have not investigated all potential influences on precision and response time. Previous studies, such as Bays et al. (2009), have shown for example that misreporting nontarget features can influence the measured value of precision, an effect that has also been shown for irrelevant features (Huang, Kahana, & Sekuler, 2009). These are additional influences that would profitably be investigated in future studies.

What specifically governs the decision process? The different slopes in Figure 4C show clearly that μ is not a function of memory precision P alone, but is also modulated by the stimulus displacement Δ. In this sense μ represents a kind of signal to noise ratio, reminiscent of signal detection theory accounts of visual WM (Wilken & Ma, 2004): probe displacement Δ, the signal, is in effect scaled relative to memory noise (though note that, as in most previous studies, the signal detection theory account of visual WM did not refer to RTs). Furthermore, the parallel relationship between 1/RT distributions in Figure 4A confirms that this ratio only affects μ. The position of the decision priors and threshold and the trial-to-trial variance in rate of rise σ (see Figure 1) remain unchanged.

Of the many models of WM retrieval developed since Sternberg, perhaps the most successful has been the Drift Diffusion Model (Ratcliff, 1978). It also views retrieval as a decision, and its rate-of-accumulation variable Drift reflects the relatedness between remembered items and the probe. But Drift, along with numerous other variables, is simply fit from RT data independently for each condition; no attempt is made to account for trends in their values using other known properties of memory (Donkin & Nosofsky, 2012b). In response, recent studies have attempted to model in great detail the influence on RT distributions of factors such as forgetting (Donkin & Nosofsky, 2012a) and the combined effect of array size and serial position (Donkin & Nosofsky, 2012b). But they have still needed to draw on assumptions about how memory and decision interact, such as that drift varies linearly with array size. A related problem affects a model that is used to describe memory resources in terms of quantized slots (Cowan, 2005; Luck & Vogel, 1997; Vogel, Woodman, & Luck, 2001; Zhang & Luck, 2008). In this model, as memory load increases so that more items have to be stored, separate compartments in memory are filled one by one. But the capacity of each slot—its resolution—does not change. None of these models specifies how a memory representation dictates what value is accrued by a decision mechanism. By using stimuli in our study whose discriminability from one another (in position or orientation) could be varied, we were able to estimate memory precision and relate measured decision parameters to it.

Our approach here has been to consider whether the resource framework provided by studies of error patterns in WM (Bays & Husain, 2008; Palmer, 1990; Salmela et al., 2012; Wilken & Ma, 2004) might make predictions for response time in the context of a decision-making interpretation. Expert consensus on a range of problems in neuroscience has begun to converge on a view of behavior in which decision variables accrue evidence in support of potential choices over time, with the first to reach a threshold chosen as the response (Carpenter & Williams, 1995; Gold & Shadlen, 2007; Laming, 1968; Ratcliff, 1978; Usher & McClelland, 2001). The findings presented here show that such an approach might also have important implications for WM.

Furthermore, a natural conclusion to be drawn from the argument developed here is that encoding, maintenance, and retrieval need not be functionally separate cognitive processes, but operations on the same noise-limited neural resource (Bays et al., 2009; Bays & Husain, 2008; Palmer, 1990; Wilken & Ma, 2004). This view has gained ground through studies showing the contents of working memory to be intrinsically bound up with sensory mechanisms. Neurons in visual area MT, as well as encoding visible motion stimuli, are known to exhibit activity during the delay after the stimulus that is related to remembered motion (Bisley, Zaksas, Droll, & Pasternak, 2004), and fMRI decoding has identified orientation information retained in WM across V1-4 (Harrison & Tong, 2009). Independent behavioral confirmation has come from work showing that the contents of WM can bias the subjective experience of orientation and ambiguous motion stimuli (Scocchia, Cicchini, & Triesch, 2013; Scocchia, Valsecchi, Gegenfurtner, & Triesch, 2013). Further studies that have looked specifically at encoding (Bays, Gorgoraptis, Wee, Marshall, & Husain, 2011) and maintenance durations (Pertzov et al., 2013; Phillips, 1974) also point to these being operations on the same noise-limited resource, providing a simple, elegant conceptual framework for unifying these different aspects of working memory.