The cognitive neuroscience of visual short-term memory

Abstract

Our understanding of the neural bases of visual short-term memory (STM), the ability to mentally retain information over short periods of time, is being reshaped by two important developments: the application of methods from statistical machine learning, often a variant of multivariate pattern analysis (MVPA), to functional magnetic resonance imaging (fMRI) and electroencephalographic (EEG) data sets; and advances in our understanding of the physiology and functions of neuronal oscillations. One consequence is that many commonly observed physiological "signatures" that have previously been interpreted as directly related to the retention of information in visual STM may require reinterpretation as more general, state-related changes that can accompany cognitive-task performance. Another is important refinements of theoretical models of visual STM.

Signal intensity-based vs. multivariate analyses of fMRI data

Reconsidering the link between delay-period activity and “storage”

For decades, a governing assumption in STM research has been that the short-term retention of visual information is supported by regions that show elevated levels of activity during the delay period of STM tasks. Thus, for example, debates over the role of the prefrontal cortex (PFC) in STM and the related construct of working memory were framed in terms of whether or not its delay-period activity showed load-sensitivity -- systematic variation of signal intensity as a function of memory set size [1–4]. Similarly, patterns of load-sensitive variation of activity in the intraparietal sulcus have been used to test and refine theoretical models about mechanisms underlying capacity limits in visual STM [e.g., 5,6]. With the advent of MVPA, however, this signal-intensity assumption has been called into question.

A fundamental difference between MVPA and univariate signal intensity-based analyses is that the former does not entail thresholding the dataset prior to analysis, but, rather, analyzes the pattern produced by all elements in the sampled space. The analytic advantages to this approach are marked gains in sensitivity and specificity [e.g., 7]. In the domain of visual STM, this was first demonstrated with the successful decoding of delay-period stimulus identity from early visual cortex, including V1, despite the absence of above-baseline delay-period activity [8,9]. Subsequently, it was demonstrated that although the short-term retention of specific directions of motion was decodable from medial and lateral occipital regions (despite the absence of elevated delay-period activity), this information was not decodable from regions of intraparietal sulcus and frontal cortex (including PFC) that nonetheless evinced robust elevated delay-period activity [10]. Further, in these posterior areas the strength of MVPA decoding, a proxy for the fidelity of neural representation, declined with increasing memory load. Importantly, these changes in MVPA decoding predicted load-related declines in behavioral estimates of the precision of visual STM [11] (Figure 1). Relatedly, an fMRI study using a forward encoding-model approach [12] has demonstrated that interindividual differences in the dispersion (i.e., “sharpness”) of multivariate channel tuning functions in areas V1 and V2v predicts recall precision of STM for orientations[13]. Thus, studies [11] and [13] indicate an important link between the fidelity of the distributed neural representation and the fidelity of the mental representation that it is assumed to support.

Figure 1.

Dissociating elevated delay-period signal from the short-term retention of information. Summary of results from [11], in which subjects were scanned with fMRI while viewing one, two, or three sample displays of moving dots, then probed to recall the direction of one.

A. Univariate statistical maps indicating regions showing load sensitivity during sample presentation, the delay period, or both.

B. Time series data from “sample-only” voxels (panel on left) or “delay-only” voxels. Teal waveform illustrates decoding performance of a classifier trained at the time point with the maximal stimulus-evoked response (indicated with dot) then swept across the remainder of the trial. Maroon and solid gray waveforms are the analogous performance of classifiers trained at a time point late in the delay period, or 2 sec prior to sample onset, respectively. Asterisks indicate better-than-chance decoding at p < .05(*) and p < .01(**). Superimposed is the trial-averaged BOLD activity, depicted in the dotted waveform and aligned with the vertical axis on the right-hand side of the plot.

C. Plots of neural precision against behavioral precision. Each color corresponds to an individual subject and each digit (3, 2, or 1) to that individual’s neural and behavioral precision at the corresponding memory load. Lines are the fit indicated by ANCOVA (r² = .35).

The localization of visual STM, and insight into mechanism

It is not the case that intraparietal sulcus and frontal cortex are inherently “undecodable” (see Box 1), nor that they are never recruited for the short-term retention of information. A determinant of whether a network will be engaged in the short-term retention of a particular kind of information is whether it is engaged in the perception or other processing of that information in situations that don’t explicitly require STM. Thus, for example, when the short-term retention of abstract visuospatial patterns [23] or dynamically morphing flow-field stimuli [24] is tested, MVPA reveals delay-period stimulus representation in intraparietal sulcus, in addition to occipital regions; the same is true for face, house, and human-body stimuli in ventral occipitotemporal regions (e.g., [20]). When the to-be-remembered stimulus affords oculomotor planning, its identity can also be decoded from oculumotor-control regions of intraparietal sulcus and of frontal cortex [25]. Indeed, [25] demonstrated that an MVPA classifier trained on only one condition -- attention to a location, planning a saccade to a location, or STM for a location – can decode the other two. This could only be possible if similar patterns of neural activity, implying similar mechanisms, underlie the behaviors that have traditionally been categorized as “attention” vs. “intention” vs. “retention”.

Population coding in PFC.

PFC shows increases in activity during difficult vs. easy conditions of many types of task, not just STM (for which load is an operationalization of difficulty) [14]. With regard to STM, MVPA of neuronal activity recorded from monkeys provides hints of what functions may be supported by the elevated activity measured in humans with fMRI. In two studies, MVPA revealed a delay-period transition from an initial representation of properties specific to a stimulus, to one of either the item’s status as a “Go” or “No-go” cue [15], or the trial’s status as a “Match” or “Nonmatch” trial [16]. In a test of STM for the color of varying numbers of objects, PFC represented the passage of time across the delay period and the location of to-be-remembered stimuli, but not the colors themselves [17] (c.f. [18]). Consistent with these unit-level findings, MVPA of human fMRI of STM has shown PFC to encode such factors as stimulus category, attentional context, and match-nonmatch status of a trial (e.g., [10], [19], [20]). Thus, in addition to its well-established role in the top-down control of neural processing (e.g., [14,20]), another function of PFC may be the processing of information that, although not explicitly being tested, is nonetheless unfolding, and of possible relevance to the organism [17,21,22].

Patterns of localization can also reflect how the brain supports the strategic recoding of information from the format presented at study into one best suited for the impending memory-guided action. One study first presented subjects with a sample object, then, early in the delay, indicated whether memory for fine-grained perceptual details or for category membership would be tested. For the former, MVPA found evidence for delay-period stimulus representation in inferior occipitotemporal cortex, but not PFC; for the latter, the converse was true [19]. Combining MVPA with univariate and functional connectivity analyses has revealed a role for frontal cortex and intraparietal sulcus in implementing such strategic shifts of mental coding in visual STM [20]. MVPA can also track the evolution of mental coding in the absence of instructions, demonstrating, for example, that the verbal recoding of visually presented information also entails the recruitment of a semantic code [26].

Neural data also provide important constraints on models of capacity limitations of visual STM ([27], [28]). One influential model holds inferior intraparietal sulcus to be important for individuating objects that are to be encoded into visual STM, whereas superior intraparietal sulcus and an area of lateral occipital cortex are responsible for identifying these objects [6]. Recently, however, although the univariate analyses of data from a follow-up experiment [29] did reproduce many of the findings from the earlier study, MVPA of the same data failed to support a model of segregated circuits performing these two operations. Instead, the study of Naughton et al. [29] produced two novel findings. First, the contrasts intended to operationalize individuation vs. identification recruited primarily overlapping regions, thereby calling into question the dissociability of these two hypothesized mechanisms. Second, many regions outside of the intraparietal sulcus regions emphasized by [6] were also sensitive to these contrasts, suggesting that broadly distributed systems underlie the control of visual STM.

Signal intensity-based vs. multivariate analyses of EEG data

Event-related potential (ERP) correlates of STM

Another neural effect that has influenced models of visual STM capacity limitation is the contralateral delay activity (CDA), an ERP component that scales monotonically with STM load, but asymptotes at the psychophysically estimated capacity of an individual [34]. The CDA is widely interpreted as an index of the short-term retention of information (e.g., [35]), such that, e.g., the presence of a CDA during visual search has been taken as evidence for “memory in search” [36,37], and the diminution of the CDA across consecutive trials requiring search for the same target as evidence for the “handoff” of the mnemonic representation of the search template from STM to LTM [38].

Not unlike with univariate analyses of fMRI data, however, there can be problems with equating a 1-D, signal intensity-based measure like the CDA with a single psychological construct (in this case, the short-term retention of information). For example, empirically, the CDA can be observed during tasks for which it is unclear that the short-term retention of information is required, such as during multiple object tracking [39], or during change detection “even when the observers know that the objects will not disappear from the visual field” [40] (p. 8257). Further, the CDA during STM and during visual search is markedly reduced after intensive visual working memory training, despite the fact that STM capacity is increased and search performance improves with training [41]. Under these conditions, a physiological marker specific to the short-term retention of information would be expected to increase in intensity. An additional challenge to the idea that the CDA is specific to the short-term retention of information comes from the proposal that it may, in fact, be the consequence of averaging across trials containing asymmetric amplitude modulation of alpha-band oscillations [42]. From this perspective, because the CDA is linked to alpha-band oscillations (and, hence, to a general aspect of neurophysiological state, such as cortical excitability or inhibitory tone), the CDA may not index a memory storage mechanism per se, but rather a “general mechanism for allocation of resources” [43] (p. 903). Perhaps relatedly, multivariate analyses of alpha-band dynamics have provided important new insights into the neural bases of the short-term retention of visual information.

Multivariate analysis of EEG in STM

Using a multivariate forward-encoding-model approach similar to [13], Anderson et al. [44] constructed channel tuning functions for two narrowly filtered components of the EEG: alpha-band oscillations that were evoked by memory-sample onset; and alpha-band oscillations whose amplitude, but not phase, was modulated by sample onset (i.e., induced). Their results indicated that spatially distributed patterns in induced – but not evoked – delay period-spanning alpha-band activity predicted both inter- and intra-subject variation in precision of STM for line orientation. Note that these results don’t necessarily implicate induced alpha-band oscillations in the delay-period representation, per se, of stimuli. Alternatively, they may reflect distributed patterns of local inhibition and/or the long-range synchronization of localized representations of features, either of which would nonetheless be unique to each stimulus (c.f. [17]). Although several oscillatory phenomena have been associated with the short-term retention of information (including, e.g., local field potential oscillations at different frequencies, local and distal cross-frequency coupling, phase-amplitude coupling, and long-distance spike-field coherence (reviewed, e.g., in [45])), their investigation with multivariate methods (e.g., [46]) will be an important step in determining their specificity for stimulus representation vs. their possible contributions to other processes engaged by STM tasks.

Do distributed patterns of activity reflect STM or attention?

The multivariate methods reviewed here draw on two longstanding assumptions about STM: First, that stimulus representation is accomplished by anatomically distributed networks; Second, that the short-term retention of these representations is accomplished via elevated activity in these networks. Most often, however, STM tasks confound the focus of attention with the short-term retention, per se, of information. Recent studies have addressed this by first presenting two sample items, then indicating with a delay-period retrocue which of the two will be relevant for the impending memory probe. (Thus, the cue designates an “attended memory item”.) Because the first memory probe will be followed by a second delay period, a second retrocue, and a second probe, the item that was not cued during the initial delay (the “unattended memory item”) must be retained in STM, because it may be cued as relevant for the second probe. Intriguingly, MVPA of fMRI [47] and EEG [48] variants of this task fail to find evidence for an active neural representation of the unattended memory item, even though its active neural representation is reinstated if it is selected by the second retrocue (Figure 2)¹. These findings provide empirical support for the possibility that elevated activity may correspond more directly to the focus of attention than to the short-term retention of information, per se. The short-term retention of information, by this account, may depend on the establishment of representations encoded in distributed patterns of transiently modified synaptic weights, a code that would not be detectible by activity-based measurements. This phenomenon has been observed directly in the PFC of monkeys performing a visual working-memory task [15], and has been simulated in many computational implementations [49]. It has also been inferred to support the short-term retention of visual information in inferotemporal cortex [50], and so needs not be assumed to be a PFC-specific phenomenon. An important focus of current study is whether there are differences between the neural representation of unattended memory items, which are presumed to passively “slip out of” the focus of attention vs. of items that are intentionally removed from STM [20,35].

Figure 2.

Neural evidence for AMIs vs. UMIs vs. absent items, on trials when the second retrocue cues the same item as had the first (“Repeat”), or the previously uncued item (“Switch”). Legend labels “cued” and “uncued” refer to an item’s status relative to the first cue.

A. MVPA of fMRI data from [47]. Circles along timeline denote sample presentation, triangles denote retrocues, and squares denote recognition probes. Circles at top of plots indicate statistical significance of a stimulus category vs. the empirical baseline of MVPA evidence for the irrelevant category. MVPA classifiers were trained on data acquired in a prior training session.

B. MVPA of EEG data from [48]. Graphical conventions are the same as in A, with the exception that statistical significance (only tested during delay periods) is denoted with color-coded asterisks. MVPA classifiers were trained and tested on the same dataset using hold-one-trial-out cross validation.

Conclusion

High-level cognition, including STM, emerges from dynamic, distributed neural interactions that unfold on multiple time scales. The adoption of methods that more closely align with these principles of brain function is leading to discoveries with important implications for cognitive models of STM and working memory (e.g., [51,52]), and is informing ongoing research into such questions as the factors that underlie capacity limitations of visual STM [27,28], and the relation between STM and attention (e.g., [53,54]).

Highlights.

• Multivariate pattern analysis (MVPA) has revealed that elevated delay-period activity is neither specific to nor necessary for storage in short-term memory (STM).

• MVPA can track the dynamics of mental coding, and its control, in STM.

• Multivariate forward encoding models reveal specific role for alpha-band oscillations in STM.

• When focus of attention and STM are unconfounded, neural activity patterns track the former.

• MVPA supports models of synaptic weight-based storage of information in STM.

Network-level dynamics in STM.

Under conditions for which a stable mental code is assumed (e.g., no instructions to strategically recode [19,26]), MVPA typically reveals a stable set of regions to represent memoranda across the duration of a delay-period. However, the activity patterns within these regions can be dynamic. E.g., with auditory STM, the frequency-specific pattern of elevated stimulus-evoked activity transitions to become a pattern of negative activity during the delay period [30]. For visual STM, a classifier trained on a time point early in the trial will often perform progressively worse as it is slid forward across the remainder of the delay period, the converse being true for a classifier trained on a late-in-the-delay time point and slid backwards (Figure 1.B). This suggests a temporal evolution of the neural code underlying the short-term retention of a subjectively “stable” mental representation [11,31]. It remains to be determined whether these observations from fMRI relate in a meaningful way to the finding of dynamic coding in populations of neurons in monkeys performing tasks requiring sustained attention to an object [32,33].