Imagery and visual working memory: one and the same?

Abstract

Although visual imagery and visual working memory are both defined by the ability to actively represent and manipulate visual information, it is not known whether they rely on common mechanisms. A recent study by Albers and colleagues directly investigates this issue, finding evidence of common internal representations in early visual areas.

Cognitive psychologists love to come up with new terms to describe specialized mental processes or constructs. Examples abound, from priming to subitizing, metacognition to mindblindness. Because thought processes cannot be directly perceived, this inferential approach is essential to our field’s advancement, but sometimes it can lead to the baffling emergence of parallel literatures, akin to divided universes that reflect one another, but scarcely interact.

One such example might be found in the parallel research domains of mental imagery and visual working memory. Mental imagery refers to the ability to access or reactivate perceptual information from memory, as well as the ability to dynamically manipulate this information for the purposes of planning, reasoning, inference, or flights of fancy [1]. In the visual-spatial domain, it can be used to bring to mind the countenance of a close friend, to plan the packing of a car trunk, or to surmise what it might be like to ride alongside a beam of light. In the 1970s, cognitive psychologists set upon the task of developing objective measures to infer subjective acts of imagery, by quantifying the time required to mentally rotate an object or to zoom across an island using the mind’s internal eye [2, 3]. Curiously, at around the same time, researchers were redefining the concept of visual short-term memory as something more than just a passive temporary store. Baddeley proposed a model of ‘working memory’, which consisted of a central executive and two subsidiary stores that allowed for the active maintenance and manipulation of phonological information and visual-spatial information [4].

Although both imagery and visual working memory depend on the ability to actively represent and manipulate visual information, the resulting research has somehow diverged into two separate literatures that rarely reference one another. Because of the different behavioral measures and tasks used in these two domains, it has proven challenging to establish direct links across these subfields. However, a possible bridge has emerged as a result of recent advances in multivariate pattern analysis of functional MRI data [5] and its successful application to decode the contents of visual working memory in early visual areas [6, 7].

Building on a visual working memory paradigm by Harrison and Tong [6], a recent neuroimaging study by Albers et al. [8] provides compelling new evidence that working memory and imagery rely on common visual representations. Participants were presented with oriented gratings at one of three possible orientations (15°, 75°, or 115°) and provided with a central cue indicating whether the relevant grating should be maintained as is or mentally rotated clockwise or counterclockwise by 60° or 120°. (To avoid potential stimulus confounds, two of the three possible orientations were presented sequentially at the beginning of a trial, and a subsequent cue indicated whether the first or second grating was the task-relevant stimulus.) After a 10s delay period, a test grating was presented, rotated slightly clockwise or counterclockwise relative to the orientation to be maintained, and the observer had to make a forced-choice discrimination judgment. Participants were somewhat more accurate on working memory trials than on trials requiring the additional step of mental rotation, suggesting the accrual of some error from performing these mental acrobatics.

The authors analyzed the data of each participant by training a linear classifier on activity patterns from early visual areas V1–V3, and then using the classifier to decode which of the three possible orientations was being internally maintained on separate test runs. Here, decoding accuracy provides an index of the amount of item-specific information contained in the cortical activity patterns.

Activity patterns in areas V1–V3 led to reliable decoding not only for orientations maintained in working memory (54% accuracy; chance level 33%), but also for orientations resulting from mental rotation (46% accuracy). Of particular importance, training on activity patterns from working memory trials proved just as effective at predicting the represented stimulus on mental rotation trials (45% accuracy), implying that the internal visual representation was very similar across imagery and working memory. Further experiments showed that stimulus-driven responses to unattended gratings could also predict the orientation represented during working memory and imagery. These findings concur with the proposal that imagery and working memory rely on similar neural representations as those used for perception [1, 6].

By analyzing performance across individual fMRI time points (collected every 2s), the authors characterized the temporal unfolding of these mental representations. On working memory trials, information about the maintained orientation emerged fairly quickly, within 4s after the start of the delay period, and this orientation preference was maintained throughout the delay period. On mental rotation trials, activity patterns were initially biased in favor of the orientation that was seen and cued, as was evident early in the delay period at time point 4s. However, by a time of 8s, these activity patterns were now biased in favor of the mentally rotated orientation. These results demonstrate a dynamic transformation of the internal visual representation as a consequence of mental rotation.

Overall, the results of Albers et al. provide compelling evidence of a common internal representation for visual working memory and mental imagery. These findings bolster the proposal that early visual areas may serve as a dynamic blackboard for both bottom-up perception and the top-down generation of visual content [1, 6, 9]. This study also raises the possibility that scientists may one day better understand why individuals vary in these abilities. In their study, Albers et al. noted that individuals who performed these tasks more accurately also exhibited better decoding in early visual areas. Similarly, a recent behavioral study found strong correlations between individual differences in imagery strength and visual working memory performance [10].

It will be of considerable interest to see if future studies can establish further empirical and theoretical links between the subfields of imagery and visual working memory. Although working memory is believed to support both the maintenance and manipulation of visual information, most behavioral studies have focused exclusively on the maintenance component [11]. By investigating the dynamic components of visual working memory, or alternatively what might be called imagery, we may come to better understand the more generative aspects of human vision and imagination.