Revisit once more the sensory storage account of visual working memory

Abstract

Recent work has highlighted the role of early visual areas in visual working memory (VWM) storage and put forward a sensory storage account of VWM. Using a distractor interference paradigm, however, we previolsy showed that the contribution of early visual areas to VWM storage may not be essential. Instead, higher cortical regions such as the posterior parietal cortex may play a more significant role in VWM storage. This is consistent with reviews of other available behavioral, neuroimaging and neurophysiology results. Recently, a number of studies brought forward new evidence regarding this debate. Here I review these new pieces of evidence in detail and show that there is still no strong and definitive evidence supporting an essential role of the early visual areas in VWM storage. Instead, converging evidence suggests that early visual areas may contribute to the decision stage of a VWM task by facilitating target and probe comparison. Aside from further clarifying this debate, it is also important to note that whether or not VWM storage uses a sensory code depends on how it is defined, and that behavioral interactions between VWM and perception tasks do not necessarily support the involvement of sensory regions in VWM storage.

Despite decades of research studying the cognitive and neural mechanisms underlying visual working memory (VWM), how VWM interacts with visual perception is still being debated. Earlier extensive work has shown the involvement of the primate prefrontal cortex (PFC) and posterior parietal cortex (PPC) in VWM storage (e.g., Goldman-Rakic, 1995; Fuster, 2001; Miller & Cohen, 2001; D’Esposito & Postle, 2015; Riley & Constantinidis, 2016; Serences, 2016; Christophel et al., 2017). Starting with the publication of Harrison and Tong (2009) and Serences et al. (2009), a host of recent fMRI studies and review papers have highlighted the role of early visual areas in VWM storage and put forward the sensory storage account of VWM (for reviews, see e.g., D’Esposito & Postle, 2015; Serences, 2016; Christophel et al., 2017). The key idea here is that the same neural substrates supporting perception in sensory regions may also support the retention of the perceived sensory information for a prolonged period of time in VWM. The idea is appealing as it reflects a potentially cortically efficient way of handling both perception and VWM. However, the experiments conducted did not take into account the continuous influx of irrelevant visual information in everyday VWM experience. When we used a distractor interference paradigm, we found that the contribution of early visual areas to VWM storage may not be essential (Bettencourt & Xu, 2016a). In a subsequent extended review of the existing behavior, neuroimaging, and neurophysiological evidence, I also failed to find robust evidence supporting a sensory storage account of VWM (Xu, 2017 & 2018a). This conclusion is in agreement with another review covering 40 years of neurophysiology studies which also found very little evidence supporting the role of sensory region in VWM storage (Leavitt et al. 2017). Since the publication of these papers, a number of studies have brought forward new evidence regarding this debate (Christophel et al., 2018; Lorenc et al., 2018; Rademaker, et al., 2019). Here I provide a detailed examination of these new studies and show that there is still no strong and definitive evidence supporting the role of sensory regions in VWM storage. Instead, these studies are consistent with early visual areas being nonessential in VWM storage. Current evidence, however, does point to a potentially different role early visual areas may play in VWM tasks. I will describe an emerging view proposing that VWM representation in early visual areas may facilitate template matching during the decision stage of the task and review some evidence that is consistent with this view. In the last part of this review, I present arguments to emphasize that whether VWM storage uses a sensory code depends on how it is defined, and that behavioral interactions between VWM and perception tasks do not necessarily support the involvement of sensory regions in VWM storage.

Early visual areas are non-essential for VWM storage

In their original fMRI study, Harrison and Tong (2009) asked human participants to retain in VWM the orientation of a grating and were able to decode the remembered orientation successfully during the VWM delay period in early visual areas (see also Serences et al., 2009). This finding is robust and has been replicated multiple times across different labs (e.g., Ester et al., 2009, 2013 & 2015; Riggall & Postle, 2012; Albers et al., 2013; Emrich et al., 2013; Sprague et al., 2014 & 2016; Bettencourt & Xu, 2016a; Yu & Shim, 2017). These results have been argued to provide neural evidence supporting an important role of sensory regions, such as the early visual areas, in VWM storage.

In the above studies, successful VWM decoding in early visual areas was obtained when the entire delay period was unfilled. This is, however, very unlike real-world visual experience that often taxes our visual system with concurrent perception and VWM demands, such as in visual search, which involves examining the content of the continuous influx of new visual input to find the target object we hold in mind in VWM. If early visual areas were to play an essential role in VWM storage in real world vision, then a concurrent processing demand from both VWM and perception should not diminish their stored VWM representation, especially when perceptual processing poses no behavioral interference to VWM storage. Otherwise, the essential role of early visual areas in VWM storage may be questionable.

To address this question, we used the paradigm developed by Harrison and Tong (2009) and compared decoding for trials with an unfilled delay and those with a filled delay with face and gazebo distractors (see Figure 1 for details of the experimental paradigm and trial timing). We made distractor presence in a trial to be completely predictable by blocking trials with and without distractors¹. In the distractor absent trials, VWM decoding in early visual areas was high and comparable to what was previously reported (Harrison & Tong, 2009). However, in the distractor present trials, VWM decoding in early visual areas dropped significantly and became indistinguishable from chance level performance (Bettencourt & Xu, 2016a). If early visual areas are essential to VWM storage, a significant drop in decoding performance would predict a significant drop in behavioral performance. However, distractor presence had no detectable impact on behavioral performance. The content of VWM was thus unlikely to have been stored in early visual areas in this situation. In a second experiment, instead of making the distractor presence predictable by blocking trials with and without distractors, we randomly intermixed the two types of trials. Again, we found no effect of distractors on behavioral performance. Interestingly, we were able to obtain significant and comparable VWM decoding in early visual areas for both trial types during the delay period. Nevertheless, decoding for trials without distractors was significantly lower for those in the second than in the first experiment². That is, even when distractors were not shown, knowing that distractors might be present significantly weakened VWM decoding in early visual areas. VWM representation in early visual areas thus appears to be heavily modulated by both the presence and the predictability of the distractors despite distractors having no effect on behavioral performance. These findings are at odds with sensory areas, such as the early visual areas, being essential for VWM storage.

Figure 1.

Trial structures and timings of the studies conducted by Bettencourt and Xu (2016a), Christophel et al. (2018), Lorenc et al. (2018) and Rademaker et al. (2019). The delay period in each study in which VWM decoding was examined is marked by a box with a bold outline.

Besides early visual areas, we also examined superior intra-parietal sulcus (IPS), a brain region whose fMRI response amplitudes have been shown to track behavioral VWM capacity (Todd & Marios, 2004 & 2005; Xu & Chun, 2006 & 2007; Xu, 2007, 2009, & 2010; Jeong & Xu, 2013). In contrast to early visual areas, we obtained successful VWM decoding in superior IPS irrespective of the presence and predictability of the distractors, mirroring behavioral performance (Bettencourt & Xu, 2016a). We additionally found a close behavior and brain correlation, with two orientations eliciting a similar fMRI response pattern in superior IPS during the VWM delay period also being more confusable in a behavioral VWM change detection task (Bettencourt and Xu, 2016a). Together, these results show that PPC contains a more robust VWM code than early visual areas when distraction is present in a VWM task.

In our study, we also found a correlation between decoding performance and behavioral RT measures in early visual areas. Because visual information is first encoded by early visual areas and then propagated to other brain regions, similar sensory representations in early visual areas would likely result in similar VWM representation in regions involved in VWM retention. Since VWM delay period signal in early visual areas could either reflect the extension of its encoding activities or feedback from higher regions (such as from PPC; see a detailed discussion on feedback signals later), it is not surprising that VWM delay period signal in early visual area may also track the representational structure seen in behavioral RT measures. Thus, showing a correlation with behavioral performance is a necessary, but not sufficient, condition for a brain region to be involved in VWM storage. The correlation result, together with how a brain region handles distraction, places PPC as playing a more significant role in VWM storage than early visual areas (see a similar argument made in Bettencourt & Xu, 2016a).

Our results do not imply that sensory regions cannot concurrently represent the content of both VWM and sensory inputs. When distractor presence was unpredictable, we were able to decode the content of the distractors and VWM during delay in early visual areas. What our results show is that VWM signals in early visual cortex could not be essential for VWM storage. In real world vision, as we move around the environment, we expect visual input to continuously change, with some events being more attention grabbing than others. While there are instances that require a continuous integration of VWM and perception, such as in forming an integrated representation of a visual scene through saccades, there are also ample occasions that require a separation of VWM and perception, such as in visual search as mentioned earlier. Our predictable distractor presence condition was set up to simulate such a situation: we expect and are prepared to be constantly distracted while holding information in VWM in real world vision. It is unclear how the unreliable VWM representations in early visual areas could play an essential role in VWM storage in such sitations in real world vision. On the other hand, separated VWM and perception processing in different brain regions as our data have shown would be more adaptive and flexible in information processing and manipulation.

Our fMRI results showing that early visual areas are nonessential for VWM storage are in agreement with a similar conclusion drawn by Leavitt et al. (2017). After reviewing four decades of over 90 neurophysiological recording studies, Leavitt et al. reported that it is rare to observe VWM-related sustained activity in early sensory areas and that, when such activity is observed, it typically does not reflect the content of VWM. In my review of additional human behavioral and transcranial magnetic stimulation (TMS) studies from the last several decades, I also found very little evidence supporting the storage of VWM in early visual areas (Xu, 2017 & 2018a).

New evidence on the role of early visual areas in VWM storage

Since the publication of our study (Bettencourt & Xu, 2016a and Xu, 2017), a number of studies have been conducted and brought forward new evidence regarding the role of early visual areas in VWM storage (Christophel, et al., 2018; Lorenc et al., 2018; and Rademaker et al., 2019). Below I describe and analyze this new set of evidence. I show that there is still no strong and definitive evidence showing an essential role of early visual areas in VWM storage. Instead, they are consistent with early visual areas being nonessential for VWM storage.

Christophel et al. (2018).

In a recent study, using a retro-cue paradigm, Christophel et al. (2018) asked participants to remember two items in VWM and then cue one item to be probed after the 1^st delay period. Either the same or the other remembered item was probed after the 2^nd delay period (see Figure 1 for details of the experimental paradigm and trial timing). During the 1^st delay period, although both items could be equally well decoded in IPS and frontal eye fields, only the cued, but not the un-cued, item could be successfully decoded in early visual areas. Behaviorally, the attended item during the 1^st delay period was better remembered after the 1^st delay period than the unattended item during the 1^st delay period that was subsequently attended during the 2^nd delay period and then recalled. Christophel et al. (2018) concluded that the superior VWM recall after the 1^st delay period was due to the engagement of early visual areas during the 1^st delay period of the attended items. In other words, they argued that early visual areas directly contributed to the higher recall precision of the attended item after the 1^st delay period. There were, however, two major differences between the 1^st and 2^nd recalled items: Besides the engagement of early visual areas during the retention of the 1^st but not the 2^nd recalled item during the 1^st delay period, there was also the occurrence of the 1^st recall task before the 2^nd item was attended and probed. Because the 1^st recall task involved attention and decision mechanisms in parietal and frontal brain regions, it could have momentarily distracted VWM maintenance of the unattended item in these brain regions, resulting in worse recall performance of the 2^nd item. This is an important event and needs to be fully taken into account before we could attribute better recall of the 1^st item solely to the engagement of the early visual areas during the 1^st delay period. Without doing so, the results of Christophel et al. (2018) should not be taken as providing evidence supporting the role of early visual areas in VWM storage. What these results did show, however, was that the content of VWM was only present in early visual areas when a probe detection task was imminent. This supports the template matching hypothesis that will be discussed in detail in the next section.

Besides using a direct pattern decoding approach to assess whether information is stored in a brain region during the delay period, two recent studies have used the inverted encoding model (IEM) to examine the effect of distractors on VWM representation in early visual areas and IPS (Lorenc et al., 2018; Rademaker et al., 2019).

Lorenc et al. (2018).

Lorenc et al. (2018) asked participants to retain the orientation of a single grating in VWM and measured orientation representation during two delay periods in early visual areas V1 to V3 and an anatomically defined IPS region. In two thirds of the trials, a distractor grating was briefly shown between the two delay periods; in the other third of the trials, no distractor was shown between the two delay periods. A memory probe was shown at the end of delay period 2 to collect a behavioral response (see Figure 1 for details of the experimental paradigm and trial timing). During delay period 1, orientation representation was found to be robust in both early visual areas and IPS. During delay period 2, in no distractor trials, while orientation representation was still present in early visual areas, it was no longer seen in IPS; in distractor present trials, however, orientation representation was found in both brain regions, with early visual areas, but not IPS, showing a significant bias towards the distractor orientation. Lorenc et al. (2018) concluded that orientation information was maintained in both early visual areas and IPS in anticipation of possible distraction; when a distractor did not appear, early visual areas alone would sustain VWM representation; however, when a distractor did appear, because VWM representation in early visual areas was susceptible to interference, IPS would take over to sustain VWM representation. In other words, IPS VWM storage was only needed when distractor presence rendered early visual areas incapable of continuously sustaining a faithful VWM representation.

There are, however, a few concerns associated with the conclusion reached by Lorenc et al. (2018). In paradigms with 100% distractor absence, we and a host of others have consistently found successful VWM representation for a variety of visual information in human PPC, including spatial position, color, orientation, shape, abstract pattern, and view-invariant abstract shape (Christophel et al., 2012, 2014, 2015, 2017b & 2018; Sprague et a., 2014, 2016; Ester et al., 2015; Bettencourt & Xu, 2016a; Yu & Shim, 2017). The conclusion reached by Lorenc et al. (2018) that PPC is not involved in VWM storage when no distraction is present is inconsistent with this large set of existing PPC findings from multiple different labs. We have previously shown that not the entire IPS is involved in VWM storage, but rather only a subregion in the superior part of IPS (Bettencourt & Xu, 2016a & 2016b; Xu & Chun, 2006 & 2007; Xu, 2007, 2009, 2010). We localized this region in each participant using an independent VWM task and defined it as tracking that participant’s behavioral VWM capacity following Todd and Marois (2004). When we examined the entire IPS in our study using either topographically defined or anatomically defined IPS regions, we did not find consistent VWM decoding across all our conditions either (Bettencourt & Xu, 2016a, Supplemental Figures 3 and 4). This raised the possibility that the anatomically defined IPS region in Lorenc et al. (2018) may be underpowered to detect IPS VWM representation. In Lorenc et al. (2018), when a distractor was shown, due to the sluggishness of the fMRI signal, VWM decoding in delay period 2 in early visual areas could reflect both distractor and target orientations (see a similar comment by Rademaker et al., 2019). Thus, whether or not VWM representation in early visual areas may be distorted by distraction requires further confirmation, possibly by a method with better temporal resolution, such as EEG and MEG. What Lorenc et al. (2018) did show, however, was that during delay period 2, IPS appeared to contain a more distractor resilient VWM code than early visual areas. This is consistent with our finding.

Rademaker et al. (2019).

Using IEM, Rademaker et al. used a paradigm very similar to ours and presented either no distractors or a continuous stream of distractors during the VWM delay period when participants retained in VWM the orientation of a grating stimulus. Trials with and without distractors were randomly intermixed but with a cue informing distractor presence/absence at the start of each trial (see Figure 1 for details of the experimental paradigm and trial timing). The IEM was trained using fMRI responses from separate sensory stimuli when participants performed an orthogonal task irrelevant to the orientation of these stimuli. In Experiment 1, distractor stimuli could be either Fourier-filtered noise or oriented gratings. No difference in behavioral performance was found across distractor present and absent trials as well as the two types of distractors. Delay period VWM representation was robust and similar across the different trial types in early visual areas, but less robust in topographically defined IPS regions, with 2 out the 3 trial types eliciting significant orientation representation in IPS0 but none in IPS1 (these were the two IPS regions examined in the main study). In Experiment 2, distractor stimuli could be either oriented gratings or face and gazebo stimuli as we used in our study. Here a decrease in behavioral performance was found in the distractor present as compared to the distractor absent trials. While orientation decoding in early visual areas mirrored behavioral performance (i.e., lower decoding with than without distractors), those in the IPS regions were at chance across all trial types. When the decoder was trained with VWM delay signals instead of those from separate sensory stimuli, a different picture emerged. Here both IPS0 and IPS1 showed similar and robust VWM orientation representation across both distractor present and absent trials in both experiments despite a drop in behavioral performance in Experiment 2 in the distractor present trials. Early visual areas only showed a drop in representation in the distractor present trials in Experiment 2. Similar results were obtained when a direct decoding approach was used, such that training on the sensory task and testing on VWM delay produced weak results in IPS across the two experiments, but training and testing both within VWM delay produced robust IPS decoding in both experiments. Rademaker et al. concluded that sensory and VWM signals may be represented together in early visual areas; delay period representation in early visual areas better tracked VWM behavioral performance than that in IPS; and IPS may contain a transformed VWM code that cannot be generalized to sensory representation. Overall Rademaker et al. argued for a significant role of IPS in VWM storage and a role of early visual areas in probe detection via a template matching process (see a more detailed discussion on this later). This is consistent with our view that PPC, rather than early visual areas, plays a more essential role in VWM storage (Bettencourt & Xu, 2016a, Xu, 2017). Meanwhile, using a design that was most similar to ours compared to the other studies, Rademaker et al. also concluded that “the fidelity of mnemonic representations in visual cortex was similar with and without distractors.” (p.1340 right column line 1), and that “high-fidelity memories rely on sustained representations in early visual cortex” (p. 1343 left column line 28). Given that these statements imply distraction resistant VWM representation in early visual areas and a significant role of this brain region in supporting VWM storage, it is important to carefully examine the evidence that led to these conclusions.

As discussed earlier, concurrent representation of VWM and sensory (e.g., distractor) information in a brain region does not in itself speak to the role of that brain region in VWM storage. In our original study, when distractor presence was unpredictable, we also obtained significant VWM delay decoding in early visual areas during distraction as well as a currently decoding of both VWM and distractor information in both early visual areas and superior IPS (see Figure 3a and results reported in Bettencourt & Xu, 2016a).

Rademaker et al. randomly intermixed trials with and without distractors and then cued participants the trial type at the start of each trial in an attempt to replicate our distractor predictable manipulation. However, the frequent switching back and forth of the different trial types could have discouraged participants from fully utilizing the cue to properly configure the brain state in some of the trials, rendering the overall task condition resembling more our distractor unpredictable, rather than our distractor predictable, trial condition. Given that we found equally good VWM decoding in early visual areas for both distractor present and absent trials when distractor presence was unpredictable (but with the performance of both being lower than that of the distractor absence trials when distractor presence was 100% predictable), the specific manipulation used by Rademaker et al. (2019) unfortunately decreased the likelihood of finding a distractor effect in early visual areas. Nevertheless, a close examination of the results revealed that in their Experiment 1, a drop was present and significant in some of the early visual areas in the distractor present as compared to the distractor absent trials (see Figures 1e, 4a and 5c and Supplementary Tables 1, 2 and 5 of Rademaker et al., 2019; results may also be combined from V1 to V3 or V1 to V4 as in Harrison & Tong (2009) and our study to increase power to better assess the effect of distraction). At the very least, given that distractors did not impact behavioral performance in Experiment 1, these results are at odds with the strong conclusion that VWM decoding was unimpaired by distraction in early visual areas and that VWM representations in this brain region tracked behavioral performance and was similar with and without distractors. If anything, these results are more consistent with what we found in our study.

Instead of using IEM, using a testing/training and a decoding method similar to what we used, Rademaker et al. found no decrease in VWM decoding performance in IPS0 and IPS1 in Experiment 1 when distractor presence did not impact behavioral performance, but a drop in IPS VWM decoding in Experiment 2 when distractor presence did impact behavioral performance (the effect was significant in IPS0 and a similar trend was present for IPS1; see Figure 5c of Rademaker et al.). These results are not entirely consistent with the strong conclusion that IPS did not track behavioral VWM performance (see a detailed discussion of the IPS IEM results in a later section).

Overall, results from Rademaker et al. (2019) share some similarities with our findings, with VWM representations in some of the early visual areas impacted when distractors had no effect on behavior and those in IPS showing no effect of distraction when distractors failed to impact behavior in Experiment 1 but a drop when distractors impacted behavior in Experiment 2 (using the measure we used). This shows that IPS VWM representations, but not those from early visual areas, may better track behavioral performance. Although these results may not provide a precise replication of our original results (due to differences in experimental paradigms as noted earlier), at the very least, they do not provide strong support that early visual areas are essential for VWM storage.

Together, recent studies have not provided robust and definitive evidence showing an essential role of early visual areas in VWM storage. The detailed analyses provided above instead show that these results are more consistent with our original finding that early visual areas may not play a key role for VWM storage. fMRI is a neural measure with limited spatial and temporal resolutions that cannot distinguish between feedback and feedforward responses. As such, it can only provide partial evidence regarding the neural mechanisms supporting VWM. Definitive neural evidence will have to come from direct neurophysiological recording studies. Forty years of neurophysiology studies have so far produced very little evidence supporting the role of sensory region in VWM storage (Leavitt et al. 2017). Although other neuronal mechanisms besides sustained neuronal firing are possible to support a VWM code, they are still under debate (Leavitt et al. 2017; Masse et al., 2020). Without strong support from neurophysiology studies and further evidence from fMRI studies, it may be wise to dial back on the sensory storage account of VWM based on the currently available fMRI results, especially given the detailed evaluation of the recent studies presented above.

One argument that has been presented against a non-sensory view of VWM storage has been the null argument: it remains possible that sensory areas play an essential role in VWM storage, but that we failed to show it (e.g., Ester et al., 2016). Note that it is theoretically impossible to prove the null, as one can never distinguish between the possibilities of there being nothing versus there being something, but one not being able to show it. For this reason, our work has never intended to prove the null, but rather to show that early visual areas can exhibit a significant drop in decoding performance when distractor presence fails to impact behavioral performance and thus do not appear to play an essential role to support VWM storage. That being said, to the extent that significant decoding can be used to support the presence of robust VWM representation in early visual areas, the lack of such a decoding or decreased decoding under distraction when no impact of distraction on behavior was found should carry equal weight to refute such a strong claim, rather than been dismissed under the null argument. After all, in statistical testing, failure to reject the null is used to reject the validity of a hypothesis. Here I evaluated recent new fMRI evidence but still failed to find strong and definitive support for an essential role of early visual areas in VWM storage. The continuous lack of such evidence despite multiple attempts makes it even harder to reject the null. Any scientific theory is only as good as the positive evidence supporting it. Without such support, a theory loses its explanation power and its utility.

In contrast to the lack of strong evidence supporting early visual areas in VWM storage, all three recently published studies provided unambiguous evidence supporting the role of PPC in VWM storage, especially under distraction. This is consistent with our original findings with distractors (Bettencourt & Xu, 2016a). This reaffirms PPC’s role in adaptive visual processing as reported in a number of other recent studies using attentional, instead of VWM, manipulations (Bracci et al., 2017; Jeong & Xu, 2017; Vaziri-Pashkam & Xu, 2017; Xu & Vaziri-Pashkam, 2019). In two recent papers, I provided further reviews of how an adaptive visual processing framework may account for the diverse functions associated with PPC (Xu, 2018b) and how representations in PPC may differ fundamentally from those of the occipito-temperal cortex in the primate brain (Xu, 2018c).

A template matching account of the role of early visual areas in VWM

Even though early visual areas may not play an essential role in VWM storage, why then do we consistently find VWM representations in this general brain region? Both fMRI and neurophysiological studies have shown that VWM signals in early visual areas largely come from feedback signals from higher brain regions (e.g., Ester et al., 2009; van Kerkoerle et al., 2017; Lawrence et al., 2018). This raised the possibility that such feedback signals may play a functional role in facilitating the comparisons between VWM content and test probe at the end of the delay period, a view that I share with others (Serence, 2016; Xu, 2017; Rademaker et al., 2019). Here I describe how existing behavioral, TMS, and fMRI findings are consistent with this view.

Early visual areas have been shown to receive at least as much feedback as feedforward input, with about 1% of the external input connections to V1 coming from the LGN and about 95% from higher visual areas (Markov et al., 2011). Consequently, feedback to early visual areas has been shown to influence neural responses in both fMRI and neurophysiology studies. For example, using fMRI decoding, Ester et al. (2009) reported a dissociation between cortical areas involved in sensory processing and VWM storage of orientation such that although sensory processing only engaged area V1 contralateral to stimulus presentation, VWM content could be decoded from both contralateral and ipsilateral V1. In a recent neurophysiological study, van Kerkoerle et al. (2017) showed that the sustained spiking activity in V1 neurons during VWM delay was mainly seen in the superficial layers and layer 5 of area V1, a laminar pattern consistent with feedback from higher areas to V1. This allowed the spiking activity in V1 during the delay period to be restored via top-down feedback when it was temporally abolished by a briefly appearing mask. Others have reported top-down influences from PFC to early visual areas (e.g., Liebe, et al., 2012; Mendoza-Halliday et al., 2014). In a recent study using high-field MRI, Lawrence et al. (2018) confirmed the laminar organization of VWM signals in human V1 and showed that VWM signals activated deep and superficial layers but avoided the middle layers. In human V2 and V3, however, they found VWM signals in all depths, reflecting the cross-wiring between different V1 layers to the different layers in V2 and V3.

What could such feedback signals to early visual areas be used for in a VWM task? One possibility is that early visual areas could serve as a template where stored VWM representation and incoming probe information are compared to facilitate decision making in VWM tasks (Serences, 2016; Xu, 2017; Rademaker et al., 2019). Indeed, even though distractor presence might not impact VWM precision, it could nevertheless slow down probe detection (Sreenivasan & Jha, 2007). Likewise, although TMS to mid delay period minimally impacted VWM performance (Cattaneo et al., 2009; van de Ven et al., 2012; Rademaker et al., 2017; van Lamsweerde & Johnson, 2017), TMS towards the end of the delay period has been shown to result in a facilitation effect with a decreased reaction time in performance (Cattaneo et al., 2009). This is likely due to an activity enhancement in early visual areas that facilitated VWM and test probe comparison. Consistent with this template matching account, in Christophel et al. (2018), early visual areas only sustained the representation of the attended, but not the unattended, item, as only the attended item would be imminently probed and thus could benefit from such a template matching operation. Similarly, when distractor presence was made to be predictable or unpredictable in our study (Bettencourt & Xu, 2016a), top-down feedback to early visual areas was likely modulated accordingly to maximally take advantage of such a template matching operation in early visual areas.

One may argue that preparatory attention, which has been shown to track the hazard rate of the appearance of target stimuli (as reviewed in Nobre & Van Ede 2018), could simply activate target representation shortly before the probe appearance, rather than having such a representation maintained throughout the VWM delay period. This should be a viable strategy given that a relatively long (~10s) delay periods was typically used in fMRI VWM studies with fixed and thus predictable trial duration. Although reasonable, such a strategy may not be implemented for two reasons. First, the cost in cognitive effort by high-level control mechanisms to reactivate and reestablish the target representation in early visual areas after it has completely dissipated via feedback from higher regions (such as PPC and PFC) could be higher than simply sustaining the representation continuously in early visual areas during the delay, especially since participants are lying in the dark otherwise unengaged in any other visual tasks. Second, although participants can modulate their behavior and brain responses according to the temporal structure of a trial when the time interval is short (4 sec or less, see Grubert & Eimer, 2018, Olmos-Solis et al. 2017, and van Ede et al., 2017), they may not be able to do so when the interval is longer, such as for 11 sec in Harrison and Tong (2009) and in our study (Bettencourt & Xu, 2016). The usage of any strategy to keep track of time would just add a taxing secondary task on top of the primary VWM task. As such, maintaining VWM representation in early visual areas continuously throughout the delay period may be cost effective and adaptive, even if doing so may not directly aid information retention in VWM.

Because sensory input and feedback signals are coded by different cortical layers in V1 (van Kerkoerle et al., 2017; Lawrence et al., 2018) and then projected to the same layers in V2 and V3 (Lawrence et al., 2018), more detailed research is needed to fully understand at the circuit level how these two types of signals are compared during template matching, and how a match/mismatch signal may be relayed to regions in PPC and PFC to facilitate decision making and motor response. Given that detailed VWM representation exists in PPC and PFC (see Christophel et al., 2017 for a detailed review of this evidence), template matching could also occur in these higher brain regions to facilitate decision making. More studies are therefore needed to test whether template matching in early visual areas uniquely contributes to VWM probe detection or that such an operation is present across multiple brain regions to aid VWM task performance.

A sensory code of VWM storage?

So far, I have shown that there is still no strong and definitive evidence supporting an essential role of the early visual areas in VWM storage. Because the key idea of the sensory account of VWM storage is that the same neural substrates supporting perception in sensory regions may support the retention of the perceived sensory information for a prolonged period of time in VWM, this account explicitly argues that the same sensory code is used for both perception and VWM in early visual areas. In fact, such evidence is essential to establish the validity of this account. On the other hand, the storage of VWM representation in other cortical regions does not require this to be true: the same code could be used for both perception and VWM storage, or a transformed code different from perception could be used for VWM storage. Thus, knowing the precise nature of the neural code for VWM storage in other brain regions does not in any way affect their role in VWM storage. It is not that we are subjecting sensory areas to a stricter set of criteria than we do to the other brain regions, but rather the premise of the sensory account of VWM storage explicitly demands this, whereas the storage of VWM in other cortical regions does not. Related to this, whether perception and VWM share the same neural code in representation and whether they can interfere with each other in behavior should not be used as evidence to support the view that the content of VWM is stored in sensory regions. I discuss these issues in more detail below.

In the original study by Harrison and Tong (2009), a linear classifier trained with sensory stimuli from a separate perceptual task could decode VWM representation in early visual areas. Similar cross-decoding results have been obtained by a few others (e.g., Serences et al., 2009; Riggall & Postle, 2012). In our study (Bettencourt & Xu, 2016a), when we trained a linear classifier using the probe stimuli at test as our perceptual stimulus, we were also able to obtain significant cross-decoding to VWM representations held during the delay. Likewise, using the IEM, a number of studies have trained the model with sensory stimuli to successfully decode VWM representation in early visual areas (e.g., for spatial location, Sprague et al., 2014, 2016; and for orientation, Rademaker et al. 2019). There is thus clear evidence that VWM representation in early visual areas share a neural code with that from perception.

Using IEM trained on an independent sensory task, Rademaker et al. (2019) found that VWM decoding in IPS was weak or at chance across the two experiments; however, when the model was trained on VWM data, robust VWM decoding was found in IPS in both experiments. These results have been used to argue that IPS contained a transformed VWM code that could not be generalized to sensory representation. IEM directly models the different orientations as discrete orientation channels using idealized functions and then linearly combines these channel functions to estimate a complex neural response. While these assumptions work well with early visual areas, it remains to be seen how they would impact decoding in IPS. Using naturel images, we and others have shown that IPS representations for non-spatial information are significantly modulated by attention (Vaziri-Paskam & Xu, 2017; Bracci et al., 2017). Rademaker et al. also acknowledged that “participants did not attend the orientations of the sensory stimuli used to train our model (instead, they performed an orthogonal task)”, and as such, a “confluence of both perception and attention might be required to get reliable sensory responses from IPS” (Rademaker et al., 2019, p.1342 right column lines 2–6). There thus remains the possibly that the IPS IEM from the sensory task was not robustly trained, potentially contributing to poor WM decoding in IPS without IPS necessarily using a different neural code for sensory and VWM processing as reported in Rademaker et al. Indeed, the sensory-trained IPS IEM even failed to decode distractor information (i.e., sensory information) during the delay period in Experiment 2 of Rademaker et al., raising significant concerns regarding the validity of the IPS IEM used.

In our study (Bettencourt & Xu, 2016a), we found that both early visual areas and superior IPS could track the similarity of the orientation information retained in VWM, such that two orientations eliciting a similar fMRI response pattern during the VWM delay period were also more confusable in a behavioral VWM change detection task. Thus, the orientation representational structure appeared to be similar between early visual areas and superior IPS in VWM. Given that early visual areas use a similar neural code for perception and VWM, this provides some indirect evidence that a similar neural code may be used in superior IPS during both perception and VWM. To directly test this, we asked participants to retain faces and gazebos in a VWM task and viewed these same stimuli in a perceptual task. We found successful cross-decoding between perceptual and VWM representations in superior IPS (Bettencourt & Xu, 2015), such that a decoder trained with perceptual representations could successfully decode the content of delay period VWM representations (Figure 2). These results suggest that there is substantial overlap in the neural code used by perception and VWM in PPC. Nevertheless, more research is needed to replicate and extend these findings to fully document the similarity as well as potential differences in the neural code used by PPC in perception and VWM.

Figure 2.

To examine the cross-decoding between VWM and perceptual representations in superior IPS, we asked seven participants (six females) to perform a VWM task and a perceptual task (Bettencourt & Xu, 2015). The VWM task adopted the same paradigm we used before (see Betterncourt & Xu, 2016a) but always with an unfilled delay period (i.e., no distractors) and with either a face or a gazebo image as the target instead of oriented gratings. In the perceptual task, participants viewed a sequential presentation of either faces or gazebos and performed a 1-back repetition detection task on the letter string shown at fixation. For within-task decoding, we performed training and testing on data within the same task; and for cross-task decoding, we trained the decoder on the data from the perception task and then tested it on the data from the VWM task. Error bars indicate s.e.m. * p < .05; *** p < .001; n.s., not significant. Stats reported here are corrected for multiple comparisons using the Benjamini-Hochberg method (Benjamini & Hochberg, 1995).

Decades of neuroscience research have reported ample empirical findings showing the involvement of parietal and frontal cortices in task-related visual processing. For example, PFC and PPC have long been shown to contain neurons selective to specific visual stimuli (e.g., Rao et al., 1997; Sereno & Maunsell, 1998). In fMRI studies, we and others have found robust representation of task relevant visual information even in tasks that did not involve a VWM retention paradigm (Bracci et al., 2017; Ester et al., 2016; Jeong & Xu, 2016 & 2017; Vaziri-Pashkam & Xu, 2017 & 2019; Vaziri-Pashkam et al., 2019; Xu & Jeong, 2015). A detailed review of PPC’s involvement in visual processing and task performance may be found in my two recent reviews (Xu, 2018b & 2018c). Thus, any interference observed in the behavioral paradigm between perception and VWM tasks could come from interference at PPC and/or PFC and the results of these studies cannot be used to support the involvement of sensory areas in VWM storage. This is an important point to emphasize as a number of researchers have dismissed this larger neuroscience literature and used such interference effects to argue for the sensory storage account of VWM or attribute such interference effects to the overlap in visual processing in early visual areas (e.g., Gayet, et al. 2018; Teng & Kravitz, 2019; see also Xu, 2017 & 2018a for a more detailed description of this line of behavioral studies and a similar argument). Such an argument may not be valid given the involvement of PFC and PPC in perception and VWM tasks. To have a full understanding of the cognitive mechanisms underlying VWM, both behavioral and neural evidence needs to be taken into account. This is a must, and not a choice, for any study that attempts to capture the nature of VWM.

Does VWM storage use a sensory code? It depends on how one defines a sensory code. If it only refers to the neural responses elicited in sensory regions during visual perception, then a sensory code appears to be nonessential for VWM storage. However, if a sensory code is defined as neural responses elicited in any brain region that track perception, then at least in PPC, there is some evidence that such a code could be used for VWM storage, although more research is needed to fully understand this.

Conclusions

While recent studies have provided new support for the involvement of PPC in VWM storage, they have not provided new and definitive evidence supporting an essential role of early visual areas in VWM storage. Instead, converging evidence suggests that early visual areas may play a role in facilitating VWM target and probe comparison at the end of the task, rather than VWM storage. Whether VWM storage uses a sensory code depends on how it is defined. Behavioral interactions between VWM and perception may not be used as evidence to support the involvement of sensory regions in VWM storage.