Towards a better understanding of information storage in visual working memory

Abstract

Chota and Van der Stigchel (this issue), Iamshchinina, Christophel, Gayet, and Rademaker (this issue), Lorenc and Sreenivasa (this issue), and Teng and Postle (this issue) each present a commentary regarding Xu (2020) where I conclude that sensory regions are nonessential for the storage of information in visual working memory (VWM). They argue instead that sensory regions are critical to VWM storage. Here I briefly reiterate some of the key evidence against this account, some of which has not been accounted by the four commentaries. I also provide a detailed reanalysis of why the main evidence supporting this account may be problematic. Collectively, existence evidence from human neuroimaging and TMS studies and that from monkey neurophysiology studies does not provide strong support for the sensory storage account of VWM. To form an accurate understanding of the distinctive role each brain region may play in perception and VWM as well as how they may interact to collectively support a VWM task, it is important that we properly survey and evaluate all the available evidence.

Proponents of the sensory storage account of visual working memory (VWM) argue that sensory regions are an important part of a distributed WM storage network (e.g., D’Esposito & Postle, 2015; Postle, 2016; Serences, 2016; Christophel et al., 2017; Chota & Van der Stigchel, this issue (CV); Iamshchinina, Christophel, Gayet, & Rademaker, this issue (ICGR), Lorenc & Sreenivasa, this issue (LS), Teng & Postle, this issue (TP)). The basic premise of this account is that “the systems and representations engaged to perceive information can also contribute to the short-term retention of that information (p. 118, D’Esposito & Postle, 2015)”. In my detailed reviews (Xu, 2017, 2018a & 2020), I find several pieces of evidence that are not compatible with this account. I argue instead that sensory regions are not essential for VWM storage. It is clear from the four commentaries generated that there are still large disagreements on this key issue. Below I briefly reiterate the main evidence against the sensory storage account of VWM (see Xu, 2017, 2018a, & 2020 for more detailed discussions) and reexamine the evidence used to support this account. I also discuss issues related to null effects, analysis methods, brain region definition, the neural code of VWM, and computational modeling. Collectively, existence evidence still does not provide strong support for the sensory storage account of VWM and it is unclear how this account may accommodate all the available evidence.

The main evidence against the sensory storage account of VWM

Although the main evidence against the sensory storage account of VWM has been reviewed extensively elsewhere (Xu, 2017, 2018a & 2020), given the critiques raised by CV, ICGR, LS and TP, it is important to briefly reiterate this evidence, some of which is either overlooked or ignored by the four commentaries. One key evidence against the sensory storage account is that TMS to early visual areas can disrupt VWM performance in the early delay period, but not during the mid to late delay period (Cattaneo et al., 2009; van de Ven et al., 2012; van Lamsweerde & Johnson, 2017). Thus, once information is encoded and consolidated into VWM for storage, further disruption of processing in early visual area does not negatively impact VWM storage. A sensory-storage view needs to accommodate this evidence for it be a viable theory.

In an fMRI decoding study, we find that, while placing distractors during the VWM delay period does not impact behavioral performance and VWM representation in human superior intraparietal sulcus (IPS), it significantly modulates VWM representation in early visual areas (Bettencourt & Xu, 2016a). The effect is reported as a significant change in the decoding performance between distractor absent and present trials when distractor presence is predictable and in the distractor absent trials between distractor predictable and unpredictable trials. Note that the behavioral task of the study requires fine orientation discrimination, with performance being at neither ceiling nor floor. While some have argued for a significant role of the early visual areas in retaining fine sensory representations during VWM delay (e.g., Christophel et al., 2017; LS), it is unclear how a significant decrease in VWM representation in early visual areas without the accompany behavioral performance drop could have supported the task.

Consistent with these findings, neurophysiological studies report that VWM delay signals are present in the feedback layers of early visual areas (van Kerkoerle et al., 2017; see also Lawrence et al., 2018), enabling VWM signals in visual areas to be restored after they are disrupted by distraction (Miller et al., 1996; van Kerkoerle et al., 2017). The content of VWM is thus either stored elsewhere or in a latent format undetectable by activation-based measurements. While we have evidence for the former, evidence for the latter has not been well established (see a later discussion).

LS note that both VWM representations in early visual areas and behavioral responses are biased toward delay-period distractors, but that the average neural bias magnitude in early visual areas is significantly larger than the corresponding behavioral bias. LS argue that parallel memory representations in regions like the IPS where such bias is not found may be used to mitigate the effect of distractors. However, the greater distractor bias in early visual area than that in behavior may indicate instead that VWM representation in early visual areas and its accompany distractor effect are nonessential to VWM storage and behavioral performance.

There is now evidence suggesting that VWM signals in early visual areas may facilitate probe detection at the VWM decision stage by acting as a matching template (Serence, 2016; Xu, 2017 & 2020; Rademaker et al., 2019). CV and LS argue that such a function could still be an essential feature of WM maintenance and ICGR argue that the interplay between early visual areas and other areas is an integral part of VWM storage. However, given that disruption of VWM signal relies on top-down feedback for restoration and that the magnitude of the distraction effect in early visual cortex exceeds that of behavior, it is unclear how a template matching process and the interplay between brain regions are essential for VWM storage. In a natural scenario, when we search for a target object held in VWM (e.g., looking for a friend’s face in a crowd), the encoding of the nontarget objects should not distort our representation of the original target object; otherwise, we would lose track and never find the intended target object. It is unclear how such a distractor prone VWM representation may be an essential part of VWM storage.

Meanwhile, we and others have found robust VWM signals for basic sensory information in posterior parietal cortex (PPC) and that such signals persist even when VWM signals are disrupted in PFC and in early visual areas (e.g., Jacob & Nieder, 2014; Bettencourt & Xu, 2016a; Christophel et al., 2018; Lorenc et al., 2018). Moreover, as described Xu (2017), neurophysiological studies have shown that the PPC VWM signal has a shorter latency than that of prefrontal cortex (PFC; Salazar, et al., 2012; Swaminathan & Freedman, 2012; Jacob & Nieder, 2014). PPC thus plays a unique functional role in VWM storage in addition to that of PFC (note that ICGR express my view as only considering PPC to be essential to VWM storage, whereas in Xu, 2017, I highlighted both PPC and PFC as being essential to VWM storage). PPC’s unique role in VWM storage argues against a view expressed by LS that PPC VWM representation is epiphenomenal. Additionally, the existence of VWM representation for basic sensory information in PFC and PPC is incompatible with a hierarchical view of VWM storage that argues that concrete sensory and abstract representations are held in posterior sensory regions and anterior regions such as PFC, respectively (e.g., Christophel et al., 2017; LC; and ICGR).

It may be argued that neural response to a particular feature does not necessarily reflect the representation of that feature; it could instead reflect attention or task related control processes. Using the exact same inverted encoding model procedure, Ester et al. (2015 & 2016) report orientation representation in PFC and PPC as they do in early visual areas in both a perceptual discrimination task and during the delay period of a VWM task. It is unclear why the same result from early visual areas can be taken to reflect the existence of representations but those in PFC and PPC cannot. Single cell recording, fMRI adaptation and fMRI pattern decoding studies have reported distinctive PPC signals for a host of visual features, such as color, shape, object category, and object identity, with the representational structures of these signals closely tracking perception and behavior (see Xu, 2018b & 2018c for detailed reviews). It is difficult to see how such content specific signals differ from those carrying representations. As Andersen and Cui (2009) once argue, overextending the concept of attention (and control as well in this regard) to include a variety of different neural processes can only weaken the concept and undercut our ability to understand other cognitive functions associated with the PPC (and the PFC).

Reevaluate evidence supporting the sensory storage account of VWM

Here I review and reanalyze the main evidence that has been used to support the sensory storage account of VWM (see also Xu, 2017; 2018a; 2020). Collectively, they do not provide strong support for this account.

Evidence I: The presence of VWM content in a brain region

One central piece of evidence that has been cited to support the sensory storage account is the presence of VWM content in sensory regions during the delay period (TP, LS, and CV; see also Christophel et al., 2017). To assume the presence of VWM content in a brain region to mean that this brain region plays a role in VWM storage, however, is a leap of logic. It ignores the fact that such information can play different roles in a VWM task. Because a VWM task necessarily engages a host of different operations, such as visual stimulus encoding, attentional selection, information retention, distractor resistance, and decision making on the probe stimulus, multiple brain regions are involved. Moreover, due to interaction and feedback among cortical regions, a host of regions may carry VWM representation without them necessarily playing a functional role in VWM storage. Thus, the mere presence of VWM representation in a brain region does not help us distinguish between key regions involved in VWM storage and other brain regions not directly involved in VWM storage.

As stated before, it is not the mere presence of VWM signal in PPC and PFC that supports their roles in VWM storage; but rather, it is the evidence from TMS, distractor and feedback studies. These studies enable us to differentiate the VWM signals from the different brain regions and inform us of the nature of these signals and whether or not a region may be essential to VWM storage.

Evidence II: The correlation between neural representation and behavioral performance.

Another piece of evidence that has been used to support the sensory storage of VWM is the presence of behavioral and neural correlation in VWM tasks (ICGR, LS). As Bettencourt and Xu (2016a) and Xu (2020) state, such a correlation could stem from the registration of the visual stimuli at the encoding stage which propagates to the retention stage, the retention of information during the delay period, and/or information associated with the decision stage of a VWM task. Thus, while a VWM storage region would necessarily show a behavioral and neural correlation, finding such a correlation is not sufficient to pinpoint a brain region as playing a significant role in VWM storage.

Evidence III: Interaction between VWM and perception

By reviewing evidence showing the crosstalk between perception and VWM, CV and TP argue that such interactions support a role of early visual areas in VWM storage. Besides visual areas, both PPC and PFC have been shown to encode basic sensory information in neurophysiology and human imaging studies in passive fixation and simple discrimination tasks (e.g., Sereno & Maunsell, 1998; Liu et al. 2011; Siegel et al., 2015; Ester et al., 2016). For example, in a simple perceptual discrimination task, Siegel et al. (2015) reported the flow of basic sensory information from visual to parietal and prefrontal cortex, and the flow of sustained task information from frontoparietal to visual cortex. Consequently, even although PPC and PFC are not involved in the initial sensory stimuli encoding, their roles in information integration, manipulation and comparison, and decision-related processing may nevertheless require detailed representations of sensory stimuli in these regions.

The representations of basic sensory information in PFC and PPC and their involvement in VWM storage indicate that behavioral interactions between perception and VWM can arise from neural signals within visual areas, PFC, PPC, and/or the interactions among these brain regions. Moreover, even if perceptual and VWM interaction can be localized within visual areas, it does not necessarily imply that VWM signals in visual areas are used for VWM storage. Such signals could reflect top-down feedback from other regions needed to support operations other than VWM storage, such as probe detection (Xu, 2020).

Evidence IV: Activity silent VWM storage in early visual areas

CV argue that early visual areas can retain the content of VWM using an activity silent or latent code that evades detection by existing activation-based neurophysiology and neuroimaging methods. Consequently, a lack of VWM signal in early visual areas due to distraction does not imply a lack of representation. This is an idea that was originally proposed to explain the drop of VWM signal in PFC (e.g., Mongillo et al., 2008; Watanabe & Funahashi, 2014; Stokes et al., 2013; Stokes, 2015). CV extend the idea to propose that such a mechanism can also exist in early visual areas (seen also D’Esposito & Postle, 2015; Serences, 2016; LS; among others).

There is presently no direct evidence supporting this memory mechanism in early visual areas. When the presence of an item is not measurable, while it may be retained in an activity-silent state, it may also be retained in an active state elsewhere. In one of the first PFC studies proposing this mechanism, Watanabe and Funahashi (2014) report the presence of weak but noticeable WM delay-period activities in PFC. They caution that “it is premature to conclude that working memory is not maintained by LPFC delay-period activity” (p. 610) and that VWM signals in PFC could be maintained by sustained activities elsewhere such as in PPC. Consistent with this view, while distractor presence abolishes WM signals in PFC, it does not impact WM signals in PPC (Jacob & Neider, 2014). Likewise, while the uncued item in early visual areas show a lack of VWM decoding (Christophel, et al., 2018) or a drop in decoding (ICGR), VWM decoding in PPC and PFC is unaffected. Similar results are found in Bettencourt and Xu (2016) as described earlier. WM signals may thus be maintained in a small set of brain regions in an active state during distraction to then enable later signal restoration in PFC and early visual areas via feedback (van Kerkoerle et al., 2017; Lawrence et al., 2018). This echoes a view expressed by Schneegans and Bays (2017). Using model simulation as a proof of concept, they show that the content of VWM needs not be stored in an activity-silent state but can be retained in a weaker but still active state that may be selectively boosted later on to make it more measurable.

Wolff et al. (2015; 2017) show that pinging can amplify VWM and make remembered items decodable. Due to poor spatial localization of ERP, however, the study does not speak to where such memory is stored in the brain. Rose et al. (2016) apply TMS to parietal cortex to reactivate VWM items that are maintained outside the focus of attention. If anything, this suggests the existence of activity-silent VWM storage in parietal and not occipital cortex. Although TMS signal can travel to multiple connected regions and thus activity silent VWM code may not reside in PPC, it certainly does not provide strong evidence that such a code exists in early visual areas.

Null effect, choice of analysis methods, and brain region definition

In Bettencourt and Xu (2016a), we report that with the presence of predictable distractors, VWM decoding performance in early visual areas was no different from chance. Both Ester et al., (2016) and LS are correct in pointing out that a null finding does not guarantee an absence of VWM representation, as it is impossible to distinguish between a lack of effect vs an inability to detect the existence of an effect (Xu, 2020). As stated earlier, we do not draw our conclusions based on this null finding; but rather, based on a positive and significant decoding drop in early visual areas due to distractor presence and predictability when distractors do not impact behavioral performance.

By asking participants to retain two items in VWM and making one item relevant for an upcoming behavioral task and the other relevant for a subsequent behavioral task, Christophel et al. (2018) report that VWM decoding for the current-relevant item is above chance while that for the prospective-relevant item is at chance in early visual areas. By training the decoder with a stronger VWM representation from the current-relevant item which results in a stronger decoder, ICGR report that VWM decoding for the prospective-relevant item in early visual areas is significantly above chance, different from the null results reported by Christophel et al. (2018). Meanwhile, the decoding difference between the current-relevant and prospective-relevant items is still present (with it being significant in Christophel et al. and with the significance level not reported in ICGR). Stepping away from the null result given its weakness as stated above, these results are informative in showing a stronger VWM representation for a current-relevant than a prospective-relevant item in early visual area. This is in line with the original conclusion reached by Christophel et al (2018), rather than being ambiguous as ICGR state.

In Bettencourt and Xu (2016a), when the presence of distractors is predictable, VWM decoding in early visual areas is greater for trials without than with distractors; however, when the presence of distractor is unpredictable, this decoding is similarly above chance for both trial types. Nonetheless, decoding for the distractor absent trials is lower when distractor presence is unpredictable than when it is 100% predictable. As stated in Xu (2020), Rademaker et al. (2019) randomly intermix trials with and without distractors. Even though a cue is present at the beginning of each trial, the frequent switching back and forth of the different trial types could have discouraged participants from utilizing the cues properly and rendered their task condition to resemble our distractor unpredictable, rather than our distractor predictable, trial condition. This nonoptimal design could account for a lack of significant decoding difference between the distractor present and absent trials in early visual areas in ICGR when training and testing are done within the VWM delay period data. Thus, an exact replication of our original design is critical if our original results are to be properly refuted. In ICGR, when training is done from independent sensory data resulting in a stronger decoder, significant decoding difference is obtained between the distractor present and absent trials in early visual areas. Thus, unlike what ICGR conclude that there is no unequivocal answer regarding how analysis should be done, by leveraging upon evidence from different analysis methods, we can form a coherent understanding of the nature of VWM representation.

Of the many research groups reporting VWM decoding in PPC, we are the only one that define a superior IPS region based on its correlation with behavioral VWM capacity at the individual subject level (Todd & Marois, 2014 & 2015). This region only shows a partial overlap with IPS topographic maps (Bettencourt & Xu, 2016b). Consequently, when we examine VWM decoding broadly across topographically and anatomically defined PPC regions, we do not find the same pattern of results as we do in superior IPS (Bettencourt & Xu, 2016a, supplementary results). Because different PPC sub-regions participate in different aspects of goal-directed visual processing (see Xu, 2018b and 2018c for an extended review), selecting PPC regions based on anatomical markers, topographic maps, probabilistic maps, or activation levels in the main task may result in the selection of a heterogenous set of regions. This could potentially account for some of the discrepancies and null effects reported in decoding studies, such as in ICGR who report no IPS decoding drop when there is a drop in behavioral performance and an absence of correlation between behavioral and IPS VWM representation. While Xu (2007) report that superior IPS tracks the amount of information retained in VWM, Emrich et al. (2013), Gosseries et al. (2018), and Cai et al., (2020) do not find this effect. These discrepancies have led TP to propose that PPC serves to bind items to context in VWM, rather than being involved in direct VWM storage. It is possible that a region adjacent to superior IPS is involved in context binding in VWM. Given PPC’s role in attentional control (e.g., Yantis and Serences, 2003) and its interaction with PFC, understanding how different PPC regions may participate in VWM tasks will help us better understand the control signals needed to protect items from distraction and to prioritize items in VWM, a broader and important issue raised by TP.

The neural code of VWM

The initial evidence used to support the sensory storage account has all reported successful cross-decoding of VWM and perception in early visual areas (e.g., Harris & Tong, 2009; and Serences et al., 2009), leading these researchers to conclude that “retaining an orientation in working memory recruits many of the same orientation-selective subpopulations as those that are activated under stimulus-driven conditions” (Harris & Tong, 2009, p. 633) and “WM representations in V1 are reasonable ‘copies’ of those evoked during pure sensory processing” (Serences et al., 2009, p.207). Thus, a common code used for both perception and VWM has been a defining feature of VWM storage in early visual areas. TP argue against this finding by citing two studies that fail to find cross-decoding between perception and VWM (Riggall & Postle, 2012; Emrich et al., 2013). However, Riggall and Postle (2012) in fact show successful cross decoding in medial occipital region (see their Figure 4H) and Emrich et al. (2013) do not specifically examine cross-decoding in early visual areas.

Recently Yu et al. (2020) report anti-correlation of the fMRI response patterns in early visual areas between the current-relevant and the prospective-relevant items in VWM. They argue that such a code transformation reflects how a prospective-relevant item is stored in VWM in sensory regions. However, such anti-correlation is not reported by ICGR. van Loon et al. (2018) also report a shared code between the current-relevant and the prospective-relevant items during the main VWM delay period in posterior fusiform object shape region. An anti-correlation in code is only seen when the current-relevant item is involved in an active visual search task. Given that anti-correlation can be driven by factors not related to VWM storage, such as fMRI response undershoot (with the magnitude of the undershoot likely dependent on the nature of the sensory stimulus) and/or stimulus-specific inhibition/suppression, more research is needed to understand and replicate this finding.

Transformed VWM codes have been reported in macaque PFC during distraction to better protect the content of VWM (Parthasarathy et al., 2017) and follow the presentation of a retro cue to allow VWM signals to better guide behavior (Panichello & Buschman, 2021). Such a transformed code, however, is not found in visual area V4 (Panichello & Buschman, 2021). Although a recent study reports that memory representation in a statistical learning task is transformed in mouse auditory cortex compared to the incoming sensory signal (Libby & Buschman, 2021), the task requires the reactivation of information from long-term associative learning, rather than from VWM, and thus likely engages mechanisms distinctive from those supporting VWM.

Verbal vs computational models

TP is correct in stating that an eventual understanding of VWM requires the development of a formal quantitative model. Nevertheless, verbal models and/or intuitive reasoning can still provide important conceptual guidance to such model development. To explain VWM capacity limitation, the three quantitative models cited by TP all propose some form of interference, whether in context representation (Oberauer & Lin, 2017), connections in the random-recurrent network (Bouchacourt & Buschman, 2019), or a “binding pool” (Swan & Wyble, 2014). Interference is an intuitive verbal concept, with the quantitative models spelling out how it may be implemented. Given that the three models propose different mechanisms to implement interference, they disagree on the exact computational mechanisms underlying interference and capacity limitation. In this regard, quantitative modeling faces the same challenges as conceptual modeling and more experimental data are needed to constrain and aid model development.

TP cite a computational modeling study by Buss et al. (2020) that conclude that IPS may play a role in change detection rather than VWM storage. Buss et al. model the fMRI responses from Todd and Marios (2004) and Magen et al. (2009). Note that IPS activations in Magen et al. are more medial than those of Todd and Marios, and are localized by a group-averaged, rather than individual participant’s, behavioral capacity measure as in Todd and Marios. Consequently, Magen et al.’s IPS regions likely encompass a broader PPC region including the attentional control mechanisms in superior parietal lobule (e.g., Yantis & Serences, 2003). This significantly challenges the conclusions reached by Buss et al. and highlights a need to carefully scrutinize experimental evidence used for a computational study.

Conclusions

Here I briefly reiterate some of the key evidence against the sensory storge account of VWM. I also provide a detailed reanalysis of why the main evidence supporting this account is problematic. Collectively, existence evidence from human neuroimage and TMS studies and those from monkey neurophysiology studies does not provide strong support for the sensory storage account of VWM. For this account to be viable, it needs to accommodate all the evidence and show how sensory storage is still necessary in light of the counter evidence. While the proponents of sensory storage have argued for its various advantages (e.g., CS and ICGR), it is just as easy to argue for a storage system that is more centralized to PPC and PFC for better distractor resistance, task control, and information integration and utilization (see Xu, 2017, 2018b, & 2018c). Because VWM representations span over multiple brain regions, I concur with LS and TP that a complete understanding of VWM requires us to comprehend the complex interplay among the different brain regions. It then becomes more critical that we form an accurate depiction of how each region may uniquely contribute to VWM by leveraging upon all the available evidence.