Spatial Specificity of Feature-based Interaction Between Working Memory and Visual Processing

Abstract

Visual working memory interacts with ongoing visual processing in a stimulus-specific manner, potentially through a common neural substrate supporting visual perception and working memory maintenance. The spatial specificity of this effect, however, remains unknown. The current study tested whether features in working memory influence perception in a spatially specific or global fashion. Across four experiments, subjects performed perceptual discrimination tasks on orientation or on contrast while concurrently holding an orientation in working memory. Experiments 1 and 2 revealed that memory content boosted the perceived contrast of the discrimination stimulus when the two matched in orientation, but only when the locations of the memorandum and the discriminandum also matched. In turn, feature-based influence on memory precision was also greater when locations matched. Experiments 3 and 4 demonstrated that the spatial specificity of this interaction was influenced by task demands. When encoding of location was discouraged in Experiment 3, memory interacted with perception in a global fashion, whereas when location was task-relevant in otherwise very similar Experiment 4, the feature-based enhancement was again modulated by location. These results suggest that context-binding demand is an important determinant of the spatial specificity of memory-perception interaction and highlight the flexible configurability of working memory representations.

Significance statement:

This study demonstrated that context-binding demands determine the spatial constraint of feature-based interaction between working memory and visual processing.

Visual working memory refers to the temporary retention of a small amount of visual information in an accessible state, for elaborative processing and/or the guidance of behavior. Information held in visual working memory has been shown to interact, bidirectionally, with ongoing perception. In one direction, nonspatial features maintained in working memory influence attentional allocation to (Olivers et al., 2006; Soto et al., 2005), awareness of (Gayet et al., 2013), and judgements about (Teng & Kravitz, 2019) subsequently perceived information. In the other direction, processing visual distractors can bias working memory representations (Rademaker et al., 2015; Teng & Kravitz, 2019). These behavioral findings are compatible with recent neuroimaging findings that visual working memory storage recruits visual processing areas, including during concurrent visual processing (Harrison & Tong, 2009; Rademaker et al., 2019; Riggall & Postle, 2012; Serences et al., 2009). The goal of the present study is to delve into mechanism by exploring how dependent these interactions might be on the relative location in the visual field of remembered versus the perceived item (overlapping vs. “near” vs. “distant”).

One possibility is that the interaction between feature-based visual working memory and real-time perceptual processing could be spatially “global,” occurring regardless of the relative proximity of the remembered versus the perceived item. This might be expected on findings that working memory of visual patterns was not impacted by location changes of the stimuli between encoding and probe display (Phillips, 1974) and that binding between nonspatial visual features in working memory can be maintained independent of their location (Logie et al., 2011). A recent study by Ester et al. (2009) used fMRI (functional magnetic resonance imaging) and MVPA (multi-voxel pattern analysis) to examine the spatial specificity of working memory representation in visual cortex. They found no difference in decoding accuracy from contralateral versus ipsilateral cortex, suggesting that the maintenance of a low-level feature in working memory might recruit feature-selective neurons globally. Therefore, it is possible that working memory is maintained with a mechanism similar to feature-based attention through a global modulation of neural gain of attended feature channels (Saenz et al., 2002; Treue & Martínez Trujillo, 1999). For our present question, such global activation of feature channels would then be expected interact with the processing of subsequent sensory input, and thereby impact behavior, regardless of the retinotopic location of that input.

An alternative possibility is that feature-based interactions between working memory and perception could be modulated by relative proximity in the visual field. Location has been proposed to hold a privileged role in working memory even when it is irrelevant to the task (Kondo & Saiki, 2012; Olson & Marshuetz, 2005; Pertzov & Husain, 2014; Treisman & Zhang, 2006). Consistent with this possibility is evidence for a positional-specificity effect, whereby behavioral performance, as well as the fMRI response to the recognition probe, are enhanced when the probe location matches the location at which that trial’s sample had appeared (Postle et al., 2013). Also relevant are the findings from a recent fMRI study that used multivariate inverted encoding modeling (IEM) to reconstruct memorized orientation with models trained with data that did or did not contain information about stimulus location. Reconstructions of orientation were stronger when generated with location-dependent relative to location-independent data, and only individual differences in the amplitude of former predicted the precision of orientation recall (Cai et al., 2019). Although these findings don’t rule out the global representation of stimulus features, they do suggest an important role for the representation of location context in nonspatial visual working memory and, for our present question, predict that memory-perception interactions should be sensitive to location overlap.

In the current study, we carried out four behavioral experiments to assess the functional relevance of location context in nonspatial visual working memory by examining the influence of a feature held in working memory on perceptual sensitivity of ongoing visual processing. We adopted a psychophysical discrimination task similar to the dual-task paradigm of Teng and Kravitz (2019), adding a manipulation of spatial location. Subjects held an orientation in mind while performing an interleaved discrimination task, and the congruity of location between memorandum and discriminandum, as well as the congruity of their orientations, were manipulated.

This design allowed us to evaluate several hypotheses:

1. The global account predicted that 1) the maintained orientation would facilitate the contrast perception of a subsequently presented orientation-congruent stimulus, and the degree of facilitation would not vary with location congruity; and 2) processing an orientation-congruent stimulus would enhance the precision of the memory representation. and the degree of enhancement would not vary with location congruity.

2. The location-specific account predicted that 1) the maintained orientation would facilitate the contrast perception of a subsequently presented orientation-congruent stimulus to a greater extent if the two were also spatially congruent; and 2) processing an orientation-congruent stimulus would enhance the precision of the memory representation to a greater extent if the two were also spatially congruent.

In Experiment 1, we found initial evidence for a spatially specific influence from working memory to the processing of the discriminandum. Experiment 2 ruled out an alternative interpretation of these findings by replicating the results despite making the task demands of the two tasks orthogonal. These findings raised the question of whether these observations of sensitivity to spatial overlap reflected a process that was obligatory, or perhaps variable with task demands. It has been suggested that spatial and nonspatial information could be represented through separate mechanisms in working memory (Logie & Marchetti, 1991), and early physiological studies also found engagement of different brain areas for spatial and feature/object mnemonic information (Goldman-Rakic, 1996; Wilson et al., 1993; c.f., Rao et al, 1997). Given the existing evidence for both the global and the spatial-specific accounts, it could be possible that the spatial component of working memory may be selectively engaged when stimulus location is task relevant, and that the extent of spatial specificity of its interaction with perception also depends on the task-relevance of stimulus location during memory encoding. Task-relevance of spatial location has been shown to influence how working memory representations flexibly handle location changes between sample and probe displays in a color change-detection task (Woodman et al., 2012). Another relevant study used fMRI and a paradigm similar to Ester et al. (2009), but made location task-relevant, and found that decoding accuracy was higher for contralateral than for ipsilateral activity in early visual areas (Pratte & Tong, 2014). This raises the possibility that context-binding demands may also modulate the importance of spatial overlap for memory-perception interactions. In Experiments 3 and 4, we directly manipulated the importance of context-binding demands by discouraging the encoding of location (Experiment 3), and then by slightly modifying the procedure to make location relevant for memory retrieval (Experiment 4).

Experiment 1

Materials and Method

Participants

Using the data reported in Teng and Kravitz (2019), a priori power analysis indicated that a sample size was 17 subjects was needed to achieve 90% power to detect an effect of Cohen’s d = 0.84. Therefore, to ensure sufficient power, we recruited and tested 20 individuals (16 female, average age 21.5 ± 2.6 years) from the University of Wisconsin–Madison community. All reported normal or corrected-to-normal vision and were provided informed consent approved by the University of Wisconsin–Madison Health Sciences Institutional Review Board. All participants were monetarily compensated for their participation.

Procedure and stimuli

All stimuli were created and presented using MATLAB (MathWorks, Natick, MA) and Psychtoolbox 3 extensions (Brainard, 1997) on a 20 inch ViewSonic CRT monitor (ViewSonic America, Brea, CA).

Subjects performed a dual-task paradigm in which a visual discrimination task was embedded in a working memory task (Figure 1). Each trial began with the display of a white fixation dot at the center of the screen (0.5 s), followed by the simultaneous display of two memory-sample Gabor patches, one in each visual field, together with a central arrow indicating which of the two would be tested at the end of the trial (0.5 s). After a 1-s inter-stimulus interval (ISI), during which the arrow was replaced by the fixation dot, a discrimination Gabor patch was presented for 0.2 s, and subjects were instructed to report whether its tilt was clockwise or counterclockwise relative to vertical (left-hand key presses, “E” for counterclockwise and “F” for clockwise responses) during the ensuing ISI of 2.3 s. Finally, recall of the cued sample was prompted by the onset of an orientation wheel appearing at the same location as the cued sample (4 s). The inter-trial interval (ITI) was 1 s.

Figure 1.

Paradigm and design of Experiment 1. A) Subjects first viewed the memory sample for 0.5 second and memorized the orientation of the Gabor patch indicated by the central spatial cue. After 1 second of inter-stimulus interval (ISI), they perform a discrimination task on a new orientation patch on one of the two sides and report whether the orientation is clockwise or counterclockwise to the vertical orientation. The congruencies in spatial location and orientation of the cued memory sample and the discrimination stimulus were manipulated. Subjects could respond during the 2.5 seconds (0.2 s of stimuli display and 2.3 s of interval) between the onset of the discrimination display and the onset of the memory probe. Then they perform the continuous report during the probe period by rotating the oriented line to match the memory orientation. B) The contrast of the Gabor patch was manipulated to derive the threshold of reliably detecting the orientation. Left: example of a high-contrast Gabor; Right: example of a low-contrast Gabor.

Gabor stimuli (radius = 5°; contrast = 0.6; spatial frequency = 0.5 cycles/°; phase angle randomized between 0° and 180°), could appear in two locations along the horizontal meridian: 8° to the right and left of fixation. The orientation of each memory sample was selected independently from a fixed set of 6 values spaced by 30° (15°, 45°, 75°, 105°, 135°, 165°), and with a jitter ranging from 0° to 3° added to each. For the discrimination task, two noise patches appeared at the same two locations, and one of the noise patches was superimposed by a Gabor patch with a contrast that varied unpredictably from trial to trial between 8 values (0.04, 0.12, 0.20, 0.28, 0.36, 0.44, 0.52, or 0.60%; so as to derive psychometric functions). On orientation-incongruent trials, the orientation of the discriminandum differed from that of the cued sample by 35° or 70°, randomly determined. At recall, subjects rotated the orientation wheel (radius = 5°) with right-handed key presses (left and right arrow keys), with each discrete click rotating the wheel by 1°, and a steady press rotating it more quickly. The data were analyzed as a function of the two conditions of interest: spatial congruity between the memorandum and the discriminandum, and orientation congruity between the memorandum and the discriminandum.

Subjects completed 8 blocks of 80 trials, resulting in 640 trials in total, with location congruity, orientation congruity, and the 8 levels of discriminandum contrast counterbalanced.

Results

Discrimination task

Contrast thresholds were estimated by fitting each individual’s data to the Weibull function through the Palamedes toolbox (Prins & Kingdom, 2018) in MATLAB:

$$\psi =\gamma +(1-\gamma -\lambda )\times (1-e^{-{\left(\frac{x}{\alpha }\right)}^{\beta }})$$

in which ψ represents the proportion of accurate responses discriminating the orientation of the stimulus at contrast x; γ is the lower asymptote of the psychometric function (guess rate) and was fixed to 0.5; λ is the lapse rate and was fixed to 0.01. α and β are the fit parameters where α represents the threshold and β represents the slope of the psychometric function. For each individual’s data, threshold values of about 82% discrimination accuracy were derived separately for the four conditions (Figure 2A and B). Trials that did not have a response or had a response time greater than 2 s were excluded.

Figure 2.

Location congruity modulated feature-based interaction between working memory and perception (Experiment 1). A) Accuracy in the discrimination task plotted as a function of contrast of the discrimination stimuli for the location congruent condition. Orange and blue lines are the fitted psychometric functions for the orientation congruent and incongruent conditions, respectively. The threshold in stimulus contrast to reliably detect its orientation was significantly lower when the orientations of the memory sample and the discrimination stimulus were congruent than when they were incongruent. Note that the psychometric curves in this and all following figures were fitted on the group-averaged value for demonstration purposes. B) Accuracy in the discrimination task for the location congruent condition, plotted separately for orientation congruent and incongruent conditions. When the location of the memory sample did not match that of the discrimination stimulus, the difference between the contrast thresholds was not significant. C) Contrast threshold plotted by location and orientation congruencies. Location congruity modulated the orientation-based facilitation in contrast threshold. D) Working memory representation was influenced by both location and orientation congruencies. Memory precision was higher when location was congruent between the memory sample and the discrimination stimulus. Congruent in orientation also facilitated the memory precision. For this and all following figures: * p < 0.05; error bars represent normalized standard error (Morey, 2008).

Inspection of the derived contrast thresholds suggested that discrimination threshold was lowered on orientation-congruent trials, but only when the discriminandum appeared at the memory-cued location (Figure 2C). This was confirmed with repeated-measure analysis of variance (ANOVA) which yielded a main effect of orientation congruity (F(1,19) = 7.20, p = 0.007, partial η² = 0.325), no effect of location congruity (F(1,19) = 2.42, p = 0.136, partial η² = 0.113), and an Orientation-congruity x Location-congruity interaction (F(1,19) = 4.47, p = 0.048, partial η² = 0.190). Follow-up pairwise comparisons demonstrated that when location was congruent, the discrimination threshold was significantly lower on orientation-congruent trials than on orientation-incongruent trials (t(19) = 3.02, p = 0.007, Cohen’s d = 0.674), but the same was not true for location-incongruent trials (t(19) = 1.72, p = 0.102, Cohen’s d = 0.384).

Working memory task

Recall precision, calculated as the inverse of the standard deviation of memory error, was superior for orientation-congruent trials at both levels of location congruity, and was superior for location-congruent trials at both levels of orientation congruity (Figure 2D). Repeated-measure ANOVA revealed significant main effects of orientation congruity (F(1,19) = 20.52, p < .001, partial η² = 0.519) and of location congruity (F(1,19) = 4.71, p = 0.043, partial η² = 0.198), with a higher precision for congruent locations than incongruent locations. The interaction between the two factors was not significant (F(1,19) = 1.13, p = 0.30, partial η² = 0.056).

Discussion

This experiment tested the spatial dependence of the influence of the contents of working memory on the contrast perception of a feature in a secondary task. We observed that when subjects held an orientation in mind, their perception of a different orientation-congruent stimulus was enhanced, as if the working memory representation boosted the contrast of the discriminandum by approximately 13%, but only when the retinotopic location of the two was the same (Figure 2A). In the other direction, the perceptual processing of discriminanda also influenced the contents of working memory, in that incongruity of orientation or of location degraded memory precision. Thus, although the effect of working memory on perceptual discrimination was location-specific, the effect of perceptual discrimination on memory precision was not.

Although the results from Experiment 1 reveal bidirectional influences between stimuli being processed for visual discrimination and stimuli being held in working memory, they leave uncertain whether these effects occurred at the level of stimulus representation per se, or, alternatively, at a later stage. That is, because both tasks required guiding action based on stimulus orientation, it is possible that the interaction occurred at a post-perceptual level, such as decision making, or strategy. Therefore, we designed Experiment 2 to generate results that, should they come out as predicted, would constitute less equivocal evidence for between-task interactions at the level of stimulus representation. The results of the additional experiment would also provide additional data about the location specificity of these effects.

Experiment 2

In this experiment, although the interposed perceptual discrimination task entailed the presentation of a Gabor stimulus, as it had in Experiment 1, the task required a response about its contrast, rather than about its orientation (Figure 3A). We hypothesized that, even though the orientation of the discriminandum was irrelevant for the contrast discrimination task, the memory representation would automatically influence early processing of all features of this subsequent sensory input, regardless of their task relevance. This would be conceptually similar to the automatic capture of attention by task irrelevant distractors that match working memory content in a visual search task (e.g. Olivers et al., 2006). Therefore, we predicted that the perceived contrast of a discriminandum would be enhanced on trials when orientation and location between it and the memorandum were congruent.

Figure 3.

Paradigm and design of Experiment 2. A) Subjects first viewed the memory sample for 0.5 second and memorized its orientation. After 1 second of ISI, they were briefly presented with another Gabor patch with varying contrast, followed by a mask. Then they directly reported the perceived contrast of this discriminandum by adjusting the contrast of the test stimulus to match. At the end of the trial, they perform the continuous report on the memorized orientation.

Materials and Method

Subjects

Experiment 2 was carried out over Amazon Mechanical Turk, which necessitated collecting fewer trials per subject. A priori power analysis carried out with G*Power (Faul et al., 2009) on the data reported in Experiment 1 (and taking into account the reduced number of trials) indicated that we needed 36 subjects to achieve 90% power to detect a partial η² of 0.09 with a two-tailed alpha of 0.05. To be conservative, we collected data from 40 subjects (24 female, average age 43 ± 11.3 years). All subjects reported normal or corrected-to-normal vision and were provided informed consent approved by the University of Wisconsin–Madison Health Sciences Institutional Review Board. All were monetarily compensated for their participation.

Procedure and Stimuli

All stimuli were created with customized JavaScript code and presented to subjects through the PsiTurk toolbox (Gureckis et al., 2016).

The stimulus locations and characteristics were the same as Experiment 1, except where noted. Each trial began with the onset of a white fixation dot that remained at the center of the screen for the duration of the trial. Next, after 0.5 s, two Gabor patches appeared for 0.5 s at each side of the screen, along with a centrally presented arrow designating which item was the memorandum. Then after a 1 s ISI, one discrimination Gabor patch was presented briefly either to the left or to the right of fixation (0.2 s), followed by a visual mask for 0.2 s. As quickly as possible, subjects reported the perceived contrast of the discriminandum by adjusting the contrast of a test stimulus to match the discriminandum through key presses. 0.5 s after this response, the memory probe was presented, and subjects rotated the orientation wheel to match the orientation of the memory sample. The ITI was 1 s (Figure 3).

The orientation of the memory sample was randomly selected from a fixed set of 6 values (same as Experiment 1), spaced by 30°, with a jitter of 0-3° added. There were five levels of contrast for the discriminanda (5, 9, 17, 32, and 60%). As with Experiment 1, the two factors that were key for implementing our hypothesis test were location congruity – whether the location at which the memorandum had been presented overlapped with that of the discriminandum – and orientation congruity – whether the orientation of the memorandum matched that of the discriminandum. Each subject completed 5 blocks of 40 trials, resulting in 200 trials in total. Location congruity, orientation congruity, and the 5 levels of contrast of the discriminandum were all counterbalanced.

Results

Discrimination Task

The reported contrast was plotted as a function of the presented contrast level, and inspection of the results (Figure 4A and 4B) indicated that congruity of orientation between the memory sample and the discriminandum increased the perceived contrast level of the discriminandum when their locations were also congruent. When averaged over all contrast levels for each condition (Figure 4C), the repeated-measures ANOVA revealed a main effect of orientation congruity (F(1,39) = 6.87, p = 0.01, partial η² = 0.15), no main effect of location congruity was not significant (F(1,39) = 0.27, p = 0.60, partial η² = 0.007), and a significant interaction between orientation and location congruity (F(1,39) = 4.48, p = 0.4, partial η² = 0.10). The difference between the orientation-congruent and -incongruent conditions was significant for the location-congruent condition (t(39) = 3.43, p = 0.001, Cohen’s d = 0.54) but not for the location-incongruent condition (t(39) = 0.59, p = 0.56, Cohen’s d = 0.09). Similarly, pairwise t-tests for each contrast level found significant difference only for the location congruent condition (Figure 4A), contrast of 5%: t(39) = 3.68, p < 0.001, Cohen’s d = 0.58; 9%: t(39) = 2.12, p = 0.04, Cohen’s d = 0.34; 32%: t(39) = 3.31, p = 0.002, Cohen’s d = 0.52.

Figure 4.

Working memory content influenced contrast perception regardless of task-relevancy. A) When the memorandum matched with the high-contrast discriminandum in location, a match in orientation between the two increased the perceived contrast of the discriminandum . B) When the memory sample did not match with the discrimination stimulus in location, no difference was observed between orientation congruent and incongruent conditions. C) Reported contrast was averaged across all contrast levels and plotted by location and orientation congruity. Location modulated the extent of orientation-based facilitation that the enhancement was only evident when locations of the discriminandum and memorandum matched, not when location mismatched. D) Working memory precision also demonstrated a location-specific effect: the congruity in orientation enhanced memory precision, only when the location of the two matched.

Working Memory Task

Repeated-measures ANOVA on the precision of the working memory report revealed a significant main effect of orientation congruity (F(1, 39) = 4.66, p = 0.04, partial η² = 0.11), no main effect of location congruity (F(1, 39) = 0.15, p = 0.70, partial η² = 0.004) and a significant location × orientation interaction (F(1, 39) = 7.51, p = 0.01, partial η² = 0.16) (Figure 4D). The difference between orientation-congruent and -incongruent conditions was significant for the location-congruent condition (t(39) = 3.13, p = 0.003, Cohen’s d = 0.50) but not for the location-incongruent condition (t(39) = 0.62, p = 0.54, Cohen’s d = 0.10).

Discussion

In Experiment 2, we tested the spatial specificity of the influence of feature-based working memory on visual perception with a design that orthogonalized the demands of the memory and discrimination tasks. Despite the irrelevance of stimulus orientation for the discrimination task, the perceived contrast of discriminanda was enhanced by orientation-congruent memoranda when the location of the two also overlapped. This spatially specific effect was further demonstrated in the opposite direction: orientation congruity modulated memory precision on trials when the location of the two matched. Thus, these results replicate and extend Experiment 1’s demonstration of spatial specificity of the interaction of perception with concurrent working memory.

How can these findings be reconciled with those from previous studies that have found that working memory content can influence visual processing at locations other than where the memorandum was encoded (e.g. Gayet et al. 2017)? One possible explanation is differences in task demand. In the studies that reported spatially global interactions, the memoranda were individual, always presented centrally, and they never overlapped with the “visual” stimuli. In Experiments 1 and 2, however, the location of the critical memory sample varied unpredictably between two potential locations, it was always presented together with a second to-be-ignored stimulus, and the interpolated “visual” stimulus could also appear, unpredictably, at the same location as had the critical sample. These differences may have resulted in a stronger tendency for subjects in Experiments 1 and 2 to incorporate information about location in their mnemonic representations of the sample. Indeed, it was demonstrated in Woodman et al. (2012) in a change detection task that task demand of spatial encoding determines the interference of location change on recognition performance, suggesting that visual working memory can flexibly adapt to different task contexts. Therefore, to explore whether the spatial specificity of memory-perception interactions can also vary with task demands, Experiments 3 and 4 explicitly manipulated the relevance of stimulus location.

Experiment 3

Experiment 3 tested whether working memory interacts with perception in a spatially global way when the relevance of the location of the item maintained in working memory is minimized. The procedure from Experiment 1 was modified such that the memory sample was presented alone, with no central arrow cue and no irrelevant stimulus in the opposite visual field, and the memory probe always appeared centrally. The intent of the changes at encoding was to deemphasize the role of location by not cuing it and by removing the competition for selection between two items individuated by their location. The intent of the change at recall was to deemphasize the role of location by removing the uncertainty of where the recall wheel would appear, removing the factor of sample-to-recall overlap, and removing the possibility that the location of the recall wheel might serve as a retrieval cue. We reasoned that the combined result of these changes would be to reduce the processing of location information and, consequently, allow for spatially global interactions of orientation processing in working memory and in visual perception.

Method

Subjects

To match Experiment 2, we recruited 40 individuals from Amazon Mechanical Turk (24 female, average age 53 ± 11.2 years). All reported normal or corrected-to-normal vision and were provided informed consent approved by the University of Wisconsin–Madison Health Sciences Institutional Review Board. All were monetarily compensated for their participation.

Procedure and Stimuli

All stimuli were generated with customized JavaScript code and presented to subjects through the PsiTurk toolbox (Gureckis et al., 2016).

The stimulus locations and characteristics were the same as those from to Experiment 1, with the following exceptions. During memory encoding, one orientation sample appeared for 0.5 s either to the left or the right of fixation. After the discrimination task (clockwise/counterclockwise relative to vertical) the recall wheel appeared centrally (Figure 5A).

Figure 5.

Sample displays for Experiment 3 and 4. A) Experiment 3 introduced low context-binding demand by making location entirely task irrelevant. Memory sample was presented along during encoding and the test probe was presented centrally during recall. Subjects performed the same discrimination task as Experiment 1 during memory delay. B) Experiment 4 created high context-binding demand by making location task relevant. The test probe appeared at the same location as the memorandum for 50% of the trials and subjects reported the exact memory orientation in this condition. For the other 50% of trials, the test probe appeared on the opposite side of the screen and subjects reported the orientation that is 90 degree orthogonal to the memorandum.

Discriminanda had five levels of contrast: 5, 9, 17, 32, and 60%, location congruity and orientation congruity between the memorandum and discriminandum were manipulated, and these were all counterbalanced across the five 40-trial blocks.

Results

Discrimination task

We estimated contrast thresholds fitting each subject’s data to the Weibull function through the MATLAB Palamedes toolbox (Prins & Kingdom, 2018). The procedure and the parameters were the same as Experiment 1. For each individual’s data, threshold values of about 82% discrimination accuracy were derived for the four conditions separately (Figure 6A and B).

Figure 6.

Low context-binding demand lead to a global interaction between working memory and visual processing. A) When the memorandum and the discriminandum were matched in location, the contrast threshold in the discrimination task was significantly lower when the orientations of the two were congruent than when they were incongruent. B) When the location of the memory sample did not match that of the discrimination stimulus, the difference between the contrast thresholds was also significant. C) Contrast threshold plotted by location and orientation congruencies. There was a main effect of orientation congruity, but feature-based facilitation was not impacted by location congruity. D) Working memory representation was influenced by orientation congruity in that congruity in orientation facilitated the memory precision regardless of location.

Inspection of the derived contrast thresholds suggested that discrimination threshold was lowered on orientation-congruent trials, and this effect was not modulated by location congruity (Figure 6C). The repeated-measure ANOVA confirmed this interpretation with a main effect of orientation congruity (F(1,39) = 17.03, p < 0.001, partial η² = 0.30), no effect of location congruity (F(1,39) = 0.12, p = 0.74, partial η² = 0.003), and no orientation × location interaction (F(1,39) = 0.01, p = 0.92, partial η² < 0.001).

Working Memory Task

Repeated-measure ANOVA on the precision of the working memory report yielded a significant main effect of orientation congruity (F(1,39) = 23.63, p < 0.001, partial η² = 0.38), no main effect of location congruity (F(1,39) = 2.90, p = 0.10, partial η² = 0.07), and no orientation × location interaction (F(1,39) = 0.66, p = 0.42, partial η² = 0.02).

Discussion

In this experiment, we modified the procedure from Experiment 1 to render the location of the memory item entirely task-irrelevant and to discourage the use of location information during maintenance and retrieval. The facilitation of contrast perception by congruity with the orientation of the memorandum did not vary as a function of location congruity. These results suggest that when the binding of stimulus identity with its location context is discouraged, the visual features held in working memory tend to be represented in a spatially global manner. To further validate the idea that task demands determine the extent of spatial specificity of working memory-perception interactions, in Experiment 4 we repeated the procedures from Experiment 3 with the exception that the location of the recall wheel (same or different from the sample) determined the response that was required.

Experiment 4

To test the idea that task instructions can determine the location specificity of working memory-perception interactions on two very similar variants of the same task, we modified the task from Experiment 3 by unpredictably varying the location of the recall wheel, and requiring a different response depending on its location relative to that of the sample: overlap required reporting the orientation of the sample, but nonoverlap required a response that was rotated by 90° from the orientation of the sample (Figure 5B). Because memory for the location at which the sample was presented is critical for this variant of the task, we predicted that, unlike for Experiment 3, the enhancement of ongoing perception by a feature match with the contents of working memory would be spatially specific.

Method

Subjects

To match Experiment 3, we recruited 40 individuals from Amazon Mechanical Turk (18 female, average age 49 ± 8.2 years). All reported normal or corrected-to-normal vision and were provided informed consent approved by the University of Wisconsin–Madison Health Sciences Institutional Review Board. All were monetarily compensated for their participation.

Procedure and Stimuli

The stimulus characteristics and timings were identical to those from Experiment 3, with the exception that the recall wheel could appear, unpredictably, at either of the two possible sample locations. On trials when the location of the recall wheel overlapped with that of the sample, subjects reported the memorized orientation of the sample, on trials when the location of the recall wheel did not overlap with that of the sample, subjects were to respond with an orientation that was rotated by from that of the sample 90°.

Congruity of orientation and of location and between memorandum and discriminandum (the factors that have been critical for all four experiments), sample-recall overlap (the factor that differentiated Experiment 4 from Experiment 3), and the 5 levels of contrast of the discriminanda were all counterbalanced. Each subject completed six 40-trial blocks.

Results

Discrimination task

For each individual’s data, threshold values of about 82% discrimination accuracy were derived for the four conditions separately with the Palamedes toolbox (Prins & Kingdom, 2018) in MATLAB (Figure 7A and B).

Figure 7.

Spatial specificity of the memory-perception interaction reemerged when location was task relevant. A) In location congruent condition, congruity of orientation between the memorandum and discriminandum lowered the contrast threshold in the discrimination task. B) When the location of the two did not match, there was also significant orientation-based enhancement. C) In addition to the significant main effect of orientation congruity, the degree of enhancement was further modulated by location congruity. The difference between orientation congruent and incongruent conditions were greater when their locations was congruent than incongruent. D) A location specific interaction was observed in memory precision when the memory probe was presented at the same location as the location where the memorandum had been presented.

Inspection of Figure 7C indicates that the discrimination threshold was lowered when the orientation of the memorandum and the discriminandum was congruent, and that this effect was modulated by location congruity. Repeated-measure ANOVA yielded a significant main effect of orientation congruity (F(1,39) = 19.11, p < 0.001, partial η² = 0.33) but not of location congruity (F(1,39) = 0.29, p = 0.59, partial η² = 0.007), as well as a significant Orientation x Location interaction (F(1,39) = 4.33, p = 0.04, partial η² = 0.10). Further pairwise comparisons showed that orientation-matching memoranda lowered the discrimination threshold at both congruent and incongruent locations (t(39) = 4.65, p < 0.001, Cohen’s d = 0.74; t(39) = 2.30, p = 0.03, Cohen’s d = 0.36).

Working Memory Task

Inspection of Figure 7D indicates that recall precision was markedly higher on trials when the recall wheel appeared in the same location as had the sample, and that within these sample-recall overlap trials, orientation congruity of (memorandum and discriminandum) only boosted precision on location-congruent trials. Repeated-measures ANOVA with factors of orientation congruity, location congruity, and sample-recall overlap, indicated no main effects of orientation congruity (F(1,39) = 1.93, p = 0.17, partial η² = 0.05) or of location congruity(F(1,39) = 0.04, p = 0.85, partial η² = 0.001), but a main effect of sample-probe overlap (F(1,39) = 63.95, p < 0.001, partial η² = 0.62), the latter confirming that memory precision was markedly lower when the recall wheel and sample did not overlap (and, of course, when a rotation of the remembered sample orientation was required). The interaction between orientation congruity and location congruity was significant (F(1,39) = 4.93, p = 0.03, partial η² = 0.11), as was the three-way interaction of orientation congruity, location congruity, and sample-recall overlap (F(1,39) = 4.26, p = 0.046, partial η² = 0.10). None of the other interactions were significant (all F < 1.93, all p > 0.17, all partial η² < 0.05) (Figure 7D).

Based on the significant three-way interaction, we then conducted post hoc pairwise comparisons between orientation-congruent and -incongruent conditions for each combination of location congruity and sample-recall overlap (the four conditions on the x-axis in Figure 7D), and found that the only significant difference was on trials when the memorandum and the discriminandum were congruent in location and when the location of the recall wheel overlapped with that of the sample (t(39) = 2.46, p = 0.02, Cohen’s d = 0.39; for other comparisons all t < 0.79, all p > 0.43, all Cohen’s d < 0.13).

Discussion

When we modified the task from Experiment 3 to make it necessary to incorporate location in the working memory representation, it had the effect of reintroducing location-sensitivity into the interaction of orientation processing between working memory and perception. Although orientation congruity enhanced sensory discrimination in a spatially general way, this enhancement was greater at the congruent location. In the other direction, the influence of orientation congruity on memory precision was spatially specific. The fact that these effects were only observed when the recall wheel overlapped with the location of the sample may be due to the greater difficulty of non-overlap trials, itself likely due to the requirement on these trials to respond with a rotation of the recalled orientation.

General Discussion

Across four experiments, we demonstrated that the extent of spatial dependence of feature-based interactions between working memory and perception is itself dependent on the task relevance of the location of the encoded feature. Experiment 1 showed that working memory content influenced the level of contrast required for making a reliable orientation judgement, and that this effect was dependent on the location congruity between mnemonic and perceptual representations. Experiment 2 confirmed that the effect found in Experiment 1 was driven by stimulus-specific interaction between working memory and perception, by showing that the orientation held in mind influenced the processing of a subsequently presented Gabor stimulus, even when orientation of this Gabor was irrelevant for the perceptual discrimination. In this experiment the influence of spatial congruity on the working memory-perception interaction was evident in both perceptual judgement and in the precision of memory recall. Experiment 3 and 4 explored the role that task demands might play in determining the spatial specificity of this effect. When the task was structured to discourage the encoding of the location of the sample (Experiment 3), memory-perception interaction exhibited a spatially non-specific pattern, but when a record of sample location was needed to determine what kind of response to make (Experiment 4), spatial specificity reemerged. Overall, these findings suggest that, for working memory for a nonspatial feature, the extent to which subjects will incorporate location context into their representation of that feature is largely dependent on the demands of the task.

Mechanism of Feature-based Working Memory Effects

These results indicate that when location context is encoded in working memory representations, that information will influence concurrent task performance. At the level of mechanism, it may be that the need to represent the location at which a nonspatial feature was presented enhances its neural representation in feature channels with spatial receptive fields that overlap with that location. By this account, when subsequently presented sensory input matches the mnemonic representation in both feature and location, the processing of this sensory input is facilitated by the fact that the relevant feature channels are already strongly engaged by working memory (c.f., Postle et al. 2013). Because of the location-specificity of this enhancement, feature-matching channels with nonoverlapping receptive fields would not exhibit the same degree of facilitation when feature-matching sensory input appears at a nonoverlapping location. Thus, in Experiment 4, although there was a spatially global facilitation effect for orientation-congruent processing, this facilitation was greater when discriminanda were also location-congruent.

Our results also indicate that location-specific effects in nonspatial working memory are not obligatory, because they are not observed when the role of location context is minimized. In these situations, such as in Experiment 3, the effects of feature congruity are spatially global. This could reflect an absence of any top-down spatial bias during the maintenance of the feature information. Previous studies have also found that feature-based working memory can have a spatially global influence on attention allocation and perceptual judgement (e.g. Olivers et al., 2006; Teng & Kravitz, 2019). In such instances, context-binding demands are typically low, most often because they involve only one memory sample always appearing at the same location.

Functionally, such flexible configuration of working memory representations could be beneficial for behavioral performance under different contexts. Oftentimes memory retrieval depends on spatial context as the retrieval cue (especially when multiple items are encoded simultaneously), and location-specific facilitation could serve to maximize performance, for example, perhaps by reducing misbinding errors. In other situations, however, binding location to memory for a feature might be detrimental, such as in feature-based visual search with varying target location. In these cases, it would be advantageous to increase sensitivity to the target feature across the entirety of the visual field.

Comparison with Previous Studies

Working memory.

Behaviorally, it has been shown that recognition performance declines when the memory probe appears in a different location from the location of memory encoding (Hollingworth, 2006, 2007; Postle et al., 2013) or when the configuration of the probe display changes (Jiang et al., 2000). Neurally, this location-specific effect is accompanied by an enhancement in neural responses, in several visual regions, to location-matching probes (Postle et al., 2013). Furthermore, this effect of encoding-to-probe change has been shown to be modulated by task relevancy of location in nonspatial working memory tasks, in that location change has been shown to have no effect on performance when location is entirely task irrelevant (Woodman et al., 2012). Our current findings provide an important extension of these previous findings by showing that the flexible incorporation of location context into nonspatial working memory is even reflected in its interaction with perceptual processing.

The present results might help explain the seeming inconsistency between the demonstration by Ester et al., (2009) of spatially global representation of (nonspatial) features being held in visual working memory, versus that of Pratte and Tong (2014) of spatially specific working memory representation in early visual cortex. One critical difference between these two is the importance of location context-binding: in Ester et al. (2009) the memory sample was presented alone, and thus there was little need for feature-location binding; in Pratte and Tong (2014), however, two memory items were encoded at two different locations and location context was used to cue subsequent memory retrieval. From this perspective, these two sets of neural evidence are consistent with our assumption that working memory-perception interactions arise from the fact that the behavioral readout of visual working memory content relies, at least in part, on representations in early visual cortex.

Feature-based attention.

It has been proposed that working memory functions as internal attention (Chun et al., 2010; Kiyonaga & Egner, 2013), and classic attentional effects such as the Stroop effect can be elicited similarly with conflicts between internal and external representations (Kiyonaga & Egner, 2014). Consistent with this idea is the suggestion from Luck and Vogel (2013), that “visual working memory may not be a memory system per se, but may instead be a general-purpose visual representation system that can, when necessary, maintain information over short delays” (p. 394). The present results, however, may reveal one dimension along which these constructs diverge: whereas feature-based attention is generally understood to operate through a spatially global biasing of channels corresponding to the selected feature (Saenz et al., 2002), attending to an “internally” represented feature (i.e., working memory) may, depending on task circumstances, include a spatially constrained component. Assessing this possibility will require head-to-head comparison of the influence of location context on feature-based attention versus on working memory in carefully matched experimental conditions.

Alternative Interpretations and Future Directions

One possible alternative interpretation of the present results is that the instances of spatially specific interactions can be accounted for by the deployment of spatial attention to the memory sample location. For example, might subjects not be engaged in attention-based rehearsal (Awh & Jonides, 2001) of the location of the memory sample, or perhaps in perceptual resampling (Woodman & Luck, 2007)? Arguing against these accounts is the fact that most of our results did not show a general enhancement at the congruent location, which would be predicted by a spatial attention account. Another possible concern relates to the predictability of the orientation of the discriminandum. Although the orientation of the memory sample predicted the orientation of the discriminandum on 50% of trials, this predictability cannot account for the current findings because the same degree of facilitation was not seen at incongruent locations in Experiments 1, 2, and 4.

An important question for the future is whether memory-perception interactions also occur with more complex visual stimuli, and if they do, whether they also demonstrate spatial specificity despite the larger receptive fields in neurons that code for these stimuli. It will also be important to determine the exact degree of spatial specificity required for these interactions. In the current experiments the two locations were always on either side of the vertical midline, so it is uncertain whether the spatial specificity that we observed is constrained to exact retinotopic overlap, or whether presentation of memorandum and discriminandum to the same hemifield is sufficient. Further studies with a close manipulation of the distance between the congruent location and the incongruent location would be required to distinguish these possibilities. A final outstanding question that we will raise here addresses the factors that underlie the flexibility of the spatial specificity effect. In the present experiments we manipulated context-binding demands of the working memory task to influence the degree of spatial specificity, but this may not be the sole determining factor. Other factors such as the configuration of the perceptual task should be investigated to fully understand the conditions when memory interacts with perception in a global or a spatially specific manner.

General Conclusion

These experiments demonstrated that the location specificity of visual working memory representations varies flexibly with context-binding demands, and furthered our understanding of the feature-based interaction between visual working memory and visual perception. We found that when location and feature information are bound in visual working memory, the influence from working memory on perception is modulated by their spatial overlap, as can be the influence in the opposite direction. This spatial specificity disappeared when context-binding was discouraged. Although this evidence is consistent with models of shared sensory representation between working memory and sensory processing, it also points to a possible difference between “internally” versus “externally” focused attention.