Working memory prioritization impacts neural recovery from distraction

Abstract

The ability to protect goal-relevant information from disruption over short intervals is a hallmark of working memory. Recent behavioral data suggest that high-priority items in working memory are more vulnerable to disruption. We used functional magnetic resonance imaging to evaluate the hypothesis that prioritization of working memories might impact the recovery of their neural representation(s) after distraction. A delay-period retrospective cue informed participants which of two memory items (a face or a scene) to prioritize during a first delay period. Consistent with prior work, and confirming successful prioritization, multivoxel pattern classifier evidence in perceptual brain regions was higher for cued versus uncued memory items. A distraction task was then imposed before a second retrospective cue informed participants to either “stay” remembering the previously cued item or “switch” to the previously uncued item. This allowed for the evaluation of recovery for high-priority items (on stay trials) and also low-priority items (on switch trials). Classifiers showed successful reinstatement of both high- and low-priority items after distraction, but only low-priority items recovered to their pre-distraction representational levels. Moreover, the degree of prioritization before distraction predicted the amount of disruption for high-priority items after distraction, suggesting that the more a participant prioritized the cued item, the greater the impact of distraction. Our data provide neural evidence that prioritizing working memory information in perceptual regions makes that information more vulnerable to disruption.

1. Introduction

Working memory—the short-term retention of goal-relevant information—is a crucial component of cognition. Despite a broad range of brain regions being implicated in working memory (Christophel, Klink, Spitzer, Roelfsema, & Haynes, 2017), the ‘sensory recruitment’ hypothesis proposes that regions involved in perception are also recruited for working memory (D’Esposito, 2007; D’Esposito & Postle, 2015; Pasternak & Greenlee, 2005; Postle, 2006). This overlap of perceptual and mnemonic resources raises the important question of how — or to what degree — a memory persists during ongoing perception and distracting events. Despite the overlap of resources, working memories still often show recovery after distraction. For example, neural representations have been shown to be reinstated after a perceptual distractor (Derrfuss, Ekman, Hanke, Tittgemeyer, & Fiebach, 2017; Lewis-Peacock, Drysdale, Oberauer, & Postle, 2012; Yoon, Curtis, & D’Esposito, 2006), and behavioral performance is often spared after brief task-irrelevant distractors (Xu, 2017). Distraction resistance for visual working memory has been proposed to occur either through active storage in parietal areas (Bettencourt & Xu, 2016) or through shifted representational patterns in occipital areas (Derrfuss et al., 2017).

However, multiple mechanisms may underlie working memory maintenance (LaRocque, Lewis-Peacock, & Postle, 2014; Myers, Stokes, & Nobre, 2017; Olivers, Peters, Houtkamp, & Roelfsema, 2011). Working memory representations follow dynamic trajectories that depend on task demands and internal attention (Stokes, 2015). When internal attention is directed towards a subset of working memories, such prioritized information is often found to be retained in perceptual regions, while other low-priority representations may be retained in parietal and frontal regions (Christophel, Iamshchinina, Yan, Allefeld, & Haynes, 2018), via activity-silent mechanisms (LaRocque, Riggall, Emrich, & Postle, 2017; Lewis-Peacock et al., 2012; Rose et al., 2016; Sprague, Ester, & Serences, 2016; Wolff, Jochim, Akyürek, & Stokes, 2017), with active low-level firing (Christophel et al., 2018; Schneegans & Bays, 2017), or in inverted representations in occipital regions (van Loon, Olmos-Solis, Fahrenfort, & Olivers, 2018). Furthermore, previous work has shown that while memories interact with subsequent perception through attentional biasing (Gayet, Paffen, & Stigchel, 2017; Soto, Hodsoll, Rotshtein, & Humphreys, 2008), the degree of interaction differs for high-and low-priority representations (Olivers et al., 2011). Evidence that high-priority representations exert greater influences on perception (Mallett & Lewis-Peacock, 2018) suggests that the reverse may also be true; subsequent perception may have a greater impact on high- than low-priority memory representations.

Recent literature suggests that task-irrelevant perceptual interference presented during a delay period has a greater negative impact on behavioral performance for prioritized items (Allen & Ueno, 2018; Hitch, Hu, Allen, & Baddeley, 2018; Hu, Allen, Baddeley, & Hitch, 2016; Hu, Hitch, Baddeley, Zhang, & Allen, 2014), providing support of a vulnerability account for prioritized memoranda. These results are in accordance with a model of working memory where privileged access is given to high-priority representations at the cost of susceptibility to interference, and is consistent with findings that a high-priority representation has more interaction with sensory processing than low-priority representations (Mallett & Lewis-Peacock, 2018; Olivers et al., 2011). However, behavioral experiments using single retrospective cue (“retro-cue”) paradigms often support a protection account, whereby cued representations are protected against interference, relative to neutral cues, both during the delay period (van Moorselaar, Gunseli, Theeuwes, & N. L. Olivers, 2015) and at test (Makovski, Sussman, & Jiang, 2008; Souza, Rerko, & Oberauer, 2016). Thus, it remains unclear as to whether high- or low-priority memory items are more protected from interference.

Here, we apply functional magnetic resonance imaging (fMRI) and multivoxel pattern analysis (Norman, Polyn, Detre, & Haxby, 2006; Tong & Pratte, 2012) to the question of working memory distraction recovery. To investigate the effects of distraction on differently prioritized memory representations, participants were shown two visual images to remember on each trial, and a retro-cue informed them which item should be prioritized. Replicating past research (LaRocque et al., 2017; Lewis-Peacock et al., 2012; Rose et al., 2016), our data show that the retro-cue was successful in separating the two items into high- and low-priority states. To investigate the vulnerability of items in these two priority states, a distracting visual change-detection task appeared during the delay period on 2/3 of trials. Then, a second retro-cue instructed participants to either “stay” with the same memory item or “switch” to the previously low-priority memory item for testing after a second delay period. We used fMRI pattern classification during this post-distraction delay period to track the recovery of previously high- and low-priority working memory representations. Our analyses revealed stronger neural recovery for low-priority memories (on switch trials) compared to high-priority memories (on stay trials). In line with recent behavioral work (Allen & Ueno, 2018; Hitch et al., 2018), our results support a model of visual working memory in which high-priority representational patterns maintained in occipital and temporal regions are preferentially susceptible to disruption from distraction.

2. Material and methods

2.1. Participants

Seventeen healthy participants between the ages of 20 and 30 years of age (M=24.8±3; 10 females) were recruited from the University of Texas at Austin community in accordance with the University of Texas Institutional Review Board. Sample size was based on previous research with similar designs and methods (e.g., (LaRocque, Lewis-Peacock, Drysdale, Oberauer, & Postle, 2013; Lewis-Peacock et al., 2012; Rose et al., 2016)). Participants provided written consent and received $20/hr in compensation.

2.2. General procedure

After task training, participants completed 1 fMRI session consisting of 3 localizer runs (18 trials and 6 min per run) followed by 10 memory task runs (12 trials and 6.1 min per run), for a total scan time of approximately 100 minutes including the structural scan. One participant completed 9 memory task runs due to time constraints. Another participant completed only 3 memory task runs due to scanner discomfort and was therefore not included in analyses.

2.3. Stimuli

Experiment displays were scripted and presented using PsychoPy software (J. Peirce & MacAskill, 2018; J. W. Peirce, 2007). All stimuli were presented on a gray background. A central fixation point (a dark gray filled circle, 0.15° radius) was always on screen unless a target or probe stimulus was being centrally presented, or during an inter-trial-interval (ITI). In all tasks, target and probe stimuli (faces/scenes/objects; 1.5° radius) were presented either centrally (localizer task and memory task probe) or above/below fixation (memory task encoding; 2° from fixation to the center of target stimulus), and surrounded by a thin white border, which turned red or green at the end of each trial to provide response accuracy feedback. The first retro-cue was a white filled rectangle (1° W × 0.2° H) placed above or below (0.55°) fixation. The second retro-cue was a centered word (“STAY” or “SWITCH”). Letter stimuli in the distraction task were random consonants, placed equidistant along an invisible circle (1.75° radius) around central fixation. All letters and text were white.

Face stimuli were from the 10k US Adult Faces database (Bainbridge, Isola, & Oliva, 2013). Of the face images with memorability scores, we first selected those images in the lowest 40% percentile of memorability scores. Then we excluded images based on idiosyncratic features (e.g., removing famous faces and faces with excess facial hair or makeup). Images from this database are vertically rectangular and framed with a white oval border. To make the face images conform to the scene images which do not have an image border, we cropped each image to be a square where each side was 75% of the original image width. Scene stimuli were both indoor/outdoor images selected from a large corpus of subcategorized scenes (Konkle, Brady, Alvarez, & Oliva, 2010). Object images were selected from a large corpus of object stimuli (Brady, Konkle, Alvarez, & Oliva, 2008). All images were converted to grayscale.

2.4. Tasks

The memory task (Figure 1A) consisted of 3 conditions (early/stay/switch), with each task run containing 4 trials per condition in a randomized order. In brief, participants encoded two memory items, and were cued retrospectively as to which item would likely be tested with an upcoming probe. On early trials, participants were probed on the cued target after a single delay period. On the other two thirds of trials, a distracting change-detection task was presented after the first delay, and then a second retro-cue informed participants whether they should “stay” with the previously cued memory target for the upcoming probe, or “switch” to the previously uncued target. Thus, on all trials, a single memory item was prioritized in the first delay period, as distraction was presented. After the second retro-cue, on stay trials participants were required to “recover” a memory target that was prioritized during the presentation of a distracting task, and on switch trials recover a previously low-priority target. To encourage adherence to the cue, early trials were included and all retro-cues were 100% valid (i.e., on switch trials participants were informed of the switch before the probe).

Figure 1:

A) Memory task design. After encoding two images, a retro-cue indicated which item would likely be tested after an 8-s delay. On 33% of trials (early), this item was tested. On the remaining trials, a distraction task appeared instead, and then a second retro-cue indicated which item (33% stay, 33% switch) would be tested after another 9-s delay. Probes were masked to cover a random half of the image. B) Localizer task design and fMRI classifier training confusion matrix. Each trial consisted of a stream of 5 images on/off (1.5/1 s, respectively) from a single category (faces/scenes/objects). MVPA classifiers could easily distinguish between faces/scenes/objects/rest in VTC (M=92±6% across all categories). C) Behavioral performance. Participants performed worse on later trials (stay and switch) than early trials, but performance did not differ for stay vs. switch trials. Dashed line represents chance performance.

Each trial began with central fixation (0.2 s) followed by an encoding period (3 s) where 1 face and 1 scene were presented simultaneously above and below central fixation. To disentangle stimulus category and location, face and scene images were equally likely to be above/below fixation. Following a pre-cue delay (0.3 s), a rectangular retro-cue appeared either above or below fixation (0.5 s), indicating the spatial location of the memory item that should be prioritized during the first delay. Retro-cues were presented equally above/below fixation within each run. After the first delay (8 s), early trials concluded with a memory probe (2 s) of the cued item. All memory probes were either same as the target (50%) or different, and participants were instructed to press a button with the index finger of their right hand if it was the same, or a button with their middle finger if it was different. Feedback was provided with a red or green border around the probe image, indicating an incorrect or correct response, respectively. On stay and switch trials, the first delay was followed by distraction in the form of a change-detection task (6 s in total) involving an array of 8 random consonants (B. Wang, Theeuwes, & Olivers, 2018). These 8 consonants were presented equidistant around fixation (1 s), followed by a delay (2.5 s) and a probe display (2 s) containing another 8 consonants, and then a brief delay (0.5 s). Each probe display had a 50% chance of including one changed consonant in any position. Participants were instructed to press a button with the index finger of their right hand if there was no change (i.e., “same”), or a button with their middle finger if there was a change (i.e., “different”). To promote task engagement, feedback was provided with a red or green border around fixation upon response. The distraction task was followed by a second retro-cue for the memory task (1 s), a second delay period (9 s), and finally a memory probe (2 s). All memory probes were 50% masked (randomly top/bottom/left/right) to enhance task difficulty by discouraging a mnemonic strategy of focusing on an idiosyncratic image feature at encoding. All trials were separated by an ITI (6 s). For both the distraction and memory tasks, if a response was provided within the allotted response time window, accuracy feedback was provided for the remainder of the window.

The localizer designed to train category-specific fMRI pattern classifiers was a 1-back memory task (Figure 1B). Each trial consisted of 5 serial stimulus presentations (1.5 s on, 1 s off), followed by an ITI (7.5 s). Participants were asked to respond with a button press whenever a stimulus repeated (33% of probes within each run). Each run consisted of 6 trials per stimulus category (faces/scenes/objects) in a pseudo-randomized order, where each trial included only images of the respective category. All participants saw the same sequence of images (see Perception-trained decoding).

2.5. fMRI acquisition, preprocessing, and region-of-interest

Participants were scanned in a Siemens Skyra 3T scanner with a 32-channel head coil. Each scan session included a single high-resolution T1-weighted anatomical image (MEMPRAGE; FoV 256 mm, 256 × 256 matrix, 176 sagittal slices; TE=1.64/3.5/5.36/7.22 s). All functional scans were acquired using the same EPI sequence (TR=2 s; 76 slices; 3×3×3 mm voxel dimensions; 2× multiband factor). For each scan session, the mean of the first fMRI memory task run was used as a reference functional image template for rigid body motion correction using FSL’s MCFLIRT (Jenkinson, Beckmann, Behrens, Woolrich, & Smith, 2012). Each run was then independently temporally detrended and z-scored using PyMVPA (Hanke et al., 2009). All primary fMRI analyses were carried out within a ventral temporal cortex (VTC) region-of-interest, based on its functional role in item categorization (Grill-Spector & Weiner, 2014). Based on recent research (e.g., (Bettencourt & Xu, 2016; Christophel et al., 2018)), we performed additional exploratory analyses in intraparietal sulcus (IPS). Masks were determined anatomically for each participant individually using surface reconstructions in FreeSurfer (Fischl, 2012). The VTC mask was a combination of the inferior temporal, parahippocampal, and fusiform labels available in FreeSurfer (Desikan et al., 2006; Fischl et al., 2004). The IPS mask was a combination of IPS0–5 using a probabilistic retinotopic map (L. Wang, Mruczek, Arcaro, & Kastner, 2015). Left and right hemispheres were included in both region-of-interest masks, totaling 2938±279 voxels in VTC and 768±69 in IPS across participants. Pycortex (Gao, Huth, Lescroart, & Gallant, 2015) was used for region-of-interest visualizations (Figure 3B).

2.6. Multivariate pattern classification

Classification analyses were implemented using scikit-learn (Pedregosa et al., 2011) and PyMVPA (Hanke et al., 2009). A multi-class L2-penalized logistic regression classifier (scikit-learn, one-vs-rest, regularization parameter C=1.0) was used for all classification analyses. Feature selection involved an ANOVA-based voxel selection process, where F values assessed whether each voxel’s activity varied significantly between conditions. The top 1000 features (highest F values) were maintained. The IPS analyses used a smaller feature count of 500 due to the generally lower number of voxels in IPS compared to VTC. Both classification analyses described below were performed on a subset of TRs that were most likely to represent memory-related brain activity, given the hemodynamic lag: the last 2 TRs from each trial epoch of interest (all data unshifted). All TRs were treated as independent samples during classification (i.e., TRs were not averaged). For the pre-distraction period (i.e., first delay), we included the 5^th and 6^th TRs (8–12 s) of all trial types. All trials were collapsed together for the pre-distraction analysis because trials did not differ across conditions (early/stay/switch) until 12 s into the task, where either a probe (early trials) or a distracting task (stay and switch trials) would appear. Analyses of the distraction period included the 8^th and 9^th TRs (14–18 s) of both stay and switch trials. To investigate memory representations during recovery from distraction (“post-distraction”), we included the 13^th and 14^th TRs (24–28 s), separately for stay and switch trials.

2.6.1. Perception-trained decoding

The main decoding analysis involved training a classifier on data from the localizer task and testing on data from the memory task. This analysis is based on the idea that patterns of voxel activity measured during stimulus perception are similar to those recruited during working memory retention (Harrison & Tong, 2009; Serences, Ester, Vogel, & Awh, 2009), and thus was performed only in category-selective VTC (Grill-Spector & Weiner, 2014) for each participant. For these perception-trained classification analyses, fMRI pattern classifiers were trained (including feature selection) on the localizer task data and then applied to the memory task data, separately for each participant. The last 3 TRs of each localizer trial (6–12 s) were used independently for classifier training, labeled according to the stimulus category of that trial. Additionally, single TRs from the end of each localizer ITI (18–20 s) were labeled as “rest” and used for classifier training. This allowed for equal training samples for each of the four classes (face/scene/object/rest). Probability estimates from the logistic regression classifiers (“classifier evidence”) were retrieved for each class during the delay periods (pre- and post-distraction) and also during the distraction period of the memory task. To prevent classifier evidence (or lack-thereof) of one trial-relevant class impacting the other, the classifiers were also trained on classes that were task-irrelevant (object/rest) and probability estimates were not normalized across classes (to prevent summing to one). To quantify the classifier’s ability to distinguish between the categories of the two memory items (one face, one scene), we calculated a separation metric defined as the classifier evidence for the most recently cued category minus the classifier evidence for the uncued category. This provided a single metric to assess the relative prioritization between cued and uncued items before distraction, and the relative recovery of these items after distraction. Note that prior to this analysis, we trained and tested our classifier on localizer data only using a leave-one-run-out cross-validation scheme to verify the classifier’s ability to distinguish categories (Figure 1B).

2.6.2. Memory-trained decoding

Recent evidence suggests there may be representational formats in working memory that are unique to the delay period, that either augment (Derrfuss et al., 2017) or replace (Yu & Shim, 2018) those which are more similar to perception. To account for this possibility, we performed a second classification analysis where classifiers were trained and tested on cued vs. uncued memory categories separately within each trial epoch of interest. Typically, this analysis might involve aggregating within-participant classifier accuracies after a leave-one-run-out cross-validation classification scheme for each participant. Due to an insufficient number of trials on the memory task per participant, we used a functional alignment procedure known as hyperalignment (Haxby, Connolly, & Guntupalli, 2014; Haxby et al., 2011) to align voxel activity for each participant into a “common space” and then combined data across participants and performed between-participant classification by using a leave-one-participant-out cross-validation procedure. This procedure, described in detail elsewhere (Guntupalli, Feilong, & Haxby, 2018; Guntupalli et al., 2016; Haxby et al., 2011), first uses a subset of data to create a “common” or reference space via an iterative participant transformation-and-averaging process, which all participants will be aligned to. After the common space is formed, a Procrustean transformation matrix is derived for each participant which rotates, scales, and shifts their temporal voxel trajectories into the common space. The transformation matrices are then applied to a held-out dataset to align all participants into a common unified space for further analyses. This provided the classifiers with more training and testing data for each cross-validation fold, and reduced the variance of the estimated null distributions from permutation testing to be more tightly coupled with theoretical chance performance (50%, Figure 3). As all participants viewed the exact same sequence of images in the localizer task, we used these data to build the common space model with the PyMVPA toolbox (Hanke et al., 2009). For each participant, we first used localizer data to perform feature selection and then iteratively derived Procrustean transformation matrices for each participant to bring them into alignment with a common space (Haxby et al., 2011). The feature selection and transformation matrices were then applied to the memory task data for each participant. Then all participants were concatenated into a single common-space dataset, and finally the concatenated dataset was z-scored across participants. We then performed the leave-one-participant-out cross-validation scheme separately for each relevant epoch of the memory trials (pre-distraction, distraction, and post-distraction).

2.7. Statistical analyses

All data in-text are presented as mean ± SD. Statistical analyses were conducted using R statistical software. For all comparisons of classification results across different trial epochs (pre-distraction, post-distraction stay, post-distraction switch), we conducted repeated-measures ANOVAs followed by planned two-way comparisons. We used one-sample t-tests to determine if perception-trained classifier separation metrics were nonzero and if behavioral performance on the distraction task was above chance (50%). Behavioral accuracy on the memory task was assessed using a repeated-measures ANOVA across trial types (early, stay, switch) with planned two-way comparisons. All ANOVAs and t-tests were two-tailed. We used permutation tests to determine if memory-trained classifier accuracy was above chance. A null distribution of “chance” accuracy scores was derived after hyperalignment by shuffling classifier labels and retraining/testing the classifier 1000 times. For each condition, the p-value represents the proportion of null distribution scores that were above the true mean accuracy of all test folds. Specifically, this is calculated as (C+1) / (N+1), where C is the number of permutation scores greater than or equal to the mean of test scores and N is the number of permutations. The +1 in the numerator and denominator accounts for the possibility of zero permutation scores being above the mean test score.

3. Results

Behavioral performance differed across trial types in accuracy (F_(2,30)=5.7, p=0.0079; Figure 1C) and reaction times (F_(2,30)=17.3, p<0.0001). Accuracy on early trials (M=91±6%) was higher than both stay trials (M=87±10%; p=0.0366) and switch trials (M=86±9%; p=0.0034), but did not differ between stay and switch trials (p=0.7215). Reaction times on early trials (M=971±107 ms) was higher than both stay (M=877±139 ms; p<0.0001) and switch (M=876±120 ms) trials, but did not differ between stay and switch trials (p=0.998). Accuracy on the change detection task (M=66±11%) was above chance (t₍₁₅₎=5.7, p=0.00004), confirming that participants were engaged in the distraction task on stay and switch trials.

To verify the prioritization of cued memory items over uncued memory items before distraction, we measured the difference between fMRI pattern classifier evidence in category-selective VTC for these items during the first delay period (Figure 2A). Consistent with previous work (LaRocque et al., 2014), the cued items were associated with higher classifier evidence than the uncued items (M=0.17±0.11, t₍₁₅₎=6.3, p=0.00002; cued=0.41±0.11, uncued=0.24±0.06). This confirmed our ability to evaluate the impact of distraction separately for high- and low-priority items in working memory. We designed our experiment such that stay trials and switch trials would require the recovery of a high-priority or a low-priority memory item, respectively. To quantify the neural recovery of these working memory representations after distraction, we measured the same classifier separation index during the second delay period.

Figure 2: Perception-trained decoding in VTC.

A) Separation of classifier evidence for cued and uncued memory items. Delay 1 showed clear separation, consistent with past literature that the cued memory has a prioritized neural representation. During recovery from distraction in the second delay (‘post-distraction’) there was greater separation of memory items on switch trials, in which a low-priority memory item had to be recovered. B) Across-participant relationship between prioritization and low-priority recovery benefit. The more a participant prioritized the cued representation during the first delay, the greater they benefited when recovering a low-priority representation on switch trials. Post-distraction cost is switch trial separation minus stay trial separation, both from the second delay.

There was no classifier separation, and thus no evidence of memory prioritization, during the distraction period (M=0.01±0.04, t₍₁₅₎=1.2, p=0.2361; cued=0.21±0.09, uncued=0.20±0.08), yet both stay trials (M=0.08±0.06, t₍₁₅₎=4.8, p=0.0002; cued=0.29±0.07, uncued=0.20±0.07) and switch trials (M=0.16±0.12, t₍₁₅₎=5.2, p=0.0001; cued=0.35±0.10, uncued=0.19±0.08) showed successful recovery of the relevant item post-distraction. Classifier separation measures differed significantly across the pre-distraction and both post-distraction epochs (F_(2,30)=6.0, p=0.0065, Figure 2A). Importantly, the classifier separation scores post-distraction were higher for switch trials than for stay trials (p=0.0153). Furthermore, recovery of high-priority items on stay trials failed to reach pre-distraction levels of classifier separation (p=0.0041), as was observed for low-priority items on switch trials (p=0.9140). These results indicate that recovering the neural representation of a low-priority memory item after distraction was more successful than recovering a high-priority item after distraction.

Despite a difference in neural measures, behavioral performance was similar across stay and switch trials. The behavioral similarity might have resulted from a variety of reasons (see Discussion), but to investigate the behavioral relevance of the separation metric, we correlated classifier separation (cued minus uncued item classifier evidence) with behavioral accuracy and reaction times across participants. When relating classifier separation in the second delay to behavioral performance on late trials (stay and switch), we found trending correlations in the positive direction for accuracy (r=0.44, p=0.087) and negative direction for reaction time (r=−0.42, p=0.107), suggesting higher classifier separation metric is indicative of better behavioral performance, even if our current sample did not detect differences in performance across stay and switch trials. There was a similar trend between classifier separation and accuracy (r=0.42, p=0.101) on early trials but not for reaction times (r=−0.30, p=0.254).

From these data, the “cost of prioritization” for distraction recovery can be estimated by comparing the neural measures of recovery (classifier separation) for high-priority items vs. low-priority items. We found that these cost estimates were correlated with the degree of prioritization prior to distraction (r=0.50, p=0.0496; Figure 2B). This indicates that the more a participant prioritized one item over another in working memory prior to distraction, the more susceptible that item’s neural representation was to distraction.

Neural results presented thus far have described the degree to which category-level fMRI pattern classifiers that were trained on data from a perceptual task could detect working memory representations reinstated after distraction. While the 1-back task used in the localizer had a memory component, the sluggish fMRI activity recorded from that fast event-related design was dominated by perceptual signals. Therefore, to investigate neural recovery of memory representations that may be unique to the delay periods (i.e., potentially separate from perceptual reinstatement), we trained and tested classifiers within each trial epoch of the memory task (see Memory-trained decoding in Methods). In VTC, the memory-trained classifiers (Figure 3A) showed successful decoding of the cued memory category pre-distraction (M=58±4%, p=0.0010) and post-distraction for switch trials (M=58±9%, p=0.0010). However, these classifiers failed to discriminate between the cued and uncued categories during the distraction period (M=49±4%, p=0.7792) and during the post-distraction period of stay trials (M=51±7%, p=0.2358). Classifier accuracy differed significantly across the pre-distraction and both post-distraction periods (F_(2,30)=5.5, p=0.0094), showing a significant drop post-distraction for stay trials (p=0.0101) but not for switch trials (p=0.9948). In line with results from the perception-trained classifiers, switch trials showed greater classifier accuracy than stay trials in the post-distraction period (p=0.0136). Together, these results provide further support that working memories held in a high-priority state are more vulnerable to distraction, or conversely, that working memories held in a low-priority state are better protected from distraction.

Figure 3: Memory-trained decoding in VTC and IPS.

After functional alignment of fMRI data from all participants into a common space, a classifier was trained and tested within each trial period. A) VTC decoding. Similar to perception-trained decoding, the information about the cued item was present before and after distraction, with more post-distraction information on switch trials than stay trials. B) Region-of-interest masks. Cortical VTC and IPS masks of a representative participant. C) IPS decoding. IPS allowed for successful decoding of cued items before but not after distraction. Error bars represent ±1 SEM across leave-one-participant-out cross-validation. Vertical light-gray distributions represent null permuted distributions created with shuffled labels. Horizontal dashed lines represent theoretical chance.

Recent work has implicated IPS in demanding working memory tasks (Bettencourt & Xu, 2016; Christophel et al., 2018; Lorenc, Sreenivasan, Nee, Vandenbroucke, & D’Esposito, 2018). Thus, although our task was not designed to detect such representations (see Discussion), we repeated the memory-trained decoding analysis in IPS for exploratory purposes (Figure 3C). Consistent with results from VTC, hyperaligning participants into a common space using localizer data from IPS allowed for successful decoding of the cued memory category pre-distraction (M=54±4%, p=0.0010). This demonstrates that IPS was sensitive, as was VTC, to the manipulation of working memory prioritization. Yet after the onset of distraction, there was no successful decoding in IPS during distraction (M=50±3%, p=0.4316) and no evidence of recovery post-distraction (stay: M=49±5%, p=0.6763, switch: M=49±7%, p=0.6404).

4. Discussion

We showed that the prioritization of a working memory item via retrospective cueing impairs the ability to recover the neural representation of that item after a distracting event. This extends the growing body of neuroscientific evidence in the working memory literature that a retro-cued memory item receives a prioritized neural representation (LaRocque et al., 2014; Myers et al., 2017; Olivers et al., 2011), and it bridges to more recent behavioral work suggesting that prioritized memory items are more susceptible to distraction (Allen & Ueno, 2018; Hitch et al., 2018; Hu et al., 2016, 2014). We trained fMRI pattern classifiers on independent data from a category localizer task to track memory representations of faces and scenes in ventral temporal cortex during a working memory task with distraction. Before distraction, classifier evidence was higher for the cued (i.e., high-priority) item, replicating past work and confirming that in our experimental design, items entered distraction in (and subsequently recovered from) different priority levels. The distraction consisted of a change-detection task for an array of letters, which disrupted the neural decoding of the memory representations, but they recovered afterwards, and we found that recovery of low-priority items was more successful (greater neural separation of relevant vs. irrelevant information) than recovery of high-priority items. Additionally, across participants, the degree of prioritization of cued items prior to distraction was predictive of the impaired recovery of these items relative to low-priority items. This suggests that the more an item was prioritized before distraction, the more susceptible it was to the negative consequences of distraction. Together, our results suggest that the performance benefit of prioritizing an item in working memory (Souza & Oberauer, 2016) comes at the cost of increasing its neural vulnerability to disruption.

The sensory recruitment hypothesis (D’Esposito, 2007; D’Esposito & Postle, 2015; Pasternak & Greenlee, 2005; Postle, 2006) has faced criticism (Xu, 2017) (although see (Gayet et al., 2017; Scimeca, Kiyonaga, & D’Esposito, 2018)), and mnemonic representational formats may differ substantially from perception (Derrfuss et al., 2017; Yu & Shim, 2018). Therefore, we also evaluated classifiers that were trained and tested within individual epochs of the working memory task, offering an avenue to detect memory representations that either mimic perception or take their own representational pattern. These memory-trained classifier results replicated those of the perception-trained classifiers, in that classification in ventral temporal cortex was more successful during the post-distraction recovery of low-priority items relative to high-priority items. This provides further support for an increased vulnerability to distraction of high-priority working memory representations in perceptual regions.

Behavioral results regarding retro-cues and interference are mixed. While the current study utilized retro-cues (Griffin & Nobre, 2003) to assign prioritization to memory items, working memory items can be prioritized in a variety of ways (Niklaus, Singmann, & Oberauer, 2019), for example via recency after serial item presentation (McElree & Dosher, 1989) or reward values (Klyszejko, Rahmati, & Curtis, 2014). Accordingly, (Hu et al., 2016, 2014) presented task-irrelevant perceptual distractors during a working memory delay after serial item encoding, and found that the most recent item was preferentially impacted by the distractor. (Hitch et al., 2018; Hu et al., 2016) extended this finding by showing that the earliest presented memory items were not impacted by distraction, unless cued for prioritization. This was the case even when multiple working memories were prioritized (Hitch et al., 2018). Using similar distractors, (Allen & Ueno, 2018) assigned varying reward values to each of four memory items, and found that the items of higher reward value were most negatively impacted by distraction. Taken together, these experiments confirm that working memories can be prioritized using a variety of methods, and in each case prioritization results in increased performance (in the absence of distraction) at the cost of increased vulnerability to distraction.

In contrast, behavioral work using a single deterministic retro-cue to direct attention to memory items has provided mixed results. When attention is directed towards a single memory item as compared to equal prioritization of the entire set, attention seems to have a protective effect (Souza & Oberauer, 2016). When a task-irrelevant distractor is presented during a delay, the negative consequences on behavior are often dampened by a retro-cue (Barth & Schneider, 2018; Krefeld-Schwalb, 2018; Makovski & Jiang, 2007; Makovski et al., 2008; Pertzov, Bays, Joseph, & Husain, 2013; Schneider, Barth, Getzmann, & Wascher, 2017; Schneider, Barth, & Wascher, 2017; Souza et al., 2016; van Moorselaar et al., 2015), but not always (Souza, Rerko, & Oberauer, 2014). More akin to the current study, when the distractor is cognitively demanding, retro-cues still aid behavior but no longer interact with distraction effects (Hollingworth & Maxcey-Richard, 2013; Rerko, Souza, & Oberauer, 2014), except in one case where retro-cues preferentially diminished swap errors (Makovski & Pertzov, 2015). The discrepancy between these opposing accounts of prioritization (protective vs vulnerable) has been attributed to different cueing procedures (Allen & Ueno, 2018; Myers, Chekroud, Stokes, & Nobre, 2018), as the representational consequences to an uncued memory item may depend on the method/reliability of the retro-cue (Atkinson et al., 2018; Dube, Lumsden, & Al-Aidroos, 2018; Gunseli, Moorselaar, Meeter, & Olivers, 2015; Lewis-Peacock, Kessler, & Oberauer, 2018; Williams, Hong, Kang, Carlisle, & Woodman, 2013). It seems that studies testing multiple states of prioritization often support the notion that prioritized items are preferentially susceptible, while this is less the case for studies that compare the prioritization of a single item against broad attention to the whole item set.

We used an engaging secondary task based on (B. Wang et al., 2018), who argued that performance of such a task removes an item from the focus of attention in working memory. They and others (Bae & Luck, 2018) have found that when the distracting task was presented during the delay and the working memory probe immediately followed, performance was impaired. In both of these experiments, the distracting task in a sense placed the (single) memory item into a low-priority state during distraction. It is possible that our task put both items in a low-priority state during distraction. Indeed, our lack of memory signal during distraction suggests this was the case. We emphasize that our results relate to the priority of working memory representations prior to distraction, and their subsequent recovery after distraction. Further research focusing on representational formats during distraction – exploring the possibility of persistent, distraction-resistant representations – could offer additional insight into the representational protection of prioritized working memories.

The notion that prioritization leads to distraction susceptibility is consistent with a common finding that working memory prioritization leads to more memory-driven attentional capture (Olivers et al., 2011). Attentional capture by memory matching items is an exemplary case of how working memory representations impact perception, and the current finding reflects the inverse of this relationship (i.e., that perception impacts working memory representations). Previous work has shown that when four items are held in working memory, only the prioritized representation impacts perception (van Moorselaar, Theeuwes, & Olivers, 2014). In a dual retro-cue paradigm imparting high- and low-priority to multiple memory items, the high-priority memory representation impacts perception to a larger degree (Mallett & Lewis-Peacock, 2018). (Xu, 2017, 2018) has argued that this interaction between memory items and perception is a distinct disadvantage of the sensory recruitment account of working memory (D’Esposito & Postle, 2015). In showing that prioritization in sensory regions promotes susceptibility to distraction, our data seem to lend support to this argument. However, perceptual biases from mental representations offer a functional benefit in many situations (Gayet et al., 2017; Kiyonaga, Scimeca, Bliss, & Whitney, 2017), and we found no negative behavioral consequences in our experiment. It is possible that our behavioral measures were not sensitive enough to detect any negative consequences of prioritization. Still, if present, such negative impacts might be a necessary cost of an otherwise efficient system (Kiyonaga et al., 2017). Furthermore, memory representations in alternate (e.g., parietal) regions might be a viable alternative in particularly demanding scenarios (Bettencourt & Xu, 2016; Christophel et al., 2018; Lorenc et al., 2018), but we found no evidence of this possibility in the current study (see below).

There have been many recent proposals as to how working memories of different priority levels are represented in the brain. Double retro-cue paradigms that track working memory representations using multivoxel pattern analysis often show classifier evidence of the high-priority representation, but not the low-priority representation (LaRocque et al., 2013, 2017; Lewis-Peacock et al., 2012; Rose et al., 2016; Sprague et al., 2016; Wolff et al., 2017) (although see (Christophel et al., 2018; Schneegans & Bays, 2017; van Loon et al., 2018)). These findings have developed alongside propositions of “activity-silent” working memory representations (Stokes, 2015) that are maintained via synaptic weight patterns (Barak & Tsodyks, 2014; Mongillo, Barak, & Tsodyks, 2008) rather than the canonical view of persistent firing (Fuster & Alexander, 1971). The persistent activity of high-priority representations is consistent with the increased interaction with perception (see above), and thus also accounts for our data suggesting that perception interacts with active working memory representations more than inactive ones. Others have argued that a lack of evidence for low-priority representations in sensory regions is not a result of activity-silent storage, but rather reduced or altered activity, which is not detectable with commonly used methods or designs (Christophel et al., 2018; Schneegans & Bays, 2017). Our data suggest only that more activity is associated with more distraction susceptibility at the neural level, and thus are in accord with either account. Another possibility is that lack of detection of low-priority memory items results from brief intermittent “bursts” of activity (Lundqvist et al., 2016), which might offer a more distraction-resistant mechanism of maintenance (Miller, Lundqvist, & Bastos, 2018). A dynamic coding framework (Stokes, 2015) where neurons switch their coding profiles before, during, and after distraction (Jacob & Nieder, 2014; Parthasarathy et al., 2017; Stokes et al., 2013) might also result in null distraction decoding with fMRI. Additional support for a vulnerability of prioritized items comes from two experiments where delay-period transcranial magnetic stimulation disrupts high- but not low-priority memory representations (Zokaei, Manohar, Husain, & Feredoes, 2014; Zokaei, Ning, Manohar, Feredoes, & Husain, 2014), potentially as a result of active maintenance of the high-priority representation.

It is possible that low-priority memory representations are maintained via persistent firing in parietal/frontal regions (Christophel et al., 2018). The use of neural resources less essential to perception might aid in distraction-resistance (Xu, 2017). Consistent with this, a recent finding suggests that pattern representations in early visual but not parietal regions are impacted by a perceptual distractor (Lorenc et al., 2018). Previous research using a single memory item (i.e., no different prioritizations) found classifier evidence for the memory item in parietal but not early visual regions during distraction (Bettencourt & Xu, 2016). In the present study, we found evidence for the prioritized memory item in both ventral temporal cortex and intraparietal sulcus prior to distraction, but only in ventral temporal cortex after distraction. These results would suggest that intraparietal sulcus is not a crucial region for working memory storage, yet for the following reasons our design may not have been optimal for decoding in parietal regions. Our main analysis relies on perception-trained decoding, and parietal memory representations are less likely to mimic perception than those in visual regions (Yu & Shim, 2018). Our memory-trained decoding analysis would be more likely to detect parietal representations, yet the hyperalignment procedure we used relied heavily on perceptual activity from the 1-back localizer task to align participants into a common space (Haxby et al., 2011), and therefore might be less effective in parietal regions (note, however, that pre-distraction decoding using this procedure was successful, Figure 3C). Further, most previous research showing successful decoding of working memory representations in intraparietal sulcus used low-level visual stimuli (e.g., orientations or spatial locations), and therefore we had no strong a priori predictions about the success of categorical face/scene decoding in this region.

A more recent proposition for low-priority representation is that of persistent firing in sensory regions, albeit with broad representational patterns that are systematically transformed until re-prioritized and reverted back to a form mimicking perception (van Loon et al., 2018). (van Loon et al., 2018) propose this format might have protective properties, which is consistent with our results. An experiment that did not investigate effects of prioritization but examined neural representations during distraction found a similar shift and re-shift of representational patterns during and after distraction (Derrfuss et al., 2017). Our lack of successful mnemonic decoding during distraction suggests that there was no active maintenance during distraction, which is consistent with previous work of a single memory item ((Clapp, Rubens, & Gazzaley, 2010; Lewis-Peacock et al., 2012), although see (Bettencourt & Xu, 2016; Derrfuss et al., 2017)). Alternatively, it is possible that the memory items were equally prioritized above baseline during distraction (i.e., the high-priority item dropped to low-priority status) because both items were equally likely to be tested after distraction. However, our study was not designed to optimize the observation of representational status during distraction. The visual stimuli presented during distraction likely degraded our ability to decode the categories of the visual memories, which is precisely why a long post-distraction delay period (10 s) was chosen to allow for observation of the recovery of these representations. Future work using an alternate design and/or analysis strategy should directly address the question of whether visual working memories persist during, rather than recovery from, a distracting event.

Our study is also unable to make claims about the specific form of representational maintenance before distraction. That is, lower classifier evidence for the low-priority representation might result from activity-silent maintenance of this item, but there are other plausible alternatives. The current experiment was not designed to quantify the specific type of alternate representation between high- and low-priority memory representations.

It remains a possibility that prioritized working memory items are susceptible to interference due to their being maintained via persistent firing (Allen & Ueno, 2018; Xu, 2017). This notion is based on potential pattern disruption or overwriting that results from overlapping cortical resources of similar representations (Franconeri, Alvarez, & Cavanagh, 2013). Lower-priority representations, through their alternate storage location (Christophel et al., 2018), transformed pattern activation (van Loon et al., 2018), or activity-silent storage (Lewis-Peacock et al., 2012; Rose et al., 2016; Wolff et al., 2017), no longer share the representational space with the perceptual input or cognitive demands of the distracting task. The conflict might arise in two ways. One possibility is that, based on the overlap of mnemonic and perceptual representations (D’Esposito & Postle, 2015), the active memory representation and incoming distraction representation might compete for resources. In our study, the memory items were faces and scenes while the distracting stimulus was a letter array. Even if verbalized, the letter array and memory targets might share a portion of cortical resources (Lewis-Peacock et al., 2012). Another (non-exclusive) possibility is that the cognitive attentional and mnemonic demands of the distracting task are the interferer. Attention and memory resources are largely overlapping (Awh & Jonides, 2001; Chun, Golomb, & Turk-Browne, 2011; Kiyonaga & Egner, 2013), and the resources recruited for the distraction task might interfere with the “online” or “active” resources being recruited for maintenance of the high-priority memory item.

Another limitation of the current design is the use of categorical fMRI pattern classifiers. A possible re-interpretation of our data is that during recovery on switch trials, participants re-instated the category of the low-priority memory representation, but not the actual memory target. While plausible, the high behavioral performance on switch trials suggests otherwise. If participants were unable to recover the low-priority representation on switch trials, we would expect performance on the probe to suffer relative to stay trials, but this did not occur. Furthermore, recent work suggests correspondence between category- and item-level fMRI decoding results (e.g., (Lewis-Peacock et al., 2012) and (LaRocque et al., 2017) respectively). Yet still, categorical decoding serves as a proxy rather than a direct measure of item-level processing. Future work using a similar design with either item-level decoding or stimulus reconstruction methods (Sprague, Saproo, & Serences, 2015) could provide additional insights into neural distraction susceptibility. Such a design might also afford more behavioral performance sensitivity.

While we found no differences in behavior between stay and switch trials, our use of faces and scenes rather than low-level feature stimuli possibly prevented any detection of subtle differences that have been previously found between high- and low-priority representations (e.g., (LaRocque et al., 2015)). Working memory experiments often use continuous response measurements of low-level features for modeling purposes, potentially offering detection of more subtle response profiles (Ma, Husain, & Bays, 2014). Moreover, our design required a lengthy post-distraction delay period to account for the hemodynamic lag involved in fMRI measurements, which is inconsistent with all behavioral work under discussion. Based on this disparity and previous research suggesting that behavioral consequences of distraction are fleeting (B. Wang et al., 2018), we did not expect a strong behavioral effect between stay and switch trials. Rather, we anticipated the likelihood that other storage mechanisms would be involved in retaining and/or recovering working memory items and thus sparing behavioral performance (Lewis-Peacock et al., 2012; Lewis-Peacock, Drysdale, & Postle, 2015). Furthermore, the existing literature effects of behavioral high-priority susceptibility observe an interaction where the negative impact of distraction on low-priority memories is greater than that on high-priority memories, while raw performance is still equal or higher for high- than low-priority working memories after distraction (Allen & Ueno, 2018; Hitch et al., 2018). Thus, it is possible that the “susceptibility” of high-priority representations is more appropriately described as a loss of benefit. Without a comparable no-distraction condition for stay and switch trials in the current study we were unable to investigate this possibility.

In summary, perception-trained fMRI pattern classifiers in ventral temporal cortex showed successful recovery of both high- and low-priority visual working memory items after distraction, but only low-priority items recovered to pre-distraction levels. Moreover, across participants the degree of prioritization before distraction predicted the degree of disruption for high-priority items after distraction. Together, our results suggest that prioritizing a working memory increases the vulnerability of its neural representation to distraction.

Significance statement.

Working memory requirements vary throughout the day. Within a brief period, one is often required to remember multiple chunks of goal-relevant information despite a barrage of disruptions from other thoughts and from the environment. Sometimes memories survive such distraction, but often they don’t. Here, we demonstrate that how we hold our thoughts in working memory can influence how they are impacted by distraction. Our data suggest that when attention is focused on an item in working memory to enhance its availability for an upcoming memory test, this renders it more vulnerable to distraction. These results highlight an important tradeoff in the flexible allocation of attention in working memory between the availability and the durability of memories.