Distractor-relevance determines whether task-switching enhances or impairs distractor memory

Abstract

Richter & Yeung (2012) recently documented a novel task-switching effect, a switch-induced reduction in “memory selectivity,” characterized by relatively enhanced memory for distractor stimuli and impaired memory for target stimuli encountered on switch trials compared to repeat trials. One interpretation of this finding argues that task-switching involves opening a “gate” to working memory, which promotes updating of the task-set, but at the same time allows for increased distraction from task-irrelevant information. However, in that study, the distractor category on a switch trial also represented the task-relevant target category from the previous trial. Thus, distractors were only intermittently task-irrelevant, such that switch-enhanced distractor memory could alternatively be due to remnant attention to the previously relevant stimuli, or “task-set inertia.” Here we adjudicated between the open-gate and the task-set inertia accounts of switch-enhanced distractor memory by assessing incidental memory for distractors that were either intermittently or always task-irrelevant. While we replicated switch-enhanced distractor memory in the intermittently-irrelevant distractor condition, this effect was reversed in the always-irrelevant distractor condition. These results speak against the open-gate account, and instead indicate that switch-enhanced distractor memory arises from task-set inertia, and will thus not be observed for truly task-relevant stimuli presented during switching.

Introduction

Cognitive control refers to a collection of mechanisms that allow us to coordinate our thoughts and actions in line with internal goals. At the core of cognitive control lies the ability to maintain and adjust working memory (WM) representations of the current “rules of the game”, such as relevant stimuli and appropriate stimulus-response mappings, in the form of a task set (e.g., Egner, 2015). The cognitive mechanisms involved in establishing and switching task-sets have therefore attracted much interest (for reviews, see Kiesel et al., 2010; Monsell, 2003; Vandierendonck, Liefooghe, & Verbruggen, 2010). A novel addition to this literature has been provided by Richter & Yeung (2012). Participants in that study switched between two tasks on trial-unique stimuli consisting of foreground words overlaid on background images. On each trial, a task cue specified the relevant task, thus defining one of the stimuli (word or image) as the task-relevant target and the other as the task-irrelevant distractor. Following a filler task, participants performed a surprise recognition memory test for the stimuli used in the switching task. The authors found a switch-induced reduction in “memory selectivity,” characterized by superior memory for distractors, and inferior memory for targets, from switch trials compared to repeat trials.

One interpretation of this result relates to the proposal that switching task-sets requires opening a “gating mechanism” to WM: When the gate is closed, an ongoing task-set is stable, as it is shielded from interference by task-irrelevant distractors, but when the gate opens (e.g., in response to a cue indicating that a different task has to be performed), WM shielding is temporarily dropped to enable task-set updating (Braver & Cohen, 2000). The upside of this gate-opening is that the task-set becomes malleable, but a downside is that it may allow distractors to enter WM (Dreisbach & Wenke, 2011). Thus, the phenomenon of a switch-induced drop in memory-selectivity may reflect an open-gate state that allows for the intrusion of task-irrelevant stimuli into WM (Richter & Yeung, 2012).

However, as pointed out by Richter & Yeung (2012), their design leaves open an alternative to the “open-gate” account. Specifically, the distractors in their protocol were only intermittently task-irrelevant, such that on a switch trial, the current distractor category (word or image), by definition, represented the task-relevant target category from the previous trial. Therefore, switch-enhanced distractor memory could instead be a consequence of “task-set inertia” (Allport, Styles, & Hsieh, 1994). That is, due to lingering activation of the previous task-set, remnant attention would facilitate encoding of “target” stimuli under that set (Wylie, Javitt, & Foxe, 2004; Yeung, Nystrom, Aronson, & Cohen, 2006).

To adjudicate between the “open-gate” and “task-set inertia” accounts of switch-enhanced distractor memory, we designed a task-switching experiment to assess incidental memory for distractors that were either intermittently or always task-irrelevant. Participants in the intermittently-irrelevant distractor group switched between categorizing words and images, replicating the design of Richter & Yeung (2012). By contrast, participants in the always-irrelevant distractor group were exposed to the exact same set of stimuli, but they switched between two categorization tasks on the foreground words, such that the background images were never task-relevant. Here, the open-gate account still predicts distractors to gain access to WM on switch trials, resulting in enhanced distractor memory; whereas the task-set inertia account predicts this effect to disappear, because the images are never part of the participants’ task set to begin with. Thus, according to the open-gate account, we should observe a main effect of trial type (switch > repeat) on subsequent memory for distractor images; by contrast, the task-set inertia account predicts an interaction between trial type and group, with switch-enhanced distractor memory only evident if images are intermittently irrelevant.

Method

Participants

Fifty-six Amazon Mechanical Turk workers provided informed consent to the study (mean age = 32.66, SD = 8.77; 25 males, 31 females). Half of the participants were assigned to the intermittently-irrelevant distractor group and half to the always-irrelevant distractor group. Participants were compensated with $2.5. The sample size was determined by a power calculation based on the effect size in Richter & Yeung’s (2012). With a power of 0.9 and a Type I error of 0.05, the estimated sample size was 23 per group. We recruited 5 more per group in anticipation of data exclusions. Data from 9 participants (5 & 4)¹ were excluded, 3 (2 & 1)¹ due to data loss and 6 (3 & 3)¹ due to their performance being further than three standard deviations from the mean.

Stimuli

We used 240 images of living and non-living objects (120 each) from Moreno-Martínez & Montoro (2012). Half of them appeared only in the recognition memory task, as the “new” stimuli, and the other half appeared first in the task-switching procedure, and then again in the recognition memory task as “old” stimuli. In addition, 60 abstract and 60 concrete nouns (from Poldrack et al., 1999) were used in the switching task. Each foreground word was rendered in black, and overlaid on a background image, forming a trial-unique stimulus (Fig. 1a). All nouns were in lowercase in the intermittently-irrelevant distractor group whereas half were in lowercase and half in uppercase in the always-irrelevant distractor group.

Figure 1.

(a) Tasks and procedure. Participants first performed a cued task-switching procedure during which they switched between two categorization tasks (e.g., gray frame: living vs. non-living on images, black frame: abstract vs. concrete on words). They then completed a filler task that provided an encoding-retrieval delay of about 5 min. Finally, participants completed a surprise recognition memory task for background images in the task-switching procedure. Note that the original colors of the task cues were red and blue, but were shown in gray scale here. (b) Mean d’ on the recognition memory task as a function of distractor irrelevance and trial type. Error bars indicate within-subjects standard error of the mean (see Franz & Loftus, 2012).* indicates p < .001.

Design and Procedure

All participants first performed 120 trials of the task-switching procedure. This was followed by a filler task (a parity categorization task on single digits), which provided a 5 min encoding-retrieval delay, and will not be further reported. Crucially, it was followed by a surprise recognition memory task (240 trials) (Fig. 1a). Only subsequent memory for background images was assessed in the memory task.

During the switching task, participants in the intermittently-irrelevant distractor group switched between categorizing words (abstract vs. concrete) and categorizing objects in the images (living vs. non-living). This design conceptually replicated Richter & Yeung (2012), as the images were irrelevant in the word task but were relevant in the object task. By contrast, participants in the always-irrelevant distractor group switched between categorizing words by their semantics (abstract vs. concrete) and by their physical appearance (uppercase vs. lowercase), such that the background images for this group were always task-irrelevant. With this design, both groups encountered exactly the same number and set of images and words. In the analyses, as we focused on the subsequent memory for images in the abstract vs. concrete word task (denoted as “semantic word-categorization task” hereafter), the two groups were furthermore equated for the type of foreground task and the number of stimuli/trials considered. Combined with the fact that the background images derived from the identical stimulus set, and that image-to-condition (switch vs. repeat) assignment was randomized across participants in both groups, there were no systematic differences between the groups in terms of stimulus material and trial numbers in these analyses. Tasks were cued by a color frame that preceded and surrounded the stimuli (Fig. 1a). The incidence of the two tasks and trial types (switch/repeat) was equated. Trial timing was identical across groups, with each trial starting with the task cue for 800 ms, followed by the addition of a picture-word stimulus inside the frame for 2 s, during which participants could make a response, and after which they received on-screen feedback (correct or incorrect) for 1 s. Participants were instructed to respond as fast as possible without sacrificing accuracy. Before the main task, participants familiarized themselves with the assigned stimulus-response mapping in one or more practice blocks (to reach >85% accuracy). Two keys, ‘G’ & ‘J’ were used to indicate the category of the stimulus. Stimulus-response mappings were counterbalanced across participants.

In the surprise recognition memory task, one image was presented until response on each trial, along with four memory ratings (definitely new, probably new, probably old and definitely old), which were mapped onto the ‘v,’ ‘b,’ ‘n,’ and ‘m’ keys, respectively, or in the reversed order.

Analysis

To compare the two groups, we focused on performance of the common semantic word-categorization task. We assessed basic task-switching performance using a 2 (distractor relevance: intermittently vs. always irrelevant) × 2 (trial type: switch vs. repeat) analysis of variance (ANOVA) on accuracy and response time (RT), with distractor relevance as a between-subjects factor and trial type as a within-subject factor. To assess subsequent memory without bias, we calculated recognition sensitivity d’ (z(hit rate) − z(false alarm rate)) by collapsing probably and definitely responses and differentiating only old/new responses². We performed the same 2×2 ANOVA on d’. Significant effects were followed up by post-hoc t tests and we report ${\mathrm{\eta }}_p^2$ for ANOVAs and Cohen’s d for t tests.

Results

Task switching performance

As expected, both groups of participants took longer to respond on switch trials than on repeat trials, F(1,45) = 39.51, ${\mathrm{\eta }}_p^2=.47$, p <.001. The magnitude of switch costs between groups was neither similar nor significantly different, as the group by trial type interaction was marginal, F(1, 45) = 3.98, ${\mathrm{\eta }}_p^2=.08$, p =.052. Accuracy was lower on switch than on repeat trials, F(1,45) = 8.00, ${\mathrm{\eta }}_p^2=.15$, p <.01, and this effect did not differ between groups, F(1, 45) = .43, p >.05. The overall mean RT did not differ between groups, F(1,45) = 1.99, p >.05. However, the overall mean accuracy was significantly different, F(1,45) = 4.16, ${\mathrm{\eta }}_p^2=.09$, p = .047, as the mean accuracy was higher in the always-irrelevant distractor group (M = 87.22, SEM = 1.23) than in the intermittently-irrelevant distractor group (M = 83.04, SEM = 1.66), t(45)= 2.02, p = .049, Cohen’s d = .61. These results suggest that performing the semantic word-categorization task was somewhat easier in the context of switching between the two foreground tasks in the always-irrelevant distractor group than in the context of switching attention back and forth between foreground and background stimuli in the intermittently-irrelevant distractor group. See Table 1 for complete descriptive statistics.

Table 1.

Mean and Standard Error for task-switching performance

	Abstract vs. Concrete		Living vs. Non-living
Intermittently-irrelevant distractor group	Repeat	Switch	Repeat	Switch
Accuracy (%)	85.51 (1.77)	80.58 (2.05)	94.49 (1.27)	94.20 (1.07)
Response time (ms)	1113.28 (32.80)	1169.53 (30.55)	897.80 (29.33)	1013.37 (35.80)
Always-irrelevant distractor group			Uppercase vs. Lowercase
Accuracy (%)	88.75 (1.40)	85.69 (1.65)	96.67 (0.81)	96.81 (0.77)
Response time (ms)	1158.32 (39.10)	1266.81 (40.20)	939.43 (29.35)	1073.81 (34.13)

Recognition Memory

Mean recognition memory sensitivity (d’) was relatively poor (M = .48, SEM = .04), which is unsurprising, given that we measured incidental memory for task-irrelevant stimuli. However, recognition memory was well above chance, as the hit rates were greater than the false alarm rates in both groups (intermittently-irrelevant group: t(22) =6.96, p <.001, Cohen’s d = 1.45; always-irrelevant group: t(23) = 6.31, p <.001, Cohen’s d = 1.32). Importantly, while we detected no overall difference in d’ between the groups, F(1,45) =1.18, p > .1, the distractor relevance (intermittently vs. always) × trial type (switch vs. repeat) interaction was highly significant, F(1,45) = 14.71, ${\mathrm{\eta }}_p^2=.25$, p <.001. This interaction was due to better memory for distractors on switch vs. repeat trials in the intermittently-irrelevant distractor group, t(22) = 2.64, p <.05, Cohen’s d = .56, accompanied by the reverse pattern in the always-irrelevant distractor group, t(23) = 2.79, p <.05, Cohen’s d = .58 (Fig. 1b). The same pattern of results was found when using memory ratings instead of d’ (cf. Richter & Yeung, 2012), significant interaction: F(1,45) = 8.59, ${\mathrm{\eta }}_p^2=.16$, p <.005. Complete descriptive statistics for d’ and memory ratings are shown in Table 2a. These results clearly speak against the open-gate account, which anticipated superior memory for distractors on switch trials in both groups. By contrast, the task-set inertia account predicted a trial type × distractor relevance interaction.

Table 2.

(a) Mean and Standard Error for subsequent memory performance
	Abstract vs. Concrete		Living vs. Non-living
Intermittently-irrelevant distractor group	Repeat	Switch	Repeat	Switch
Sensitivity (d')	.45 (.07)	.63 (.09)	1.52 (.13)	1.42 (.10)
Rating	2.19 (.09)	2.33 (.08)	3.14 (.08)	3.03 (.09)
Always-irrelevant distractor group			Uppercase vs. Lowercase
Sensitivity (d')	.54 (.08)	.32 (.08)	.64 (.10)	.62 (.10)
Rating	2.44 (.07)	2.31 (.08)	2.56 (.09)	2.56 (.07)

(b) Mean and Standard Error for the proportion of responses for each memory rating.
		Memory rating
	Stimuli	Definitely new	Probably new	Probably old	Definitely old
Intermittently-irrelevant distractor group	Repeat	32.17 (5.02)	31.88 (4.37)	20.14 (2.21)	15.65 (2.09)
	Switch	29.28 (4.45)	28.26 (3.68)	21.01 (2.56)	21.45 (2.60)
	New	42.07 (5.00)	36.41 (4.74)	15.43 (1.53)	5.98 (0.77)
Always-irrelevant distractor group	Repeat	24.72 (4.50)	30.28 (4.04)	23.33 (3.00)	21.67 (2.65)
	Switch	27.50 (4.99)	34.58 (4.48)	18.89 (2.50)	18.75 (2.39)
	New	33.72 (5.86)	38.78 (4.67)	18.26 (2.58)	9.13 (1.86)

Note that instead of the memory advantage for distractors on switch trials simply disappearing in the always-irrelevant condition, we instead observed a cost in distractor memory on switch trials. One plausible explanation for this finding is that the control processes involved in switching (e.g., overcoming task-set inertia) are resource-demanding, such that completely task-irrelevant distractors are less likely to be encoded on switch trials than on less resource-demanding repeat trials. If this explanation were correct, we would expect memory for irrelevant distractors to be generally reduced as a function of load or difficulty in the relevant task. Our task design allowed us to test this, because the two foreground tasks that the always-irrelevant distractor group performed differed in difficulty: mean accuracy was higher, and mean RT was shorter for the physical appearance task (accuracy: M = 96.74, SEM = .67; RT: M = 1006.62, SEM = 150.30) than for the semantic task (accuracy: M = 87.22, SEM = 1.23; RT: M = 1212.56, SEM = 190.74), accuracy: t(23) = 7.02, p <.001, Cohen’s d = 1.46; RT: t(23) = 8.03, p <.001, Cohen’s d = 2.00 (Table 1). Accordingly, we performed a paired t test on distracter memory (d’) between the two categorization tasks. Memory for distractors was indeed modulated by the difficulty of the relevant task, t(23) = 3.04, p <.01, Cohen’s d= .63, as it was better for distractors presented in the easier physical appearance task (M = .63, SEM = .09) than in the harder semantic word-categorization task (M = .43, SEM = .07) (Table 2a). Together, these findings show that the encoding (or attentional selection) of truly irrelevant distractors is reduced during switch compared to repeat trials, and under a more difficult compared to an easier foreground task.

Discussion

The present experiment adjudicated between two plausible interpretations for the switch-induced reduction in memory selectivity reported by Richter & Yeung (2012): (a) that switch-trial distractors were better remembered because switching opens a (non-selective) gate to WM, or (b) that distractors were instead better remembered on switch trials because they had been task-relevant in the preceding trial. Using a between-subject design, we contrasted incidental memory between distractors that were either intermittently or always task-irrelevant. Even though the two groups performed exactly the same foreground task while encountering the identical set of distractors, strikingly, we found switch-enhanced distractor memory in the intermittently-irrelevant distractor condition but switch-impaired distractor memory in the always-irrelevant distractor condition. This finding is consistent with the hypothesis that the switch-enhanced distractor memory observed in Richter & Yeung (2012) arose from task-set inertia and lingering attention on the distractor category, rather than from an open-gate state that would permit even completely irrelevant distractors to intrude WM.

In addition, we found that the switching operation does in fact reduce memory for distractors, as long as these distractors are truly task-irrelevant. This finding is likely due to the fact that switching consumes attentional resources that are no longer available for encoding concurrent information (Reynolds, Donaldson, Wagner, & Braver, 2004). We corroborated this interpretation by showing that overall memory for task-irrelevant distractors was better when the relevant task was easier, which accords well with findings in the attentional load literature (see Lavie, 2005 for a reivew; Rissman, Gazzaley, & D’Esposito, 2009). Finally, it should be noted that our result do not refute the existence of a WM gating mechanism per se; rather, our data suggest that even an open-gate state may retain a certain degree of selectivity, and that this selectivity is at least in part dependent on the currently available processing resources.