Neural and Behavioral Evidence for an Online Resetting Process in Visual Working Memory

Abstract

Visual working memory (VWM) guides behavior by holding a set of active representations and modifying them according to changes in the environment. This updating process relies on a unique mapping between each VWM representation and an actual object in the environment. Here, we destroyed this mapping by either presenting a coherent object but then breaking it into independent parts or presenting an object but then abruptly replacing it with a different object. This allowed us to introduce the neural marker and behavioral consequence of an online resetting process in humans' VWM. Across seven experiments, we demonstrate that this resetting process involves abandoning the old VWM contents because they no longer correspond to the objects in the environment. Then, VWM encodes the novel information and reestablishes the correspondence between the new representations and the objects. The resetting process was marked by a unique neural signature: a sharp drop in the amplitude of the electrophysiological index of VWM contents (the contralateral delay activity), presumably indicating the loss of the existent object-to-representation mappings. This marker was missing when an updating process occurred. Moreover, when tracking moving items, VWM failed to detect salient changes in the object's shape when these changes occurred during the resetting process. This happened despite the object being fully visible, presumably because the mapping between the object and a VWM representation was lost. Importantly, we show that resetting, its neural marker, and the behavioral cost it entails, are specific to situations that involve a destruction of the objects-to-representations correspondence.

SIGNIFICANCE STATEMENT Visual working memory (VWM) maintains task-relevant information in an online state. Previous studies showed that VWM representations are accessed and modified after changes in the environment. Here, we show that this updating process critically depends on an ongoing mapping between the representations and the objects in the environment. When this mapping breaks, VWM cannot access the old representations and instead resets. The novel resetting process that we introduce removes the existing representations instead of modifying them and this process is accompanied by a unique neural marker. During the resetting process, VWM was blind to salient changes in the object's shape. The resetting process highlights the flexibility of our cognitive system in handling the dynamic environment by abruptly abandoning irrelevant schemas.

Introduction

Objects around us change constantly as they move and interact with each other, yet our visual sensation is of a stable and continuous world. One mechanism that helps us make sense of this “object chaos” is visual working memory (VWM), which stores a limited amount of information in an online state (Goldman-Rakic, 1995; Cowan, 2001; Luck and Vogel, 2013). VWM updates its representations according to the dynamic status of the objects so that when the location, form, or even the interpretation of an object changes, VWM modifies the corresponding representation (Blaser et al., 2000; Drew et al., 2011; Schlegel et al., 2013; Luria and Vogel, 2014; Balaban and Luria, 2016a). We suggest that updating relies on a distinctive correspondence between a specific object in the environment and a unique VWM representation (Kahneman et al., 1992; Levillain and Flombaum, 2012). This objects-to-representations mapping enables the updating process to access and modify the correct representation.

Occasionally, with dramatic or pervasive changes, instead of updating an existing representation, it is preferable to discard it and start anew. Several theories proposed the possibility of removing no-longer-relevant items from working memory (Hasher et al., 1999; Oberauer et al., 2012). However, to date, no direct neurophysiological evidence for this process was presented, a gap that we aim to fill.

Importantly, we also propose the mechanism and the triggering conditions of this removal. We argue that, whenever the correspondence between objects and their representations breaks, a novel online process resets the contents of VWM (instead of updating the existing representations) and then creates a new correspondence between the environment and VWM representations.

To observe online resetting in VWM, in Experiment 1, participants tracked complex shapes that moved as integrated objects before separating into their comprising parts [e.g., a “roofed-house” shape that separated into a triangle (“roof”) and a rectangle (“house”); Figure 1a]. Participants memorized the shape-parts for a change-detection task (Luck and Vogel, 1997; Zhang and Luck, 2008; Gao et al., 2009). The initial integrated movement phase conveyed a strong common-fate Gestalt cue intended to create one integrated object representation in VWM (Luria and Vogel, 2014; Balaban and Luria, 2015; Balaban and Luria, 2016b), forming a link between the moving shape and a single VWM representation. Once the shape separated into its comprising parts, the correspondence between the object and the unique VWM representation broke: now, there were two moving items that had to be followed and represented independently to perform the change-detection task (Xu, 2002). Therefore, instead of an updating process, we expected a resetting process in which the existing single-object representations are abandoned (instead of being modified), followed by an individuation of the shape-parts and their reencoding as two separate novel VWM representations.

Figure 1.

Stimuli and results of Experiment 1. a, Examples of trials in the different conditions (gray arrows indicate movement directions and were not presented). Top to bottom, Separating shapes, Two shape-halves, Four shape-halves, and Integrated shapes. b, Grand-averaged CDA waves (averaged across the P7/8, PO3/4, and PO7/8 electrodes) time locked to memory array presentation. Negative voltage is plotted upward. The vertical dashed line depicts the time of separation. Analyzed time windows are depicted by colored rectangles: Drop (pink), in which the amplitude of the Separating shapes condition dropped, and Pre- and Post-Drop (gray).

To determine the online status of VWM, we recorded EEG and isolated an event-related potential (ERP) component that tracks the contents of VWM, the contralateral delay activity (CDA) (Vogel and Machizawa, 2004; McCollough et al., 2007). The CDA is a sustained posterior activity whose amplitude rises, reflecting the number of items in VWM. The CDA amplitude reaches a stable asymptote that is strongly correlated with individual VWM capacity (Vogel and Machizawa, 2004; Drew and Vogel, 2008; Tsubomi et al., 2013; Kang and Woodman, 2014; for review, see Luria et al., 2016). Notably, the CDA does not reflect the number of attended locations (Ikkai et al., 2010; Luria and Vogel, 2014; Balaban and Luria, 2016a; Balaban and Luria, 2016b), eye movements (Kang and Woodman, 2014), or other low-level perceptual factors (Ikkai et al., 2010).

If the CDA is a valid marker of VWM, it should distinguish updating from resetting. Specifically, unlike the steady rise in amplitude observed in updating when items are added (Vogel et al., 2005), resetting should cause the CDA to decrease as the original representations are abandoned, and only then rise to reflect the encoding of new representations (because, after the separation, there are four instead of two shape-parts).

Materials and Methods

Participants

Subjects were Tel Aviv University students (age range: 18–30 years) with normal or corrected-to-normal visual acuity and normal color vision, who gave informed consent following the procedures of a protocol approved by the local ethics committee. Subjects received partial course credit or 40 NIS (∼$10 US) per hour for participation. Each EEG experiment included 12 fresh subjects in the final analysis (9, 9, 8, 4, and 10 females in Experiments 1–5, respectively). Subjects with a >25% rejection rate due to blinks or eye movements were replaced (none, 2, 1, none, and 2, in Experiments 1–5, respectively). Experiments 6 and 7 (behavior only) included 20 fresh subjects each (13 females in Experiment 6, and 11 in Experiment 7).

Stimuli and procedure

EEG experiments

A bilateral change-detection task was used. After a 750 ms fixation display (a white cross in the center of a gray screen, 0.4° × 0.4° of visual angle from a viewing distance of ∼60 cm), two white (1.9° × 0.4°) arrows appeared for 200 ms, indicating the relevant side in the current trial (left or right with an equal probability) to which subjects had to attend while holding fixation. Only the fixation cross remained visible for 300, 400, or 500 ms (randomly determined with an equal probability) and was followed by the memory array with randomly chosen (without replacement) items. Stimuli appeared in randomly chosen locations inside an imaginary 4.5° × 3.5° rectangle (one in each side of the screen), with the constraint that the distance between items would not be smaller than 2°. The two sides were always equated in terms of the number and type of stimuli. After a 900 ms retention interval, the items reappeared and subjects indicated (by pressing the “Z” or “/” buttons on a keyboard) whether one item in the attended side changed relative to the memory array (50% probability). On change trials, a single item (i.e., a single shape-half in Experiments 1–4; a single colored square in Experiment 5) changed to a new item that was not included in the memory array.

Experiment 1.

Stimuli were 4 top-half and 4 bottom-half black shapes (0.8° × 1.6°), which could form 16 different shapes. Items in the memory array moved for 1000 ms and then remained stationary for 300 ms. Four conditions were randomly mixed: two integrated shapes separating to four shape-halves after 400 ms, four shape-halves moving as two integrated shapes without separating, and two or four shape-halves moving separately. Twelve practice trials were followed by 15 experimental blocks of 60 trials each, for a total of 900 trials.

Experiment 2.

This experiment was similar to Experiment 1 except that stimuli were 6 left- and 6 right-sided polygon-halves (1.6° × 0.8°) and the conditions were a polygon separating into two polygon-halves, an integrated polygon that did not separate, and one and two separately moving polygon-halves. Twelve practice trials were followed by 15 experimental blocks of 60 trials each, for a total of 900 trials.

Experiment 3.

Stimuli were similar to Experiment 2. A single stationary polygon-half appeared for 500 ms, followed by a 50 ms blank interval. Then, this polygon-half could reappear alone (the single polygon-half condition) or with a second polygon-half that either appeared in a new location (forming two separate items; the “Add” condition) or completing the first half to an integrated polygon (forming one integrated item; the “Switch” condition). Twelve practice trials were followed by 12 experimental blocks of 60 trials each, for a total of 720 trials.

Experiment 4.

This experiment was similar to Experiment 2 except that items moved for 1400 ms and conditions included one and two separately moving polygon-halves, and two polygon-halves moving toward each other for 400 ms, moving as an integrated polygon for 400 ms, and separating again to two halves. After 1400 ms of movement, the items remained stationary for 300 ms. Twelve practice trials were followed by 12 experimental blocks of 60 trials each, for a total of 720 trials.

Experiment 5.

This experiment was similar to Experiment 1 except that stimuli were large (1.3° × 1.3°) or small (0.8° × 0.8°) colored squares and conditions included two color–color conjunctions (i.e., sets of a small colored square on top of a large colored square) that separated to four independently moving colors after 400 ms, two color–color conjunctions that moved without separating, and two or four independently moving colors. The items moved for 1300 ms (without a stationary display). Twelve practice trials were followed by 15 experimental blocks of 60 trials each, for a total of 900 trials.

Behavioral experiments

An “online change-detection” task was used, in which the change occurred while the items were visible on the screen, so there was no retention interval. Items moved and, on 50% of the trials, a single item (i.e., a single polygon half, or a single color, see below) changed during the movement. Participants had to detect whether this online change had occurred. The hit rate and 95% confidence intervals are reported (Hollands and Jarmasz, 2010).

Experiment 6.

Separate blocks (order was counterbalanced across subjects) included polygons or colors. Each trial included two items that moved together (i.e., a single integrated polygon or a color–color conjunction) for 600 ms and then either continued to move together or separated and moved separately for an additional 800 ms. Changes happened either 350, 600, or 850 ms from trial onset. For the conditions including separation, these times corresponded to 250 before the separation, during the separation, and 250 ms after the separation. Twelve practice trials were followed by 14 experimental blocks of 60 trials each, for a total of 840 trials.

Experiment 7.

This experiment was similar to Experiment 6 except that it included only polygon blocks and conditions were a single polygon separating after 800 ms, two halves meeting after 400 ms and moving as a single polygon for 400 ms before separating again, and a single polygon that moved without separating. Twelve practice trials were followed by 11 experimental blocks of 60 trials each, for a total of 660 trials.

EEG recording and analysis

In Experiments 1–5, EEG was recorded inside a shielded Faraday cage using a BioSemi ActiveTwo system from 32 scalp electrodes at a subset of locations from the extended 10–20 system (including mostly occipital and parietal sites in which the CDA is most pronounced: Fp1, Fp2, AF3, AF4, F3, F4, F7, F8, Fz, FCz, C3, C4, Cz, T7, T8, P1, P2, P3, P4, P5, P6, P7, P8, Pz, PO3, PO4, PO7, PO8, POz, O1, O2, and Oz) and from two electrodes placed on the mastoids. EOG was recorded from two electrodes placed 1 cm from the external canthi and from an electrode beneath the left eye. Data were digitized at 256 Hz.

Offline signal processing was performed using the EEGLAB Toolbox (Delorme and Makeig, 2004), the ERPLAB Toolbox (Lopez-Calderon and Luck, 2014), and custom MATLAB (The MathWorks) scripts. All electrodes were referenced to the average of the mastoids. The continuous data were segmented into epochs from −200 from memory array onset to the end of the retention interval (+2200 in Experiments 1, 2, and 5; +1950 in Experiment 3; +2600 in Experiment 4). Artifact detection was performed using a sliding window peak-to-peak analysis, with a threshold of 80 μV for the EOG electrodes and 100 μV for the analyzed electrodes (P7, P8, PO3, PO4, PO7, and PO8). This procedure resulted in a mean rejection rate of 9.5% in Experiment 1, 7.7% in Experiment 2, 5.6% in Experiment 3, 9.6% in Experiment 4, and 7.2% in Experiment 5. Only trials with a correct response were included in the analysis. For illustration purposes, the epoched data displayed in all figures were low-pass filtered using a noncausal Butterworth filter (12 dB/oct) with a half-amplitude cutoff point at 30 Hz. All statistical analyses were performed on the unfiltered data.

Epoched data were averaged separately for each condition and the CDA difference wave was calculated by subtracting the average activity at electrodes ipsilateral to the memorized side from the average activity at electrodes contralateral to the memorized side. As the dependent measure, we used mean amplitude at three time windows: “Pre-Drop,” 100–200 ms after the separation (or, in Experiment 3, after the presentation of the second memory array); “Drop,” 200–300 ms after the separation (or, in Experiment 3, after the presentation of the second memory array); and “Post-Drop,” during the entire range of the retention interval. Note that 200 ms is the usual time that it takes the CDA to develop after the items are first presented in the memory array (Vogel and Machizawa, 2004), so the influence of item separation or replacement is expected to occur around that time. We present only the results from the average of 3 electrode pairs (P7/8, PO3/4, and PO7/8), but we found the same patterns of activity in each pair separately.

Statistical analysis

EEG experiments

For each of the three time windows (Pre-Drop, Drop, and Post-Drop), a one-way ANOVA was conducted, with condition as a within-subject variable on the CDA mean amplitude as a dependent measure. An additional ANOVA with condition as a within-subject variable was conducted on accuracy. We focus on the results of planned comparisons (contrasts) between the different conditions.

Behavioral experiments

In Experiment 6, a three-way ANOVA was conducted, with block-type (polygon vs color), condition (separating item vs integrated item), and time of change (−250, 0, and 250, relative to separation) as within-subject variables on hit rate as a dependent measure. In Experiment 7, we conducted a two-way ANOVA with condition (separating polygon, joining-and-separating polygon, and integrated polygon) and time (−250, 0, and 250, relative to separation) as within-subject variables on hit rate as a dependent measure.

Results

Experiment 1

In Experiment 1, we aimed to observe online resetting in VWM by breaking the correspondence between objects and their representations. In the critical condition, two moving integrated shapes separated into four shape-halves that continued to move independently. We reasoned that, before the separation, each shape would be represented as a single object in VWM, with a unique mapping between each moving item and a single VWM representation. Critically, after each shape separated into two shape-halves, the unique mapping marking a single complete object was no longer relevant, causing a lack of correspondence between the contents of VWM and the items in the environment. We hypothesized that this lack of correspondence would prevent the possibility of updating the representations after the change and thus trigger a resetting process.

Two and four separately moving shape-halves served as baselines for VWM activity. In an additional control condition, two integrated shapes moved throughout the trial without separating so that the separation in the critical condition was unpredictable. Because resetting involves abandoning existing representations and then individuating and encoding new objects, we expected the CDA amplitude in the separating shapes condition to drop sharply after the separation (during resetting) and then to gradually recover until it reflected the new number of items. In contrast, if VWM can simply update after the separation, then we should observe only a gradual change in CDA amplitude until it reflects the new object status, without a drop (Vogel et al., 2005; Drew et al., 2011; Luria and Vogel, 2014; Balaban and Luria, 2015; Balaban and Luria, 2016b); more specifically, the amplitude should rise because more items are present after the separation.

The results were consistent with the resetting hypothesis: the separation was followed by a drop in CDA amplitude (Fig. 1b) after ∼200 ms, which is the usual time it takes for the CDA to develop (Vogel and Machizawa, 2004). We defined the Drop time window (in all experiments) as 200–300 ms after the shapes separation. We analyzed two additional time windows: Pre-Drop, 100–200 ms after the separation, and Post-Drop, during the entire range of the retention interval. Importantly, the Drop CDA amplitude was lower than that of two shape-halves (F_(1,11) = 20.46, p = 0.0009), suggesting that the original representations were abandoned. The Pre-Drop amplitude of the separating shapes was the same as that of the two shape-halves (F_(1,11) = 1.49, p = 0.25), indicating that, before their separation, the parts were represented as integrated objects. The Post-Drop amplitude was similar to four shape-halves (F < 1), suggesting that, after resetting, VWM recovered and represented the shape-halves as independent objects.

Accuracy in the separating shapes condition (0.75, SE: 0.02) was lower than accuracy for two shape-halves (0.92, SE: 0.02; F_(1,11) = 151.11, p = 0.000001) and not significantly different from four shape-halves (0.74, SE: 0.02; F < 1).

Experiment 2

To support our claim regarding VWM resetting, we replicated Experiment 1 with different stimuli and a different number of items. Experiment 2 used polygon-halves, which, when combined, created integrated random polygons (Fig. 2a). As in Experiment 1, we reasoned that the joint movement of the two polygon-halves would support an integrated interpretation, meaning that the moving polygon is represented as a single object in VWM, along with a single mapping between this polygon and its VWM representation. When the polygon separates into two halves, it should cause a correspondence mismatch because a single mapping pointing to an integrated representation cannot support the two independently moving halves. This, in turn, should prevent VWM from updating the representations to accommodate the novel situation because updating relies on the mapping to access the correct representations. Therefore, we reasoned that VWM would reset; that is, abandoning the existing representation and reencoding the two independent polygon-halves as two separate representations, with the appropriate objects-to-representations mappings. This resetting process should be marked by a drop in CDA amplitude at the same time window as in Experiment 1. Conversely, if VWM can update after the separation, then the CDA amplitude should rise steadily without a drop because more objects are present after the separation.

Figure 2.

Stimuli and results of Experiment 2. a, Examples of trials in the different conditions (gray arrows indicate movement directions and were not presented). Top to bottom, Separating polygon, One polygon-half, Two polygon-halves, and Integrated polygon. b, Grand-averaged CDA waves (averaged across the P7/8, PO3/4, and PO7/8 electrodes) time locked to memory array presentation. Negative voltage is plotted upward. The vertical dashed line depicts the time of separation. Analyzed time windows are depicted by colored rectangles: Drop (pink), in which the amplitude of the Separating polygon condition dropped, and Pre- and Post-Drop (gray).

When a single integrated polygon separated into halves, we again found a decrease in VWM contents (Fig. 2b), supporting the resetting hypothesis. The Drop CDA amplitude was lower than the amplitude of a single half (F_(1,11) = 20.73, p = 0.0008), indicating a loss of VWM contents. Conversely, the Pre-Drop amplitude was similar to just a single half (F < 1), indicating that, before separation, the two halves were represented as a single VWM unit (Balaban and Luria, 2015). After the separation was completed, VWM represented the polygon-halves as independent objects, producing a Post-Drop amplitude not significantly different from two polygon-halves (F_(1,11) = 1.58, p = 0.23).

Accuracy in the separation condition (0.68, SE: 0.02) was lower than accuracy for a single polygon-half (0.85, SE: 0.02; F_(1,11) = 122.48, p = 0.000001) and even lower than accuracy for two independently moving polygon-halves (0.73, SE: 0.03; F_(1,11) = 6.91, p = 0.02).

Experiment 3

In Experiments 1 and 2, we observed evidence supporting the existence of a novel VWM resetting process: when an object separated into its comprising parts, a drop in the CDA amplitude indicated that VWM did not update its representations (which would have been manifested by a steady rise in CDA amplitude to account for the additional objects), but instead abandoned the old representations and reencoded the new information. We argue that the reason for this resetting process is the loss of a correspondence between VWM representations and the objects in the environment. However, in Experiments 1 and 2, resetting was confounded with object separation, increasing the set size, and a need to track moving items. The goal of Experiment 3 was to demonstrate a resetting process without these factors, providing compelling evidence that resetting is a general process occurring when the objects-to-representation mapping breaks.

In the critical condition, an object was presented and encoded in VWM, but then was abruptly replaced by a different object that appeared at the same position, breaking the initial mapping between the old object and the corresponding VWM representation. We presented a stationary polygon-half and, after 500 ms, added a second half that completed the first half to create one integrated polygon (Fig. 3a). Importantly, we inserted a brief (50 ms) blank interval between the two presentations so that the first half seemed to be replaced by a different integrated polygon. We hypothesized that this “Switch” between the old and new representations should trigger a resetting process, reflected in a CDA-drop. In contrast, if VWM can update the representation after the switch, then no change in CDA amplitude should be observed because only a single object is present also in the second memory array (Balaban and Luria, 2015).

Figure 3.

Stimuli and results of Experiment 3. a, Examples of trials in the different conditions. Top to bottom, Switch, Add, and Half-polygon repeat. b, Grand-averaged CDA waves (averaged across the P7/8, PO3/4, and PO7/8 electrodes) time locked to memory array presentation. Negative voltage is plotted upward. The vertical dashed line depicts the time of the presentation of the second memory array. Analyzed time windows are depicted by colored rectangles: Drop (pink), in which the amplitude of the Switch condition, but not the Add condition, dropped, and Pre- and Post-Drop (gray).

Aside from a control condition including only a single half (necessary to ensure that subjects encode the first half), we compared the Switch condition to an “Add” condition in which the second half was presented at a new location, rather than attached to the first half. In this Add condition, the correspondence between the visual input and the relevant representation remained valid, so the second half should trigger a typical updating process (to account for the additional object) that maintains the still-relevant mapping (Vogel et al., 2005).

Consistent with the resetting hypothesis, in the Switch condition, we found a drop in CDA amplitude, suggesting a resetting of VWM (Fig. 3b). The Drop CDA amplitude was lower than a single half (F_(1,11) = 13.27, p = 0.004). The Post-Drop amplitude was similar to a single half, indicating that the integrated polygon was represented as a single object (and not as two independent halves; Balaban and Luria, 2015; F_(1,11) = 1.67, p = 0.22).

Conversely, in the Add condition, we observed a typical VWM-updating process, such that the CDA amplitude steadily rose from one to two items. In the Add condition, amplitude in the Drop time window was not lower than a single half (F_(1,11) = 1.10, p = 0.32), suggesting no loss of VWM contents in this condition, and the Post-Drop amplitude was higher than a single half (F_(1,11) = 16.17, p = 0.002), indicating that the information was added successfully to VWM. Therefore, we were able to observe both resetting and updating within a single experiment using similar stimuli.

Importantly, the different patterns of results for the Switch and Add conditions were found despite the fact that the same information (i.e., an identical polygon half) was added to the memory array in both conditions. We suggest that the difference between the two conditions lies in the ability to maintain the mapping between the first polygon-half (presented in the first memory array) and its original VWM representation. In the Add condition (which resulted in an updating process), the old polygon-half could be maintained as a distinct object throughout the trial because the new polygon-half appeared in a new location. Conversely, in the Switch condition (which resulted in a resetting process), the old polygon-half was no longer maintained in VWM because the new polygon-half completed it to a whole polygon, creating the appearance of a seemingly new object replacing the original one, thus making the original object-to-representation mapping irrelevant. Importantly, the different pattern of results in the Add and Switch conditions strongly suggests that the critical factor determining whether a resetting process or an updating process will take place is the validity of the correspondence. Once the original correspondence is deemed invalid, whether due to object separation, as in Experiments 1 and 2, or to object replacement, as in the current experiment, a resetting process is initiated, the old representations are dropped from VWM, and new representations and correspondences are established.

Interestingly, although the onset of the drop in the Switch condition of the present experiment occurred at approximately the same time as in the previous two experiments, it was wider relative to them. Note that, whereas the previous experiments involved constant movement monitoring, the current experiment included two presentations of stationary objects separated by a blank interval of 50 ms. Therefore, unlike in the separation condition of Experiments 1 and 2, in which the items were visible throughout the trial, in the switch condition of Experiment 3, the discontinuity presumably resulted in subjects experiencing a completely new object. Supporting this idea, when asked directly during the postexperimental debriefing, 10 of 12 subjects did not notice that the integrated polygon in the Switch condition included the original polygon-half presented in the first memory array. Namely, subjects missed that the second integrated polygon was never a new item because it was composed of the old polygon-half from the first display that was attached to a new polygon-half, creating an integrated polygon that was actually only “half” new. This happened despite observing the Switch condition take place ∼240 times during the experiment. This means that subjects encoded the second item as if it was novel, similar to encoding the item in the first memory array. Accordingly, the slow recovery from the drop in the Switch condition is comparable to the initial rise of the CDA. Obviously, more research is needed to clarify this point.

Accuracy in the Switch condition (0.76, SE: 0.01) was lower than in the single half condition (0.84, SE: 0.02; F_(1,11) = 29.73, p = 0.0002) and higher than in the Add condition (0.68, SE: 0.02, F_(1,11) = 19.97, p = 0.0009).

Experiment 4

Experiment 3 supported the claim that resetting occurs when the correspondence between an object in the environment and its unique VWM representation breaks and that the resetting process does not depend on object separation, the need to track moving items, or an increase in the number of encoded items. To complement this, our goal in Experiment 4 was to show that, when the objects-to-representations mapping is maintained, a separating polygon (which triggered resetting in Experiment 2) then results in updating. We used the stimuli from Experiment 2, but added an initial separate movement phase in which the two polygon-parts started moving independently toward each other, then met and moved as an integrated shape, only to separate again (Fig. 4a). Although the joint movement and the separation phases were identical to Experiment 2, the initial independent movement should allow for the formation of a mapping between each polygon-half (instead of the complete polygon) and a unique VWM representation. Furthermore, we removed the condition in which an integrated polygon moved as a single unit throughout the trial (without separating) to make the separation of the halves completely predictable.

Figure 4.

Stimuli and results of Experiment 4. a, Examples of trials in the different conditions of (gray arrows indicate movement directions and were not presented). Top to bottom, Joining-and-separating polygon, One polygon-half, and Two polygon-halves. b, Grand-averaged CDA waves (averaged across the P7/8, PO3/4, and PO7/8 electrodes) time locked to memory array presentation. Negative voltage is plotted upward. The vertical dashed line depicts the time of separation. Analyzed time windows are depicted by colored rectangles: Drop (pink), in which there was no significant reduction in amplitude in the Joining-and-separating polygon condition, and Pre- and Post-Drop (gray).

If the CDA-drop found in Experiments 1–3 indeed reflects a resetting process due to the loss of an object-to-representation mapping, then we should not observe this effect in the present experiment because the correspondence can be maintained after the separation due to the initial separate movement phase. However, if the CDA-drop was caused by the visual cue of separation and not by the cognitive process of resetting, then we should observe a resetting process in the present experiment as well because the joining-and-separating polygon condition of this experiment was identical to the separating polygon condition of Experiment 2 during the critical time window.

Consistent with the resetting hypothesis, the first 400 ms of separate movement was enough to make the CDA-drop in the present experiment to disappear (Fig. 4b). The CDA was similar to a single polygon-half during both the Pre-Drop and Drop time windows (both F < 1). After their separation, the halves were held in VWM as independent items, as evident from a Post-Drop (i.e., retention) amplitude that was not significantly different from that of the two separate halves (F_(1,11) = 1.71, p = 0.22), indicating that VWM could update its contents instead of resetting.

Accuracy for the joining-and-separating polygon (0.68, SE: 0.02) was lower than for a single polygon-half (0.86, SE: 0.03; F_(1,11) = 191.04, p = 0.000001) and also lower than for two polygon-halves (0.72, SE: 0.03; F_(1,11) = 10.12, p = 0.009).

The lack of a CDA-drop in the present experiment rules out a purely stimulus-driven account of our results and shows that identical situations (i.e., identical separation) can lead to updating when the objects are easily individuated (thus the mapping involves the polygon-halves), but to resetting when they are difficult to individuate, forcing VWM to break the mapping between the old representations and the novel objects to represent the new situation. This further demonstrates that online resetting only follows changes that break the correspondence between objects and their VWM representations.

Experiment 5

Experiment 5 used color–color conjunction items that moved as integrated units, but then separated into their comprising colors (Fig. 5a). The individual colors are easy to identify already during the integrated movement and before the separation, making it possible to form a correspondence using the individual colors already during the common motion phase, well before the colors separated. Therefore, after the separation, we hypothesized that VWM would be able to access the correct representations and update them, maintaining the original mapping. Therefore, we expected an updating process, indicated by a steady rise in CDA amplitude without a drop, to account for the additional items. Conversely, if the CDA-drop found in Experiments 1 and 2 is due to object separation or to adding more to-be-remembered items, then we should observe it in the present experiment as well. We compared the separating-colors condition with two or four separately moving colors and also included a control condition in which the color–color conjunctions continued to move together (to make the separation unpredictable).

Figure 5.

Stimuli and results of Experiment 5. a, Examples of trials in the different conditions (gray arrows indicate movement directions and were not presented). Top to bottom, Separating colors, Two colors, Four colors, and Integrated colors. b, Grand-averaged CDA waves (averaged across the P7/8, PO3/4, and PO7/8 electrodes) time locked to memory array presentation. Negative voltage is plotted upward. The vertical dashed line depicts the time of separation. Analyzed time windows are depicted by colored rectangles: Drop (pink), in which there was no significant reduction in amplitude in the Separating colors condition, and Pre- and Post-Drop (gray).

The results did not reveal any evidence for a drop in CDA because the amplitude gradually increased after the separation, reflecting an updating process rather than resetting (Fig. 5b). CDA amplitude in the separating colors condition was similar to two separate colors during both the Pre-Drop (F < 1) and the Drop (F_(1,11) = 2.35, p = 0.15) time windows, indicating no loss of VWM contents. After separation, the colors were represented as independent items, as indicated by the Post-Drop amplitude not being significantly different from four separate colors (F_(1,11) = 2.06, p = 0.18). Therefore, resetting is not a necessary outcome of separation given that the correspondence between the new visual input and the existing representations could be maintained (or easily reestablished).

Accuracy for the separating colors (0.86, SE: 0.02) was lower than for two colors (0.96, SE: 0.02; F_(1,11) = 30.10, p = 0.00009) and not significantly different from four colors (0.86, SE: 0.02; F < 1).

Eye movements

To make sure that the observed effects were not driven by eye movement artifacts, we examined the horizontal EOG (HEOG) in all experiments. We created difference waves of the left EOG electrode minus the right EOG electrode (so that any leftward eye movement creates a negative deflection and any rightward eye movement creates a positive deflection). In Figure 6, we plotted the HEOG waveforms by condition and cued side (left or right). As a reference point, we used 8 μV, which correspondence to ∼0.5° of drifting eye movement (Hillyard and Galambos, 1970).

Figure 6.

HEOG waveforms (left − right) for all experiments by condition and attended side. The Drop time window is depicted by a pink rectangle. a, The Separating shapes and Integrated shapes conditions of Experiment 1. b, The Separating polygon and Integrated polygon conditions of Experiment 2. c, All conditions of Experiment 3. d, All conditions of Experiment 4. e, The Separating colors and Integrated colors conditions of Experiment 5.

Inspection of these waveforms suggests some drifting eye movements, but it is unlikely that this was responsible for the observed effects for several reasons. First, overall eye movements were not large: the HEOG did not exceed 4 μV, which translates to ∼0.25°, and were usually much lower (around 1 μV, which corresponds to 0.06°) during the Drop time window (for a similar scale of eye drifting in a VWM task involving moving items, see Drew and Vogel, 2008). Second, in Experiments 1–3, eye movements were similar between conditions, whereas a CDA-drop was observed only in one condition. Third, experiments in which resetting was observed were not accompanied by increasing eye movements. For example, in Experiment 2 (in which resetting was observed), there were actually fewer eye movements than in Experiment 4 (in which updating was observed).

As an additional way to ensure that eye movements were not responsible for the observed pattern of results, we reanalyzed all of the experiments using a lower threshold of 50 μV for the external eye electrodes. This analysis led to identical results. We chose to present the original analysis (with a threshold of 80 μV) because it is based on more trials.

Experiment 6

The goal of Experiment 6 was to examine the behavioral sensitivity of VWM to detect salient changes during the resetting process. We argue that resetting is caused by the loss of a valid mapping between the object in the environment and its VWM representation. Therefore, we hypothesized that, during the resetting process, changes in the object status would not be translated to the corresponding representation.

To test this prediction, in Experiment 6 participants tracked one moving polygon that separated into halves. On 50% of the trials, one polygon-half was replaced by a different half during the movement and participants indicated whether they noticed this significant shape change (Fig. 7a). The CDA results indicated that this separating polygon triggers resetting (see Experiment 2), so we investigated whether VWM would miss the change during resetting. The shape change could happen during separation (when VWM should be most vulnerable), 250 ms before separation, or 250 ms after separation. On half of the trials, the polygon continued to move as an integrated object, making the separation unpredictable. In different blocks, we presented color–color conjunctions (Fig. 7b) that moved as integrated objects and subjects had to indicate whether they noticed a color change. On 50% of trials, the colors separated and moved independently. The CDA results of Experiment 5 suggested that this type of separation was updated without resetting VWM contents. We therefore expected the cost to be smaller in color blocks.

Figure 7.

Stimuli and results of Experiment 6. a, Example of a Separating polygon trial (gray arrows indicate movement directions and were not presented). b, Example of a Separating colors trial. c, Hit rate by condition and time of change (error bars represent 95% confidence intervals). Performance decreased during resetting (time 0) for polygons, whereas for colors, no resetting was needed and the cost was smaller.

The results indicated that performance (evaluated by hit rate) was lower when a change occurred for separating items and, critically, this effect was larger for polygons than for colors. There was an interaction of block type, condition, and time (F_(2,38) = 5.26, p = 0.01) reflecting a larger cost for polygons than for colors during separation (F_(1,19) = 6.54, p = 0.02; Fig. 7c). This behavioral cost is especially impressive when considering the overall simplicity of the task and the high baseline ceiling performance to detect a shape change in a condition not involving resetting.

Experiment 7

In Experiment 6, we found a larger behavioral cost for resetting compared with updating. Although both conditions included separating items, the stimuli categories were different: polygons versus color–color conjunctions, leading to the possibility that the different costs were due to the different stimuli. However, our EEG analysis clearly showed that both separating and resetting were possible in identical separating situations with identical stimuli depending on their individuation (see Experiment 2 vs Experiment 4). Therefore, we predicted that the behavioral costs would show a similar trend, meaning a smaller cost associated with identical polygon separation when the parts are easier to individuate.

To test this, in Experiment 7, we compared trials in which two polygon-halves moved as an integrated polygon and then separated with trials in which the two halves first moved independently and only then met and moved as an integrated polygon that later separated (see Fig. 8a,b; the experiment also included a control condition in which an integrated polygon moved as a coherent object without separating). In the latter condition, no resetting is needed because the independent halves can be identified and individuated already before their integrated movement (see Experiment 4). Importantly, apart from the initial 400 ms, these conditions were identical, both including similar events of polygon separation, and the changes could only happen after this initial phase.

Figure 8.

Stimuli and results of Experiment 7. a, Example of a Joining-and-separating polygon trial. b, Example of a Separating polygon trial. c, Hit rate by condition and time of change (error bars represent 95% confidence intervals). The decrease in performance during separation (time 0) was larger when resetting occurred (i.e., in separation trials) compared with when identical stimuli could be updated without resetting (i.e., in joining-and-separating trials).

Hit rate was lower when a change occurred at the time of polygon separation relative to before or after separation. Critically, this cost was larger when the separation triggered resetting (i.e., when the polygon started moving as an integrated object) than when the same separation did not trigger resetting (i.e., when the polygon-halves were first presented separately), which created a significant interaction of condition and time (F_(4,76) = 3.95, p = 0.006; Fig. 8c). Specifically, changes occurring at the moment of polygon separation were harder to identify when the separating polygon first appeared as an integrated object, triggering a resetting, than when the separating polygon previously moved as two separate halves, which did not trigger a resetting (F_(1,19) = 6.80, p = 0.02), despite the fact that these conditions were perfectly identical at this time, only differing in their movement history. This lower cost was only present at time 0, whereas before and after separation, the conditions did not differ significantly (both F < 1). These results show that the behavioral cost is specifically triggered by resetting and not by object separation in general.

Notably, the behavioral cost in the separating polygon condition was smaller in Experiment 7 than in Experiment 6. This might be due to the fact that in a third of the trials (i.e., in the joining-and-separating condition), subjects saw the polygon-halves separately before separation. This could have allowed them to create a correspondence between each separate half and a different VWM representation in the separating polygon condition as well, meaning that VWM did not have to reset on some of the trials, presumably making the behavioral cost smaller.

Discussion

The present study demonstrates that the ability of VWM to update critically depends on an ongoing mapping between objects and their representations. Breaking this correspondence triggers a resetting process: existing VWM representations are abandoned and new representations are encoded, reestablishing a mapping between the novel input and VWM representations (for a summary, see Fig. 9). Experiments 1 and 2 demonstrated that when objects move as a coherent unit and then separate, VWM resets, as indicated by a drop in the CDA amplitude, the electrophysiological marker of VWM contents. We argue that this is because VWM initially mapped the integrated object into a single representation and the separation broke this correspondence, forcing VWM to abandon the old representations and encode new ones. Experiment 3 demonstrated that resetting could also occur when the old objects were abruptly replaced by novel objects. Experiments 4 and 5 complemented this finding by showing that resetting is not a necessary outcome of separation as long as the objects-to-representations mapping can be maintained because it involved the individual parts instead of the integrated objects. Finally, Experiments 6 and 7 demonstrated that, during resetting, VWM is fragile, struggling to detect changes in the objects, presumably because, at the time of the change, the correspondence between these objects and the representations is gone.

Figure 9.

A summary of the differences between updating and resetting. There is a unique correspondence between each VWM representation and an object in the environment. When items change, VWM representations can be updated, that is, accessed and modified, as long as the mapping holds (left), but, if the mapping breaks, VWM has to reset (right). We demonstrated that this can happen in object replacement situations (Experiment 3) and in object separation situations (Experiments 1 and 2), but we suggest that resetting will take place whenever the objects-to-representation correspondence breaks. Critically, similar changes (e.g., separation) that allow the mapping to hold result in updating rather than resetting (Experiments 4 and 5). The resetting process involves abandoning the no-longer-relevant representations and reencoding the novel information, along with a new correspondence. It is accompanied by a sharp drop in CDA amplitude (unlike updating, which involves a steady change in CDA amplitude) and produces a pronounced behavioral cost (Experiments 6 and 7).

Several findings could serve to rule out possible confounds. First, in Experiments 1 and 2, resetting was triggered when subjects tracked moving items that separated. However, resetting was triggered without object separation in Experiment 3 by replacing one static object with another one. Furthermore, Experiments 4 and 5 demonstrated that object separation, which does not break the objects-to-representation mapping, results in updating instead of resetting. Specifically, Experiment 4 used exactly the same object separation as Experiment 2 and only manipulated the items' movement history (i.e., allowing the two polygon-halves to move independently before meeting and subsequently separating again) to make their individuation easier. Because the pre-separation mapping could now be maintained, no resetting was observed, in contrast to Experiment 2. This shows that the relevant factor determining whether updating or resetting will occur concerns not the separation per se, but rather the ability to maintain the original object-to-representation mapping.

Second, in Experiments 1 and 2, resetting followed an increase in the number of items. However, resetting cannot simply be a result of adding more items because the Add condition of Experiment 3 also included the addition of an item, but because the correspondence between the original item and its representation was still valid, VWM could update without resetting. Furthermore, the Switch condition of Experiment 3 resulted in resetting without increasing the number of items in VWM because only one object was present both before and after the resetting.

Third, resetting always followed a strong perceptual signal. Importantly, Experiment 4 ruled out the possibility that resetting reflects an effect of a pronounced or surprising perceptual event because object separation that was visually identical to that of Experiment 2 did not result in resetting once the object-to-representation correspondence could be maintained due to the separate movement history of the items. In addition, in Experiment 5, object separation did not result in resetting with color stimuli, which included parts that were easy to identify before separation, thus allowing the mapping to be maintained. Similarly, the resetting that followed the presentation of the second polygon half in the Switch condition of Experiment 3 could not be due to the transient signal because the same polygon-half appeared in the Add condition, triggering an updating process without a resetting since the original mapping was maintained.

Finally, because the process triggering resetting (i.e., object separation or object “switching”) always happened at the same time throughout each experiment, another concern is that our results reflect some sort of expectation. For example, the separation in Experiments 1 and 2 always happened 400 ms after trial onset and it might be argued that the CDA-drop is related to subjects preparing for this separation. However, whereas the separation timing was completely predictable, whether a separation took place was unpredictable: all experiments included a condition that was identical except that the items continued to move as an integrated shape without separating. Therefore, subjects could prepare for the separation, but an effect for this should have been revealed in the integrated shapes condition as well. A similar argument exists as for the Switch and Add conditions of Experiment 3. Although the timing of the second item's presentation was perfectly predictable, on half of the trials, it appeared in a novel location and did not trigger resetting. Furthermore, Experiment 4 demonstrated that a predictable separation does not trigger resetting as long as the objects-to-representations mapping can be maintained after separation.

Previously, several studies focused on the ability of VWM to update its representations in different situations, for example, when moving items change location (Drew and Vogel, 2008; Drew et al., 2011; Drew et al., 2012) or when stationary items change their features (e.g., color or orientation; Blaser et al., 2000). Aside from changes in the items themselves, interactions between items can also lead to VWM updating. Specifically, representations of independent items can be integrated into one chunk according to different objecthood cues (Wilson et al., 2012; Luria and Vogel, 2014; Balaban and Luria, 2015; Peterson et al., 2015; Balaban and Luria, 2016b). This demonstrates the dynamic nature of VWM representations, which is maintained through their ability to update in a wide range of situations from basic perceptual changes to complex changes regarding task demands (Vergauwe and Cowan, 2015; Balaban and Luria, 2016a).

The present work shows that this updating process is limited by the demand for a constant mapping between objects and representations. The situations in which VWM cannot update and has to reset are not characterized by a more perceptually demanding change (see above), but rather by the breaking of the original objects-to-representations correspondence, which could happen in different situations such as object separation or object replacement. Importantly, we demonstrated that the difference between the processes of updating and resetting is not quantitative, but rather qualitative. That is, breaking the correspondence between objects and their representation in VWM does not result in “more” updating, but rather in a qualitatively different neurobehavioral adaptation process of resetting.

Most theories proposing a resetting-like process (Gernsbacher, 1991) did not discuss the role of working memory in this process (but see SOB-CS; Oberauer et al., 2012). Remarkably, seminal working memory models (e.g., Baddeley and Hitch, 1974) did not focus on the online aspects of the representations themselves, but on their short-term maintenance (e.g., after the encoded items disappear from view). However, recent findings suggest that maintaining active and accessible representations is very similar for items that are not visible (e.g., classic change-detection tasks) and for items that are still within view in terms of both capacity limit (Tsubomi et al., 2013) and the brain regions that are involved (e.g., in both cases, the CDA originates from temporal areas; Becke et al., 2015).

The present work highlights the dynamic processes that operate on active representations. We argue that the resetting process can be easily integrated within existing theories of working memory. Generally, regardless of the nature of the representations, no-longer-relevant information must be removed from the limited workspace. This is true for nonvisual information as well and we suggest that resetting is not limited to VWM. Furthermore, it is possible that well documented deficits of working memory in clinical populations such as in schizophrenia (Gold et al., 2003) are actually due to a resetting problem. Interestingly, VWM deficits sometimes coincide with flexibility problems in rule switching (Fey, 1951), which could be linked to resetting, and further work is needed to clarify this possibility.

What happens in working memory during resetting, causing the observed behavioral cost and CDA-drop? We suggest that, once the original representation loses its correspondence to a specific object in the environment, it is erased from VWM because it can no longer be accessed in a useful way (Fig. 9). This explains the CDA-drop because the amplitude of the CDA reflects the number of items held in VWM (or perhaps even the number of “pointers”). This deletion account also explains the behavioral cost because the original representations are unavailable and thus cannot be accessed and changed.

Resetting enables flexibility in the function of VWM, which can discard mental schemes that are no longer relevant, an ability that is likely at the heart of our capacity to learn new information. Aside from establishing a novel process in VWM, our results thus provide a stable neural marker that allows studying the online dynamics of how the cognitive system handles the ever-changing environment.