Goal-directed modulation of neural activity during working memory maintenance

Abstract

Visual working memory is crucial for goal-directed thoughts and behaviors. However, it is not clear how goals modulate working memory maintenance, as previous models often considered stimulus encoding to be the endpoint of goal-directed control. To address this gap, eighty adults performed delayed estimation tasks while their brain activity was recorded using electroencephalography. In each trial, participants memorized three visual gratings varying in orientation and location and were instructed which attribute to recall. Based on recall success, trials were classified as successful or unsuccessful. We examined the effects of task instructions and their behavioral relevance using event-related potentials and multivariate pattern analysis during encoding and maintenance. The orientation task elicited larger contralateral delay activity than the location task. Moreover, the two tasks were decodable from brain patterns during the maintenance phase, and these patterns did not generalize to the encoding phase, suggesting that goal-directed modulation during maintenance was not merely a consequence of selective encoding. We further found that goal-directed modulation involves two functionally distinct processes that unfold dynamically over time, with the latter beginning even before stimulus offset and continuing throughout the entire maintenance phase. Finally, task decoding accuracy was consistently higher for successful than unsuccessful trials during maintenance.

Introduction

Visual working memory refers to the ability to hold and manipulate visual information over short periods of time. This ability plays a crucial role in visual perception by maintaining a subjective sense of continuity between saccadic eye movements (Irwin 1992; Irwin and Andrews 1996). It also facilitates a wide range of goal-directed thoughts and behaviors. A prominent question is how goals affect the way information is encoded and maintained in working memory. In this study, we investigated the neural mechanisms underlying goal-directed control in visual working memory tasks.

In typical delayed response tasks for studying visual working memory, participants are instructed to encode an array of items. Following a short maintenance interval where no visual stimuli are present, participants are cued to recall information and provide a response based on a memory probe. Evidence suggests the existence of goal-directed modulatory mechanisms at multiple stages of visual working memory, including the encoding phase (eg Lepsien and Nobre 2007; Bollinger et al. 2010; Chadick and Gazzaley 2011; Gazzaley and Nobre 2012). For example, studies using electroencephalography (EEG) have shown that the amplitude of the P1 component, an early event-related potential (ERP) evoked by visual stimuli, is modulated by different task instructions (eg “remember face” vs. “remember scene”) even though the visual displays (eg overlapping face and scene stimuli) remained identical across trials (Zanto and Gazzaley 2009; Rutman et al. 2010). The degree of goal-directed modulation of the P1 is associated with subsequent working memory recognition (Zanto and Gazzaley 2009; Rutman et al. 2010). For instance, Rutman et al. (2010) reported that a larger P1 amplitude difference between face and scene trials predicted higher retrieval accuracy for both stimulus types.

A similar amplitude modulation has been observed for the N2-posterior-contralateral (N2pc) component during selective encoding of stimuli. The N2pc manifests as a negative deflection peaking at 200 to 300 ms after stimulus onset, characterized by increased negativity at posterior electrodes contralateral to the attended visual field relative to ipsilateral electrodes (Luck and Hillyard 1994; Eimer 1996; Luck et al. 1997). Salahub et al. (2019) investigated the effect of goal-directed prioritization on working memory. In their experiment, participants encoded the colors of four items: two presented laterally and two presented vertically. They found that, even with identical encoding displays, N2pc amplitude increased as the probability of probing the lateral stimuli increased, suggesting that goal-directed prioritization impacts how sensory input is encoded in working memory. Additionally, a larger N2pc was associated with more precise color recall of the probed item. These findings suggest that the N2pc tracks the up-weighting of goal-relevant over -irrelevant information during working memory encoding (Salahub et al. 2019).

While previous findings have revealed a modulatory mechanism that supports goal-directed control during encoding, it is less clear whether the maintenance phase is subject to goal-directed control. It has been argued that, once encoded, information is consolidated and remains stable, with the subsequent maintenance period involving only passive retention of this information (Sternberg 1966). This view has been challenged by studies showing that goal-directed processing continues during working memory maintenance, particularly as evidenced by studies using the “retro-cueing” technique (Griffin and Nobre 2003; Landman et al. 2003; Sligte et al. 2008). In these studies, participants receive a cue during the maintenance phase, which retroactively modifies the task relevance of maintained information. This process can involve, for example, cueing a location of the item (Griffin and Nobre 2003) or a category of stimuli (eg faces) that will be probed (Lepsien and Nobre 2007). Retro cues confer robust benefits to working memory performance. While retro-cueing studies suggest that goal-directed control can occur during the maintenance phase, such effects have largely been demonstrated in tasks that explicitly require changes to internal representations. However, it remains unclear whether goal-directed control extends to situations where there is no explicit need to prioritize or update the maintained representations. Here, we address this gap by testing whether internal goals continue to influence the fidelity and stability of encoded representations even when passive retention would suffice.

To address this aim, we employed a delayed estimation task in which participants were required to encode and maintain three visual gratings that varied in both their spatial locations and orientations (see Fig. 1). On each trial, participants needed to remember the specific combination of orientation and location for all three gratings. Based on the task instruction given prior to each block of trials (ie “report ORIENTATION” vs. “report LOCATION”), participants were later asked to recall either the orientation or the location of a probed item after the maintenance period. In the orientation task, a circle at the location of the target served as the probe, while in the location task, the probe was a grating at the centre of the screen displaying the orientation of the target. This manipulation ensured that both the orientation and location of all three gratings remained task-relevant during encoding and maintenance. In other words, participants were expected to encode and maintain the same sensory information on every trial, with task instructions indicating which feature they would later need to report. This design allowed us to test whether goal-directed control, prompted by the task instruction, could still play a role during the maintenance phase.

Fig. 1.

Schematic illustration of the delayed estimation tasks. Each block started with a task instruction to indicate which visual working memory task to perform. At the beginning of each trial, an arrow cue appeared to remind participants to encode items presented on either the left or right side of the screen. Six differently oriented gratings were then presented during the encoding period, followed by a maintenance period. In the orientation task, the location of one of the memorized items was presented as a probe to indicate which item would be retrieved. During the response period, participants reported the orientation of the target item. In the location task, the orientation of one of the memorized items was presented as a probe, and participants reported the location of the target item during the response period.

Our first goal was to test whether goal-directed control occurred during the encoding period by examining amplitude differences in the N2pc elicited by the orientation and location tasks. We hypothesized that the N2pc amplitude would vary between tasks despite using identical encoding displays across different tasks. However, we did not make a specific prediction about the direction of this difference, given that all three gratings were task-relevant and previous findings of item prioritization from Salahub et al. (2019) may not directly translate to our working memory design. To complement the univariate approach, we examined task differences using multivariate pattern analysis (MVPA). Unlike traditional ERP methods, which focus on a small subset of electrodes, MVPA leverages spatially distributed patterns across the scalp to characterize differences in neural activity between experimental manipulations (Haxby et al. 2014; Grootswagers et al. 2017). Any above-chance classification accuracy suggests that the manipulation of interest is decodable from patterns of brain activity. We used MVPA to characterize time-resolved changes in brain activity during encoding and maintenance as a function of task instructions. This combination of univariate and multivariate approaches provides a more sensitive and comprehensive assessment of goal-directed modulations of different stages of visual working memory. We expected that patterns of brain activity would be modulated by task demands and, therefore, distinguishable between tasks during the encoding period.

Next, we aimed to address our key question: whether goal-directed control continues during the maintenance period even when there is no explicit need to prioritize or update working memory representations. If so, we would expect to observe similar modulation of brain activity during the maintenance period as during the encoding period. In delayed-response working memory tasks, contralateral delay activity (CDA) is observed consistently during the maintenance period (Luria et al. 2016). The CDA is a negative slow wave that sustains throughout the maintenance period, with greater negativity observed at posterior electrodes contralateral to the attended side relative to ipsilateral electrodes. The CDA amplitude increases as working memory load increases (Ikkai et al. 2010; Luria et al. 2010; Luria and Vogel 2011; Brady et al. 2016; Hakim et al. 2019) and is sensitive to the type of information held in working memory, with certain visual attributes eliciting a stronger CDA response than others (McCollough et al. 2007; Woodman and Vogel 2008; Luria et al. 2010). Here, we tested whether the CDA was also sensitive to goal-directed modulation when the to-be-memorized content was identical across tasks. We did not make a specific prediction about the direction of this effect, given the lack of prior research directly comparing CDA amplitudes for spatial location and orientation. We also employed MVPA to examine task-related differences in patterns of brain activity during the maintenance period. We expected the orientation and location tasks to exhibit different CDA amplitudes and elicit distinct patterns of brain activity.

Any such findings, however, would not rule out the possibility that neural differences during maintenance were merely a consequence of the encoding period. If differential encoding of sensory information guided by task instructions were to occur, we would also expect these neural differences to persist into the maintenance phase, even if the representations were only passively maintained. To investigate whether goal-directed control occurred during the maintenance phase independently of any encoding differences, we conducted a multivariate temporal generalization analysis to determine whether task-specific brain patterns remained stable or evolved dynamically from encoding to maintenance (King and Dehaene 2014). This approach allowed a direct comparison between the two phases, allowing us to examine how well neural patterns trained during encoding generalized into the maintenance period. If neural differences during maintenance were merely a consequence of selective encoding, then task decoding should strongly generalize between the encoding and maintenance periods. By contrast, if neural differences during maintenance arose from both selective encoding and ongoing goal-directed control, then generalization between the two periods should be substantially lower than would be expected if maintenance were simply a continuation of encoding.

Lastly, we examined whether goal-directed modulations were predictive of working memory performance measured behaviorally. Specifically, we investigated how task-induced modulations differed depending on response type, as well as how response-related differences varied as a function of task. To this end, we first categorized individual trials as either “successful” (if the response resulted from the accurate retrieval of the target item), or “unsuccessful” (if the response was based on the retrieval of nontarget items or random guessing; Bays et al. 2009). We then performed repeated-measures ANOVAs on the peak amplitude of N2pc and the mean amplitude of CDA, with task (orientation vs. location) and response type (successful vs. unsuccessful recall) as within-subject factors. Given prior research suggesting that goal-directed modulations in brain activity are closely related to working memory performance (Zanto and Gazzaley 2009; Rutman et al. 2010; Salahub et al. 2019), we predicted a significant interaction between task and response type. Specifically, we expected that the extent of task modulations in both ERP components would differ between successful and unsuccessful recall and that recall-related modulations would vary across tasks. We also used MVPA to decode the task on successful and unsuccessful trials separately, as well as to decode response type within each task. We hypothesized that the strength of task decoding would differ between response type, and the strength of response decoding would vary between tasks.

Materials and methods

Participants

Eighty-seven healthy adult humans were recruited from The University of Queensland through an online volunteer system. Participants performed the visual working memory experiment concurrently with EEG recording. Seven participants were excluded due to incomplete data acquisition. The final sample included 80 participants aged 18 to 38 years (M = 24.24, SD = 4.61; 39 females). All participants completed safety screening questionnaires and provided written informed consent before the experimental sessions. Participants were reimbursed at a rate of $20 per hour. The study was approved by the Human Research Ethics Committee of The University of Queensland (2018001427). The experiment was undertaken with the understanding and written consent of each participant.

Experimental procedure

The delayed estimation paradigm has been described in detail in previous papers (Li et al. 2023; Li et al. 2024; Fig. 1). Briefly, the experiment comprised two blocks of trials of each task, presented in random order. Each block started with the presentation of a task instruction (ie “report ORIENTATION” and “report LOCATION”) to indicate which task the participant should perform in the following trials. Each block contained two runs in which participants were cued to encode items on the left side of the screen and two runs in which participants were cued to encode items on the right side, and these alternated with each other. The runs were counterbalanced across participants. Each block comprised 120 trials, with 30 trials per run. A total of 480 trials were collected from each participant.

Each trial started with a central fixation cross, above which was an arrow pointing to the left or right. This manipulation was designed to cue participants’ covert attention to the task-relevant visual hemifield and to permit computation of the N2pc and CDA components. The cue was followed by an encoding phase for 400 ms, during which six differently oriented gratings were presented simultaneously and bilaterally (ie three in each visual hemifield). The gratings were randomly arranged on an invisible circle with respect to their centre. Grating locations were sampled independently in the left and right visual hemifields between 15° and 165°, with a minimum separation of 37° between any two gratings. Grating orientations were sampled from 0° to 179°, with pairwise orientation differences ranging from a minimum of 20° to a maximum of 90°. Based on the direction of the cue arrow, participants had to focus covertly only on the three gratings presented in the left or right hemifield. The encoding phase was followed by a 900-ms maintenance phase, during which only the central cross remained on the screen. Note that during both the encoding and maintenance periods, participants were required to maintain fixation on a central cross. If any eye movements were detected during these periods, a trial was discarded and repeated at the end of each run. Next, a response probe appeared showing either the location or orientation of one of the memorized gratings for 700 ms. In the orientation task, a probe stimulus was displayed at the location of a randomly chosen grating. As the probe disappeared, participants were instructed to report the orientation of the target item in a white circle. In the location task, by contrast, the probe indicated the orientation of a randomly chosen grating. As the probe disappeared, participants were instructed to report the location of the target item on a white circle. For both tasks, participants had maximally 3,500 ms to respond. After the response, or at the end of the response period, feedback was provided for 1,000 ms in the form of a green line to show the correct orientation, or a green circle to show the correct location.

The experiment was implemented in MATLAB R2018a (MathWorks, Natick, MA) using Psychtoolbox (Brainard 1997; Pelli 1997). Visual stimuli were presented on an LCD monitor (VG248QE) with a resolution of 1920 × 1080 pixels and a refresh rate of 60 Hz. Participants were seated approximately 60 cm from the monitor in a dimly lit room, with their head supported by a chinrest. Eye position was recorded with a desk-mounted eye-tracking system, sampled at 120 Hz (iView RED-m infrared eye tracker, SensoMotoric Instruments, Teltow, Germany). The eye tracker was calibrated and validated before each experimental block using a five-point calibration grid.

Behavioral analyses

To quantify behavioral performance in each trial, response errors were computed as the angular difference between the participants’ response and the correct orientation or location of the probed item. In the orientation task, response errors ranged between 0° and ± 90°. In the location task, response errors ranged between 0° and ± 180°, with errors larger than ± 90° indicating that participants selected a location in the uncued hemifield. Initial data inspection indicated that participants never made such “hemifield-swap” errors. Thus, in later analyses, item location was treated similarly to item orientation, with the range of response errors being between 0° and ± 90°. Response errors in both tasks were transformed from degrees to pi radians (πrad) with 0° and ± 90° mapped to 0 πrad and ± 1 πrad, respectively. A histogram of response errors was constructed for each participant and each task.

To identify outlier participants, error distributions per task were compared against a uniform distribution (ie a uniform distribution is expected if a participant guessed in a majority of trials) for each participant, using the Kolmogorov–Smirnov test (Massey 1951). Participants whose error distributions were uniformly distributed at the level of P < 0.05 were removed from further analyses. Based on this criterion, eight participants were excluded, leaving a total of 72 participants (36 females; 18 to 38 years, M = 24.31, SD = 4.77) for the following analyses.

To identify different sources of behavioral variability in the visual working memory tasks, a mixture distribution modeling approach introduced by Bays et al. (2009) was applied, which attributes response errors to a mixture of three components. Briefly, the model is defined as the probability of reporting the target item (P_T), the probability of reporting the nontarget items (P_NT), the probability of random guessing (P_G), and the concentration parameter κ of the von Mises distribution that described the variability around the target value. Maximum likelihood estimates of the parameters were obtained separately for each participant in each task using an expectation–maximization algorithm. The fitted von Mises κ was converted to circular standard deviation (σ_vM) as defined by Fisher (1995), giving an inverse measure of “memory precision” that reflects the precision of representations stored in visual working memory (Bays et al. 2011a; Bays et al. 2011b; Pratte et al. 2017). P_NT measures “swap errors,” which describe the proportion of responses arising from feature binding anomalies in working memory where a nontarget feature is “swapped in” for the target feature (Bays 2016; Schneegans and Bays 2017). P_G measures “random guesses,” which reflect the proportion of responses originating from attention lapses, poor task compliance, or other motivational factors. These analyses were performed using the Analogue Report Toolbox in MATLAB R2021b (Bays et al. 2009; Schneegans and Bays 2016).

The mixture distribution model also returned trial-by-trial posterior probabilities, referred to here as “weights,” which represent the likelihoods of a response originating from the target, nontarget, and random guess distributions. Trials with weights for all three components falling between 0.29 and 0.39 were considered ambiguous and excluded from further analysis. The remaining trials were classified as “successful” or “unsuccessful” by comparing the weight assigned to the target distribution with the weights assigned to the nontarget and random guess distributions. Three participants were excluded as all their trials fell exclusively into a single trial type: Two had only successful trials, and one had only unsuccessful trials.

E‌EG analyses

E‌EG recording

Electrophysiological recordings were acquired from 63 Ag/AgCl electrodes positioned according to the 10-20 system using a waveguard original cap. The reference electrode was placed at CPz, and the ground electrode was placed at AFz. All EEG signals were recorded with an eego amplifier (ANT Neuro, Enschede, Netherlands) and digitized at a sampling rate of 1024 Hz. Electrode impedances were kept below 20 kΩ.

E‌EG preprocessing

EEG signals were analyzed using the EEGLAB (v2021.1; Delorme and Makeig 2004) and ERPLAB toolboxes (v8.10; Lopez-Calderon and Luck 2014). The data were down-sampled to 256 Hz and re-referenced offline to the average electrode. High-frequency noise and slow voltage changes were attenuated using a second-order IIR Butterworth band-pass filter with half-amplitude cut-offs at 0.1 and 30 Hz. The 50 Hz line noise was removed using a Parks–McClellan Notch filter. The preprocessed signal was segmented into epochs of 1,300 ms, spanning from the onset of stimulus encoding to the onset of the probe period, and baseline-corrected relative to the interval from −200 to 0 ms before stimulus onset. The segmented data were further analyzed using independent component analysis (ICA) to detect and remove eye and muscle artifacts. Trials that contained extreme amplitudes over ±100 μV or peak-to-peak deflection over ±200 μV were automatically identified and rejected from further analyses. One participant with 76% rejected trials was removed. For the remaining 68 participants (33 females; M = 24.51, SD = 4.81), the average proportion of rejected trials was 12.7%, ranging between 0.7% and 35.8% of total trials per participant. For three participants, we identified and excluded excessively noisy channels prior to average referencing. These channels were subsequently interpolated using the superfast spherical interpolation after artifact rejection.

Univariate ERP analysis

To characterize differences in neural activity between the orientation and location tasks, we first conducted univariate ERP analysis, focusing on the N2pc and CDA components identified in the contralateral-minus-ipsilateral difference waves as per convention (Luck et al. 1997; Vogel and Machizawa 2004). Given that maximal amplitude of N2pc and CDA are typically observed over the posterior parietal and occipital cortex (Stormer et al. 2013; Kang and Woodman 2014; Luck 2014; Luria et al. 2016), we analyzed eight pairs of posterior electrodes (P1/P2, P3/P4, P5/P6, P7/P8, PO3/PO4, PO5/PO6, PO7/PO8, O1/O2). For each trial, contralateral-minus-ipsilateral difference waves were calculated for each electrode pair, and signals were averaged across all pairs to improve the signal-to-noise ratio. The peak N2pc amplitude was measured as the amplitude of the largest negative peak observed between 200 and 300 ms poststimulus onset. The mean amplitude of CDA was measured between 600 and 1,200 ms. Paired-samples t-tests were used to assess differences between tasks for each component, with a significance level set at P < 0.05.

To examine whether goal-directed modulations in brain activity predicted working memory performance, we extracted contralateral-minus-ipsilateral difference waves for successful and unsuccessful recall trials separately in each task. We then conducted two repeated-measures ANOVAs on the peak amplitude of N2pc and the mean amplitude of CDA, with task (orientation vs. location) and response type (successful vs. unsuccessful recall) as within-subject factors and a significance level at P < 0.05. Significant interactions were followed up with a series of paired-samples t-tests.

Multivariate pattern analysis

In addition to the univariate analyses, we applied MVPA to the timeseries data to test whether modulations induced by task instructions were decodable from dynamic brain patterns across different regions during the encoding and maintenance phases. The analysis was performed using the CoSMoMVPA (Oosterhof et al. 2016) and LIBSVM (Chang and Lin 2011) toolboxes in MATLAB.

For each participant, a radial kernel support vector machine (SVM) was trained at each time point to find the decision boundary that discriminates between patterns related to successful and unsuccessful recall using signals across all scalp electrodes. The raw voltage values were spatially filtered with surface Laplacian (v1.1 CSD Toolbox; Kayser and Tenke 2006) and temporally smoothed using a 20-ms Gaussian-weighted running average. To improve signal-to-noise ratio while avoiding overfitting, classification was performed on each time point and its 4 neighboring time points (Grootswagers et al. 2017). All trials per participant were partitioned into 10 subsets using a balanced partitioning approach to ensure that each task occurred equally often in each subset. The classifier at each time point underwent a 10-fold cross-validation scheme, where it was trained on nine subsets of the data and tested on the remaining subset in each iteration. Classification accuracies from all iterations were then averaged at each time point for each participant, with adjustments made for the average accuracy across all time points during the 200-ms baseline prior to the onset of stimulus encoding.

To investigate whether brain patterns that discriminated between tasks remained stable or evolved dynamically over time, we performed temporal generalization analyses (King and Dehaene 2014). Instead of being trained and tested at the same time point, a classifier was trained at one time point and tested at all other time points to determine whether neural patterns identified at a specific time could be generalized to another time. This procedure produced a temporal generalization matrix for each participant, with the diagonal representing train-test pairs from the same time point. Classification accuracies were baseline-corrected by subtracting the average accuracy of the diagonal values within the 200-ms period preceding stimulus onset.

To further test whether goal-directed modulations in brain activity were predictive of working memory performance, we performed two sets of multivariate decoding analyses. First, to examine how brain activity modulation induced by task instructions differed as a function of behavioral responses, we conducted task decoding analyses separately for successful and unsuccessful recall trials. We then used a cross-condition validation scheme to test whether the brain patterns distinguishing between orientation and location tasks for one response type could generalize to another. This involved training classifiers on successful trials and testing on unsuccessful trials, as well as training on unsuccessful trials and testing on successful trials. Second, to determine whether and how behavioral responses, if decodable through whole-brain activity patterns, varied depending on the task, we performed response decoding analyses separately for the orientation and location tasks. Cross-condition generalization analyses were performed to test whether the brain patterns distinguishing between successful and unsuccessful responses in one task could generalize to the other task.

The above results were tested for statistical significance using one-sample t-tests and Monte Carlo permutations at the group level to determine whether the observed accuracies were significantly above chance (ie 50%). The t-statistics were corrected for multiple comparisons by computing the threshold-free cluster enhancement (TFCE) statistics (Smith and Nichols 2009), as implemented in the cosmo_montecarlo_cluster_stat function of the CoSMoMVPA Toolbox. We specified TFCE with the standard parameters recommended by Smith and Nichols: extent exponent E = 0.5, height exponent H = 2, and threshold step (dh) = 0.1. The null distribution was acquired over 10,000 iterations of randomly flipping the sign of the statistic values across time points. The observed TFCE statistic at each time point was considered significant if its value was larger than the 95th percentile of the null distribution, corresponding to P < 0.05 for a one-tailed test, corrected for multiple comparisons.

Results

Behavioral results

Figure 2a shows the empirical error distributions for the orientation and location tasks, and the corresponding model fits predicted by the mixture distribution model. The presence of nonuniform, bell-shaped distributions centred on zero suggests that participants were able to perform the tasks as expected. The distribution of response errors was wider for the orientation task than for the location task. The mixture distribution model provided good fits to the empirical data for both tasks, with the estimated σ_vM being significantly higher, and thus memory precision being significantly lower, for the orientation task than for the location task [σ_vM = 0.74/0.03 [M/SEM] vs. 0.40/0.01, t(71) = 10.40, P < 0.001; Fig. 2b]. The error distributions for both tasks also displayed long tails, indicating the presence of swap errors and random guesses. Figure 2c shows the differences in the model-estimated probabilities between tasks. The estimated probabilities of reporting the target item were similar across tasks [P_T = 0.61/0.02 vs. 0.64/0.01, t(71) = −1.41, P = 0.652], whereas the probability of swap errors was significantly lower in the orientation task than the location task [P_NT = 0.03/0.01 vs. 0.34/0.01, t(71) = −28.10, P < 0.001]. The probability of random guessing, on the other hand, was significantly higher in the orientation task than in the location task [P_G = 0.36/0.03 vs. 0.02/0.01, t(71) = 13.20, P < 0.001].

Fig. 2.

Behavioral results for orientation and location tasks. a) Histograms of response deviation relative to the target value for both tasks. Solid lines show the mean of model fits across participants, with shading areas showing ±1 SEM. b) The estimates of memory precision as measured by the circular standard deviation of von Mises (σ_vM) and (c) probabilities of reporting the target (P_T) and nontarget items (P_NT), and random guessing (P_G) across participants for both tasks. Error bars denote ±1 SEM.

Goal-directed modulations of brain activity

The univariate ERP analysis showed a negative peak at around 250 ms at parietal–occipital electrodes for both orientation and location tasks during the encoding phase (Fig. 3a). A paired-samples t-test revealed that there was no significant difference in the peak amplitude of the N2pc between the orientation (M/SEM = −0.73 μV/0.07) and location tasks [−0.78 μV/0.10), t(67) = 0.71, P = 0.479 (Fig. 3b]. Throughout the maintenance period, a sustained slow wave at parietal–occipital electrodes was found for both tasks (Fig. 3a). A paired-samples t-test revealed that the CDA mean amplitude was significantly more negative for the orientation task (−0.23 μV/0.09) than for the location task (0.02 μV/0.11), t(67) = −4.25, P < 0.001 (Fig. 3b).

Fig. 3.

Results of ERP and MVPA for the two tasks. a) Grand average difference waves identifying the N2pc and CDA for both tasks. The average waves were calculated using eight pairs of posterior electrodes: P1/P2, P3/P4, P5/P6, P7/P8, PO3/PO4, PO5/PO6, PO7/PO8, and O1/O2. The encoding phase is highlighted by a light gray background in plots a) and c); the colored shading around lines in these plots represents ±1 SEM. Topographies inserted on the right show the instantaneous activity difference between the contralateral and ipsilateral electrodes at 250 and 500 ms for both tasks. b) N2pc peak amplitude and CDA mean amplitude for both tasks. c) Decoding performance of MVPA on task. The black arrows highlight the peaks in accuracy during encoding. Time periods showing above-chance decoding at TFCE-corrected P < 0.05 are highlighted by the colored dotted line. d) Temporal generalization matrix of task decoding. The baseline, encoding, and maintenance phases are denoted by patterned, light gray, and dark gray bars, respectively. Significant above-chance temporal generalization is highlighted by white contour lines. The small and large black squares highlight above-chance decoding in early encoding and in late encoding through maintenance, respectively. The black arrows highlight the drop in generalization accuracy when training and testing at 200 ms.

The MVPA revealed a rapid increase in decoding accuracy shortly after the onset of the encoding display, reaching the first peak of 57.4% at 150 ms, a second peak of 60% at 250 ms, and a broader peak of 62.5% around 710 ms. To evaluate the reliability of these peaks, we conducted a peak detection analysis at the individual-participant level using a moving-window approach. All three peaks were detected in every participant, indicating that they are robust at both the individual and group levels. Decoding accuracy remained relatively stable during the maintenance phase. The decoding accuracy was significantly above chance from 95 ms onward (TFCE-corrected Ps < 0.020; Fig. 3c). This suggests that the orientation and location tasks were reliably distinguishable on the basis of brain activity patterns throughout both encoding and maintenance.

To test whether such brain patterns remained stable or evolved dynamically from the encoding to the maintenance period, we conducted a temporal generalization analysis (Fig. 3d). The analysis revealed distinct patterns of temporal generalization in decoding performance before and after 200 ms. Starting from 95 ms, there was an above-chance, diagonal-shaped pattern indicating that classifiers trained at each time point primarily generalized to neighboring time points. It is worth noting that there is a clear drop in generalization performance when training and testing at around 200 ms, which likely reflects a genuine transition in the underlying processes before and after this time point. From 200 ms to the end of the maintenance phase, an overall square-shaped pattern emerges, with classifiers trained within this window generally showing above-chance generalization across much of it. This pattern is consistent with a sustained process for goal-directed control, although generalization in the 200 to 550 ms range and later time points is less consistently above chance. These results suggest that two distinct processes support goal-directed control during stimulus encoding and maintenance, with one process occurring very early during encoding and another starting later in the encoding phase and continuing into the maintenance phase.

Behavioral relevance of goal-directed modulations

Consistent with the grand average results shown in Fig. 3a, ERP analyses performed separately for successful and unsuccessful recall trials also revealed prominent N2pc and CDA components for both the orientation and location tasks (Fig. 4a).

Fig. 4.

Results of ERP and MVPA for each task and for different responses. a) Difference waves obtained from successful and unsuccessful recall trials show an N2pc and CDA for both tasks. b) N2pc peak amplitudes and CDA mean amplitudes for each response type in each task. Topographies on the right show the instantaneous activity difference between contralateral and ipsilateral electrodes at 250 and 500 ms for both tasks. c) Task decoding performance for successful and unsuccessful trials (left). Task decoding performance of the cross-condition generalization analysis from successful to unsuccessful trials, and vice versa (right). d) Response decoding performance for the orientation and location tasks (left). Response decoding performance of the cross-condition generalization analysis from orientation to location task, and vice versa (right). The encoding phase is highlighted by a light gray background in a) to d); the colored shading around lines in these plots represents ±1 SEM. Time periods showing above-chance decoding at TFCE-corrected P < 0.05 are highlighted by the colored dotted line in c) and d).

To test whether goal-directed modulations predicted working memory performance measured behaviourally, we conducted two separate repeated-measures ANOVAs on the peak amplitude of the N2pc and the mean amplitude of the CDA, with task (orientation vs. location) and response type (successful vs. unsuccessful recall) as within-subject factors. For N2pc peak amplitude (Fig. 4b, left), the analysis revealed a significant main effect of task, F(1, 67) = 112.56, P < 0.001, and a significant main effect of response type, F(1, 67) = 133.05, P < 0.001. There was also a significant interaction of task and response type, F(1, 67) = 107.53, P < 0.001. Follow-up analyses revealed that the N2pc peak amplitude was significantly more negative for unsuccessful trials compared with successful trials, in the orientation task, t(67) = 2.36, P = 0.021, whereas no significant difference was found between successful and unsuccessful trials in the location task, t(67) = 0.72, P = 0.472. By contrast, analyses of the mean amplitude of the CDA (Fig. 4b, right) revealed no significant main effects of task, F(1, 67) = 0.20, P = 0.658, and response type, F(1, 67) = 1.67, P = 0.201, and no significant interaction between the two, F(1, 67) = 0.14, P = 0.705. Overall, the ERP analyses reveal that task-specific modulation during the encoding phase (ie N2pc), but not during the maintenance phase (ie CDA), was predictive of recall success.

The task decoding analyses, performed separately for successful and unsuccessful trials, revealed broadly similar temporal trends but clear differences in decoding strength between response types. In both trial types, there was a rapid increase in decoding accuracy after stimulus onset, which remained stable throughout the encoding and maintenance periods (Fig. 4c, left). Significant decoding was observed from the late encoding phase onward. For successful trials, decoding accuracy reached significance at 102 ms poststimulus onset (TFCE-corrected Ps < 0.001), with a mean accuracy of 56%. For unsuccessful trials, decoding accuracy became significant at 86 ms (TFCE-corrected Ps < 0.019), with a mean accuracy of 54%. Notably, decoding accuracy was consistently higher for successful compared with unsuccessful trials throughout both the encoding and maintenance periods. Paired-samples t-tests indicated significant differences at 30% of time points (ts < 3.16, Ps > 0.002), and one cluster spanning from 900 to 1,000 ms remained significant after correction for multiple comparisons using a cluster-based permutation approach. The cross-conditions generalization analysis revealed that brain patterns trained to distinguish between the orientation and location tasks in successful trials generalized effectively to unsuccessful trials, with decoding accuracy consistently above chance from the late encoding phase onward (TFCE-corrected Ps < 0.002, Fig. 4c, right). Similarly, classifiers trained to decode the tasks in unsuccessful trials also showed above-chance decoding when tested on successful trials (TFCE-corrected Ps < 0.033).

The response decoding analyses, conducted separately for orientation and location tasks, revealed clear differences in decoding performance between the two tasks (Fig. 4d, left). For the orientation task, successful and unsuccessful recall were distinguishable only during encoding, in time windows of 130 to 200 ms and 285 to 410 ms (TFCE-corrected Ps < 0.040). In contrast, for the location task, successful and unsuccessful trials were not decodable from patterns of brain activity during either the encoding or the maintenance phases (TFCE-corrected Ps > 0.097). The cross-condition generalization analyses further support this distinction between tasks (Fig. 4d, right). Specifically, classifiers trained on the orientation task did not generalize to the location task (TFCE-corrected Ps > 0.296), nor did classifiers trained on the location task generalize to the orientation task (TFCE-corrected Ps > 0.500).

Discussion

We investigated the role of internal goals, prompted by task instructions, in modulating brain activity during visual working memory tasks involving encoding and maintenance of orientation and location information. The results confirmed our hypothesis that goal-directed control occurred during the encoding period of the delayed estimation tasks. Although no difference between tasks was observed in N2pc peak amplitude, the multivariate analysis revealed that different tasks could be decoded significantly during the encoding period. Consistent with our hypothesis, we found task-related differences in the mean amplitude of the CDA. Additionally, we observed that the orientation and location tasks were decodable based on multivariate brain patterns during the maintenance phase. Importantly, these brain patterns were not fully generalizable into the encoding period. This suggests that the task-specific modulation of brain activity during the maintenance phase was not merely a consequence of the encoding differences between tasks. Finally, our findings provide clear evidence that goal-directed control is closely related to visual working memory performance. From the late encoding period onward, task decoding accuracy was consistently higher for successful trials compared with unsuccessful trials. Moreover, the N2pc peak amplitude was significantly larger for unsuccessful than successful trials, but this effect was specific to the orientation task and absent in the location task. This task-specific difference was further supported by the response decoding analyses, which revealed that successful and unsuccessful recall trials were distinguishable based on neural activity patterns during working memory encoding in the orientation task only.

The modulation of activity by internal goals during working memory encoding has been shown to occur at an early stage of sensory processing (Gazzaley 2011). Previous EEG studies showed that the amplitude of the P1 and N1 components, ERPs evoked by early visual processing around 100 to 200 ms after stimulus onset, was modulated by different task instructions (eg a larger P1 amplitude for “remember face” vs. “remember scene” trials), even though the encoding displays were identical in form across trials (Zanto and Gazzaley 2009; Rutman et al. 2010). Although we did not directly assess early ERP components, our task decoding analysis revealed findings consistent with the idea of early modulation. Specifically, the orientation and location tasks became distinguishable as early as 95 ms after the onset of the encoding display, with the first peak in decoding accuracy occurring around 150 ms. Goal-directed modulation has also been observed in the N2pc, whose amplitude varies with item prioritization, and the degree of this modulation is closely related to working memory performance (Salahub et al. 2019). While our study found no difference in N2pc peak amplitude between tasks, significant task decoding from 200 ms to stimulus offset indicated differential modulation of activity during the late encoding phase, with the highest decoding accuracy occurring around 250 ms.

As expected, we found a difference in CDA amplitude between the orientation and location tasks, with a larger amplitude for the orientation task. McCants et al. (2019) also compared CDA amplitudes across tasks involving spatial and nonspatial visual attributes, specifically location and color. Interestingly, they found the opposite pattern, with larger CDA amplitude in the location task compared with the color task. This discrepancy may be due to the fact that, in McCants et al., the location task required participants to maintain precise spatial information sampled from 240 possible angular positions, whereas the color task involved remembering only five distinct colors. In contrast, our experiment was designed to match task parameters across conditions. We controlled not only for the number of items and the relevance of visual attributes, both known to affect CDA amplitude (McCollough et al. 2007; Woodman and Vogel 2008; Ikkai et al. 2010; Luria et al. 2010; Luria and Vogel 2011; Brady et al. 2016; Hakim et al. 2019), but also for the fidelity with which participants needed to remember the different visual attributes. Specifically, the orientations and locations both varied over the same ±90° range within the attended visual field. These aspects of our designs allow us to rule out stimulus-specific explanations for the observed CDA difference in our study.

Significant task decoding was sustained throughout the entire maintenance period, suggesting that the orientation and location tasks remained distinguishable, even though the visual stimuli were no longer visible in the display. This finding extends previous MVPA results by showing that whole-brain activity patterns can be used not only to decode low-level feature-specific contents, such as orientation, color, and spatial frequency (Lee and Baker 2016), but also more abstract representations resulting from selective encoding or even the ongoing control “itself” over the maintained representations.

Evidence of goal-directed control during the maintenance period was reflected in our finding that brain patterns distinguishing between the orientation and location tasks were not fully generalizable between the encoding and maintenance phases, suggesting that neural modulations during maintenance were not a trivial consequence of selective encoding. However, the timing of control does not strictly match the expected separation between encoding and maintenance in our delayed estimation task. As revealed by the temporal generalization analysis, two distinct processes might be involved: one that begins around 95 ms and ends at 200 ms and another that continues from the late encoding period through to the end of the maintenance period. The early process, which aligns temporally with ERP components associated with visual processing in previous studies (Zanto and Gazzaley 2009; Rutman et al. 2010; Gazzaley 2011), may influence resource allocation to low-level representations. It could involve the enhancement of to-be-reported visual features and down-weighting of the probe feature during stimulus encoding. The later process, which continues throughout the rest of the trial, may involve the representation of abstract goals, presumably task-set information necessary for regulating downstream processing. A similar finding was reported in a previous EEG study using a cued task-switching paradigm, where different task sets (orientation task vs. color task) were decodable with high accuracy for nearly the entire duration of the trial (Hubbard et al. 2019). In summary, goal-directed control occurs during the maintenance phase, presumably achieved by a process occurring before stimulus offset, which is responsible for differentiating between tasks.

In line with previous research (Zanto and Gazzaley 2009; Rutman et al. 2010; Hubbard et al. 2019; Salahub et al. 2019), our findings suggest that goal-directed modulations in brain activity are closely related to behavioral performance. When task decoding was performed separately for successful and unsuccessful trials, we observed consistently higher decoding accuracy for successful trials from the late encoding period onward. However, this effect should be interpreted with caution, as paired-samples t-tests revealed significance only within a single time window (900 to 1,000 ms) after correction for multiple comparisons, and the difference was further reduced when the analysis was repeated with a balanced number of trials across response types. The cross-condition generalization analyses showed that brain patterns distinguishing between tasks in successful trials generalized to unsuccessful trials, and vice versa, indicating that a shared process is engaged in both trial types, which could be the overarching task-set configuration. In addition, we found evidence that response-related differences in brain activity were modulated by the specific task performed. A significantly more negative N2pc peak amplitude was observed for unsuccessful compared with successful trials, but only in the orientation task, with no such difference in the location task. Consistent with this, the response decoding analyses revealed above-chance classification of response type in the orientation task at around 130 to 200 ms and 285 to 410 ms poststimulus onset. These findings likely reflect both the selective encoding process and the ongoing configuration of abstract internal goals, which directly affect recall success within each task. The reason why such response-related differences in brain activity were observed only for the orientation task, but not the location task, may relate to differences in task demands. As shown by the behavioral results, working memory responses in the orientation task were less precise than those in the location task, suggesting that the orientation task is generally more cognitively demanding. This increased demand may require more resources to enhance or “up-weight” the fine-grained orientation information of items, compared with the relatively less demanding process of up-weighting spatial location in the location task. A related alternative explanation is that the observed differences reflect the distinct error profiles of the two tasks. The orientation task was characterized by a much higher proportion of random guesses and fewer swap errors, whereas the location task showed the opposite pattern. Swap errors in the location task may produce neural signatures that are similar to those of correct responses, making them more difficult to decode. In contrast, random guesses in the orientation task are likely to yield neural patterns that differ more markedly from correct responses, thereby supporting above-chance decoding.

The current study offers two main contributions to our understanding of goal-directed control of working memory. First, by using a multivariate decoding approach, we were able to detect task-related neural differences during encoding that were not evident in the conventional univariate ERP analyses, and to more precisely characterize when this modulation emerged. Second, through the use of temporal generalization analysis, we directly compared neural representations between encoding and maintenance phases, revealing that goal-directed control during maintenance is not merely a continuation of earlier processes. Instead, the results point to two temporally distinct control mechanisms: one linked to early sensory prioritization and another reflecting sustained, abstract goal representation. These results align with emergent theoretical shifts away from the concept of a unitary central executive (Baddeley and Logie 1999) toward models that characterize executive control as an ensemble of functionally distinct subprocesses (Vandierendonck et al. 2007; Logie 2016). Our findings further extend this framework by showing that these subprocesses are not only functionally distinct but also temporally structured, unfolding dynamically across different stages of the task.

Several limitations should also be considered when interpreting our findings. First, temporal generalization can be affected by asymmetries in signal-to-noise ratio (SNR) between time points (van Hurk and Op de Beeck 2019). Although we did not estimate SNR directly, we used task decoding performance as a proxy for SNR (Fig. 3c). Following a sharp rise at around 95 ms, decoding accuracy remained relatively stable with only small fluctuations across the trial. This was confirmed by a cluster-based permutation test, which identified no significant consecutive increases in decoding accuracy across time points after 113 ms. We therefore conclude that large SNR differences are unlikely to account for the temporal generalization patterns observed in this study. Second, signal mixing in EEG can produce above-chance temporal generalization even when no shared neural representations are present. A more conservative approach, proposed by Sandhaeger and Siegel (2023), is to compare empirical generalization with the expected generalization under identity, which estimates the level of generalization that would be expected if the underlying representations at two time points were truly identical. This method, however, requires symmetric, distance-based measures such as the cross-validated Mahalanobis distance, and is not directly applicable to the classifier accuracy used in the present study. Accordingly, while we interpret the sustained generalization from 200 ms to the end of the maintenance phase as indicative of a unitary control process, we acknowledge that measurement-related factors cannot be fully ruled out. Future studies should consider Sandhaeger and Siegel’s method, as it provides a more rigorous benchmark for assessing representational stability. A third limitation concerns our decision to classify trials into binary categories of successful or unsuccessful recall. While this reduces the continuous nature of angular error to discrete outcomes, we believe this framework is justified. Our prior work has shown that mixture model components are differentially associated with white matter tracts and resting-state networks (Li et al. 2023, 2024), indicating that these categories capture meaningful neural processes rather than arbitrary divisions of a continuous distribution. In the present study, this approach allowed us to separate the majority of trials into target versus non-target responses in a principled way, while maintaining consistency with our earlier work. Nonetheless, we recognize that this categorization may overlook more subtle, trial-by-trial variations in precision, and we highlight this as a limitation. Future studies that analyze empirical response errors directly may be better positioned to assess trial-level neural dynamics in relation to continuous variations in recall precision.

Conclusion

In summary, our findings demonstrate that goal-directed control, induced by task instructions, occurs during both stimulus encoding and maintenance, even in the absence of an explicit need to prioritize or update the representations held in working memory. Specifically, this control likely involves two functionally distinct processes: one that emerges during early encoding, and another that begins later and persists throughout the maintenance phase. These goal-directed processes are closely linked to subsequent recall success, offering new insight into how internal goals shape the fidelity and stability of representations in visual working memory.

Author contributions

Xuqian Li (Conceptualization, Investigation, Methodology, Software, Visualization, Writing—original draft, Writing—review & editing), Michael O’Sullivan (Conceptualization, Investigation, Methodology, Writing—review & editing), Jason B. Mattingley (Conceptualization, Methodology, Writing—review & editing), and Dragan Rangelov (Conceptualization, Methodology, Software, Supervision, Writing—review & editing).

Funding

J.B.M. was supported by a National Health and Medical Research Council (NHMRC) Investigator Grant (2010141).

Conflict of interest statement: None declared.

Data availability

The data and code supporting the findings of this study will be made available upon reasonable request.