Reconstructions of information in visual spatial working memory degrade with memory load

Summary

Working memory (WM) enables the maintenance and manipulation of information relevant to behavioral goals. Variability in WM ability is strongly correlated with IQ [1] and WM function is impaired in many neurological and psychiatric disorders [2, 3], suggesting that this system is a core component of higher cognition. WM storage is thought to be mediated by patterns of activity in neural populations selective for specific properties (e.g., color, orientation, location, motion direction) of memoranda [4–13]. Accordingly, many models propose that differences in the amplitude of these population responses should be related to differences in memory performance [14, 15]. Here, we used functional magnetic resonance imaging (fMRI) and an image reconstruction technique based on a spatial encoding model [16] to visualize and quantify population-level memory representations supported by multivoxel patterns of activation within regions of occipital, parietal and frontal cortex while participants precisely remembered the location(s) of 0, 1, or 2 small stimuli. We successfully reconstructed images containing representations of the remembered – but not forgotten – locations within regions of occipital, parietal, and frontal cortex using delay-period activation patterns. Critically, the amplitude of representations of remembered locations and behavioral performance both decreased with increasing memory load. These results suggest that differences in visual WM performance between memory load conditions are mediated by changes in the fidelity of large-scale population response profiles distributed across multiple areas of human cortex.

Results

To assess the functional role that population codes in different visually-responsive occipital, parietal, and frontal regions of interest (ROIs) play in spatial WM, we presented participants (n = 4, 4 scanning sessions each) with two target stimuli (Fig. 1A) followed by a post-cue instructing them to remember the location(s) of 0 (R0), 1 (R1), or 2 (R2) stimuli. In behavioral testing sessions performed outside of the scanner participants used a mouse click to indicate the exact position of the remembered target. During scanning participants performed a two-alternative forced-choice (2AFC) discrimination task in which they compared the position of a probe stimulus to that of the corresponding remembered target stimulus (Fig. 1A). We chose to test precise memory for spatial positions using either a recall task (outside the scanner) or a “same/different” task (during scanning) so that participants were required to encode exact spatial positions rather than use a verbal code or only encode a single dimension (e.g., “8 o’clock”, “far to the left”).

Figure 1. Visual spatial working memory (WM) task and behavioral performance.

(A) Participants (n = 4) viewed 2 target stimuli and were post-cued to passively fixate for the remainder of the trial (Remember 0, R0), remember the precise position of a single target stimulus (Remember 1, R1), or remember the precise position of both target stimuli (Remember 2, R2). After a 8 s delay, participants either determined whether a probe stimulus was in exactly the same or a slightly different position as the corresponding target (during fMRI scanning sessions, 0.1–1.5° offset) or precisely recalled the remembered position using a computer mouse (during behavioral sessions). (B) To discourage implementation of a “digital” encoding strategy each target was presented within one of 8 discs with uniform jitter equally spaced around fixation and offset from horizontal and vertical meridians. (C) During behavioral testing sessions outside the scanner, spatial positions were remembered less precisely with larger memory load as indicated by increased behavioral recall error distance (p < 0.001), and this is qualitatively observed for each participant. Each symbol is a single participant, and symbols match those presented in panels D–E and Figures S1–S2. (D) During scanning, behavioral accuracy was approximately equal across set sizes (p = 0.174). (E) Response times inside the scanner were significantly longer for larger memory load trials (p < 0.001). Throughout all figures, unfilled symbols refer to single-participant data; filled symbols refer to across-participant means. Asterisks reflect significant across-participant resampling tests, see Experimental Procedures.

Behavioral performance on the analog recall task performed outside the scanner revealed lower mnemonic precision when two target locations were remembered compared to when a single target location was remembered (Fig. 1C; p < 0.001, resampling test). During scanning, response accuracy did not significantly differ across set size conditions, although 3 out of 4 participants performed slightly worse with increasing set size (Fig. 1D, p = 0.174, resampling test, see Experimental Procedures). However, response times (RT) were significantly longer when two stimuli were remembered compared to when a single stimulus was remembered (Fig. 1E; p < 0.001, resampling test). Increased RTs during scanning suggest that memory representations in the R2 condition were degraded and thus less accessible during behavioral report, consistent with previous observations of increased RTs following manipulations that impair spatial WM (e.g., [17]). Together, the behavioral data recorded inside and outside of the scanner are consistent with a degraded representation of each remembered location in the R2 condition compared to the R1 condition.

To characterize neural responses associated with WM maintenance we first compared averaged blood oxygenation level dependent (BOLD) fMRI responses in a set of functionally defined occipital (V1-hV4, V3A), parietal (IPS0–3), and frontal (sPCS; thought to be the human homolog of macaque frontal eye fields [18, 19]) ROIs as a function of memory load. We replicated previous reports that BOLD responses in frontal and parietal ROIs were larger on R2 trials compared to R1 trials [6, 20, 21] (Fig. S1). Interestingly, in early visual areas (V2–V3A, hV4) we observed a larger mean BOLD amplitude on R0 trials compared to R1 or R2 trials (Fig. S1B, p < 0.001, resampling test). We also observed similar results using a complementary exploratory analysis in which we searched for any voxels with increased activation for larger memory loads (Fig. S1C).

Next, we used a multivariate image reconstruction technique based on a spatial encoding model [16] to reconstruct remembered locations in spatial WM based on the pattern of activation across all voxels within each ROI (Fig 2). In contrast to analyses that focus solely on mean signal intensity (Fig. S1), neural firing rates, or multivariate classification accuracy, this analysis uses an independently-estimated model of the spatial sensitivity profile across all voxels in each ROI to transform BOLD activation patterns into an image of the remembered stimulus position(s) carried by those patterns (Fig. 2; Experimental Procedures). Importantly, this analysis provides additional information compared to some other methods such as univariate population receptive field (pRF, [22]) estimation or multivariate linear classification [23]: by yielding a reconstructed image of the remembered stimulus location(s), covert information held in WM can be directly visualized, quantified, and related to behavior [16]. These reconstructions can be thought of as an image of the spatial WM contents in visual field coordinates (rather than coordinates relative to the cortical surface), and we interpret the focal bright spots found at target positions as target representations.

Figure 2. Inverted spatial encoding model for reconstructing the contents of spatial WM.

(A) Each participant was scanned for 3–4 independent spatial mapping runs for encoding model estimation per session (see Supplemental Experimental Procedures). Participants performed a challenging spatial WM task in which they determined whether a probe stimulus (500 ms) was in the exact same position or a slightly different position from a remembered target position (500 ms; 2AFC, see [16]). During the brief delay period (3,000 ms) a flickering checkerboard stimulus was presented nearby the remembered target position. This stimulus was irrelevant to the task performed by the participant, but was used to drive large sensory responses to estimate a voxel-level encoding model used for computing reconstructions in the main task (see C–E). We adjusted difficulty on a run-by-run basis to maintain vigilance and equate performance across participants and sessions (73.738 ± 1.819% accuracy, mean ± SEM). (B) We presented the mapping stimulus at each of 36 positions arrayed across a 6 × 6 square grid (1 trial per position per run). (C) To estimate spatial sensitivity profiles for each voxel we predicted the response of each of 36 hypothetical “information channels” (spatial filters) to each stimulus used in the training runs [16]. Then, we took the measured response of each voxel and the predicted hypothetical channel responses to each stimulus position and used ordinary least squares linear regression to estimate the contribution of each information channel to the signal observed in each voxel. This step is performed on each voxel independently (see Supplemental Experimental Procedures, Equation 3). (D) For each collection of voxels for which we computed reconstructions (ROIs, Figs. 3 & 4, all voxels from all ROIs, Fig. 4) we computed a mapping from voxel space into channel space (Supplemental Experimental Procedures, Equation 4). In contrast to ‘population receptive field (pRF)’ analyses [22], this step is multivariate, and must be performed using all voxels that contribute to the image reconstruction. Using the computed linear mapping, the measured activation pattern across all voxels is transformed into “information space” – the amount each channel must have been active in order to produce the measured voxel activation pattern. A “raw” reconstruction can be computed for any single observation (e.g., one fMRI volume from area V1) by computing a sum of the spatial filters that define the information channels weighted by the estimated channel responses (right panel). (E) When computing average reconstructions across all trials (Fig. 4C, S2B), we coregistered different target positions on each trial to a common location by first rotating the spatial filters around the fixation point such that the target lies along the Cartesian x axis, then shifting the filter centers horizontally such that the target is positioned 3.25° from fixation along the x axis (white dot in reconstructions shown in Figs. 4C, S2B). For R0 and R2 trials, this is done for each remembered target and the coregistered reconstructions aligned to each target are averaged. Importantly, this coregistration procedure enables us to average the representations of spatial WM targets that appeared at different positions in the display on different trials.

Spatial WM reconstructions computed based on patterns of delay-period activation from occipital (V1-hV4v/V3A), parietal (IPS0–3) and frontal (sPCS) cortex revealed highly robust representations of remembered target positions on R1 trials, but not on R0 trials (Fig. 3, see Fig. S2A for data from individual participants), suggesting that these images reflect memory-related activation changes rather than lingering sensory signals. Furthermore, reconstructed images contain representations of both remembered target locations on R2 trials that were robust in many occipital and posterior parietal ROIs (Fig 3C; V1–V3A, hV4, IPS0 and IPS1), but became less separable in anterior parietal and frontal ROIs (IPS2–3, sPCS). The relative decline in separability of R2 target representations in these anterior parietal and frontal ROIs may reflect the rather small screen size we used relative to the large size of spatial RFs typical of these ROIs [24, 25]. Finally, we examined the temporal structure of WM reconstructions from all ROIs over the course of the entire trial. We could readily reconstruct images of both remembered locations during target presentation when the positions were encoded into WM, but we could only reconstruct images of locations held in WM during the delay interval (Movie S1A–C).

Figure 3. Reconstructed contents of spatial WM measured using delay-period patterns of activation.

Image reconstructions for all target position arrangements during Remember 0 (A), Remember 1 (B) and Remember 2 (C) conditions from each ROI. Each reconstruction is computed using spatial filters that have been rotated around the fixation point and flipped over the horizontal meridian such that there are four possible target arrangements (top panel). Targets appeared uniformly within each of these 4 windows. Early visual (V1-hV4) and parietal (IPS0–1) ROIs carry precise target representations over the delay interval of a single remembered position (Remember 1, B) or both remembered positions (Remember 2, C). Reconstructions from anterior parietal (IPS2–3) and frontal (sPCS) ROIs carry moderately precise target representations when a single position is maintained in WM, but they are not as disjoint when both positions are simultaneously held in WM (IPS: intraparietal sulcus, sPCS: superior precentral sulcus, human homolog to macaque frontal eye fields [18, 19]). Additionally, despite a significant reduction in average BOLD response during the delay period in occipital ROIs (Fig. S1), reconstructions contain robust representations of remembered stimuli. See Figure S2A for spatial reconstructions from each individual participant and Movies S1A–C for temporal unfolding of reconstructions across the duration of the trial. Color map identical across all panels.

Next, we sought to quantify how spatial WM reconstructions differ across ROIs and under different memory loads. To do so, we rotated and shifted the reconstruction on each trial to a common reference location such that the target positions were in alignment and averaged all coregistered reconstructions together (Fig. 2E; Movie S2 for coregistered reconstructions through time). Then, because the target position across all trials was now aligned, we quantified attributes of the averaged target representation by fitting a 2-D surface (Fig. 4A–B) characterized by several independent parameters (see Fig. S4A–D for a demonstration that these parameters reflect dissociable properties of target representations). The size parameter reflects the spread (full-width half-maximum, FWHM) of the delay-period target representation: an increased fit size would reflect a less spatially-precise representation of the remembered target location (note that here and elsewhere, we use “spatial” with reference to visual field space, not cortical space). The amplitude parameter reflects the height of the target representation over baseline: increased fit amplitude would correspond to a more prominent representation of the target over baseline activation not related to the target location. The baseline parameter reflects the non-spatially-selective response amplitude (i.e., a constant offset across the entire reconstructed visual field): a change in baseline reflects a change in mean signal amplitude across an entire ROI which does not carry spatial information and thus does not directly change the spatial information content of the reconstruction.

Fig. 4. Target representations within WM reconstructions are less informative with greater memory load.

(A) To quantify the topography of the reconstructed images averaged across trials within each memory load condition, we fit a surface to the average reconstruction that was centered at its global maximum by allowing the size (full-width half-maximum, FWHM), amplitude and baseline of the surface to freely vary. (B) Example surface used for fitting. (C) All reconstructions from each ROI and memory condition (Remember 1 and Remember 2), rotated and shifted such that the exact target position is aligned to the small white dot (see Fig. 2E). We combined trials across participants and resampled all trials, with replacement, from each memory condition and ROI and quantified the averaged reconstruction on each resampling iteration (see Fig. S2 for reconstructions and best-fit parameters for each participant individually). Black + and dotted circle indicate the average best-fit smooth surface to the target representation within the reconstruction (+ indicates the center and the dashed line is drawn at FWHM of fit surface). For Remember 2, representations of each target are averaged together before fitting. See Movie S2 for temporal evolution of coregistered reconstructions across the duration of the trial. (D) Parameters describing best-fit surfaces to target representations from each ROI and memory condition. Target representation size remains constant in early visual areas (V1-hV4), but amplitude decreases with larger memory load, suggestive of a less informative population code (Fig. S3). Anterior parietal and frontal ROIs have larger target representations with increasing memory load, as well as trends towards higher amplitude representations, though some size increases are introduced during the coregistration and averaging procedure (see Supplemental Experimental Procedures: Simulating and fitting representations with known parameters, Fig. S4). The spatially non-selective baseline parameter remains largely constant across memory load conditions, except in IPS0 and IPS1. Black asterisks indicate significant differences at p < 0.05, Bonferroni-corrected within each parameter across 10 comparisons (ROIs). Gray asterisks indicate trends defined as p < 0.05, uncorrected for multiple comparisons. All tests performed using resampling procedures (see Experimental Procedures). Error bars 95% CI computed via resampling data pooled across participants. See Table S1 for p values.

Increasing memory load did not change the size of the best-fit surfaces to the target representations within WM reconstructions that were based on activation patterns in occipital and posterior parietal ROIs (Fig. 4D; V1-IPS0; all statistics computed via non-parametric resampling methods and Bonferroni corrected for multiple comparisons; Table S1; see Experimental Procedures). However, fit surface size did increase with memory load in anterior parietal (IPS2–3) and frontal (sPCS) ROIs. Note that in these ROIs we did not observe strongly disjoint target representations during R2 trials (Fig 3C), so these size increases may partially reflect an inability to separately quantify the representation of each location. It is likely that a larger display and more stimulus separation would enable a more accurate reconstruction and quantification of each remembered target representation in these anterior parietal and frontal areas (like in the early visual and posterior IPS ROIs). We evaluated the possibility that observed size increases may be partially an artifact of coregistering reconstructions and averaging over target positions on R2 trials, even if the “true” target representations are constant in size, by simulating reconstructions under the null assumption that target representations were equal in size across memory load conditions and performing an identical coregistration and quantification procedure as that used in Figure 4. These simulations determined that fit target representation size is artificially inflated by 8.62% on average due to the coregistration and averaging procedure. Importantly, our empirically observed size expansion in these regions (IPS2: 24.8%, IPS3: 32.7%, sPCS: 19.6%) was substantially larger than that induced by the analysis procedure itself (see Fig. S4E, Supplemental Experimental Procedures), suggesting there are still important changes in target representation size across memory load conditions.

The amplitude of best-fit surfaces decreased with increasing memory load in striate and extrastriate occipital (V1-hV4) and posterior parietal (IPS0–1) ROIs, consistent with predictions from a model in which increasing memory load results in lower gain of population-level representations of remembered stimuli [14, 15]. In contrast, fit amplitude trended towards increasing with greater memory load in anterior parietal (IPS2–3) and frontal (sPCS) ROIs (trend defined as p < 0.05, uncorrected for multiple comparisons). This latter result is consistent with previous demonstrations that average delay-period activation levels increase in frontoparietal ROIs with memory load [6, 20, 21] (Fig. S1). Furthermore, simulations confirm that the fit amplitude parameter captures changes in the amplitude of the target representation and is independent of changes in baseline or size (Fig. S4).

Finally, the non-spatial baseline parameter significantly increased with memory load in posterior parietal ROIs (IPS0–1). The fact that non-spatial baseline levels increased only in IPS0–1 with greater memory load suggests that previously documented univariate BOLD response increases in the more anterior parietal and frontal ROIs (Fig. S1A, IPS2–3, sPCS) likely correspond to a spatially focal change in target representation amplitude as opposed to spatially uninformative baseline modulations.

We observed population codes for remembered spatial positions in all of the ROIs we examined, and the representations of remembered locations within these reconstructed images changed in different ways with increasing memory load (Figs. 3–4). However, the activation pattern across all these ROIs may provide additional information above and beyond the activation pattern within any individual ROI, and reconstructions computed using all these across-ROI modulations may be more closely associated with behavioral memory load effects than reconstructions computed from individual ROIs alone (on the assumption that mnemonic fidelity is a function of information represented across multiple brain regions). We tested this by computing reconstructions as before (Fig. 2), but using all voxels from the 10 ROIs in each participant (importantly, because this is a multivariate analysis, this is not equivalent to averaging reconstructions across all ROIs, see Supplemental Experimental Procedures). Comparing target representations within these WM reconstructions computed using the combined ROI (Fig 4C–D, “All voxels combined”) across memory load conditions revealed each of the significant results found in the ROIs when analyzed individually (Fig 4C–D): size broadened, amplitude decreased, and baseline increased when two items were remembered compared to when one item was remembered (all p < 0.001, resampling test). As an additional exploratory analysis, we evaluated how these target representations (Fig. 4C–D) were related to behavioral performance by computing and quantifying target representations within WM reconstructions as described above using data from each participant, ROI, and memory load individually. These results are presented and discussed in Figure S2B–C.

Discussion

Here, we employed an image reconstruction approach implemented using a multivariate inverted encoding model [8, 16, 26–29] to reconstruct the contents of spatial WM based on activation patterns in occipital, parietal, and frontal regions of human cortex. Prior studies have used measures like classification accuracy to correlate behavioral performance with the discriminability of neural activation patterns [6, 13]. Although these analyses have many advantages due to a relative lack of model assumptions, changes in decoding accuracy may result from many different types of neural response pattern modulation [26, 30]. In contrast, by assuming a set of spatial basis functions, our method allows us to assess whether each region encoded information about the location of a remembered stimulus (e.g. [5, 31]), as well as visualize and quantify the characteristics of these covert representations of target locations and relate different aspects of these quantified representations to behavioral performance (e.g., [8, 16, 26, 28]). In addition, these findings reinforce the importance of measuring the effect of cognitive manipulations on population-level estimates of mnemonic representations rather than on particular properties of the underlying neural generators, as these population-level representations can be more closely associated with cognition and behavior than activity changes in single neurons or voxels [8, 16, 26–30, 32–34].

These image reconstruction and quantification analyses revealed lower amplitude and, in some anterior parietal and frontal ROIs, broader target representations with increasing memory load (Fig. 4). From an information theoretic perspective, response variability (i.e., inter-trial variability in the reconstructed images) has two components: signal entropy, which is variability associated with experimental manipulations (remembered location), and noise entropy, which is variability not associated with experimental manipulations. The decrease in target representation amplitude under increased memory load should lead to less variability that is related to the remembered location(s) and thus to a decrease in the signal entropy and information about the remembered location. An increase in target representation size should also decrease signal entropy, as increased size leads to more overlap between target representations for different locations which would decrease the ability of the population code to discriminate between locations. In contrast, baseline shifts should not strongly influence information content as an additive shift in the entire reconstruction does not change signal entropy [14, 16, 35] (Fig. S3). Thus, the observation of higher amplitude target representations corresponds to higher information content of population codes about a spatial position [14–16, 33–35] (Fig. S3), and may be a consequence of changes in delay-period neural gain associated with neurons tuned to remembered locations [14]. In addition, modest increases in target representation size in anterior IPS and sPCS may reflect poorer mnemonic fidelity within particular ROIs, echoing previous results that the dispersion (analogous to size here) of reconstructed profiles of remembered features (e.g., orientation) correlates with behavioral performance [8, 26, 28]. However, future work using larger spatial stimulus arrays may help to more accurately disentangle and characterize multiple WM representations in anterior IPS and sPCS.

We were able to reconstruct the covert contents of spatial WM not only in occipital [4, 6–10, 13] and posterior parietal regions [10, 13], but also in anterior parietal and frontal cortex [5, 11]. These widespread modulations raise the possibility that distributed WM representations can be optimized to differentially contribute to complementary sensory (e.g., target localization) and motor (e.g., eye movements, reaches) behaviors. Consistent with this idea, a recent demonstration that induced alpha oscillations measured with scalp EEG (which are often thought to reflect synchronized activity of large-scale cortical networks [36]) can be used to reconstruct remembered orientations also suggests that long-range, interacting representations across much of human cortex support the maintenance of information in WM [28]. The successive representations of spatial position reported here may thus allow for a common coordinate system with which low-level stimulus features (such as spatial position and color) that are represented in occipital cortex are bound with spatial motor plans (such as eye movements and arm reaches, [37]) that are more closely associated with representations in parietal and frontal cortex.

Experimental Procedures

Functional magnetic resonance imaging

We scanned each participant for 4 sessions, each lasting 2 hrs. Each session included runs of the spatial WM task (Fig. 1), an independent spatial “mapping” task (Fig. 2A–B, Supplemental Experimental Procedures, [16]), and a visual localizer task (5 mins each).

Encoding model: reconstructing contents of spatial WM

We modeled the response of each voxel as a linear combination of 36 spatially selective information channels (see [16]; Fig. 2; Supplemental Experimental Procedures). Using a separate set of training data during which we presented a flickering checkerboard “mapping” stimulus at different locations on the screen (Fig. 2A–B), we estimated the relative contribution of all 36 information channels to the observed signal in each voxel using ordinary least squares regression (Fig. 2C). Then, using all of these measured “channel weights” across a given ROI, combined with the multivariate pattern of activation measured from that ROI during performance of the main spatial WM task (Fig. 1A), we computed the channel responses which were most likely to produce the measured pattern of activation (Fig. 2D). We combined these computed channel responses and the spatial filters (information channels) to produce reconstructed images of the spatial WM contents within each ROI for each measured pattern of activation (Figs. 3–4: activation patterns measured 6.75–9 s after target onset, Movies S1–S2: activation patterns measured at each time point during the trial).

Quantifying target representations in WM reconstructions

We fit a surface to each reconstruction that was allowed to vary in its size, amplitude and baseline (Fig. 4 A–B). Its center was constrained to be the position, in visual field coordinates, with the highest local reconstruction amplitude (local average within a 0.5° radius).

Statistics

For group-level analyses (Fig. 1C–E, Fig. 4D, Fig. S1B), we combined data from all participants within a given ROI and memory load condition and resampled all trials with replacement and computed a mean measurement value for that resampling iteration (Fig. 1C, behavioral recall error; Fig. 1D, behavioral accuracy; Fig. 1E, response time; Fig. 4D, target representation fit parameters; Fig. S1B, mean BOLD signal). We repeated this procedure 1,000 times to produce a resampled distribution of each measured value for each memory load condition. We computed p-values for each ROI and each parameter as the two-tailed probability of observing an effect in the opposite direction of the mean effect observed. Comparisons are Bonferroni-corrected across ROIs for each parameter (Fig. 4D,10 comparisons) or across all comparisons performed (Fig. S1B, 30 pairwise comparisons). All error bars are 95% confidence intervals derived from these resampled distributions unless indicated otherwise (Fig. S1A).

For exploratory individual-participant analyses (Fig. S2C), we performed an identical procedure but resampled only across each participant’s data when computing confidence intervals.

Highlights.

• We reconstructed images of remembered visual stimuli

• We found mnemonic representations in occipital, parietal and frontal cortex

• When multiple items were remembered, target representations decreased in amplitude

• Changes in amplitude echoed changes in behavioral performance