Multiple spatial frames for immersive working memory

Abstract

As we move around, relevant information that disappears from sight can still be held in working memory to serve upcoming behaviour. How we maintain and select visual information as we move through the environment remains poorly understood since most laboratory tasks of working memory rely on removing visual material while participants remain still. We used virtual reality (VR) to study visual working memory following self-movement in immersive environments. Directional biases in gaze revealed the recruitment of more than one spatial frame for maintaining and selecting memoranda following self-movement. The findings bring the important realisation that multiple spatial frames support working memory in natural behaviour. The results also illustrate how VR can be a critical experimental tool to characterise this core memory system.

Introduction

Imagine gaining possession of the ball in your favourite game and turning to approach the goal. As you turn, team members disappear from sight. Checked by defenders, and with no time to look around, you make a no-look pass (blind pass) to one of those team members. Your success depends on two core cognitive abilities: to maintain previously visible team members in memory (after turning) and to select the appropriate team member (given the defence).

We have poor understanding of how we maintain and select visual information in working memory ¹ in the context of natural behaviour ^2–7, when information disappears out of sight due to self-movement. This is because, in the laboratory, maintenance ^8–12 and selection ^13–15 of visual information in working memory are traditionally investigated in static settings in which visual material is briefly “flashed” on a computer monitor while participants remain still. In such constrained settings, visual material is removed from view artificially, and the spatial relations between the observer and the memoranda remain fixed. In contrast, following self-movement, visual material usually remains present in the environment, but is rendered out of sight because the spatial relations between observer and environment change ^16,17.

Such cases raise interesting questions regarding the nature of the spatial reference frames^16,18–25 that support the maintenance and selection of information in working memory in service of upcoming behaviour. For example, it is well-established from studies in long-term ^26,27 and trans-saccadic memory ^28–31 that multiple spatial frames (such as egocentric and allocentric) cooperate. This literature has brought the important realisation that usually not a single, but multiple spatial frames are constructed which enable efficient actions in our environment ³². However, little is known about the natural use of and reliance on equivalent spatial frames in immersive working memory – when memorised information disappears from view due to our own movements but where this information continues to hold relevance for guiding ensuing behaviour.

Do memoranda in visual working memory following self-movement remain anchored in their spatial positioning relative to other memoranda at encoding, or are memorised locations updated ‘in mind’ to reflect locations in the external environment? Moreover, might more than one spatial frame be used for maintaining and selecting visual contents in working memory, and does it matter how the visual contents disappear from sight (removed artificially vs. following self-movement)?

Here, we used virtual reality (VR) to answer these questions, bypassing traditional laboratory-task constraints to investigate working memory in an immersive context following self-movement.

The recent discovery that small directional biases in gaze track attentional focusing of memorised visual locations ^33,34 provided an exciting new sensitive and continuous metric for investigating the spatial frame(s) used for maintaining and selecting visual information in working memory following self-movement. One particular strength of this approach was that it allowed us to do so without having to explicitly ask participants about the locations of their memory content.

Results

Internal selective attention biases gaze towards memorised item locations in virtual reality

As a first step, we confirmed that gaze bias could be measured when selecting visual contents in working memory in VR. Participants viewed two coloured tilted bars that appeared briefly on the front wall of the virtual room and reproduced the orientation of one bar after a working-memory delay (Fig. 1A). To cue which bar (item) to report, the colour of the central fixation cross changed to match that of the colour of either memorised bar. Participants then used a hand-held controller to reproduce the precise orientation of the cued bar on a dial appearing around fixation.

Figure 1. Basic working memory task in VR and gaze signature of selection in working memory.

A) Twenty-four participants were immersed in a virtual room and memorised the tilt of two coloured bars. After a delay, a change in fixation-cross colour cued participants to report the colour-matching item using the hand-held controller. B) Horizontal gaze after cues for left/right memory items (zero denotes perfect fixation; ±2 cm = ±0.57°). Shaded areas indicate ± 1 SEM. C) Gaze bias, expressed as “towardness”. Horizontal line indicates cluster significance ³⁵. D) Difference in gaze density following cues for right minus left items (400-1000 ms post-cue). Circles indicate item locations at encoding (centred at ±100 cm / ±28.7°). Inset for close-up. Data from no-rotation condition Experiment 1. See Supplementary Figure 1 for corresponding data from the no-rotation condition in Experiments 2-4.

Despite the cue and the dial appearing centrally (and despite us never asking participants about the location of the cued memory item), participants’ gaze became biased in the direction of the memorised location of the cued (selected) memory item. When the cued item had previously occupied the left position, gaze was biased to the left, whereas when the cued item had occupied the right position, gaze was biased to the right (Fig. 1B). Our measure of towardness captured this “gaze bias” in a sensitive and time-resolved manner (Fig. 1C; cluster-based permutation analysis ³⁵; cluster p < 0.001; BF₁₀ = 24738.79 for the 400-1000 ms post-cue “selection” period). A density map of gaze positions (contrasting selection of items encoded on left vs. right) confirmed that this bias was constituted by small displacements in gaze toward the original location of the selected memorandum (Fig. 1D), in line with our prior report ³³. Note how this phenomenon is distinct from prior reports of gaze shifts all the way to the original location of the memorised objects, such as those observed in the context of long-term memory retrieval and imagery ^36–38. Rather, they resemble (related) reports of small biases in fixational gaze behaviour, as also seen during attentional focusing in perceptual tasks ^39–41. All four working-memory experiments replicated this basic pattern of results in the corresponding no-rotation condition (Supplementary Figure 1).

The sensitivity and reliability of the spatially indexed gaze bias in VR pave the way for investigating spatial working-memory frames in immersive contexts involving self-movement. In what follows, we present three manipulations (https://doi.org/10.6084/m9.figshare.16859656.v1), which directly and systematically varied the spatial relations between the observer and the environment in an increasingly realistic manner. The series of experiments (Supplementary Table 1 for an overview) culminated in the central discovery that multiple spatial frames support immersive working memory during maintenance and selection.

Gaze biases reveal multiple spatial frames for immersive working memory following rotation

In the first experiment, we manipulated the relative spatial positions of the observer and environment by rotating the room during the working-memory delay. The front wall, on which the bars had been presented at encoding, moved either to the left or the right of the participant during the memory delay (Fig. 2A; https://doi.org/10.6084/m9.figshare.16859728.v1). Participants themselves remained in place and were asked to hold fixation on a central cross, which remained fixed. We confirmed that participants could still perform the working-memory task well following a room rotation (Supplementary Figure 2).

Figure 2. Spatial frames for maintaining and selecting visual information following self-movement.

A) Horizontal gaze following cues for left/right items, separately for left/right room-rotation conditions in Experiment 1 (https://doi.org/10.6084/m9.figshare.16859728.v1). B-C) Gaze bias relative to whether the left/right item was cued (B), or whether the items had been presented on the left/right wall prior to rotation (C). D-I) Equivalent data following participantrotation conditions in Experiments 2 (D-F, https://doi.org/10.6084/m9.figshare.16859743.v1) and 3 (G-I, https://doi.org/10.6084/m9.figshare.16859857). In comparison to Experiment 2, in Experiment 3 items remained visible during rotation, becoming “out-of-view” because of self-movement (https://doi.org/10.6084/m9.figshare.16859656.v1). Horizontal purple line (I) indicate comparison between Experiments 2 and 3. Grey timeline boxes (top left) indicate rotation-manipulation and cue-locked-analysis windows. Rotation ended 1750-2250 ms prior to cue-onset in Experiment 1, and 1150 ms in Experiments 2-3. The earliest sample (at -500 ms) in our analysis window in the post-rotation memory period was therefore always 650 ms after the fixation cross had stabilised in its new position (and always 835 ms after the bars were removed in Experiment 3 in which the bars remained present during the first part of the rotation). Shaded areas indicate ± 1 SEM.

What is the relevant spatial frame for working memory following rotation? Toward what location will gaze be biased? For example, following a rightward room rotation, the environmental location associated with the “left” memory item will now be to the “right” of the participant. To formalise this question, we considered two possible spatial frames used for working memory. One pertains to the items’ relative positioning as encoded into memory (i.e., the left vs. right item, regardless of rotation). The other pertains to their locations in the external environment (e.g., at the left vs. right wall, contingent on rotation). We refer to these spaces as “internal-content space” and “external-environment space”, respectively (illustrated in Fig. 2B,C). Because we manipulated left/right-item cues and left/right rotations orthogonally, we were in the position to track evidence for both spatial frames independently.

Experiment 1 tested whether gaze bias remained anchored to the relative original positions of the items, or instead was updated to track the assumed current location of the item in the environment. To quantify this, we constructed separate measures of towardness for each of the aforementioned spaces: one indexing gaze bias toward the left/right cued item (internal-content space) and one indexing gaze bias toward the wall containing the items before the room rotated to the left or right (external-environment space). Following room rotation, we observed a clear gaze bias in internalcontent space (Figure 2B; cluster p < 0.001; BF₁₀ = 3442.982 for the 400-1000 ms post-cue selection period). Even after the room had rotated, participants’ gaze after the cue was still pulled toward the relative left/right positions in internal-content space (Fig. 2A-B, cf. Fig. 1B-C). In contrast, there was moderate evidence (BF₁₀ = 0.26) for a lack of gaze bias in the external-environment space during the post-cue selection period (400-1000 ms) and anecdotal evidence (BF₁₀ = 0.35) for no bias during the - 500-0 ms pre-cue “maintenance” period (Figure 2C; p > 0.612 for all clusters of the full time-window). The same pattern was observed when the items remained visible during the room rotation (Supplementary Figure 3).

In Experiment 2, we increased the ecological validity of the manipulation. Rather than rotating the room, we instructed participants to turn themselves during the working-memory delay (Fig. 2D; https://doi.org/10.6084/m9.figshare.16859743.v1). We asked participants to follow the fixation cross, which moved from the front wall to the wall on the left or right. This yielded equivalent changes in the spatial relations between the memorised information at visual encoding and after rotation as in Experiment 1, but this time induced by self-movement. Participants were still able to perform the task well above chance (Supplementary Figure 2).

We again found a clear gaze bias in internal-content space, after the cue to select the left vs. right item (Fig. 2D-E; cluster p < 0.001; BF₁₀ = 28.61 for the 400-1000 ms post-cue selection period). Interestingly, we now also started to see signs of a gaze bias in the external-environment space (Fig. 2F). However, the evidence for this this bias was anecdotal in Experiment 2 (cluster p = 0.046; BF₁₀ = 1.94 for the 400-1000 ms post-cue selection period and BF₁₀ = 0.5 for the -500-0 ms pre-cue maintenance period).

In Experiment 3, the final critical step was taken to simulate real-world visual working memories formed by self-motion in the environment. Like in most natural situations, the memory items remained in the environment, rather than only being flashed briefly (Fig. 2G; https://doi.org/10.6084/m9.figshare.16859857) (differing from Experiments 1-2, as well as from traditional laboratory tasks of visual working memory). The memory items remained visible on the front wall until after participants turned to an adjacent wall as prompted. Accordingly, in Experiment 3, the self-movement (rotation) of the participant made the visual information disappear from sight (i.e., “memory because we move away”). To avoid any “looking back” as a potential strategy ^5,42, however, the bars were removed from the wall eventually (after they had disappeared from view through selfmovement), which was made explicit to the participants. Thus, during and after the rotation, the impression was that the items remained on the wall from which they rotated away (even if they knew that the items were removed after becoming out of view). We again found that participants could perform the task well (Supplementary Figure 2).

As in Experiments 1-2, we observed a clear gaze bias in internal-content space, after the cue to select the left or the right memory item (Fig. 2G; towardness in Fig. 2H; cluster p < 0.001; BF₁₀ = 1447.6 for the 400-1000 ms post-cue selection period). Critically, however, this time, we also observed a pronounced gaze bias in the direction of the location of the memory items in the external environment (Fig. 2G; towardness in Fig. 2I; cluster p < 0.001). Interestingly, the bias in external-environment space was already present prior to the selection cue (BF₁₀ = 16.25 for the -500-0 ms precue maintenance period), during the maintenance of both visual items in working memory after the rotation. It did however persist throughout the selection period, suggesting that the two spatial frames cooperated when visual content was selected (BF₁₀ = 6.93 for the 400-1000 ms post-cue selection period). If the participant rotated to the wall on the left, gaze was pulled slightly rightward, back toward the central wall that had contained the two bars at encoding (Fig. 2G, left panel). Likewise, if the participant rotated to the right, gaze was pulled leftward (Fig. 2G, right panel). This bias in external-environment space comprised a “pull” of gaze (Figure 3), similar in magnitude to that subsequently observed for the internal-content bias during item selection (Fig. 1D; Supplementary Figure 1).

Figure 3. Density plot for Exp. 3.

Difference in gaze density between rotations to the left vs. right wall in the pre-cue period (-500 to 0 ms) as indicated in the grey box in the timeline at the top. The inset shows a close-up, overlaid on the fixation cross. Similar to the gaze bias in internal-content space (Fig. 1D) this bias in space comprised a “pull” of gaze (as opposed to large saccades) toward the wall that had contained the memorised items (external-environment space).

We could compare data from Experiments 2 and 3 directly (grey and black time courses in Fig. 2I), because we ran these experiments as a within-participant manipulation (Exp. 2 vs. Exp. 3) with the same locomotive demands. We found that the bias during the maintenance period in external-environment space was significantly larger when the items disappeared from sight because of the self-movement, as opposed to before self-movement (Experiment 3 vs. 2; Fig. 2I, purple horizontal line; cluster p = 0.01; BF₁₀ = 14.9 for the -500-0 ms pre-cue maintenance period; for the 400-1000 ms post-cue selection period there was moderate evidence, BF₁₀ = 0.3, in favor of no bias). The more realistic nature by which the memory items disappeared from view (during vs. preceding the participant’s rotation) thus appeared to increase the reliance on external-environment space – as reflected in our gaze measure – during memory.

These biases were observed after the visual array was no longer available and during a period with no stimulation. The first point in the pre-cue period we analysed and visualised (at -500 ms before the cue) was always 835 ms after the items were removed from the wall and 650 ms after the fixation cross stabilised in its new position (at 1335 and 1150 ms pre-cue respectively). Participants also knew that the items were removed after becoming out of view. Thus, we associated gaze behaviour in our experiment in this period to directing attention in memory, rather than actually looking back at physically present items ^5,42.

We could replicate the reliance on the two spatial frames from Experiment 3 in replication-Experiment 3b (Supplementary Figure 4). This further allowed a within-subject comparison of the bias in external-environment space between Experiment 3b and its matched control Experiment 3c, from which we stripped the necessity to use working-memory while retaining the same locomotive demands (Supplementary Figure 4A). In a similar vein as the comparison between Experiment 2 and 3, this confirmed that this bias in the external environment space was modulated by memory demands (Supplementary Figure 4C; cluster p = 0.031; BF₁₀ = 5.63 for the -500-0 ms pre-cue maintenance period), being stronger in rotation trials with (Experiment 3b) vs. without (Experiment 3c) demands on memory. Importantly, while these data do not rule out the possible contribution of other factors to the overall bias in the external-environment space (such as VR-related ⁴³ or other rotation-related effects on eye movements), the comparisons between experimental conditions differing only in memory demands reveal that at least part of this bias must be attributed to the memory-related demands of the task.

Memory load increases reliance on multiple spatial frames for immersive working memory

If the biases in gaze reflect memory reliance, then we would expect them to increase with memory load ¹¹. To test this, we conducted an experiment in which we manipulated memory load (Experiment 4; Fig. 4). Participants maintained two or four items and rotated away from the memory material that remained visible throughout the rotation (as in Experiments 3 and 3b). Moreover, by using item-configurations as depicted in Figure 4A, we ensured that the last visible item during the rotation was matched between load 2 and 4 conditions.

Figure 4. Increasing memory demands increases reliance on both spatial frames.

A) In Experiment 4, we manipulated the number of to-be-maintained items (2 vs 4), while again asking 24 participants to rotate away from the memory material that remained visible throughout the rotation. Four bars were always presented (top, left, right, bottom), and participants were asked to remember only the coloured bars. In load 4 all bars were coloured. In load 2, only the left and right bar were coloured, and the top and bottom bars would be grey (serving as fillers). B-C) Gaze bias relative to whether the left/right item was cued (B), or whether the items had been presented on the left/right wall prior to rotation (C). Horizontal purple lines (B, C) indicate comparison between memory load 2 and 4. Shaded areas indicate ± 1 SEM. D) Bias in external-environment space averaged for the -500-0 ms pre-cue maintenance period and the 400-1000 ms post-cue selection period. Lines represent individual participants and error bars indicate SEM.

The results confirmed a dependence of the towardness biases on memory demands. In addition to replicating the reliance on two spatial frames, gaze biases were significantly modulated by memory load.

First, there was a clear gaze bias in internal-content space, after the cue to select the left or the right memory item (Fig. 4B) for both memory load 2 (grey time course in Fig. 4B; cluster p < 0.001; BF₁₀ = 3598.309 for the 400-1000 ms post-cue selection period) and memory load 4 (black time course in Fig. 4B; cluster p < 0.001; BF₁₀ = 13737.02 for the 400-1000 ms post-cue selection period). The comparison of gaze biases in loads 2 and 4 confirmed that increasing memory load increased the reliance on internal-content space (Fig. 4B, purple horizontal line; cluster p < 0.001; BF₁₀ = 27.119 for the 400-1000 ms post-cue selection period).

We also again observed a gaze bias in the direction of the location of the memory items in the external environment for both memory load 2 (grey line in Fig. 4C; towardness in Fig. 2I; cluster p < 0.001; BF₁₀ = 4.145 for the 400-1000 ms post-cue selection period and BF₁₀ = 61.539 for the -500-0 ms pre-cue maintenance period) and memory load 4 (black line in Fig. 4C; towardness in Fig. 2I; cluster p < 0.001; BF₁₀ = 22772.24 for the 400-1000 ms post-cue selection period and BF₁₀ = 889.342 for the -500-0 ms pre-cue maintenance period). Critically, the comparison of gaze biases in loads 2 and 4 confirmed that increasing memory load also increased the reliance on external-environment space (Fig. 4C-D, purple horizontal line; cluster p < 0.001; BF₁₀ = 16607.71 for the 400-1000 ms post-cue selection period and BF₁₀ = 46.082for the -500-0 ms pre-cue maintenance period). These patterns were observed despite the fact that rotation demands (and the last visible item during rotation) were equated between loads, thus providing further evidence for a link between these biases and memory-related processes.

Discussion

Using VR and systematically approximating the immersive nature of everyday working-memory demands, we learned that visual working memory relies on not one, but at least two spatial frames. These frames were observed while participants maintained and selected visual contents in working memory. We speculate that the two spatial frames may reflect (at least in part) complementary roles served by both maintenance and selection following self-movement in our tasks – pertaining to, respectively, a global frame for remembering where memorised information is (in the environment), and a specific frame that allows to discriminate and select which information (in memory) is currently needed to guide behaviour.

We found clear reliance on the internal-content space during the selection period in all tested cases. However, reliance on the external-environment space – as reflected in gaze – varied according to the perceived availability of the items and self-movement. Whereas participants engaged in memory maintenance in each experiment, the towardness gaze bias was especially pronounced in naturalistic cases following self-movement when items seemed to remain available in the environment. We believe that the use of these two spatial frames for specific working-memory operations will ultimately depend on the nature of the task at hand. Indeed, when memory load increased from two to four items, gaze bias in the external-environment space became amplified during both maintenance and selection. These results directly show that internal selection can rely on multiple spatial frames, and that this joint reliance is increased with higher memory demands.

From related research on long-term memory, we have come to appreciate how multiple reference systems cooperate in complex ways to support natural long-term memory representations ^23,25,26,44. Amongst the various critical determinants on the relative importance of each frame of reference are landmarks, spatial structure, and viewpoints at encoding and retrieval. We expect these factors to carry over their importance in the realm of working-memory representations, though this remains to be addressed in future research. To this end, the current approach provides an ideal starting point for also probing the dynamics of maintenance and selection of working-memory content in the context of even richer environments, with even richer spatial contexts, to better approximate the type of situations faced by our working-memory system in everyday life.

In the study of the spatial frames used for memory, a common approach is to ask participants explicitly where they remember something to be and, for example, to point to it ^44,45. Such approaches can uncover the (multiple) spatial frames that come into play when deliberately reporting a memorised-object’s location. Such approaches, however, do not address whether these frames are also used during everyday behavior when space is an implicit component of the task at hand. In our study, we never asked participants about the memorised location of their working-memory contents (we only asked about the orientation of the memory content whose colour matched the cue; regardless of its spatial position at encoding). Yet we found clear spatial biases in gaze behavior. Through these implicit biases, we were able to interrogate and uncover the natural use of two complementary spatial frames for immersive working memory following self-movement.

Studying working memory after self-movement is important not only because it better approximates reality ^4,5,46; it can also expose phenomena that otherwise go unnoticed. The gaze bias in external-environment space became apparent particularly when the visual information disappeared from view as a result of the participant’s own rotation. This, in turn, helped reveal that overt gaze behaviour is sensitive not only to internal processes of content selection (as per our previous report of this gaze bias ³³) but may also be sensitive to content maintenance in immersive visual working memory.

In recent years, VR ⁴⁷ has become a go-to tool for investigating long-term memory and spatial navigation ^21,48–53. In contrast, the use of VR for investigating working memory remains rare⁵. Our study makes clear that VR – here combined with eye-tracking – is indispensable for understanding the nature and mechanisms of this core memory system in the context of natural behaviour.

Methods

Results of Experiments 1-4 are presented in our main text, while the remaining experiments are presented in our Supplementary Information.

Participants

We ran seven experiments, each with a sample size of 24 healthy human volunteers. We separately recruited participants for Experiment 1, for Experiments 2-3, for Experiments 3b-3c as well as Experiment 4 and the Control Experiment (Exp. 1: mean age = 25.8, range = 18-40, 20 female, all right-handed; Exp. 2-3: mean age = 26.5, range = 19-36, 19 female, all right-handed; Exp. 3b-3c: mean age = 25.5, range = 19-37, 13 female, 1 left-handed; Exp. 4 & Control Experiment: mean age = 26.7, range = 21-31, 14 female, all right-handed). Sample size was set based on our prior study that contained 4 experiments using a similar outcome measure ³³ and revealed robust results with 20 to 25 participants. Here we set sample size to 24 to enable counterbalancing the order of tasks in Experiments 2-3 and 3b-3c. In total (across all experiments), four participants dropped out during the practice session because of nausea ^43,54,55. All participants had normal or corrected-to-normal vision and reported no history of neurological or psychiatric disorders. Participants received financial compensation (£10/h) and provided informed consent prior to participating in the experiment. Protocols were approved by the local ethics committee (Central University Research Ethics Committee #R64089/RE001).

Apparatus and Virtual Environment

Participants wore an HTC Vive Tobii Pro VR integration with a built-in binocular eye tracker with an accuracy of approximately 0.5° visual angle. We tracked gaze position in 3D space at a sampling rate of 90 Hz. Gaze position in 3D space was obtained by intersecting the gaze vector with items in the environment (https://docs.worldviz.com/vizard/latest/#Tobii.htm). The head-mounted display (HMD) consisted of two 1080 × 1200 pixel resolution OLED screens (refresh rate = 90 Hz, field-of-view = 100° horizontally × 110° vertically). Locations of the headset and the hand-held controller were tracked with sub-millimetre precision using two Lighthouse base stations that emitted 60 infrared pulses per second, which were detected by 37 infrared sensors in the HMD and 24 in the controller. Tracking was further optimized by an accelerometer and a gyroscope embedded in the HMD. A trigger button (operated with the index finger) and a touchpad button (operated with the thumb) on the wireless controller were used for response collection.

The virtual environment was presented and rendered with Vizard 6 by WorldViz on a high-performance laptop running Windows 10. The environment consisted of a 420 x 420 cm room with a ceiling height of 250 cm. The same homogenous texture (https://osf.io/stf4w/) was applied to all walls, creating a concrete-like appearance (Fig. 1A; https://doi.org/10.6084/m9.figshare.16859656.v1). Participants were situated in the centre of the cubic room. Three-dimensional cylinder stimuli (which we will refer to as “bars” for simplicity) were presented at a distance of 200 cm from the centre of the room, 10 cm in front of the wall.

Procedure and tasks

Upon arrival, participants were familiarized with the HMD, the wireless controller, and the lab space. Participants completed a short practice session, including twenty-four practice trials, during which they were accustomed to both the HMD and the testing procedure.

All experiments involved the same basic procedure (Fig. 1A). Each trial began with the presentation of a fixation cross for 500 ms. The fixation cross subtended approximately 3.4° of visual angle (12 cm) in both width and height. Two oriented bars then appeared 100 cm to the left and to the right of the central fixation cross (200 cm in front of the participant). Bars were 50 cm (14.25°) in length and 5 cm (1.425°) in width and depth. One of the bars was always red, and the other always blue, which was randomly sampled. Bars were presented for 250 ms in Experiment 1, for 1000ms in the Control Experiment, for 500 ms in Experiment 2 and for 1600 ms in Experiments 3-3b (no bars were presented in Experiment 3c). In Experiment 4, four bars were presented (green, purple, blue, orange, or grey) for 2100 ms.

The participant had to memorise the orientation of the bars (drawn independently at a random orientation), in order to reproduce the orientation of either bar after a working memory delay. At the end of the trial, a colour change of the central fixation cross (the “memory cue”) instructed participants to select and report back the corresponding bar.

The critical manipulations occurred during the working-memory delay, after encoding the visual material but well before we cued either memory item for report (see Supplementary Table 1 for an overview). The environment either rotated around the participant (Exp. 1) or the participant was asked to rotate to face an adjacent wall, by following the movement of the fixation cross (Exp. 2-4). Video material of example trials are presented in https://doi.org/10.6084/m9.figshare.16859656.v1. In all experiments, one third of trials contained a left rotation, one third of trials contained a right rotation, and one third of trials contained no rotation (randomly intermixed).

In Experiment 1, the environment rotated around the participant (i.e. Euler rotation) at 180 deg/sec for 500 ms for a total rotation of 90° (https://doi.org/10.6084/m9.figshare.16859728.v1). The rotation occurred immediately after the visual offset of the items. In a Control Experiment (Supplementary Figure 3), the items remained on the wall during the rotation, rendering the bars out of sight due to the rotation, rather than due to a sudden visual offset. In Experiment 1, the fixation cross remained positioned centrally in front of the participant, and participants performed this task seated. In the Control Experiment (Supplementary Figure 3), participants were standing to better match the critical Experiments 3-3b where the items also remained visual until after the rotation.

In Experiments 2-4, the environment remained stable and participants were asked to rotate instead (https://doi.org/10.6084/m9.figshare.16859743.v1 and https://doi.org/10.6084/m9.figshare.16859857). In rotation trials, the fixation cross started moving 500 ms (1000 ms in Exp. 4) after the onset of the visual bars, from the centre of the front wall to the centre of the left or right adjacent wall. Participants were in upright (standing) position and were instructed to follow the fixation cross, which required a rotation of head and torso. The movement of the fixation cross occurred at a speed of 220 cm/sec, which took ~1285 ms to travel a distance of ~283 cm. In Experiment 2 the visual bars were removed from sight 500 ms after encoding. In contrast, in Experiments 3, 3b and 4, the bars remained present on the front wall for an additional 1100 ms during the rotation. This created the visual impression that the items remained on the wall from which participants rotated away, rendering the bars out of sight due to the self-movement, rather than due to a sudden visual offset. The participants, however, were informed that the items would eventually disappear (after the rotation) and also experienced the disappearance during non-rotation trials.

Experiments 3 and 3b were identical in set-up, except that Experiment 3 was paired with Experiment 2, while replication-Experiment 3b was paired with the perceptual-control Experiment 3c (in both cases enabling within-subjects comparisons). In Experiments 2-3 participant were asked to follow the fixation cross with no further instructions, while in Experiments 3b-3c participants were explicitly instructed to follow the fixation cross by only rotating their head and torso. Despite this difference, the results support the same conclusion, regardless of whether we based our inference on the critical Experiment 3 (“participant rotates, items remain”) or the otherwise identical Experiment 3b. To confirm the pre-cue/post-rotation maintenance results from Experiments 3-3b depended on memory-demands, we compared working-memory Experiment 3b to perceptual-control-experiment Experiment 3c (Supplementary Figure 4A). Experiment 3c was procedurally identical to Experiment 3b; except participants were not required to maintain and select items from memory, as no bars were presented during the trial. Instead, participants were instructed to press a button as soon as the fixation cross changed colour – with the two potential colours associated with a different instructed button. Thus, what served as the memory cue in Experiments 1-3b, served as a perceptual target in Experiment 3c. The assignment of button (touchpad vs. trigger) to target-colour (blue vs. red fixation cross) was counterbalanced across participants.

Experiment 4 was similar in set-up to Experiment 3 but to provide further evidence for the influence of ‘memory demands’, we manipulated the number of to-be-maintained items (Fig. 4). Four bars were always presented (always at the top, left, right, bottom; see Fig. 4A), and participants were asked to remember only the coloured bars. In load 4 all bars were coloured. In load 2, only the left and right bar were coloured, and the top and bottom bars (that now served as fillers) would be grey. Load 2 and load 4 trials were randomly intermixed. We chose to add a central top and bottom item to move from load 2 to load 4, as this configuration allowed us to match load 2 and load 4 conditions with regard to the last visible item during rotation – that would always be the left or the right memory item.

The room rotation in Experiment 1 was followed by a further memory delay between 1750-2250 ms (uniformly sampled) before the memory cue was presented (1150 ms in the Control Experiment to match the critical Experiments 3-3b). In Experiments 2-4 the movement of the central fixation cross was always followed by an additional memory delay of 1150 ms before the memory cue (in Experiments 2-4) or perceptual target (in Experiment 3c) was presented. The earliest sample (at 500 ms prior to the cue) in our analysis window in the post-rotation memory period was thus always 650 ms after the fixation cross had stabilised in its new position (and always 835 ms after the bars were removed in Experiments 3, 3b and 4 in which the bars remained present during the first part of the rotation).

In each trial, both bars were equally likely to be cued for report, independently of their colour, orientation, or location. Participants had unlimited time to initiate a response after the memory cue onset. The response dial consisted of a circular torus (50 cm/14.3° in diameter + 1 cm tube radius) with two small spherical handles (3 cm radius) that could be re-aligned to match the memorised orientation of the cued bar. Dial-up was performed with the wireless controller, operated with the dominant hand. The orientation of the controller was directly linked to the response dial allowing for an intuitive orientation report. The response dial appeared on the screen only at response initiation (i.e. movement initiation) and was always positioned around the central fixation cross. Participants had a maximum of 2000 ms to complete their dial-up report and could complete the report earlier by pressing the trigger button with their index finger. Feedback followed immediately after response termination by presenting the performance (reproduction error scaled between 1 and 100, with 100 reflecting perfect reproduction) above the fixation cross for 500 ms. Finally, after feedback presentation, either the environment (Exp. 1, Control Experiment) or the participant (Exp. 2-4) rotated back to the initial position (or remained in position in no-rotation trials), lasting 500 ms in Exp. 1 and the Control Experiment, and 700 ms in all other experiments. The next trial always started after an inter-trial interval that was randomly drawn between 1.5 to 2 s.

Each experiment (except Exp. 4) contained 300 trials, separated into 5 blocks of 60 trials (50 trials overall per condition, given the factorial combination of rotation (left, right, no-rotation) x item side (left, right)). Experiment 4 contained 360 trials, separated into 5 blocks of 72 trials (15 trials overall per condition for memory load 4, given the factorial combination of rotation (left, right, no-rotation) x item location (left, right, up, down) and 30 trials per condition for memory load 2 (given there were only 2 possible locations: left and right). This resulted in an equal number of trials in both memory load conditions. In load 4 all four items were equally likely to be probed, just like in load 2 both items were equally likely to be probed. For the bias in internal-content space, we only analyzed trials in which the left/right item was probed. For the bias in external-environment space, we also included load-4 trials in which the top/bottom item would be probed.

A built-in eye-tracking calibration was conducted before the start of each block. Experiment 1 contained a single session of approximately 1 hour, whereas Experiments 2-3, 3b-3c and 4-Control Experiment each involved two consecutive sessions of approximately 1 hour each. For Experiments 2-3 and 3b-3c, the order of experiments was counterbalanced across participants.

Data analysis

Eye tracking samples were collected and saved into a csv file. All data are publicly available at https://osf.io/cj97y/.

Pre-processing and analysis procedures were straightforward and followed previous work on “gaze bias” during attentional focusing in memorised visual space ³³. The data from the left and the right eye were averaged, yielding a single time course of horizontal gaze position (x position) and a single time course of vertical gaze position (y position). We removed trials in which horizontal gaze position exceeded ±50 cm (14.35°) from fixation (i.e., further than half the distance to either item at encoding). We were confident that this procedure would not restrict our analysis of the “gaze bias”, as we had previously reported that this bias is constituted by changes in gaze position close to fixation ³³ – a finding we also confirmed in the current data (Fig. 1D). At this stage, one participant from Experiments 2-3 had to be removed from the analysis due to a high number of excluded trials (>50%). In the remaining participants, this number was much lower (Exp. 1: 7.1±1.8%; Exp. 2-3: 8.7±1.9%; Exp. 3b-3c: 9.0±2%; Exp. 4: 6.0±1.6%; Control Exp.: 13.9±3%; [M±SE]). Data were epoched from 500 ms before to 1000 ms after cue onset. Gaze time courses were smoothed by a 4-sample (44 ms) average moving window.

We compared trial-averaged gaze-position time courses between conditions in which the cued memory item occupied the left or the right position during encoding, separately for trials with no rotation, and for trials in which the items had been presented on the wall that was now left or right of the current wall (after rotation). To increase sensitivity and interpretability, we constructed two measures of “towardness”. One expressed gaze bias toward the relative location of the cued memory item in reference to the other memory item (“internal-content space”); the other expressed gaze bias (in rotation trials) toward the items’ locations in the external environment (i.e. toward the wall on which the items had been presented at encoding; “external-environment space”). Specifically, towardness in internal-content space was defined as the average horizontal gaze position following cues prompting the selection of right minus left memory items, divided by 2. As the items were displayed at a horizontal distance of 100 cm, the towardness values can be interpreted as a percentage towards the item. Towardness in environment space was calculated by subtracting the average horizontal gaze position when items had been presented on the wall that was to the right vs. left after rotation, divided by 2. For the load-4 condition in Experiment 4, we also included trials in which the top/bottom item were probed when considering the external-environment space, because the towardness in this space captures how horizontal gaze position relates to the position of the wall on which the items were presented prior to rotation. That is, in external-environment space, after self-rotation, all items are either to the left or right wall relative to the observed, irrespective of the relative vertical position of the items. Because cued item location and rotation direction were manipulated independently (i.e., a left/right rotation was followed equally often by a cue to select the left/right item), we were able to quantify gaze biases in internal-content space and in external-environment space independently.

For time series, statistical results were subjected to a “cluster mass” cluster-based permutation procedure ³⁵ implemented in R ^56,57, which controls for multiple comparisons across time points, while retaining high sensitivity. To follow-up findings (including null findings), we also averaged gaze towardness values within the pre-defined “selection” time window of 400-1000 ms after cue onset (as in ³³) as well as in the “maintenance” window -500-0 ms before cue onset, and conducted Bayesian t-tests ⁵⁸ with the default settings of the Bayes Factor package⁵⁹. Bayes factor values either indicated anecdotal (BF₁₀ > 1 and BF₁₀ < 3), moderate (BF₁₀ > 3 and BF₁₀ < 10), or strong (BF₁₀ > 10) evidence in favor of the alternative hypothesis ^60,61. Bayes factor values could also indicate anecdotal (BF₁₀ > 0.33 and BF₁₀ <1), moderate (BF₁₀ > 0.1 and BF₁₀ <0.33), or strong (BF₁₀ <0.1) evidence in favor of the null hypothesis.

For visualization purposes we constructed heat-maps of gaze position for samples within the time window of 400-1000 ms ³³. We created histograms without removing trials with gaze positions (>±50 cm) and without averaging samples. Two-dimensional kernel density estimation with a 20x20 cm bandwidth with an axis-aligned bivariate normal kernel was evaluated on a 200x200 cm square grid (the distance between the positions of the items on the wall of the environment). This procedure was performed separately for trials in which the probed memory item had different locations (left vs. right). We subtracted gaze-density values that were shared between these two conditions which allowed us to focus selectively on the differences between the conditions.

Data availability

All data are publicly available at https://osf.io/cj97y/.

Code availability

Code is available at https://osf.io/cj97y/.