Orthogonal-Rotational Dynamics Supports Efficient Encoding and Updating for Streaming Information in Working Memory

Abstract

We rely on the working memory (WM) to organize, store, and process the perpetual stream of information. Efficient encoding and processing of WM requires a framework that (1) separates individual memory items while accurately maintaining their temporal rank and (2) updates the sequence by discarding no-longer-needed items and accommodating newly arrived ones. To investigate the computational mechanisms underlying this functional implementation of WM, we analyzed the neural information representation in both a recurrent neural network (RNN) model and human subjects (n = 28, 18 males) under the same N-back WM task, which necessitates continuous encoding and updating of memory items. We discovered that an orthogonal-rotational dynamical framework facilitates memory encoding and updating, allowing both the RNN and brain to organize memory items efficiently. In the RNN model, we identified an orthogonal coding space where each memory item occupies a subspace corresponding to its ordinal rank. A rotational operation dynamically transfers information across these subspaces, updating memory while preserving their internal order. Overall, this orthogonal-rotational framework enables the network to store the information in a “first in, first out” manner. Remarkably, we also observed similar orthogonal-rotational dynamics in EEG signals recorded from the prefrontal areas of human participants engaged in the same task. These findings suggest a novel mechanism underlying the brain's ability to efficiently organize information stream for “online” processing and indicate that this strategy may be utilized by both biological and artificial neural networks for optimal information storage and updating.

Significance Statement

In our interactions with the world, the brain receives information as continuous streams, which we temporarily store and process using the working memory (WM). Efficient “online” processing in WM is therefore crucial, which requires preserving the order of incoming information and discarding no-longer-needed items to accommodate new ones simultaneously. The underlying mechanisms remain an open question. To address this, we conducted a comparative analysis of the neural activity in both a recurrent neural network (RNN) model and human brain under a N-back WM task and found the orthogonal-rotational dynamics in both of them. These findings revealed a novel mechanism of the brain to efficiently organize information streams for efficient “online” processing.

Introduction

In the daily experience, we are surrounded by various types of information steams organized as continuous sequences, such as language, music, or spatial routes (Dehaene et al., 2015). In order to treat them in cognition rapidly and efficiently for better interaction with the external environment, we rely on the working memory (WM), referring to the ability to hold the external information in mind temporarily (Curtis and D’Esposito, 2003; Mongillo et al., 2008; Hansel and Mato, 2013). Such capability to process information flow based on WM in the neocortex allows human and animals to integrate temporal information from past to present (Baddeley, 2003). However, because of the limited memory capacity (Eriksson et al., 2015; Constantinidis and Klingberg, 2016), the WM cannot directly store the long, successive information steam, but process information parts orderly in its workspace in real-time. This requires (1) efficient encoding to separate stimuli reaching at nearby times and (2) flexible updating to discard the previous information and to accommodate reaching one (Fig. 1A). How the WM system achieves such “online” cognitive memory processing is still an open question.

Figure 1.

Task and model. A, The schematic of the WM “online” processing in the information flow. When processing the streaming information (e.g., ABCDE) via WM, the brain needs to memorize and encode part of the information (e.g., BCD) in WM buffer, discarding the ones no longer needed (e.g., A) and accommodating the new ones (e.g., E). B, RNN model. The input is a 10 × 1 vector representing digits stimuli with one-hot encoded. The corresponding label is a 3 × 1 vector representing 0 (Mismatching), 1 (Matching), and 2 (Initialization) with one-hot encoded. The output is a 3 × 1 vector representing the probability distribution of model responses, taking the choice with the highest probability as the final decision. C, The test accuracy and the change of training loss in the 2-back and 3-back task. D, Left panel, Visualization of the averaged hidden state h₁ in the 2-back model. Right panel, Visualization of the averaged hidden state h₁ in the 3-back model. The hidden units performed mixed selectivity.

It has been proposed that the WM works as a short-term memory buffer in the neocortex, responsible for the organization and processing of memory information, as well as interacting with the hippocampus to accomplish information storage to the long-term memory (Jensen and Lisman, 2005; D’Esposito and Postle, 2015). At the low-dimensional representational level embedded in the neuronal population activity, the buffer encodes memory items in an orthogonal manner, reducing interference across information with different ordinal rank (Xie et al., 2022; Chen et al., 2024; Tian et al., 2024). However, previous studies mainly focused on the encoding of static sequence of items that did not involve constant memory updating. How the brain encodes and updates memory information rapidly in its buffer for continuous information stream still remains unclear.

The recurrent neural network (RNN) model provides a robust tool for investigating neural mechanisms of the brain. It has been demonstrated that various higher-order cognitive functions, such as memory, decision-making, or inhibitory control, can be effectively stimulated by the RNN model (Yang et al., 2019). In addition, previous studies have revealed similarities of neural representations between the RNN model and the brain when engaged in the same cognitive tasks. For example, during the decision-making process, dynamic trajectories and line attractors emerge in the activity of both the prefrontal cortex and the RNN model, reflecting the information integration and decision selection (Mante et al., 2013). The reward-based learning in the brain driven by the reward prediction error signal, and dopamine release can be modeled as a RNN-based reinforcement learning (RL) architecture (Wang et al., 2018). Recent advancements in cognitive simulation using the RNN model have provided new insights into more complex cognitive behaviors, such as learning-to-learn (Goudar et al., 2023) or the functional modulation of language (Riveland and Pouget, 2024). These results suggested the rationality of using the RNN model to investigate the neural mechanism of the brain cognition, inspiring us to explore the functional implementation in the context of the continuous sequential memory task in both artificial and biological neural networks.

Thus, to address the questions about the computational mechanism of the brain to encode and update memory information efficiently and rapidly during the successive information stream, we analyzed the information representation of both an RNN model and human subjects in the same sequential WM task—N-back task (Owen et al., 2005; Lamichhane et al., 2020; Pahor et al., 2022), requiring them to encode and update memory information continuously. In the low-dimensional representational space, we found a common computational framework of orthogonal-rotational dynamics implemented in both artificial and biological neural network. It supports information to be stored in the network in a “first in, first out” (FIFO) manner, allowing memory items in the information stream to be organized and “online” processed in an iterative manner and minimize the interference between them.

Materials and Methods

Task

We trained the network model and compared its representation of memory items with human participants in the N-back task. This task requires temporary encoding and storage of each stimulus and a continuous updating in the memory space. In each trial of the task, one digit was presented for the model or participants. They needed to memorize recent, adjacent N stimulus and determine whether the current stimulus matched the stimulus in the previous N trials back, where N is set to 2 or 3 in the study. If matching, the model should output 1 or the participants need to press the space on the keyboard for response as soon as possible. If mismatching, the model should output 0 or the participants did not respond. There is a special case that at the first N trials of the task, where the model or the participants cannot determine matching or mismatching, so the model should output 2 or the participants did not require to respond either under such initial situation.

In the model task, the digit input ranging from 0 to 9 and the output of 0, 1, 2 were one-hot encoded. Each trial corresponded with one time step in the model, and the model received an input stimulus and gave an output. The condition of matching and mismatching accounts for 50% equally. In the participant task, the black digit stimuli ranging from 1 to 9 were presented on a computer screen with white background (Alienware, resolution 1,920 × 1,080, 24.5 inches, 240 Hz refresh rate). The screen was placed in front of the participants ∼60 cm and the height of the screen was adjusted to align the participants’ straight sight. The stimulus of each trial lasted for 4 s and if the participants pressed the space for response, the stimulus would disappear instantly. The intertrial interval (ITI) between two adjacent trials was 0.5 s, during which a black cross fixation was presented. Correspondingly, the condition of matching and mismatching occupied 50% equally. The presentation of the stimuli was controlled by MATLAB (MathWorks) using the Psychophysics Toolbox Version 3 (Brainard). The accuracy was evaluated and EEG was recorded during the participants’ task.

RNN model and training

The RNN model comprised a fully connected population of 100 hidden units and received 10 × 1 one-hot encoded digits stimuli. The activity of hidden units followed the dynamical equation:

$${\mathit{h}}_{\mathit{t}}=f(W_{ih}\times {\mathit{x}}_{\mathit{t}}+b_{ih}+W_{hh}\times {\mathit{h}}_{\mathit{t}-1}+b_{hh}),(1)$$

where ${\mathit{h}}_{\mathit{t}}$ is a 100 × 1 vector representing the activity of hidden units and ${\mathit{x}}_{\mathit{t}}$ is a 10 × 1 vector representing the input stimuli at time step t. $W_{hh}$ is an 100 × 100 matrix of the recurrent connection weights and $W_{ih}$is an 100 × 10 matrix of the input connection weights. The activation function $f(x)$ was chosen tanh function $f(x)=(e^x-e^{-x})/(e^x+e^{-x})$, constraining the activity within −1 to 1.

The output responses were read out from the hidden activity as following:

$$\mathit{y}=g(W_{\mathrm{out}}\times {\mathit{h}}_{\mathit{t}}+b_{\mathrm{out}}),(2)$$

where $\mathit{y}$ is a 3 × 1 vector representing the probability distribution of model responses. $W_{\mathrm{out}}$ is a 3 × 100 matrix of the output weights. The activation function $g(x)$ was chosen softmax function $g(x)=e^x/{\sum }_{j=1}^3{e}^{x_j}$, generating the probability of each response choice. The choice with the highest probability was selected as the final decision.

Before the training, all parameters of the model were initialized with random values drawn from the standard Gaussian distribution N(0,1). The 2-back and 3-back model were trained separately. For each task, we generated 1,000 random stimuli sequences, each containing 100 digits stimuli corresponding with 100 trials as the training set. The training batch of each training iteration contained one sequence and the model was trained through 10,000 training iterations via the backpropagation through time (BPTT) algorithm with the learning rate 2 × 10⁻³. In addition, we applied the Adam optimizer for training (Kingma, 2014). The parameters were adjusted to minimize the objective function of cross-entropy across all trials:

$$L_{\mathrm{CE}}=-{\displaystyle \frac{1}{T}{\sum }_{t=1}^T{\sum }_{c=1}^3{\stackrel{⌣}{y}}_{tc}\log (y_{tc}),(3)}$$

where $T$ indicated the number of trials, i.e., 100 in this study. $\stackrel{⌣}{y}$ indicated the specified target response, which comprise 0 (Mismatching), 1 (Matching), 2 (Initial situation) and was one-hot encoded. y is the probability distribution of model responses. The model was considered to train sufficiently when the $L_{\mathrm{CE}}$ converged to nearly 0. The model implementation and parameter updating were performed through PyTorch (Paszke et al., 2019) in the Python programming.

After the model training, we generated another 1,000 random 2-back and 3-back stimuli sequences for testing the performance of 2-back and 3-back model, respectively. The activity of hidden states ${\mathit{h}}_{\mathit{t}}$ of each time step t among all test sequences was recorded.

Participants, EEG recording, and preprocessing

Twenty-eight participants (18 males and 10 females, mean age ± SE = 23.4 ± 1.7) were recruited for the working memory task. All participants were right-handed with normal or corrected-to-normal vision and without any neurological disorders. They were asked to take a good rest before the day of the experiment. The experiment was approved by the institution's ethics committee. All participants signed the informed consent document.

EEG data was recorded through EasyCap containing 64 Ag/AgCl sintered ring electrodes and connected with BrainAmp DC amplifiers (Brain Products). The position of electrodes was arranged according to the standard 10–20 system. The ground was placed at AFz and the reference was placed on the apex of the nose. The eye movements and eyeblinks were monitored by a vertical electrooculogram (EOG). The EEG signals were amplified and digitized at a sampling rate of 5,000 Hz (0.016–100 Hz bandpass filtering) with the impedance of each electrode below 20 kΩ during recording. Data acquisition was controlled through the software of Brain Vision Recorder (version 1.03, Brain Products). EEG preprocessing was performed in EEGLAB (version 14.1.1; Delorme and Makeig, 2004) installed in Matlab (MathWorks). Raw EEG data was channel-located to the standard electrode location in the software, and the sampling rate was changed to 1,000 Hz. One electrode was used for recording the EOG, and four other electrodes (FT9, FT10, TP9, TP10) distributed at the boundary of the EEG cap were excluded from analysis, so totally 59 channels of the EEG data were left. The data was 1–80 Hz bandpass filtered followed by a notch filter at 50 Hz using three-order zero-phase noncausal Butterworth filter. The data was divided into 4 s segments relative to the stimulus onset, with prestimulus 1 s and poststimulus 3 s. Segments with obvious movement artifacts were excluded. Then data segments were corrected for baseline drifting, and the parts with eye movements and blinks were removed by using independent component analysis (ICA).

Phase-locking value

In order to obtain the neural representation during the working memory task, we considered phase-locking value (PLV) that measures the phase synchronization between two signals. Previous studies found the phase of neural oscillation reflected the neuronal firing patterns, as well as the phase coupling of LFP coordinated the firing of neuronal assembles, suggesting that the synchronization of EEG phase can characterize the neuronal firing features (Canolty et al., 2010; Ng et al., 2013). The PLV was calculated as follows (Aydore et al., 2013):

First, a pair of signals $s_1(t)$ and $s_2(t)$ were bandpass filtered to a specific narrow band and the analytic signals $z_i(t)=A_i{e}^{j{\phi }_i(t)}$,where $i=1,2$ and $j=\sqrt{-1}$ were calculated from $s_i(t)$ using Hilbert transform:

$$\mathrm{HT}(s_i(t))={\displaystyle \frac{1}{\pi }{\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}{\displaystyle \frac{s_i(t)}{t-\tau }d\tau },(4)}$$

Then the phase difference was computed as follows:

$$\mathrm{\Delta }\phi (t)=\arg \left({\displaystyle \frac{z_1(t)z_2^{*}(t)}{|z_1(t)||z_2(t)|}}\right),(5)$$

where * represented conjugation and $|\cdot |$ represented the modulus of the complex number. So the PLV was defined as follows:

$$\mathrm{PLV}=\left|{\displaystyle \frac{1}{N}{\sum }_{t=1}^N{e}^{j\mathrm{\Delta }\phi (t)}}\right|.(6)$$

PLV ranged from 0 to 1, where 0 represented no phase synchrony and 1 represented perfect phase synchrony. In this study, we obtained the PLV matrices of the frontal network (Fp1, Fp2, Fpz, F3, F4, F7, F8) in five frequency bands: delta (1–4 Hz), theta (4–8 Hz), alpha (8–12 Hz), beta (12–30 Hz), and gamma (30–45 Hz). The PLV was calculated using EEG during ITI, where memory items were encoded and maintained in the neural activity. Because of the symmetry of the PLV matrix, we only selected the upper triangular of each matrix and flattened and concatenated them among all frequency bands into a 105 × 1 PLV vector.

Dimensional reduction analysis

We analyzed the neural representations in the low-dimensional space. In the model of 2-back or 3-back task, we combined all hidden states ${\mathit{h}}_{\mathit{n}}$ of time step n, obtaining an 100 × 1,000 hidden state set X [100 represented the dimension of hidden state (number of hidden neurons) and 1,000 represented the number of test sequences] and performed principal component analysis (PCA; Ringnér, 2008). The main goal of PCA is to identify and highlight the most important features or variables that contribute the most to its variability in the low-dimensional space. In detail, hidden state set X was subtracted from its mean of columns and calculated the covariance matrix C:

$$C={\displaystyle \frac{1}{n}X{X}^T.(7)}$$

Then the eigenvalues and eigenvectors of C were obtained using eigenvalue decomposition. The quotient between the individual eigenvalue and the sum of all eigenvalues indicate the ratio of explained variance (RV), representing the importance of each component. Eigenvectors corresponding to the largest three eigenvalues were selected to construct the projection matrix P with the dimension of 3 × 100. So the low-dimensional representation Y of X was computed as $Y=PX$, where Y had the dimension of 3 × 1,000. The projection matrix P was obtained from the combined set of ${\mathit{h}}_1$ and ${\mathit{h}}_2$ and was shared among the hidden state sets of all time steps in order to align all low-dimensional representations in the same space. Because the representations of memory items were organized orthogonally but not along the PC in such space, the new axes were defined in order to highlight the optimal orthogonal representation of memory items. In the 2-back space, the axes were defined as [1,0] (axis T) and [0,1] (axis T-1) in the PC1–PC3 space. In the 3-back space, the axes were defined as [3.26, −5.61, 0] (axis T), [0, 0, 1] (axis T-1), and [0.43, 0.25, 0] (axis T-2). Such axes obtained via PCA were verified by the generalized linear model (GLM). The projection of subsequent ${\mathit{h}}_{\mathit{n}}$ was under the same projection matrix of PCA to ensure that the representations of different trials were in the same low-dimensional space.

It was assumed that ${\mathit{h}}_{\mathit{n}}$ comprised the information of stimuli across different ordinal ranks, i.e., not only the current but also previous N − 1 stimuli. For a specific ordinal rank of stimuli, after projecting the low-dimensional representation of ${\mathit{h}}_{\mathit{n}}$ onto one axis and calculating the distance between the projection and the origin, a series of distance curves corresponding with different stimulus’ digits could be obtained. If one axis represented the memory information of a certain ordinal rank properly, the structure of the distance curves was horizontal parallel with small variability within each distance curve and large disparity across different distance curves, indicating the separability of stimulus’ digits on such axis and ordinal rank. Specifically, we designed a metric to measure this characteristic. For two distance curves $y_1$ and $y_2$, the difference $s=y_1-y_2$ between them and the absolute mean value $|\mathrm{mean}(s)|$, as well as the standard deviation of the distance curves $\mathrm{std}(y_1)$ and $\mathrm{std}(y_2)$, were calculated. The metric was defined as $k=|\mathrm{mean}(s)|/(\mathrm{std}(y_1)+\mathrm{std}(y_2))$ (Fig. 2D), measuring the separation of projected distance distribution of different stimulus. The higher k indicates the higher separability of the projections of different stimulus on a certain axis, suggesting such axis is the encoding axis. Further, we verified the validity of metric k and analyzed the representation of memory items on different axes from two other aspects: First, we analyzed the decoding performance of each memory item on different axes via support vector machine (SVM); next, we calculated the axes by fitting the representation with different ordinal rank and analyzed the angle between them.

Figure 2.

Representation analysis of memory information. A, Top panel, Representation of h₂ in the two-dimensional space composed by PC1 and PC3 in 2-back task, colored by the digits of input 2. Bottom panel, Representation of h₂ in the two-dimensional space composed by PC1 and PC3 in 3-back task, colored by the digits of input 2. Different digits are distributed along the axis T. B, Top panel, The same representation with the top panel in A, colored by the digits of input 1. Bottom panel, The same representation with the bottom panel in A, colored by the digits of input 1. Different digits are distributed along the axis T-1. C, The illustration of the distance curves of h₂, showing the distance curves of different digits of input 2 on axis T with the digits of input 1 as the independent variable in the 2-back task (I: input; D: digit). D, The diagram to calculate the metric k based on the distance curves. E, Left panel, The representational metric k₂ and k₁ of input 2 and input 1 on axis T and axis T-1 of h₂ in the 2-back task, respectively. Right panel, Similar with the left panel in the 3-back task. Error bar represents SEM; two-paired t test. ***p < 0.001.

In EEG analysis, we extracted the neural representation of memory information via PLV referring to the methods in Xie et al. (2022). Similarly, the previous memory items were one-hot encoded to input vectors. For example, if the previous memory items were [5,3] in the 2-back task, the input vector was encoded as a 18 × 1 vector [0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0]. Then the matrix of linear regression coefficients of single trial between the vector of input and PLV were calculated with the dimension of 105 × 18. The coefficients were flattened and concatenated among all trials. The neural representation of memory information was analyzed under the low-dimensional space of combined regression coefficients via PCA. In the 2-back space, the axes were defined as [−8.00, 11.01] (axis T) and [1.68, 1.98] (axis T-1) in the PC1–PC3 space. In the 3-back space, the axes were defined as [1.14, 0.19, −1.38] (axis T), [4.55, 1.83, 4.01] (axis T-1), and [−2.02, 6.66, −0.76] (axis T-2). Correspondingly, the projection distances, distance curves, and k were calculated.

Two-sided paired t test was applied on each comparison between the metric k of the same memory representation on different axes for analyzing the orthogonality. All significances were judged as p < 0.05. Pearson’s correlation coefficient and their significance between representations across different subspaces were tested for analyzing the rotational dynamics. The significances of the correlation were judged as p < 0.05.

Results

Orthogonal representation of memory items in order

An RNN model, composed of a group of recurrently connected hidden neurons, was trained to perform the N-back task (Fig. 1B; N = 2 or 3), which demands dynamic encoding and updating information in memory. The model was required to determine whether the current input (a digit ranging from 0 to 9) matched the previous Nth input. The responses were read out from the population's state via a fully connected layer to the output units, indicating mismatching or matching response. The models performed well in both of the 2-back and 3-back tasks after training (Fig. 1C), with higher test accuracy achieved in the 2-back task, which are consistent with human performance (Fig. 5B). Figure 1D visualizes the hidden state h of the RNN, showing the mixed selectivity of memory items in the RNN hidden neurons. Clustering analysis identified three types of neuronal populations, one without selectivity and the other two exhibiting mixed selectivity with opposite activation patterns.

Figure 5.

Orthogonal and rotational representations in EEG. A, The diagram of N-back task for the human subjects. B, Performances of human subjects in the 2-back and 3-back task. C, RT analysis. Left, The individual trial-averaged RT in both 2-back and 3-back task. Right, The distribution of single-trial RT across all subjects. D, The schematic of extracting the low-dimensional representation. E, Top panel, Representation of memory items in the low-dimensional space in the 2-back task. Bottom panel, Representation in the 3-back task. F, Trial-averaged representational metric k_n and k_n − 1 of stimulus n and stimulus n−1 in d_n in the 2-back task. G, Left panel, Trial-averaged representational metric k_n and k_n − 1 of stimulus n and stimulus n−1 in d_n in the 3-back task. Right panel, Trial-averaged representational metric k_n and k_n − 2 of stimulus n and stimulus n−2 in d_n in the 3-back task. H, The distance correlation of stimulus n between d_n and d_n + 1 on axis T and axis T-1, respectively, among all trials in the 2-back task. I, Left panel, The distance correlation of stimulus n between d_n and d_n + 1 on axis T and axis T-1, respectively, among all trials in the 3-back task. Right panel, The distance correlation of stimulus n between d_n + 1 and d_n + 2 on axis T-1 and axis T-2, respectively, among all trials in the 3-back task. Two-paired t test. Error bar represents SEM; *p < 0.05.

First, we considered how multiple memory items with different ordinal ranks are represented in the memory encoded by the hidden state h. At the beginning, we focused on h₂, which indicates h observed after the 2nd input. We note that h₂ encoded not only the memory information of the current input (input 2) and also the previous one (input 1; compare with Eq. 1). We found that h₂ exhibited specific structure in the two-dimensional space composed of the 1st and 3rd principal components, i.e., PC1 and PC3. Specifically, the contents of inputs 2 and 1 were encoded along two orthogonal axis, termed axis T and T-1, respectively (Fig. 2A,B). The change of the learning rate or the ground truth (e.g., from the one-hot output to the graded output [1,1,1;2,2,2;3,3,3]) for the training of RNN did not affect the representation. However, reducing the number of hidden neurons in the RNN will disrupt the representation compared with increasing the number, suggesting that the storage of memory items relies on the population encoding.

In order to examine this orthogonality more comprehensively, we projected the representation of h₂ onto axis T and T-1 and calculated the distance between the projected point to the origin on these two axes. As an example, in the 2-back task, the distances of different contents of the input 2 on axis T were presented, taking the content of the input 1 as an independent variable (Fig. 2C). The distance curves among different contents of the input 2 was nearly horizontal parallel, which meant the distance of the same digit of the input 2 occupied the similar distance regardless of the input 1 (Fig. 2C, Fig. S1A,C). In contrast, the distance curves of different digits of the input 2 on axis T-1 did not exhibit such structure (Fig. S1B,D). Conversely, the distance curves of the input 1 exhibited the horizontal parallel structure only on axis T-1 (Fig. S1B,D), while did n't on axis T (Fig. S1A,C). These results reconfirm that the content of the input 2 and 1 was encoded along the axis T and T-1, respectively.

Next, a metric, k, was defined among the distance curves of different digits to quantify the orthogonality of representation. k measures the degree of distance distribution separation of among different distance curves (see Materials and Methods for details; Fig. 2D). Consistent with above analysis, the k₂ of input 2 was higher than k₁ of input 1 in h₂, while the k₁ of input 1 was higher than k₂ of input 2 in h₂ in both 2-back and 3-back task (Fig. 2E, left panel: 2-back, axis T: t₍₄₄₎ = 4.9597, p < 0.001; axis T-1: t₍₄₄₎ = 7.5087, p < 0.001. Right panel: 3-back, axis T: t₍₄₄₎ = 9.4325, p < 0.001; axis T-1: t₍₄₄₎ = 12.3153, p < 0.001.), indicating the orthogonal representation between input 2 and 1 in h₂.

The above analysis was focused on the structure of h₂, regarding the coding of input 2 and 1. Next, we extended it to more generalized conditions, regarding the coding of input n (the current input) and n−1 (the previous input), n−2 (the second previous input), etc., using the same projection matrix of PCA. We first determined how many dominant dimensions were required to complete the 2-back and 3-back tasks. To this end, we trained five models and analyzed the RV of individual PCs (Fig. 3A) and the difference in RV between the adjacent PCs (Fig. 3B) in the 2-back and 3-back task. The first two/three differences were significantly larger in the 2-/3-back tasks, indicating 2/3 dimensions are needed to represent the model activity. Consistently, we found that the input n and n−1 of h₂ in the 2-back task (Fig. 3C), as well as the input n, n−1, and n−2 of h₃ in the 3-back task (Fig. 3D), was encoded along mutually orthogonal axes. The representation of h_n in different ordinal ranks showed similar orthogonal patterns. Quantitative analysis confirmed that in the 2-back task, trial-averaged k_n of input n was higher than k_n − 1 of input n−1 on axis T and trial-averaged k_n − 1 of input n−1 was higher than k_n of input n in h_n on axis T-1 (Fig. 3E, t₍₉₇₎ = 94.4337, p < 0.001; t₍₉₇₎ = 118.5724, p < 0.001; Fig. 3E). Correspondingly, in the 3-back task, when considering the projection space of input n (current input) and input n−1 (previous-first input), trial-averaged k_n of input n was higher than k_n − 1 of input n−1 on axis T, and trial-averaged k_n − 1 of input n−1 was higher than k_n of input n in h_n on axis T-1 (Fig. 3F; t₍₉₇₎ = 166.4978, p < 0.001; t₍₉₇₎ = 91.0632, p < 0.001); when considering the projection space of input n (current input) and input n−2 (previous second input), trial-averaged k_n of input n was higher than k_n − 2 of input n−2 on axis T, and trial-averaged k_n − 2 of input n−2 was higher than k_n of input n in h_n on axis T-1 (Fig. 3G; t₍₉₇₎ = 141.5627, p < 0.001; t₍₉₇₎ = 191.0204, p < 0.001). Further, the angle between the fitted axes of the representation with different ordinal ranks were nearly 90° in both 2-back and 3-back task (Fig. 3H,I). Decoding analysis showed the similar results. In the 2-back task, the information of input n could be decoded on axis T and the information of input n−1 could be decoded on axis T-1, respectively, while axis T-2 corresponding with PC2 did not encode any information (Fig. S2A–C). In the 3-back task, the information of input n could be decoded on axis T, the information of input n−1 could be decoded on axis T-1, and the information of input n−2 could be decoded on axis T-2, respectively (Fig. S2D–F). These results suggested that the representation of memory items occupies the fixed, orthogonal subspace according to the relative ordinal rank among the items. The schematic of orthogonality was shown in Figure 3J, showing that the input n occupied axis T, input n−1 occupied axis T-1 in both 2-back and 3-back task, as well as input n−2 occupied axis T-2 in the 3-back task.

Figure 3.

Orthogonal representations of memory items. A, Ratio of explained variance (RV) and (B) adjacent differences in the 2-back and 3-back task. C, Trial-averaged representation of memory information in the 2-back task. D, Trial-averaged representation of memory information in the 3-back task. E, Trial-averaged representational metric k_n and k_n − 1 of input n and input n−1 in h_n in the 2-back task. F, Trial-averaged representational metric k_n and k_n − 1 of input n and input n−1 in h_n in the 3-back task. G, Trial-averaged representational metric k_n and k_n − 2 of input n and input n−2 in h_n in the 3-back task. H, The angle between the vectors of the representations of different memory items in the 2-back task. I, The angle between the vectors of the representations of different memory items in the 3-back task. J, The schematic of orthogonal representation. Error bar represents SEM; two-paired t test. ***p < 0.001.

Rotation of representation across different subspace

When the new input is coming, the representation of items in the memory space will be reorganized for keeping the memory information in order. Next we investigate how this reorganization was accomplished. In the model, the recurrent connection weight W_hh is important for transforming the hidden state, which played a key role in reorganizing of memory (compare with Eq. 1). So we started with focusing elucidating the role of W_hh. The W_hhW_hh performed the characteristic of orthogonal matrix similar with the rotational matrix in geometry, i.e., the W_hh*.W_hh’ was nearly an identity matrix where the diagonal elements were close to 1 (Fig. S3), suggesting that the W_hh plays the role in rotating the representation. In the two-dimensional space spanned by PC1 and PC3, the low-dimensional representation of input 1 in hidden state h₁ was distributed on axis T (Fig. 4A). However, when operated by W_hh, the representation of input 1 was encoded in W_hh× h₁ and was distributed on axis T-1 that is orthogonal to axis T (Fig. 4A,B). Moreover, the rotation angle between the fitted axes of the representation before and after the W_hh operation was close to 90° at every ordinal rank (Fig. 4C,D), indicating that the operation of W_hh rotated the memory information in the representation space orthogonally with scaling. In addition, the b_hh and b_ih did not affect the representation, and the new memory information was introduced by W_ih and the dimension of representation was increased.

Figure 4.

Rotational dynamics during the memory updating. A, Rotation between axis T and axis T-1 of the representation from h₁ to W_hh× h₁ in the 2-back task. B, Rotation between axis T and axis T-1 of the representation from h₁ to W_hh× h₁ in the 3-back task. C, The rotation angle in the 2-back task. D, The rotation angle in the 3-back task. E, Top panel, The distance correlation of input 1 between h₁ and h₂ on axis T and axis T-1, respectively, in the 2-back task. Bottom panel, The distance correlation of input n between h_n and h_n + 1 on axis T and axis T-1, respectively, among all trials in the 2-back task. F, Top panel, The distance correlation of input 1 between h₁ and h₂ on axis T and axis T-1, respectively, in the 3-back task. Bottom panel, The distance correlation of input n between h_n and h_n + 1 on axis T and axis T-1, respectively, among all trials in the 3-back task. G, Top panel, The distance correlation of input 1 between h₂ and h₃ on axis T-1 and axis T-2, respectively, in the 3-back task. Bottom panel, The distance correlation of input n between h_n + 1 and h_n + 2 on axis T-1 and axis T-2, respectively, among all trials in the 3-back task. H, The schematic of rotational dynamics.

In addition, we examined if the content structure of information was well preserved during the transferring of representation between different subspaces. We analyzed the correlation of representation across different axes, where the absolute value of coefficients nearly 1 indicates the rotational characteristic, i.e., the memory information was rotated from one axis to another, while the sign of the coefficients indicates the direction of the rotation. We found the positive correlation between the representation on axis T in h₁ and the representation on axis T-1 in h₂ (Fig. 4E,F). What's more, we found the negative correlation between the representation on axis T-1 in h₂ and the representation on axis T-2 in h₃ (Fig. 4G). These results could be extended to the general case of information transferring from h_n to h_n + 1 during the entire task period. These results not only indicated the rotational dynamics through W_hh but also suggested the maintenance of the content structure during the rotational process. The schematic was shown in Figure 4H.

Taken together, these results suggested that the rotational dynamics with scaling mediated by the operation of W_hh transform the information of input n from axis T to axis T-1 in both 2-back and 3-back tasks and transform the information of input n−1 from axis T-1 to axis T-2 in the 3-back task. Such mechanism forms a “first in, first out” (FIFO) WM buffer to process and update the sequential information.

Orthogonal-rotational characteristic in the brain

To investigate if the orthogonal-rotational dynamics reported above in the RNN model also hold true for the brain, we analyzed the neural representation measured by EEG signals when human subjects (n = 28, 18 males) were engaged in the 2-back and 3-back WM tasks similarly to the ones completed by the network model.

Specifically, we analyzed signals recorded in the intertrial intervals (ITI) between two adjacent memory items (Fig. 5A), during which the memory items were encoded and maintained. Similar with the performance of the model, the accuracy of participants in the 2-back task was higher than the 3-back task (Fig. 5B). The mean reaction time (RT) of individual subject (Fig. 5C, left panel) and most single trials (Fig. 5C, right panel) was lower than 2.5 s, suggesting that the EEG segments containing poststimulus 3 s incorporated the neural activity of memory encoding. To extract the brain dynamics representing the underlying information processing of the task, we computed the PLV across the frontal channels from 5 frequency bands: delta (1–4 Hz), theta (4–8 Hz), alpha (8–12 Hz), beta (12–30 Hz), and gamma (30–45 Hz). The coefficients of linear regression between the vector of stimulus and PLV were extracted for subsequent analysis under the low-dimensional space via PCA (Fig. 5D). Correspondingly, we specified two orthogonal axes [axis T (red), axis T-1 (purple)] in the 2-back task (Fig. 5E), as well as three orthogonal axes [axis T (red), axis T-1 (purple), axis T-2 (blue)] in the 3-back task (Fig. 5E).

Importantly, we found orthogonal-rotational characteristic that is qualitatively similar to that observed in the RNN model. Specifically, in the 2-back task, k_n of input n was higher than k_n − 1 of input n−1 on axis T, and k_n − 1 of input n−1 was higher than k_n of input n on axis T-1 (Fig. 5F; t₍₃₅₎ = 2.9892, p = 0.022; t₍₃₅₎ = 2.3621, p = 0.0375), with the positive correlation between the distance representation of stimulus n on axis T and the representation of stimulus n−1 on axis T-1 (Fig. 5H). In the 3-back task, when considering the projection space of input n (current input) and input n−1 (previous-first input), k_n of input n was higher than k_n − 1 of input n−1 on axis T, as well as k_n − 1 of input n−1 was higher than k_n of input n on axis T-1 (Fig. 5G, left panel; t₍₃₅₎ = 2.8193, p = 0.0102; t₍₃₅₎ = 2.4261, p = 0.035), with the positive correlation between the distance representation of stimulus n on axis T and the representation of stimulus n−1 on axis T-1 (Fig. 5I), as well as the negative correlation between the distance representation of stimulus n−1 on axis T-1 and the representation of stimulus n−2 on axis T-2 (Fig. 5I). Such representations were disrupted in the error trials (Fig. S4), suggesting the necessity of orthogonal-rotational dynamics for memory encoding and updating.

Our results suggested the existence of similar orthogonal representation and rotational operation of memory storage in the brain dynamics detected by EEG and an artificial neural network model (summarized in Fig. S5), suggesting that the such a framework might be optimal for memory storage and updating in neural networks.

Discussion

As the foundation of most complex cognitive behaviors, the WM refers to the ability to store and process the external information in mind temporarily in a neural network buffer (Baddeley, 2003; Curtis and D’Esposito, 2003; Mongillo et al., 2008; Hansel and Mato, 2013). The functional implementation of such a buffer relies on the neural activity, e.g., the coordination of theta and gamma rhythm in the brain oscillation, where theta activity serves as the activation of memory encoding and gamma activity serves as the item separation and preservation (Jensen and Lisman, 2005). The function of WM encompasses various ways to maintain and process memory information, such as integrating information of multi-model sensory input (Prabhakaran et al., 2000), organizing the memory items in different forms of hierarchy (Nie et al., 2017; Dotson et al., 2018), manipulating the priorities of representations (Van Ede et al., 2017; de Vries et al., 2020), or extracting the memory information to guide behavior (Van Ede et al., 2021). In the natural environment, we rely on the WM to receive and process various types of real-time information around us, most of which are organized as continuous sequences (Dehaene et al., 2015). This necessitates the capability WM to “online” process sequential parts efficiently and rapidly. While WM can meet the challenge of encoding and updating memory information simultaneously, the mechanism by which it achieves such function is still unclear.

In an effort to better understand how the brain encodes and updates memory information in real-time when processing continuous sequences, we sought to employ a computational model, i.e., RNN model and combining it with EEG analysis on humans to address this question. Optimizing artificial neural networks (ANNs) to perform the human-like task is a powerful tool to reveal the mechanisms in neuroscience (Kanwisher et al., 2023). Specifically, several studies used RNN to explain the computational mechanisms underneath the execution of cognitive functions, such as indicating the reuse of schema to explain the brain mechanism of learning-to-learn (Goudar et al., 2023) or revealing the geometry with the semantic representations to interpret the function of linguistic instructions in the human brain (Riveland and Pouget, 2024). These results suggested that ANN is a powerful model to investigate the functional mechanism of the brain cognition.

It poses a challenge to uncover and compare the neural representations across the RNN model and the brain at the level of original network activity, but the neural manifold provides an alternative perspective. It has been proposed that the underlying patterns of neural population activities are confined on a low-dimensional manifold that captures the critical modes of population dynamics (Kobak et al., 2016; Pandarinath et al., 2018). Neural manifolds are characterized in a wide range of brain areas, playing a crucial role to depict the characteristics of neural dynamics and their relation with behavior (Duncker and Sahani, 2021). In the motor task, the characteristics of neural manifold structure in the motor cortex can represent the locomotion parameters, such as position, speed, and direction (Gallego et al., 2017, 2018). The low-dimensional neural dynamics in LIP reflects the process of perceptual decision-making (Hanks and Summerfield, 2017; Latimer and Freedman, 2023). A toroidal manifold of spatial position can be uncovered in the medial entorhinal cortex (MEC; Ginosar et al., 2021). These findings provide evidence of the neural representations in the low-dimensional manifold, which is applicable for both artificial network and the brain (Mante et al., 2013). Importantly, in the same cognitive task, neural manifolds can capture similar modes from neural activities across different individuals, suggesting that it can capture fundamental, intrinsic characteristics of neural dynamics (Gallego et al., 2020; Safaie et al., 2023). Building on this, we analyzed and aligned the neural representations and dynamics of both the RNN model and the brain at the low-dimensional manifold space, indicating the shared mechanism for “online” processing the streaming information dynamically in WM.

In detail, we proposed an orthogonal-rotational framework for WM to encode and update information simultaneously, thereby constructing a “first in, first out” (FIFO) WM buffer. Firstly, we extended the findings of orthogonal sequential memory encoding in static sequences (Xie et al., 2022; Chen et al., 2024; Tian et al., 2024) to dynamic sequences. Based on this, we identified a rotational dynamic transferring memory information across subspaces, supporting the memory updating as well as keeping the order of memory information. In addition, we found that the memory capacity was highly associated with the number of subspaces, enabling this mechanism to generalize to situations with different memory loads. More importantly, this framework is universal for both biological and artificial neural network, suggesting its functional advantage for memory encoding and updating.

Both orthogonality and rotational characteristics in our framework have significant functional meaning to dissociate and eliminate the interference across distinct dimensions of information (Flesch et al., 2022; Pu et al., 2024). Meanwhile, diverse studies have demonstrated the widespread presence of the orthogonal-rotational dynamics in the brain activity. In the sequential WM task, not only the individual memory items with different ordinal ranks are organized orthogonally in the prefrontal cortex (Xie et al., 2022; Chen et al., 2024; Tian et al., 2024), but also the global and local rank are orthogonal in the hierarchical organization of memory chunks (Fan et al., 2024). During the visual decision-making procedure, the representation of stimulus information and decision process is separated with each other in an orthogonal manner in the primary visual cortex (Yiling et al., 2024). When encoding the abstract knowledge, different aspects embedded in the information, such as context and content, or multiple conceptual characteristic of knowledge, are represented in separated mutual-orthogonal dimensions in prefrontal cortex, hippocampus, or medial temporal lobe (MTL; Nieh et al., 2021). During the implicit sequence learning, the sensory space is gradually rotated to be orthogonal to the memory space for dissociating the information of sensory and memory (Libby and Buschman, 2021), suggesting that the rotational dynamics contributes to the form of orthogonal representation. These findings, in conjunction with our study, involve different cognitive functions and brain regions, indicating that orthogonality and rotation is a universal framework for information representation and processing in the brain.

Our results suggested an efficient computational framework for iterative WM information processing in the brain, but some issues still require further elucidation. For instance, in the RNN model, we indicated that the structure W_hh plays a pivotal role in executing the orthogonal-rotational operation for encoding and updating WM information dynamically. Yet, the specific biological mechanism to implement such W_hh structure in the brain still remains to be explored. In addition, various neurodegenerative and psychiatric diseases impairs the function of WM (Amici et al., 2007; Heck et al., 2014). It will be informative for future studies to investigate the detailed dynamical process affected by these disorders, in the light of the framework proposed here, to shed light on the mechanisms underlying corresponding WM deficits.

One limitation of our study is that the orthogonal-rotational representations analyzed in EEG were not clear and significant as we analyzed in the RNN model. Based on our analysis, we suggested that EEG and RNN model had similar representational patterns, implying they shared the common orthogonal-rotational dynamics to accomplish the sequential memory task. Limited by the technology, these results and comparisons are indirect. In addition, the information representation extracted from the EEG PLV were relatively rough because the memory information will be interfered by the neural noise during the periods of memory storage in the brain, and the EEG signals are relatively rough compared with the spiking activity. Further studies can provide more direct evidence based on the invasive spiking analysis.