A Basal Ganglia model for understanding working memory functions in healthy and Parkinson’s conditions

Abstract

Working memory (WM) is considered as the scratchpad for reading, writing, and processing information necessary to perform cognitive tasks. The Basal Ganglia (BG) and Prefrontal Cortex are two important parts of the brain that are involved in WM functions, and both structures receive projections from dopaminergic nuclei. In this modelling study, we specifically focus on modelling the WM functions of the BG, the WM deficits in Parkinson’s disease (PD) conditions, and the impact of dopamine deficiency on different kinds of WM functions. Though there are many experimental and modelling studies of WM properties, there is a paucity of models of the BG that provide insights into the contributions of the BG in WM functions. The proposed model of BG uses bistable flip-flop neurons to model striatal up-down neurons, a network of nonlinear oscillators to model the oscillations of the Indirect Pathway of BG and race-model for action selection. Five different WM tasks are used to demonstrate the generalisation ability of the proposed model. Experimental data from the four tasks are compared with model performance in both control and PD conditions. The model is extended to predict the response time of subjects and in the PD version of the model, the effect of dopaminergic medication on WM performance is also simulated. The proposed model of BG is a unified model that can explain the WM functions of the BG over a wide variety of tasks in both normal and PD conditions, and can be used to understand why specific WM functions are impaired whereas others remain intact in PD.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11571-023-10056-y.

Introduction

Basal Ganglia (BG) is long known for its critical role in Working Memory (WM) along with its attributions to Reinforcement Learning (RL) and motor action selection (Goldman-Rakic 1995; McNab and Klingberg 2008). There are many observations of high activity in BG and Prefrontal Cortex (PFC) during WM tasks, but the BG and dopaminergic projections from BG to PFC are considered to perform gating of items and saliency computation to execute WM tasks (Braver and Cohen 1999; Dipoppa et al. 2016; Lisman et al. 1998; Volle et al. 2005). In attractor-based models of working memory, gating plays a key role in state transitions, persistence of memory and enhancement of stability against noise (Baddeley 2010; Gonon 1997; Gruber and von Cramon 2003). Gating is thought to be implemented in BG by dopamine projections to striatal medium spiny neurons whose bistable dynamics are modulated by dopamine (Kawagoe et al. 2004; Nakahara et al. 2004). Thus, the dopamine gating in BG essentially attenuates less-salient noise and latches salient items to perform WM functions like storage, updation and manipulation of memory items (Kawagoe et al. 1998; MINK, 1996). Many studies demonstrate WM deficits in animal models with loss of dopamine neurons and report improvement in performance after delivery of dopamine agonist drugs (Miyoshi et al. 2002; Rusu et al. 2013). In Parkinson’s disease (PD), depletion of dopaminergic cells in Substantia Nigra pars Compacta (SNc) affects the patient in different motor modalities including tremor, rigidity, and bradykinesia in arm movements, micrographia in handwriting generation and, dysarthria in speech (Fekete and Jankovic 2011; Magdoom et al. 2011; Mandali et al. 2016; Nair et al. 2022; Oliveira et al. 1997).

PD patients, like patients with frontal lobe lesions, exhibit significant impairments in terms of executive cognitive functions with attentional control deficits (Engle 2002; Gick et al. 1988). Cognitive tasks involving medial temporal-lobe structures are shown to be less impacted among PD patients (Owen et al. 1992). The ‘frontal-like’ Working Memory (WM) tasks are considered as the benchmark to analyse the severity of the disease and a discriminant tool to analyse subgroups of PD patients (Owen et al. 1993). However, some studies have shown that decrease of performance in spatial WM tasks is more pronounced than in non-spatial WM tasks, among moderate disease level patients (Owen et al. 1993; Takahashi and Hatakeyama 2011). There are many experiments in literature that show WM deficits among PD patients such as, backward digit span and ordering task (Bublak et al. 2002) and, random number generation task (Robertson et al. 1996). Studies show that the difference in storage capacities between control and PD patients are not significant but the difference is more prominent in executive functions (Cooper et al. 1991; Dalrymple-Alford et al. 1994). Executive function impairments of WM are found to be very specific and it is still unknown why some WM functions are intact among PD patients but others are affected (Belleville et al. 2003; Miyake et al. 2000; Owen 2000). (Gabrieli et al. 1996) suggested that reduced psychomotor speed observed among PD patients indicates the inability to perform complex cognitive functions that are necessary for performing executive WM tasks.

Computational and mathematical models at various levels of abstraction from detailed, biophysical single neuron to abstract networks of neuron-like units have been proposed to understand the underlying mechanism of BG and dopamine’s influence on WM (Balasubramani et al. 2015; Frank et al. 2001; Gruber et al. 2006; Nair et al. 2022; O’Reilly and Frank 2006). Using a conductance-based neuronal network model of prefrontal cortex, (Durstewitz and Seamans 2002) showed that dopamine’s modulation of the network via D1-receptors, resulted in deepening and widening of the attractor basins of WM states. The gating property of dopamine in WM is also modelled using spiking neurons (Brunel and Wang 2001). Dopamine precursors like Levodopa administration to PD patients have shown improvement in various kinds of WM tasks (Floel et al. 2008; Simioni et al. 2017).

In more abstract versions, models showing the gating mechanism of dopamine on striatum are proposed without the biological details of gating (Dreher and Grafman 2002; O’Reilly et al. 1999). There are also more abstract, systems-level models to replicate the key mechanisms of BG in WM (Frank et al. 2001; Gruber et al. 2006; Nakahara et al. 2004; Schroll et al. 2012; Trutti et al. 2021). However, abstract models also have limitations such as inability to show the relation between genetic/molecular level and systems level, and the inability to generalise to novel tasks (Gilbert et al. 2005; Lewis et al. 2003a, b; Schroll and Hamker 2013). These limitations hinder the understanding of trends observed by the same subject in different experimental tasks and functions, and impedes us to understand the characteristic outcome of neurodegenerative diseases. Experimental data from different cognitive tasks performed by PD patients, under medicated and unmedicated conditions, show specific WM impairments and the reasons for which are still not clearly known, when compared to other modalities like motor, speech, and gait and balance (Cai et al. 2022; Ye et al. 2022). There are only a handful of BG models that incorporate the PD condition in WM tasks; and even they are often not general enough to describe PD-related WM deficits in a variety of WM tasks described in the literature (Goulding et al. 2022; Hazy et al. 2007; Schroll et al. 2012). Table 1 lists the highlights of the proposed model and compare it with the previous works.

Table 1.

Highlights of the present work that solves the limitations of the previous works

Work	Highlights of the work	Relative merits of the proposed model
Hazy et al. (2007)	The work proposes model of PFC-BG that performs key functions of WM system Simulates a single type of WM task that involves simple storage of information, without any sequence transformation	Simulates a wide range of WM tasks that combine storage, manipulation and updating aspects Permits a detailed neurobiological interpretation of the model architecture
Vitay (2010)	Models three categories of visual WM tasks: delayed matching-to-sample (DMS), delayed nonmatching-to-sample (DNMS) and delayed pair-association (DPA) WM tasks require only storage and recall, and WM functionality is modelled using a cortico-BG loop and retrieval using thalamic stimulation The model does not account for the BG indirect pathway and therefore does not learn WM control	Storage and recall functionality by enabling the gating mechanism using the striatal flip-flop neurons in BG Uses WM tasks that require manipulation along with storage and updating capabilities, and it is important because experimental studies (Gilbert et al. 2005; Lewis et al. 2003a, b) show specific manipulative impairments among PD patients Model accounts for both direct and indirect pathways, together with striatal neurons to learn WM control
Schroll et al. (2012)	Multiple BG loops (cognitive and motor) are considered and shows that the usage of multiple parallel loops provide flexibility in achieving WM, motor control and response selection Striatum and its associated direct pathway allow for WM maintenance in PFC loops	Q-learning to learn striatopallidal connections and Value Function computation using nigrostriatal pathway Allows cortico-BG circuitry to perform manipulation and updating tasks, along with standard storage and recall Uses direct and indirect pathways, along with nigrostriatal connections to achieve WM, motor control and response selection
Nair et al. (2023)	Defines a generalized framework to learn diverse cognitive tasks and operations Maps individual subject responses by tuning the model meta-parameters	Study WM processing of BG in terms of storage, manipulation, updating and recall operations Consistent with the experimental data that manipulation is specifically impacted in PD condition PD condition modelling by clamping dopamine signal Compared the model performance to experimental data under both control and PD conditions

The proposed model tries to fill this gap by proposing a generalized computational model of BG with both control and PD versions and use it to explain the experimentally reported results of a variety of WM tasks. The model is trained by Reinforcement Learning (RL). The D1R and D2R-expressing Medium Spiny Neurons (MSNs) of the striatum are modelled using flip-flop neurons (Kumari et al. 2023) whose bistable dynamics is compared to the up-down states of MSNs (Wilson and Groves 1981). Dopaminergic activity of SNc is modelled as the reward prediction error (Schultz 2016). Nonlinear oscillators of the Lienard type are used to simulate the coupled STN-GPe populations to inject noise in the system for better exploration (Chakravarthy and Balasubramani 2014). The proposed model is validated in terms of its ability to show the specific executive WM impairments observed among PD patients. The model is validated against different WM tasks and in some tasks, the model performance is compared against experimental data reported in the original studies for both control and PD conditions. The model is also showcased to predict response times in different WM tasks. The subsequent section titled Methods and Materials describes the various tasks modelled to assess the WM functions of the BG model proposed. This is followed by the Results section where performance of the model in both control and PD conditions on the tasks described in the previous section is described. In the Results section, we also showcase the prediction capacity of the model in terms of model response times for tasks that do not have experimental response time and levodopa study on PD model to present its effects on WM performance. We then conclude by discussing the results, limitations, and the future scope in Sect. “Conclusion and discussion”.

Materials and methods

Experimental tasks

The proposed BG model can simulate a variety of experimental paradigms that can test sequence learning and working memory functions. Behavioural research uses a variety of tasks to assess sequence processing and working memory capabilities in humans. Sequence processing often implies working memory capability, since many sequence processing tasks require the sequence information to be stored for a finite duration, in order to produce the output sequence. Therefore, in this study we only use sequence processing tasks that involve working memory functions. Taking these aspects into consideration we modelled five WM tasks (Fig. 1). Of the five tasks, 1AX2BY task is chosen to test the sequence processing capacity of the model; Digit span task to validate the generalisation of the model in standard WM function; and Alphabetical Manipulation, Updating and 4-Consonants tasks are selected to check whether the PD version of the model can show specific impairment in executive WM function but intact manipulation of WM function.

1. 1AX2BY task The task involves sequences that comprise two types of stimuli - numeric stimuli and alphabetical stimuli (Frank et al. 2001). When the numeric 1 is encountered first in the input sequence, action ‘Right’ is the desired action only when A followed by an X also appears in the sequence. For all other items in the sequence, action ‘Left’ is the desired action. Similarly, when the numeric 2 is encountered first, action ‘Right’ is the desired action only when B followed by Y appears in the sequence. Similarly, for all other alphabetical sequences, action ‘Left’ is the desired action. Along with these items, in the sequence three noise input stimuli (3, C & Z) are also presented, which should be ignored.

2. Digit Span task In case of the Digit Span task, the subjects are presented with a sequence of numbers (Gilbert et al. 2005). The length of the sequence is incrementally changed starting from two till the maximum length that can be successfully recollected. The length of the sequence that can be reliably recalled by the person is considered as the span of that person. The span of the model is evaluated by a procedure similar to the one described in (Gilbert et al. 2005). Starting with sequence length of 2, the model is expected to reproduce at least two sequences in six trials to move to length(sequence)+1. If the model fails to reproduce at least two sequences of six trials then the previous sequence length is taken as the span capacity.

3. Alphabetical Manipulation task In case of the alphabetical recall task, the subject is presented with a sequence of alphabets (Gilbert et al. 2005). There are two variations of this task. In the first case (direct), the sequence presented is to be recalled in the same order. In the second case (alphabetical) the sequence of alphabets presented is to be recalled in the sorted order (A-Z).

4. Updating task In this task, the subject is presented with a sequence of alphabets and must recollect the last k-items (Gilbert et al. 2005). Usually, ‘s’ (number of span items) is chosen to be the same as the span of the subject and appended with ‘n’-extra items (n- number of updates). In this task, the difficulty depends on the number of items added to the baseline sequence as it is commensurate with the number of updates required in the memory. The model is trained using sequence length of s along with n items, and however during retrieval the model is trained to recall the last s number of items. For example, when the sequence of length s+n is given as input, the model should recall last s items.

5. 4-Consonants Task (BQNR Task) Here three scenarios are considered. In the first case (same), the subject must recall the sequence in the same order as presented (Lewis et al. 2003a, b). In the second case (ends), the subject should recall based on the (3412) rule (for example, the input sequence BQNR is to be retrieved as NRBQ). In the third case (middle), the middle elements should be swapped while recalling (for example sequence BQNR is retrieved as BNQR). For modelling, four consonants are randomly chosen from seven consonants for each trial and each consonant is encoded to a 7 bit on-hot encoder. The sequence has four parts: i) 4 items are input consonants, ii) d items of empty input (7-bit vector with all zeros), iii) 1 item of instruction cue to mention the type of manipulation to be performed and followed by iv) 2 items representing two choices among which one must be selected.

Fig. 1.

Schematic explanation of different working memory tasks modelled in this study. A 1AX2BY task—when the string AX (BY) follows 1(2) then the “Right” option should be chosen. B Digit span task—a sequence of digits should be recalled and the maximum spanning capacity of the model is calculated. C Alphabetical manipulation task—in “direct” case, the same sequence of letters presented to be recalled and in “alphabetical” case, the sequence should be recalled in dictionary order. D Updating task—recall the last ‘k’ number (k-span capacity) of items with different sequence lengths containing ‘n’ extra items (n-number of updates). (E) 4 consonants task—recall a sequence of 4 consonants based on the instruction given during instruction cue time step

The model description

We present a model of the BG that can simulate the experimental tasks mentioned above Fig. 2 (Nair et al. 2023). In this model, the input subsystem, Striatum, receives cortico-striatal projections from the cortical neurons, and the output nucleus—Globus Pallidus interna (GPi) produces the output sequence. The Subthalamic nucleus (STN) and the globus pallidus externa (GPe) together constitute the Indirect Pathway, whereas the direct striatal projections to GPi, constitute the Direct Pathway.

Fig. 2.

Block diagram of the proposed BG model. Input from the cortex (CTX) is presented to the Striatum (STR)—the input port of Basal Ganglia (BG); Striatum consists of D1 and D2 neurons modelled using Flip Flop (FF) neurons. The STR projects to the output port, Globus pallidus interna (GPi) via the direct pathway and indirectly via the indirect pathway. The Globus pallidus externa (GPe) and the subthalamic nucleus (STN) subsystems constitute the indirect pathway. Substantia Nigra pars compacta (SNc) consists of dopamine producing neurons, which regulate the information flow through the direct and indirect pathways

The Cortico-Striatal projections

Cortical projections to the BG are received at the input nucleus Striatum. The striatum consists of two blocks labelled: D1 and D2. The D1(D2)—striatum consists of neurons that represent D1(D2)- receptors expressing Medium Spiny Neurons (MSNs). The MSNs are known to exhibit up/down states that are thought to underlie working memory functions (Plenz and Kitai 1998; Wilson and Groves 1981). In the present model, the up/down dynamics of the neurons of D1- and D2-striatum are modelled using flip-flop neurons (Kumari et al. 2023). The flip-flop neurons, that are fashioned after the flip-flops of digital systems, have proven memory capabilities (Kumari et al. 2023). The flip-flop neurons have two inputs (J and K) and a single output (V). J and K are competing inputs that try to set and reset the information respectively in the output (V). JK flip-flops are considered as universal flip-flops as they can perform three essential memory functions: set, reset & toggle, and removes undefined state problem observed in SR flip-flop. The algebraic equation used is an extension of digital Boolean counterpart by making J and K continuous in the range [0, 1]. Inputs to the flip-flop neuron essentially performs gating function on different memory items and computes their saliency to whether store, maintain or delete. The dynamics of the flip-flop neurons for striatal inputs are given by Eqs. (1–3) below.

$$J_i^{D1/2}\left(t\right)=W_i^{D1/2}{X}_C\left(t\right)\left(1-V_i^{D1/2}\left(t-1\right){\lambda }_{D1/2}\right)$$ 1

$$K_i^{D1/2}\left(t\right)=\left(1-W_i^{D1/2}{X}_C\left(t\right)\right)V_i^{D1/2}\left(t-1\right){\lambda }_{D1/2}$$ 2

$$V_i^{D1/2}\left(t\right)=J_i^{D1/2}\left(t\right)+K_i^{D1/2}\left(t\right)$$ 3

$${\lambda }^{D1/2}\left(t\right)=\left(\frac{1}{1+e^{a_{1/2}\left({\delta }_v\left(t\right)\right)}}\right)$$ 4

where $a_1$< 0 and $a_2$>0 and are the sigmoid gain parameters, $W_i^{JD1/2}$ and $W_i^{D1/2}$ represent the weights connecting cortex to J and K in D1/D2 of striatum respectively, $X_C\left(t\right)$ denotes the input to cortex at time ‘t’, $V_i^{D1/2}\left(t\right)$ denotes striatal flip-flop neuron output at time ‘t’,${\lambda }^{D1/2}$ is the dopamine gating variable for D1/D2 receptors and, Value function (VF) computation and $W_i^{D1\to SNc}$ updating is as given by Eqs. (5–8) below.

$$VF\left(t\right)=\sum _{i=1}^n{W}_i^{D1\to SNc}{V}_i^{D1}\left(t\right)$$ 5

$$VF\left(t+1\right)=VF\left(t\right)+\eta \left(rew\left(t\right)+\gamma VF\left(t+1\right)-VF\left(t\right)\right)$$ 6

$$T{D}_v\left(t\right)=rew\left(t\right)+\gamma VF\left(t+1\right)-VF\left(t\right)$$ 7

$$\mathrm{\Delta }{W}_i^{D1\to SNc}=\eta T{D}_v{V}_i^{D1}$$ 8

where $W_i^{D1\to SNc}$ is the weight projecting from SNc to D1 striatum, $V_i^{D1}\left(t\right)$ is the output from D1 striatal flip-flop neurons at time ‘t’, $\gamma$ is the discount factor and $\gamma =0$, $T{D}_v$ denotes TD error, $rew\left(t\right)$ is reward at time ‘t’, $\mathrm{\Delta }{W}_i^{D1\to SNc}$ denotes change in $W_i^{D1\to SNc}$ while updating by learning parameter ($\eta$). Since D1- and D2-MSNs are modulated by dopamine, the flip-flop neurons are modulated by dopamine via the slope variables D1 and D2 as given in Eqs. (1–4). The quantity, change of VF, ${\delta }_v\left(t\right)$ is given as,

$${\delta }_v\left(t\right)=VF\left(t\right)-VF\left(t-1\right)$$ 9

The output, GPi

GPi is the output of the BG and receives inputs from the striatum via both the direct and the indirect pathways. In the case of direct pathway, the D1 striatal neurons directly project to the GPi. The weights ($W_{i,j}^{D1\to GPi})$ between the D1 MSNs and the GPi neurons are updated using Q Learning Eqs. (19–22). The STN-GPe subsystems constitute the indirect pathway and are inter-connected via excitatory and inhibitory projections, respectively. Thus, STN-GPe system forms an excitatory-inhibitory loop and has been shown to exhibit different dynamical regimes. It has been thought to act as an exploration engine by adding a controllable level of noise to GPi (Chakravarthy and Balasubramani 2014). STN-GPe dynamics are modelled by the following equations,

$${\tau }_{\mathit{GPe}}\frac{d{V}_i^{\mathit{GPe}}}{\mathit{dt}}=-V_i^{\mathit{GPe}}+\sum _{j=1}^n{W}_{\mathit{ij}}^{\mathit{GPe}}{V}_j^{\mathit{GPe}}+W_i^{STN\to GPe}{V}_i^{\mathit{STN}}-V_i^{D2}\left(t\right)$$ 10

$${\tau }_{\mathit{STN}}\frac{d{U}_i^{\mathit{STN}}}{\mathit{dt}}=-U_i^{\mathit{STN}}+\sum _{j=1}^n{W}_{\mathit{ij}}^{\mathit{STN}}{V}_j^{\mathit{STN}}-W_i^{GPe\to STN}{V}_i^{\mathit{GPe}}$$ 11

$W_{\mathit{ij}}^{GPe/STN}$ lateral weights with connection strength ${ϵ}_{GPe/STN}$ within GPe/STN are initialised by,

$$W_{\mathit{ij}}^{GPe/STN}=\text{exp}(-\frac{D_{i,j,k}^2}{{ϵ}_{GPe/STN}^2})$$ 12

$$D_{i,j,k}^2=\left({\left(i_{GPe/stn}-k_{GPe/STN}\right)}^2+{\left(j_{GPe/STN}-k_{GPe/STN}\right)}^2\right)$$ 13

$$V_i^{\mathit{STN}}\left(t\right)=\text{tanh}\left({\lambda }^{D2},\left(t\right),U_i^{\mathit{STN}}\right)$$ 14

$$V_j^{\mathit{DGPi}}\left(t\right)=\sum _{i=1}^n{W}_{i,j}^{D1\to GPi}{V}_i^{D1}$$ 15

$$V_j^{\mathit{IGPi}}\left(t\right)=\sum _{i=1}^n{W}_{i,j}^{STN\to GPi}{V}_i^{\mathit{STN}}\left(t\right)$$ 16

where $i_{GPe/stn}$, $j_{GPe/stn}$ and $k_{GPe/stn}$ are neuron indices to calculate the distance $D_{i,j,k}$. $V_i^{\mathit{GPe}}$ and $U_i^{\mathit{STN}}$ represent the internal states of GPe and STN neurons respectively, $V_i^{D2}$ is output from D2 striatal flip-flops, $V_i^{\mathit{STN}}$ represents the output of the STN neurons (Eq. 14), ${\tau }_{\mathit{GPe}}$ and ${\tau }_{\mathit{STN}}$ are the time constants of GPe and STN, respectively and, $W_i^{GPe\to STN}$ is the connection strength from GPe to STN, $W_i^{STN\to GPe}$ is the connection strength from STN to GPe,. $V_j^{\mathit{DGPi}}\left(t\right)$ and $V_j^{\mathit{IGPi}}\left(t\right)$ are the inputs arriving at GPi via direct and indirect pathways (Eqs. 15, 16).

GPi neurons combine the outputs of D1-striatum and STN, and generate the predicted output as follows.

$${\tau }_{\mathit{GPi}}\ast \frac{d{V}_i^{\mathit{GPi}}}{\mathit{dt}}=-V_i^{\mathit{GPi}}-{\lambda }^{D1}\left(t\right)V_i^{\mathit{DGPi}}\left(t\right)+{\lambda }^{D2}\left(t\right)V_i^{\mathit{IGPi}}\left(t\right)$$ 17

where ${\tau }_{\mathit{GPi}}$ is the time constant for GPi. The GPi output ($V_i^{\mathit{GPi}}$) is calculated as mentioned in Eq. (17) is compared against the threshold, $V_{\mathit{th}}$, as a race model (Eq. 18). Race models are used in modelling action selection in behavioural tasks, where $i^{\mathit{th}}$ neuron whose output crosses the threshold first, is considered the winner, and the corresponding $i^{\mathit{th}}$ action is selected (Kumari and Chakravarthy 2022). The number of steps taken to cross a predefined threshold is equivalent to the response time of the model. Based on the action taken the critic will update the value function and it acts as the surrogate for reward as mentioned in Eqs. (19–21). After an actual reward is received the system will compute the reward expectation error and using that striatum will modulate its inhibition (the projections from striatum to SNc are GABAergic) towards the SNc. Following that, SNc will release the dopamine. Hence at an abstract level the reward prediction error or the temporal difference error can be compared analogous to the dopamine signal from SNc.

The output from direct pathway $V_j^{\mathit{DGPi}}\left(t\right)$ acts as Q-value. The model performs hill-climbing in the landscape of the value function by weight update as mentioned in Eq. (22). Overall, two kinds of value functions are computed at striatum ($VF\left(t\right))$ and GPi, and corresponding weights $W_i^{D1\to SNc}$ and $W_{i,j}^{D1\to GPi}$ are updated respectively. $W_i^{D1\to SNc}$ casts its modulation effect in computing ${\delta }_v\left(t\right)$, and ${\delta }_v\left(t\right)$ in turn on $V_i^{D1/2}\left(t\right)$, thus $W_i^{D1\to SNc}$ and $W_{i,j}^{D1\to GPi}$ are updated to achieve gating function on memory items and action selection respectively to perform the WM task. Such multi-objective function optimisation under RL framework is observed in BG and thought to play an important role in WM functions (Balasubramani et al. 2014; Chakravarthy et al. 2010; Doya and Kimura 2014). Similarly, such hierarchical learning rules are also proposed in the context of PFC-BG working memory models (Hazy et al. 2007; Schroll et al. 2012).

This RL framework to compute Q-value at time ‘t’ ($Qva{l}_t(s,a)$) for state ‘s’ and action ‘a’. In the proposed model, ‘s’ denotes the D1-striatal activation ($V^{D1}$) and ‘a’ denotes the winner GPi neuron ‘j’ whose value is $V_j^{\mathit{GPi}}\left(t\right)$.

$$If,V_j^{\mathit{GPi}}\left(t\right)<V_{\mathit{th}};\text{then}i\text{is}\text{the}\text{selected}\text{action}$$ 18

$$Qva{l}_t(s,a)=V_j^{\mathit{DGPi}}\left(t\right)$$ 19

$${\delta }^{\mathit{GPi}}(j,t)=rew\left(t\right)+\gamma Qva{l}_{t+1}\left(s,,,a\right)-Qva{l}_t(s,a)$$ 20

$$Qva{l}_{t+1}\left(s,,,a\right)=Qva{l}_t(s,a)+\eta (rew\left(t\right)+\gamma Qva{l}_{t+1}\left(s,,,a\right)-Qva{l}_t(s,a))$$ 21

where, $\gamma$ is the discount factor and $\gamma =0$, ${\delta }^{\mathit{GPi}}$ denotes TD error, and $rew\left(t\right)$ is reward at time ‘t’. $W_{i,j}^{D1\to GPi}$ is updated by learning parameter ($\eta$) using,

$$\mathrm{\Delta }{W}_{i,j}^{D1\to GPi}=\eta {\delta }^{\mathit{GPi}}(j)V_i^{D1}$$ 22

The PD condition

Loss of dopaminergic neurons in PD condition led to reduction in dopamine signal to striatum than required. To mimic this, our model uses an abstract representation of dopamine $({\delta }_v\left(t\right)$) by clamping the dopamine equivalent ${\delta }_v\left(t\right)$ in Eq. (9) to a lower value ${\delta }_{\mathit{lim}}$ than ideal value as mentioned in Eq. (23).

If PD = 1 then,

$${\delta }_v\left(t\right)=\text{min}({\delta }_v\left(t\right),{\delta }_{\mathit{lim}})$$ 23

If PD_ON = 1 (administration of Levodopa) then,

$${\delta }_v\left(t\right)=\text{min}({\delta }_v\left(t\right),{\delta }_{\mathit{lim}}+{\delta }_{\mathit{lmed}})$$ 24

Parameter selection

The parameters used in the model are tuneable and are selected based on the combination of pervious works and that give desired performance close to the experimental results under normal condition. Table 2 shows the model parameters and their values used in this work. Physiological significances of the model parameters mentioned in Table 2 is mentioned in Supplementary Information (Section S2). For three parameters ${ϵ}_{GPe/STN}$, $V_{\mathit{th}}$ and ${\delta }_{\mathit{lim}},$ we perform a parameter selection method with the suitable criterion for each of the parameters and use the obtained values for all simulations in this work. Table 3 shows the criteria for choosing suitable values for the three parameters.

Table 2.

Model parameters

Constant parameters	Symbol	Value
Gain parameters for the Sigmoid function	$a_1,a_2$ (Eq. 4)	− 1, 1
Time constants for the GPe, STN and GPi dynamics	${\tau }_{\mathit{GPe}},{\tau }_{\mathit{STN}},{\tau }_{\mathit{GPi}}$ (Eqs. 10,11,17)	6, 2, 1
Threshold parameters for the race model	$V_{\mathit{th}}$ (Eq. 18)	0.45
Strength of the Lateral connectivity of STN and GPe	${ϵ}_{\mathit{GPe}},{ϵ}_{\mathit{STN}}$ (Eq. 12)	0.05
Interconnectivity Strength between STN and GPe	$W_i^{GPe\to STN},W_i^{STN\to GPe}$ (Eqs. 10, 11)	3
Learning Parameter used for Q-Learning	$\eta$ (Eqs. 6,8,21,22)	0.05
Clamping parameter for Dopamine inorder to simulate PD condition	${\delta }_{\mathit{lim}}$ (Eq. 23)	0.36
Levodopa medication factor	${\delta }_{\mathit{lmed}}$ (Eq. 24)	0.3
Flag to define Healthy and PD condition	PD (Eq. 23)	0 = Healthy 1 = PD condition
Flag to define PD condition with administration of levodopa	PD_ON (Eq. 24)	0 = Healthy/PD without levodopa 1 = PD with levodopa

Table 3.

Parameters and their selection criterion

Parameter	Criterion
${ϵ}_{GPe/STN}$	Maximum standard deviation of noise from STN-GPe
$V_{\mathit{th}}$	Closely matching response time from experimental data of 4-consonants (same) task in control condition
${\delta }_{\mathit{lim}}$	Closely matching accuracy from experimental data of 4-consonants (middle) task in PD condition

${ϵ}_{GPe/STN}$ controls the variance in noise from STN-GPe and acts as an exploration unit for learning or performing a task. Thus, the ${ϵ}_{GPe/STN}$ value is chosen to generate the maximum noise variance. $V_{\mathit{th}}$ controls the GPi race model iterations and thus the response time of the model. Small $V_{\mathit{th}}$ may not be sufficient to give accurate predictions whereas large $V_{\mathit{th}}$ will unnecessarily make the race model loop to run for the maximum number of iterations. Therefore, experimental data of control response time from 4-consonants task (same—condition) is used as the criterion for choosing $V_{\mathit{th}}$. ${\delta }_{\mathit{lim}}$ controls the clamping of dopamine and therefore, the suitable value is obtained from the matching of PD experimental data accuracy of 4-consonants task (middle—condition). The values obtained from the parameter selection process is used in all the experiments. Figure 3 shows the plots of the cost function for corresponding parameter values.

Fig. 3.

A Standard deviation of noise from STN-GPe as a function of the lateral weights of STN-GPe ${ϵ}_{GPe/STN}$. B Response time in seconds when varying $V_{\mathit{th}}$. Black dashed line marks the response time given in the experimental data of the control subject in performing 4-consonants task (same—condition). C Accuracy of 4-consonants task (middle—condition) when varying ${\delta }_{\mathit{lim}}$. Black dashed line marks the accuracy of PD subject provided in experimental data

Simulation protocol

Input data $X_C\left(t\right)$ to the model is flattened grayscale image of dimension 24 × 24. White pixels are used for the text region with black background and normalised between [0, 1]. Cortex is modelled as the input layer to the network with weights connecting cortex to striatum (D1/D2) acts as an encoder to learn features of the input image to use it in further layers of the network. Throughout the model, weights (randomly initialized in (0, 1)) connecting between any two layers are all-to-all (i.e., layers $L_1\&L_2$ with $N_1\&N_2$ number of neurons respectively are connected by weight matrix of dimension $N_2$ × $N_1$). Lateral connections within STN/GPe are modelled using Gaussian function as mentioned in Eqs. (12, 13) (Nair et al. 2022) and the connection strength (${ϵ}_{GPe/STN}$) is obtained from parameter selection method as in Sect. "The PD condition". Method of varying ${ϵ}_{GPe/STN}$ as a parameter to simulate PD conditions by varying intra-neuronal synchrony has long been used in the literature (Chakravarthy and Balasubramani 2014; Mandali et al. 2016). However, in this work we only clamp dopamine value ${(\delta }_v\left(t\right)$) to simulate PD condition as that proved to be sufficient to match with the experimental data. Table 4 shows the network architecture sizes for different tasks. Number of neurons in each layer for a particular task is selected based on experimentation trial-and-error (Stathakis 2009). Number of neurons in GPi (output layer) is equivalent to number of possible actions for the task. Since striatum acts as an encoder to the input image, number of neurons in striatum is usually more than in other layers (Stathakis 2009). Task specific simulation protocol is followed as mentioned in Sect. "Experimental tasks". Sample t-test with sample size (n) 30 is used for statistical analysis between two groups and for 4-consonants task, One-way ANOVA is used to compare differences between three groups. For all the tasks, Mean Squared Error (MSE) and Adam (adaptive learning rate algorithm) are used as cost function and optimizer respectively. All the implementations are done using Tensorflow-2 Python library. Pseudo-codes for estimating WM capacity and model’s performance metrics (accuracy and loss) are given below.

Table 4.

Network architecture sizes for different tasks

Task	No. of Striatum D1/D2 neurons	No. of STN-GPe neurons	No. of GPi neurons/No. of possible actions
1AX2BY	20	20	2
Digit span task	200	50	9
Alphabetical manipulation	400	100	26
Updating task	400	100	26
4—consonants task	200	50	2

Pseudo-code to calculate working memory capacity of the model:

Pseudo-code to calculate accuracy and error of the model

Results

In order to validate the learning capacity of the proposed model, we show the performance in terms of accuracy and loss function for the 1AX2BY task. On extending the same, to check whether the proposed model can replicate the performance of PD subjects on WM tasks, we use four different tasks: digit span, alphabetical manipulation, update, and 4-consonants tasks, and compare model performance with experimental data for both control and PD conditions.

1AX2BY task

The model is trained for 50 epochs with a total number of 1860 data samples (batch size: 32, validation split: 0.3) and the sequence length of each sample is 20. Figure 4 shows the accuracy and loss (mean and standard deviation) curves for both training and validation data with 10 repeated trials. The results indicate that the model can learn the task and generalise well.

Fig. 4.

1AX2BY task. A Accuracy. B MSE loss of the model for train and validation data. Accuracy and MSE loss are calculated as mentioned in Sect. 2.3

Digit span task

The standard digit span task is used to showcase the WM storage capacity of the model. The model is trained for 50 epochs for each sequence length, with 32 samples of the particular sequence length. The experiment is carried out in both control and PD conditions for 10 repeated trials. Figure 5 shows the span of the proposed model and experimental data as reported by (Gilbert et al. 2005). The results validate that in both control and PD versions of the model, there is no statistical difference (sample t-test p > 0.05, n = 30) between span of control and PD, and is consistent with the experimental observation.

Fig. 5.

Spanning capacity of model compared with experimental data in both control and PD condition

Alphabetical manipulation task

The alphabetical manipulation task is used to test whether the model can show a significant difference between control and PD in performing alphabetical recall task, whereas in direct recall, the performance is not significantly different between the two conditions. The span capacity of 8 obtained from the digit span task is set as the sequence length. The model is trained for 200 epochs, with English alphabets. Figure 6 shows the accuracy of the proposed model and experimental data for direct and alphabetical recall as reported in (Gilbert et al. 2005) for both control and PD conditions. The significant difference observed in alphabetical recall task performance (sample t-test p < 0.05, n = 30) and non-significant difference (sample t-test p > 0.05, n = 30) in direct recall tasks are consistent with experimental observations.

Fig. 6.

Comparison of model performance in terms of accuracy with experimental task for both control and PD conditions. A Shows comparison for direct recall task, B shows comparison for alphabetical recall task

Updating task

Updating task requires updating of items in the stored list. The model is trained for 200 epochs with English letters, where each letter is transformed to a one-hot vector similar to the alphabetical manipulation task. Figure 7 shows the percentage of items recalled by the model and experimental data reported by (Gilbert et al. 2005) when k number of updates (k = 0, 2, 4, 6) are required for both control and PD conditions. The result shows the model performance is consistent with the experimental data and the differences between update groups is significant (p < 0.01).

Fig. 7.

Accuracy of recalled items in updating task compared with model and experimental data for four different numbers of updates. A Shows the accuracy plots for control, B shows the accuracy plots for PD condition

Consonants task

The model performance on 4-consonants WM task in terms of accuracy and reaction time is compared against the experimental data reported by (Lewis et al. 2003a, b). In the experimental data, two parts of reaction times (R1-time taken by the subject to prepare the correct response mentally and R2-motor response delay) are reported. In order to compare with the model, both R1 and R2 in experimental data are added and compared. The response time of the model is equal to the number of steps taken by the winning GPi neuron to cross the threshold as in Eq. (18). Figure 8. shows the model performance and experimental data for both control and PD conditions in terms of accuracy and reaction time (in seconds) grouped by different manipulation instructions. Manipulation required to obtain the desired output in three instructions (same, ends and middle) are mentioned in Sect. "Experimental tasks". The results are consistent with experimental observations that all three manipulation groups are significantly different within each other (One-way ANOVA p < 0.01, n = 30).

Fig. 8.

Comparison of model with experimental data for both control and PD conditions. A Shows comparison in terms of accuracy, B shows comparison in terms of reaction time. Same, end and middle are the instruction cues to mention the desired output should be arrived using no swap, swapping end items, and swapping middle items of the sequence respectively

Response time prediction

With the proposed model, we also predict the response time for digit span, alphabetical manipulation, and updating tasks. This shows that the model not only has the potential to match the experimental data but also can give the prediction of metrics missed in experimental data. Similar to the 4-consonants task, response time of the model is calculated based on the number of steps elapsed in the GPi race model, and the same experimental setup is used. Figure 9 shows the predicted response time for the three tasks. It can be inferred that predicted response times are inversely proportional to the performance accuracy.

Fig. 9.

Comparison of predicted response time for different tasks in both control and PD conditions. A Shows the predicted response times for digit span, direct recall and alphabetical recall tasks, B shows the predicted response times for updating task with four different number of updates

Levodopa effect

Using the proposed model, we study the effects of administering levodopa in 4-consonants task in terms of performance accuracy. Figure 10 shows the performance of the model in three different conditions (control, PD OFF and PD ON). PD OFF is same as the PD condition of model (Eq. 23) used in all tasks. PD ON condition is modelled using clamping ${\delta }_v\left(t\right)$ by ${\delta }_{\mathit{lim}}{+\delta }_{\mathit{lmed}}$ as mentioned in Eq. (24). Details on working mechanism of Levodopa medication and its incorporation in the current model is mentioned in Supplementary Information (S3). It can be inferred the administration of levodopa improved the model performance in all three cases (same, middle, end). Thus, model can be used as a potential test bench for testing the effect of administering medication to the patient in a clinical setting.

Fig. 10.

Effect of levodopa on PD model in 4-consonats task in terms of model performance accuracy. PD OFF and PD ON denote PD condition of the model without and with levodopa medications respectively. Same, ends and middle are the instruction cues that are given during the task

Conclusion and discussion

The proposed work presents a generalized basal ganglia (BG) architecture to model the WM functions of BG under normal and Parkinson’s conditions. The model architecture is cast in Reinforcement Learning (RL) framework and includes the memory feature implemented by the bistable flip-flop neurons in D1 and D2 Striatum, the input port of the BG; the output nucleus, Globus Pallidus interna (GPi) which implements action selection using a race model; the coupled system of non-linear oscillators that represent the STN-GPe oscillations in the indirect pathway, which was earlier proposed to supply exploratory drive necessary to implement RL. The proposed model could successfully implement the WM functions as well as capture the specific WM impairments observed among PD patients. The model can also predict response times of subjects in different WM tasks.

We chose five different tasks and the model results closely match with the experimental data reported in the original studies. The 1AX2BY task showed the learning capacity of the model on standard sequence processing and while the other four tasks Digit span, Alphabetical manipulation, Updating and 4-Consonants—were used to compare the performance in control and PD conditions. From our results, we could see that the digit spanning and direct recall tasks did not show any significant difference in the performance between the control and the PD cases, which is in line with the experimental results. However, there is a significant reduction in accuracy for the PD case for the alphabetical recall, updating and 4-consonant tasks and a significant increase in response time with respect to the 4-consonant tasks in PD. We could compare the response time only for the 4-consonant task due to the unavailability of response time for other tasks in experimental data. However, the model is used to predict the response time of tasks for which experimental data are unavailable. The 4-consonants task’s trend in three different instructions (same, end, middle) can be physiologically matched to primacy and recency effects in memory tasks (Morrison et al. 2014). Usually in memory tests, primacy effect dominates because of increased attention during sequence starting and recency effect strengthens due to better recallability of the tail items. It would be interesting to extend the current work further to study the influence of primacy and recency effects on degree of manipulation required to complete the WM task. Overall, our model results closely resembled the trends in the experimental results.

From the wide range of literature that we discussed in relation to the role of BG in WM tasks we could infer that there is a lot of scope in exploring the working memory properties of BG and the related impairments caused due to PD. Computational and mathematical studies have explored dopamine’s influence and role of BG in WM functions (Balasubramani et al. 2015; Frank et al. 2001; Gruber and von Cramon 2003; Nair et al. 2022; O’Reilly and Frank 2006). There are also modelling studies representing the circuitry at more abstract levels and more detailed levels also (Braver and Cohen 1999; Brunel and Wang 2001; Dreher and Grafman 2002). However, in this wide range of studies there is a conspicuous paucity of models that address alterations of WM functions under disease conditions such as PD. When it comes to cognitive aspects and WM deficits of PD there are only a few studies (Goulding et al. 2022; Hazy et al. 2007; Schroll and Hamker 2013), and there is still a huge scope in studying different aspects of WM deficits in detail.

Our current work is a step in this direction. In this study, we have successfully modelled different types of WM with varying complexities. This helps us in comprehensively understanding working memory functions under different conditions and which aspects are impacted when the dopaminergic circuitry is disrupted. Currently, we only confine ourselves to the involvement of BG in WM and its impairments in PD, although WM involves different parts of the brain including PFC (Barbey et al. 2013). However, even within PFC it has been suggested that dorsolateral PFC supports spatial WM whereas ventrolateral PFC supports non-spatial WM (Barbey et al. 2013). In the current model, we only focus on non-spatial WM. Studies have shown evidence of spatial WM in the posterior brain also, particularly in the parietal cortex (Curtis and D’Esposito 2003). Thus, an integrative model of WM will necessarily include the task-relevant prefrontal, parietal structures in the cortex, and the BG in the subcortex. Creation of such an integrative model will define the goal of our future efforts in this area.

Supplementary Information

Below is the link to the electronic supplementary material.

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.