A neural model of hippocampal-striatal interactions in associative learning and transfer generalization in various neurological and psychiatric patients

Abstract

Building on our previous neurocomputational models of basal ganglia and hippocampal-region function (and their modulation by dopamine and acetylcholine, respectively), we show here how an integration of these models can inform our understanding of the interaction between the basal ganglia and hippocampal region in associative learning and transfer generalization across various patient populations. As a common test bed for exploring interactions between these brain regions and neuromodulators, we focus on the acquired equivalence task, an associative learning paradigm in which stimuli that have been associated with the same outcome acquire a functional similarity such that subsequent generalization between these stimuli increases. This task has been used to test cognitive dysfunction in various patient populations with damages to the hippocampal region and basal ganglia, including studies of patients with Parkinson’s disease (PD), schizophrenia, basal forebrain amnesia, and hippocampal atrophy. Simulation results show that damage to the hippocampal region—as in patients with hippocampal atrophy (HA), hypoxia, mild Alzheimer’s (AD), or schizophrenia—leads to intact associative learning but impaired transfer generalization performance. Moreover, the model demonstrates how PD and anterior communicating artery (ACoA) aneurysm—two very different brain disorders that affect different neural mechanisms—can have similar effects on acquired equivalence performance. In particular, the model shows that simulating a loss of dopamine function in the basal ganglia module (as in PD) leads to slow acquisition learning but intact transfer generalization. Similarly, the model shows that simulating the loss of acetylcholine in the hippocampal region (as in ACoA aneurysm) also results in slower acquisition learning. We argue from this that changes in associative learning of stimulus-action pathways (in the basal ganglia) or changes in the learning of stimulus representations (in the hippocampal region) can have similar functional effects.

Introduction

As a common test bed for neurocomputational exploration of the interactions between the basal ganglia (and dopamine) and the hippocampal region (and acetylcholine), we focus on the acquired equivalence task, an associative learning paradigm in which stimuli that have been associated with the same outcome acquire a functional similarity such that subsequent generalization between these stimuli increases (Bondi et al., 1993; Grice & Davis, 1960). Both human neuropsychological (Bodi, Csibri, Myers, Gluck, & Keri, 2009; Keri, Nagy, Kelemen, Myers, & Gluck, 2005; Myers et al., 2008; Myers et al., 2003; Weiler, Bellebaum, Brune, Juckel, & Daum, 2009) and animal lesion (Coutureau et al., 2002) studies show that both the hippocampal region and the basal ganglia are important for acquired equivalence.

The acquired equivalence task has two phases: acquisition (learning to associate two stimuli) and transfer generalization (learning that cues become equivalent when they were previously associated with the same response). Several neuropsychological studies from our lab have argued that the associative learning and transfer generalization processes rely on different neural structures (Myers et al., 2008; Myers et al., 2003): initial associative learning relies on the integrity of the basal ganglia, whereas transfer generalization relies on the integrity of the hippocampal region.

For example, patients with mild Alzheimer’s disease, hippocampal atrophy (HA), and hypoxia are impaired at the transfer generalization phase of the task (Bodi et al., 2009; Myers et al., 2008; Myers et al., 2003) (See Table 1). Mild Alzheimer’s disease is associated with dysfunction to the medial temporal lobe and hippocampal region (de Leon, George, Stylopoulos, Smith, & Miller, 1989). Similarly, hypoxic brain injury causes bilateral neuropathology of the hippocampus and associated medial temporal areas (Kesner & Hopkins, 2001). Recently, Di Paola (Di Paola et al., 2008) reported hippocampal dysfunction in hypoxic patients. These results suggest that the hippocampal region plays an important role in transfer generalization performance.

Table 1.

Acquired equivalence performance in various patient populations. “X” means impairment while “-“signifies performance comparable to healthy controls.

	Acquisition	Transfer
Hippocampal atrophy/Hypoxic/mild Alzheimer’s (Bodi et al., 2009; Myers et al. 2003; 2008)	-	X
Parkinson’s patients (medicated) (Myers et al. 2003)	X	-
ACoA aneurysm (Basal forebrain amnesia) (Myers et al. 2008)	X	-
Schizophrenia (Keri et al., 2005)	-	X

Keri et al. (2005) also found that schizophrenic patients are impaired at transfer generalization in the acquired equivalence task. These results were also confirmed in a recent study (Weiler et al., 2009). It is likely that impaired transfer generalization performance in schizophrenic patients is due to hippocampal dysfunction, which has been reported in the literature (Goldman & Mitchell, 2004; Goldman et al., 2007; Lodge & Grace, 2008; Spoletini et al., 2009). Schizophrenia is a psychiatric disorder which is mainly associated with positive symptoms (e.g., delusions and hallucinations) and negative symptoms (e.g., apathy). It has been shown that schizophrenic patients have mediotemporal lobe and hippocampal dysfunction (Bogerts, Meertz, & Schonfeldt-Bausch, 1985; Boyer et al., 2007; Goldman & Mitchell, 2004; Heckers, 2001; Keri, 2008; Weinberger, 1999). Bogerts et al. (1985) also reported a decrease in hippocampal size but an intact basal ganglia structure in schizophrenic patients as seen in structural brain imaging. Schizophrenic patients also show declarative memory deficits, which suggest hippocampal-region dysfunction (Aleman, Hijman, de Haan, & Kahn, 1999; Cirillo & Seidman, 2003). Lesioning the hippocampus in animals is also used as a model of schizophrenia (Tseng, Chambers, & Lipska, 2009). In addition, Rametti et al. (2009) reported decreased hippocampal activity in schizophrenic patients performing declarative memory tasks. Our model argues that hippocampal dysfunction is responsible for transfer generalization deficits in schizophrenic patients.

Unlike mild Alzheimer’s disease, schizophrenia, and hypoxia, patients with Parkinson’s disease and ACoA (anterior communicating artery) aneurysm are impaired at the acquisition phase of the acquired equivalence task. Parkinson’s disease is a neurodegenerative disorder associated with reduced dopamine levels in the basal ganglia (Jellinger, 1999; Kish, Shannak, & Hornykiewicz, 1988). On the other hand, the ACoA is one of the most common sites of aneurysm in the brain. The ACoA sends projection to the prefrontal cortex and the basal forebrain (Wright, Boeve, & Malec, 1999), and an aneurysm to ACoA is found to affect basal forebrain functioning (Wright et al., 1999). Patients with ACoA aneurysm have basal forebrain damage, which affects cholinergic input to the hippocampus. Patients with ACoA aneurysm have amnesia (O’Connor & Lafleche, 2004), executive dysfunction (Simard, Rouleau, Brosseau, Laframboise, & Bojanowsky, 2003), and other cognitive deficits (Bondi, Kaszniak, Rapcsak, & Butters, 1993; DeLuca, 1993; Diamond, DeLuca, & Kelley, 1997; Mavaddat, Sahakian, Hutchinson, & Kirkpatrick, 1999). We argue that ACoA amnesia results from basal forebrain damage that disrupts learning in the hippocampal region (Myers et al., 2001; Myers et al., 2002). Cholinergic treatments are used to treat patients with ACoA aneurysm (Benke, Koylu, Delazer, Trinka, & Kemmler, 2005), which is consistent with a dysfunction to the cholinergic system in these patients. See Table 1 for summary of patient populations’ performance on the acquired equivalence task.

Our model seeks to explain how various brain disorders affect acquisition and transfer generalization performance by simulating the interactions between changes in dopamine and acetylcholine in the basal ganglia and hippocampal-region, respectively, as well as from damage to either or both regions. The model integrates features from our existing models of the basal ganglia (Moustafa & Gluck, 2010) and hippocampal region (Myers et al., 1995; Moustafa et al., 2009). In addition, the model explains how disruption to the dopaminergic (as in Parkinson’s disease) or cholinergic (as in ACoA aneurysm) systems affects acquisition performance. Dopamine is produced in the midbrain and is projected to the basal ganglia and prefrontal cortex. Several studies show that phasic dopamine is important for stimulus-response learning (Schultz, Dayan, & Montague, 1997; Wickens, Begg, & Arbuthnott, 1996). Our model assumes that phasic dopamine is key for stimulus-response learning through synaptic modification in the basal ganglia, as we have done in our past models of the basal ganglia (Moustafa & Gluck, 2010; Guthrie et al., 2009; Moustafa & Maida, 2007).

The basal forebrain is an important source of the neurotransmitter acetylcholine throughout the cortex, with the medial septum in particular sending acetylcholine to the hippocampal region (Hasselmo, 1995; Nolte, 1993; Nauta & Feirtag, 1986). In a recent study, Kukolja (Kukolja, Thiel, & Fink, 2009) found that human subjects taking cholinergic medications show enhanced hippocampal functions such as encoding. Septal lesions disrupt hippocampal function and impair acquisition of conditioned eyeblinking in rabbits (Berry & Thompson, 1979; Salvatierra & Berry, 1989; Powell, Milligan, & Buchanan, 1976). Similarly, studies show that scopolamine (an acetylcholine antagonist) impairs encoding of new information in humans and animals, a behavioral task that relies on the integrity of the hippocampal region (Carli, Luschi, & Samanin, 1997; Mewaldt & Ghoneim, 1979). Furthermore, rodent studies have shown that acetylcholine is important for synaptic modification in the hippocampal region (Huerta & Lisman, 1993). In agreement with these experimental studies, our model assumes that acetylcholine plays a critical role in learning in the hippocampal region, much as in our earlier models of septo-hippocampal function in associative learning (Rokers et al., 2002; Myers et al., 1996).

Our integrated model of basal ganglia and hippocampal-region function also attempts to explain how patients with ACoA aneurysm or hippocampal atrophy —brain disorders that affect the hippocampal region —show different performance in the acquired equivalence task. The model assumes that hippocampal atrophy (and also mild Alzheimer’s disease and hypoxia) impairs hippocampal function, while ACoA aneurysm slows down learning in the hippocampal region. The model also shows how patients with ACoA aneurysm or Parkinson’s disease —brain disorders that affect different brain systems—show similar performance in the acquired equivalence task. The model shows that decreasing learning rate parameter values either in the hippocampal region or basal ganglia modules slows down acquisition but does not affect transfer generalization performance. See Table 1 for summary of various patient groups’ performance in the acquired equivalence task.

The model has two modules: basal ganglia and hippocampal region (Figure 1). In agreement with most models (Frank, 2005; Moustafa & Gluck, 2010), the basal ganglia is key for stimulus-response learning. Also in agreement with computational models and experimental data (Dusek & Eichenbaum, 1997; Gluck & Myers, 1993), we assume that the hippocampal region is important for stimulus-stimulus representational processes. In the model, dopamine is key for learning in the basal ganglia, while acetylcholine is key for learning in the hippocampal region (see Appendix for more details on model simulations).

Figure 1.

Model architecture. Acetylcholine (Ach) projected from basal forebrain is key for learning in the hippocampal module as in Myers et al. (Myers et al., 1996), while dopamine (DA) is key for learning in the basal ganglia module. Weight update in the hippocampal module is based on Hebbian learning. The basal ganglia and dopamine are modeled using the TD model and actor-critic architecture (also see Moustafa & Gluck, 2010). Dotted lines represent neuromodulatory effects. Abbreviations: DA, dopamine; Ach, acetylcholine.

Results

Below, we present simulation results of healthy control subjects, subjects with hippocampal damage, and lastly Parkinson’s disease and ACoA aneurysm patients. As in all experimental studies with the acquired equivalence task, we present simulation results in terms of number of errors in the acquisition and transfer (including retention and transfer trials) phases.

Healthy controls

As with experimental results from healthy controls in various studies, simulation results show that simulated controls make more errors in retention than in transfer trials of the transfer phase (see healthy control data in Figures 3–5): In the model, hippocampal representations of antecedents and consequents of the “retention” pairs are more overlapped than the representations of “transfer” pair. Accordingly, the model makes more errors during the performance of transfer trials. In other words, the model suggests that the strength of acquired equivalence effect is related to the degree of overlap of representations of stimuli in the hippocampal region: The greater the overlap, the stronger the acquired equivalence effect. Simulation results also show that increasing the number of training trials in the acquisition phase leads to a stronger acquired equivalence effect (not shown).

Figure 3.

Performance of various patient groups with hippocampal damage in the acquired equivalence task. (a–d) Patients with hippocampal atrophy, hypoxia, or schizophrenia, were impaired at transfer generalization performance, particularly during the performance of transfer trials (Keri, Juhasz et al., 2005; Myers et al., 2008; Myers et al., 2003). (e) Simulated hippocampal damage. Simulation results are quantitatively similar to all other groups with hippocampal damage.

Figure 5.

(a) As similar to Parkinson’s disease, Patients with ACoA aneurysm are impaired at acquisition performance. (b) Simulated patients with ACoA aneurysm show similar results.

Hippocampal lesion

We simulated lesioning of the hippocampal region by running the simulations without the hippocampal module (see Appendix). This results in stimulus-response learning in the absence of dynamic modification of stimulus representations. Simulation results are very similar to the performance of patients with hippocampal damage, including patients with hippocampal atrophy, hypoxia, schizophrenia, and mild Alzheimer’s disease (Figure 3). Without a hippocampal module, the model only uses the basal ganglia for stimulus-response learning. Accordingly, model’s performance in the transfer trials is at chance level.

Dopaminergic and cholinergic lesions

As noted earlier, our model argues that dopamine is important for synaptic modification in the corticostriatal pathway, and acetylcholine is important for learning in the hippocampal region module. We simulated Parkinson’s disease by decreasing the value of the learning rate parameter in the basal ganglia module, as we had previously done in Shohamy et al. (2008). Simulation results show that decreasing the learning rate parameter value in the basal ganglia leads to slow acquisition learning and intact transfer generalization performance (Figure 4). Similarly, we simulated ACoA aneurysm by decreasing the value of the learning rate in the hippocampal module capturing the functional effect of reducing cholinergic inputs to the hippocampal region. As in simulated Parkinson’s disease (with loss of dopamine inputs), our modeling results show that decreasing the learning rate in the hippocampal region leads to slow acquisition learning, as in ACoA aneurysm patients (Figure 5).

Figure 4.

Parkinson’s disease patients’ performance in the acquired equivalence task. (a) Patients with Parkinson’s disease are impaired at acquisition performance but showed similar performance to healthy controls on the transfer phase. (b) Simulated Parkinson’s disease show similar results.

Discussion

We present a new integrated neural network model that simulates functional roles of the basal ganglia and hippocampal region in associative learning and transfer generalization performance. This model integrates various features from our past models of the basal ganglia (Moustafa & Gluck, 2010; Guthrie et al., 2009) and hippocampal region (Gluck & Myers, 1993; Myers et al., 1995). The model simulates performance in the acquired equivalence task in various patient groups, including patients with Parkinson’s disease, mild Alzheimer’s disease, hypoxia, ACoA aneurysm, and schizophrenia. Simulation results show that lesioning the hippocampal region or disrupting the dopaminergic or cholinergic signal interferes with behavioral performance of the acquired equivalence task. First, the model simulates behavioral results in healthy controls. In the transfer phase, the model makes more errors during the performance of transfer than retention trials, as found in all experimental studies using the acquired equivalence task (Bodi et al., 2009; Keri, Nagy et al., 2005; Myers et al., 2008; Myers et al., 2003).

The basal ganglia module in the model is important for stimulus-response learning, in agreement with many modeling and experimental studies (Frank, 2005; Houk, 1995a; Houk, 2005; Jog, Kubota, Connolly, Hillegaart, & Graybiel, 1999; Shohamy, Myers, Geghman, Sage, & Gluck, 2006). The hippocampal module in the model is important for stimulus-stimulus representational learning and responds to the presentation of antecedents and consequents, as in our past models (Gluck & Myers, 1993). This is in agreement with existing neurophysiological studies reporting that hippocampal neurons respond to the presentation of conditioned and unconditioned stimuli (McEchron & Disterhoft, 1997, 1999). Specifically, the simulated hippocampal region in our model is responsible for compressing stimulus inputs that repeatedly co-occur, as we have done in our past models (Myers et al., 1995; Moustafa et al. 2009). Supporting evidence that subjects compress the representation of stimuli in the acquired equivalence paradigm, Meeter, Shohamy, and Myers (2009) found that subjects tend to confuse stimuli that were previously associated with the same response in the acquired equivalence paradigm. This study suggests that subjects’ confusion of stimuli is likely due to overlapped representations of these stimuli, as proposed in our model. In addition, research shows that hippocampal lesioned rats are impaired at performing the sensory preconditioning paradigm (Port & Patterson, 1984), which also relies on compression processes. Lesioning the hippocampal region in the model weakens stimulus-stimulus representational learning (see Figure 2), and leads to making more errors during the performance of transfer than retention trials in the transfer phase of the acquired equivalence task.

Figure 2.

Representational (and compression) processes of the hippocampal region. (a) Compression in intact hippocampal region. During associative learning, the hippocampal region forms compressed representations of antecedents and consequents. In this example, the hippocampal region compresses the representations of A1, A2, X1, X2 when trained on A1→ X1, A2→X1, and A1→X2. This allows the model to associate A2 with X2 during transfer test (and thus acquired equivalence effect). (b) Lesioning the hippocampal region interferes with compression processes. Using the same example in (a), the model here does not properly compress the representations of input stimuli, and thus does not show acquired equivalence effect (i.e., it does not learn to associate A2 with X2).

Furthermore, our modeling results show that different manipulations of different components within the hippocampal module lead to different behavioral performance, which is in agreement with existing experimental studies. Our simulations showed that lesioning the hippocampal-region leads to weak compression (and thus impaired transfer performance). However, our modeling results show that decreasing the learning rate parameter value in the hippocampal module (which simulates the decreased acetylcholine levels in the hippocampal region) leads to slower compression of stimuli. This consequently leads to slow learning in the basal ganglia module as well, thus slow stimulus-response learning. Similarly, decreasing the learning rate in the basal ganglia module, capturing a loss of dopamine inputs, leads to impaired acquisition learning, which models Parkinson’s disease patients’ performance in the acquired equivalence task.

Unlike our model, most existing computational models focus on simulating the role of one neuromodulator to performance (Frank, 2005; Hasselmo, Wyble, & Wallenstein, 1996; Moustafa & Gluck, 2010; Myers et al., 1996), but for some exceptions see Cox and Krickmar (2009), Frank et al. (2007), and Doya (2002, 2008). Our model captures the interaction between two neuromodulators in two brain regions, arguing that dopamine is key for stimulus-response learning in the basal ganglia, while acetylcholine is key for stimulus-stimulus learning in the hippocampal region.

Acetylcholine, dopamine, and learning

A wide range of experimental studies show that acetylcholine is important for learning in the hippocampal region, whereas dopamine is important for learning in the basal ganglia.

Orsetti, Casamenti, and Pepeu ( 1996) found that stimulus-response learning performance is correlated with acetylcholine levels in the hippocampal region. These data argue against the view that basal ganglia alone is key for stimulus-response learning (Houk, 1995b). In our model, the hippocampal region forms representations of stimuli and send these signals to the basal ganglia for further stimulus-response learning, as in the earlier Gluck and Myers (1993) model of cortico-hippocampal function.

Studies also show that galantamine (a cholinesterase inhibitor) enhances eye-blink conditioning learning performance in both young (Simon, Knuckley, & Powell, 2004) and old (Weible, Oh, Lee, & Disterhoft, 2004) rabbits, a behavioral task that has been argued to rely on the integrity of the hippocampal region. Interestingly, using rabbits, Woodruff-Pak, Lehr, Li, and Liu-Chen (2008) found that enhanced eye-blink conditioning learning correlates with nicotinic acetylcholine receptors binding in the hippocampal region. These results suggest that increase in acetylcholine levels in the hippocampal region is important for enhanced eye blink conditioning learning. These experimental data also are consistent with model assumption that acetylcholine is key for learning in the hippocampal region.

As for nigrostriatal dopamine, synaptic plasticity was found in the corticostriatal pathway (Reynolds & Wickens, 2002; Reynolds, Hyland, & Wickens, 2001; Wickens, 1997; Wilson, 2004). It has been reported that dopamine projected to the striatum is necessary for modifying synaptic strength in the corticostriatal pathway (Cepeda et al., 1993; Wickens, Begg, & Arbuthnott, 1996; Wilson, 2004). In an in vitro study on spiny neurons, Wickens et al. (1996) found that dopamine plays an important role in inducing long-term potentiation (LTP) in the corticostriatal pathway. We simulated the effects of dopamine on learning using the 3-factor rule of learning, as we have done in our earlier models (Guthrie, Myers, & Gluck, 2009; Moustafa & Gluck, 2010; Reynolds & Wickens, 2002).

Brain disorders and behavior

Below, we discuss behavioral performance in the acquired equivalence and other related tasks in various patient groups.

Parkinson’s disease

Our simulation results show that Parkinson’s disease patients are impaired at acquisition but show intact performance at the transfer phase of the acquired equivalence task, as reported in Myers et al. (2003). This is consistent with experimental findings of Shohamy et al. (2006) showing similar results albeit using a different behavioral task. In addition, Parkinson’s disease patients’ impaired performance in stimulus-response learning tasks has been shown in various experimental studies (Frank, Seeberger, & O’Reilly R, 2004; Jahanshahi, Wilkinson, Gahir, Dharminda, & Lagnado, 2009; Knowlton, Mangels, & Squire, 1996; Moustafa, Sherman, & Frank, 2008; Shohamy et al., 2006).

ACoA aneurysm

Consistent with experimental results (Myers et al., 2008), our simulation results show that patients with ACoA aneurysm are impaired at the acquisition phase of the acquired equivalence task. This is consistent with existing data showing that patients with ACoA aneurysm also show impairment learning other behavioral tasks, including eye blink conditioning (Myer, Bryant, DeLuca, & Gluck, 2002; Myers, Deluca, Hopkins, & Gluck, 2006; Myers et al., 2001): Performing both of the acquisition phase of the acquired equivalence task and the eye blink conditioning task relies on associative learning processes.

Schizophrenia

Consistent with experimental studies (Keri, Nagy et al., 2005), our simulation results show that schizophrenic patients are impaired at transfer generalization but show intact acquisition performance. The neural dysfunction in schizophrenia is heterogonous, with various studies reporting hippocampal, prefrontal, and striatal dysfunction. Some argue that positive symptoms are due to basal ganglia dysfunction (Keri, 2008), while others argue that positive (and negative and cognitive) symptoms are due to prefrontal (Rolls, Loh, Deco, & Winterer, 2008) or hippocampal (Chen, 1995; Lisman & Otmakhova, 2001; Oertel et al., 2007) dysfunction. The model presented here argues that impaired transfer performance in the acquired equivalence task in schizophrenic patients is due to a dysfunction to the hippocampal region. Hippocampal dysfunction has been reported in schizophrenic patients (Goldman & Mitchell, 2004; Keri, 2008). Given that schizophrenic patients’ performance is very similar to patients with mild Alzheimer’s disease, hippocampal atrophy, and hypoxia, it is very likely that transfer generalization impairment in schizophrenic patients is due to hippocampal dysfunction.

With regard to the acquisition phase of the acquired equivalence task, it is debatable whether schizophrenic patients show feedback learning deficits. Waltz, Frank, Robinson, and Gold (2007) found that patients are impaired at learning from reward but not from punishment. In our own prior studies (Polgar et al., 2008; Farkas et al., 2008), we found that deficit patients (i.e., schizophrenic patients with negative symptoms) show feedback learning impairments, arguing that the basal ganglia might be responsible for negative symptoms in schizophrenia. In Keri et al. (Keri, Juhasz et al., 2005; Keri et al., 2000) and Wickert et al. (2002), we found intact performance in schizophrenic patients on a multi-cue probabilistic learning task (the “weather prediction” task). Overall, consistent with many existing experimental studies, our model assumes that schizophrenic patients show intact feedback learning.

Alzheimer’s disease

Hippocampal dysfunction has been consistently reported in Alzheimer’s disease patients. For example, several studies report dysfunction to different segments of the hippocampus in Alzheimer’s disease patients (Apostolova et al., 2006; de Leon et al., 1989; Jack et al., 2000). In addition, fMRI studies report decreased hippocampal activity in Alzheimer’s disease patients (Allen et al., 2007; Wang et al., 2006). Our model argues that transfer generalization impairment in Alzheimer’s disease patients may be due to hippocampal dysfunction.

Relation to existing models

Below we compare our model to existing models that simulate performance in similar behavioral tasks and/or model of brain disorders that we have simulated in this project.

Our model is novel in that it simulates performance in the acquired equivalence task through integration of our past modeling of the basal ganglia and hippocampal region. To our knowledge, no existing neural network model simulates performance in this task. Our previous models (Gluck, Myers, & Meeter, 2005; Gluck & Myers, 1993; Myers, Gluck, & Granger, 1995; Rokers, Mercado, Allen, Myers, & Gluck, 2002) simulated performance in other associative learning paradigms, such as latent inhibition and sensory preconditioning, but did not simulate performance in the acquired equivalence task, because these past models did not integrate the functionality of both brain regions. Turnock and Becker (2007) proposed a model that assumes that the hippocampus plays a role in contextual processes and is important for gating prefrontal cortex information into the basal ganglia. This model did not simulate functional roles of dopamine and acetylcholine in learning and performance.

As noted earlier, our current integrated model, builds on our earlier models of septo-hippocampal function in associative learning. In particular, as in Rokers et al. (2002), we showed that cholinergic input to the hippocampal region is key for learning classical conditioning tasks. The Rokers et al. model builds on our earlier model (Hasselmo et al., 1996), which argues that acetylcholine is key for encoding of new information in the hippocampus. Unlike existing models, Rokers et al. simulate a role for the hippocamposeptal pathway in learning. Rokers et al. argue that the hippocampus is key for novelty detection, and that the hippocampus projects this novelty detection signal to the septum. In this model, septal acetylcholine increases for novel stimuli, and thus learning to encode information is faster for novel than familiar stimuli. The model presented here does not simulate the function of the hippocamposeptal pathway. However, the Hasselmo et al. and Rokers et al. models do not simulate performance in acquired equivalence and do not incorporate a basal ganglia module. Another novel feature of our newer integrated model is that it simulates functional roles of both dopamine and acetylcholine in behavioral performance and learning.

Unlike most existing models, the model presented here also simulates performance in various neurological and psychiatric patient groups. Most existing models focus on simulating behavioral performance in various tasks in one patient group (Cohen, Braver, & O’Reilly, 1996; Frank, 2005; Moustafa & Gluck, 2010). One exception, however, is a computational model by Amos (2000) which simulates performance in the Wisconsin Card Sorting Task in Parkinson’s disease, Huntington’s disease, and schizophrenic patients. We believe that both classes of models are important. Accounting for performance in various tasks in one patient group explains similar neural mechanisms for seemingly different behavioral tasks. An example is a basal ganglia model proposed by Frank (Frank, 2005; Frank, Loughry, & O’Reilly, 2001) which shows how the basal ganglia and prefrontal cortex interact during the performance of both working memory and decision making tasks (also see Moustafa & Maida, 2007). Simulating performance in one behavioral task in various patient groups helps explain how Parkinson’s disease and ACoA aneurysm—which affect different brain structures—are associated with similar behavioral performance (as we show in our simulation results).

Models of brain disorders

Below, we present a summary of existing models of Parkinson’s disease, schizophrenia, and Alzheimer’s disease, and discuss the similarities and differences between these models and ours.

Parkinson’s disease

Most, if not all, existing models of Parkinson’s disease focus on simulating the functional contribution of the basal ganglia and/or prefrontal cortex to motor and cognitive processes (Amos, 2000; Frank, 2005; Guthrie et al., 2009; Moustafa & Gluck, 2010; Moustafa, Gluck, & Myers, 2009). These models are consistent with our model in that nigrostriatal dopamine is important for stimulus-response learning. For a review of existing models of Parkinson’s disease, see Discussion section of Moustafa and Gluck (Moustafa & Gluck, 2010).

Schizophrenia

As for schizophrenia, most existing models focus on simulating the contribution of a single brain area to behavioral performance (Cohen & Servan-Schreiber, 1992; Rolls et al., 2008; Talamini & Meeter, 2009). However, it is experimentally known that various brain structures are affected in schizophrenia (Keri, 2008; Perlstein, Carter, Noll, & Cohen, 2001; Snitz et al., 2005), including prefrontal cortex (Perlstein et al., 2001; Snitz et al., 2005), mediotemporal lobe (Keri, 2008), and basal ganglia (Waltz et al., 2007). Most computational models of schizophrenia focus on simulating the role of prefrontal cortex in cognitive performance (Cohen & Servan-Schreiber, 1992; Rolls et al., 2008). Some other models focus on simulating dopaminergic (Schmajuk, 2005) or mediotemporal lobe (Talamini & Meeter, 2009) dysfunction. Chen (1995) proposed a one-layer attractor network which addresses how hippocampal dysfunction in schizophrenia leads to psychotic symptoms. This model assumes that psychosis is related to aberrant retrieval of information from memory. This model shows that an increase of correlation of encoding inputs in the hippocampus does interferes with retrieval processes, such that the model will retrieve wrong information at wrong time (which arguably corresponds to psychotic symptoms).

The Amos model mentioned above simulates performance in Parkinson’s disease and schizophrenia, using mathematical techniques similar to those used by Moustafa and Gluck (2010). The Amos model showed that decreasing activity of the prefrontal cortex explains working memory deficits in schizophrenia, and it also shows that decreasing activity in basal ganglia and prefrontal cortex modules simulates impaired performance in Parkinson’s disease patients. Unlike our model, the Amos model is not a learning model and does not simulate stimulus-stimulus or stimulus-response learning processes.

Alzheimer’s disease

Unlike Parkinson’s disease and schizophrenia, there are fewer models of Alzheimer’s disease in the literature. For example, Hasselmo (Hasselmo, 1994; Hasselmo, 1997) provided a model of Alzheimer’s disease which shows how damage to the hippocampus can affect functional processing in efferent cortical structures. This model argues that memory symptoms in Alzheimer’s disease are related to impaired encoding of new information in the hippocampus, such that new and existing information share similar representations. This consequently makes it difficult to retrieve information from memory, which arguably equivalent to amnesic symptoms. McAuley et al. (2009) built a computational model that addresses how changes in cortisol levels affects hippocampal functioning, which consequently leads to Alzheimer’s disease symptoms. The model shows that increase in cortisol levels inhibits hippocampal function and leads to amnesia, as found with Alzheimer’s disease patients. Meeter and Murre (2005) built a model showing that damage to the hippocampus can lead to anterograde amnesia, a main symptom of Alzheimer’s disease. As in our current model, the Meeter and Murre model simulates amnesia by disabling hippocampal neurons. This model argues that consolidation of declarative memories takes place in the cortex, and thus damaging the hippocampus leads to anterograde amnesia. It is not clear if these models can simulate performance in the acquired equivalence task. In Gluck, Myers, Nicolle, and Johnson (2006), we provided a computational analysis (though not a simulation model) of how Alzheimer’s disease might affect hippocampal functioning and behavioral performance, especially in learning and transfer. The model presented here in this paper partially builds on ideas proposed by Gluck et al., including how damaging the hippocampal region might affect transfer generalization performance. For a recent review of models of Alzheimer’s disease, see Duch (2007).

Model limitations

Even though our current integrated model simulates functional roles of the hippocampal region and basal ganglia in various patient populations, it has some limitations.

For example, the model does not simulate the different symptoms in patients with hippocampal damage (e.g., hypoxia, mild Alzheimer’s disease, and hippocampal atrophy patients). The model simply assumes that these disorders interfere with computational processes of the hippocampal region, and we simulated these disorders by running the simulations without the hippocampal module. It is likely that these disorders affect the hippocampal structure in different ways, such that hippocampal structural changes in these brain disorders lead to different behavioral changes that were not captured by the acquired equivalence task. Such speculations are beyond the scope of our model. Future work should explain how damage to the hippocampal region can lead to either amnesia or psychosis in one integrated model, as reported in different models of these symptoms (Chen et al., 1995; Meeter & Murre, 2005; Hoffman et al., 2001, 2006).

Another limitation of the current model are the findings that in addition to the hippocampal region and basal ganglia, other brain structures may be key for acquired equivalence including the prefrontal cortex (Iordanova, Killcross, & Honey, 2007). Further, Shohamy and Wagner (2008) found that dopaminergic projections to the hippocampus are involved in acquired equivalence. They found that correlated hippocampal and ventral tegmental area activity during the acquisition phase of the acquired equivalence task predicts subject’s performance in the transfer phase. Shohamy and Wagner argue that the correlated activity between these structures suggests that dopamine projected to the hippocampus is key for learning the acquired equivalence task. Similarly, Lisman and Grace (2005) argued that dopamine is key for learning in the hippocampus. As in the Rokers et al. model, Lisman and Grace argue that the hippocampus is key for novelty detection. However, unlike Rokers et al, Lisman and Grace assume that hippocampal projection to the ventral tegmental area is important for driving dopamine responses for novel stimuli, and dopamine projection to the hippocampus is important for encoding of novel stimuli. The model presented here does not include this dopamine-hippocampus pathway. It is possibly the case that both dopamine and acetylcholine interact during learning in the hippocampal region. Similarly, acetylcholine may be key for attentional processes (Cox & Krichmar, 2009) where the role of the acetylcholine in attention may be mediated by cholinergic projections to the cortex (Bucci, Holland, & Gallagher, 1998). Further modeling work to address these data needs to include a cholinergic projection to the cortex, and to address how attention affects cognitive performance in the acquired equivalence task.

In a recent study from our lab, we have shown that subjects with cocaine addiction exhibit a specific deficit on performing the new consequents phase (the third phase of acquisition, see Table 2) of the acquired equivalence task (Vadhan et al., 2008). It is possible that they are specifically impaired in this phase due to increased memory load. The new consequents phase has more trial types than those of the previous phases (see Table 2), and it is possibly the case that subjects with cocaine addiction are impaired at maintaining the various trials types in that phase in working memory. Our model does not account for this finding because it does not have a working memory mechanism. Further modeling to address these data needs to include a working memory mechanism that maintains representation of trial types, and also should explain how cocaine addiction impairs working memory, as reported in experimental studies (George et al., 2008).

Table 2.

Acquired Equivalence task. The task has 2 phases: acquisition and transfer. In the transfer phase, trials that have”?” are trials that were not previously presented to the subjects (Transfer Trials); the rest are trials that were previously presented to the subjects (Retention Trials). In data presented, we will look at performance in each type separately. A’s and B’s are called stimuli (cues), while X’s and Y’s are outcomes (or consequents).

Acquisition: Shaping	Acquisition: Equivalence Training	Acquisition: New Consequents	Transfer
A1→X1	A1→X1	A1→X1	A1→X1
	A2→X1	A2→X1	A2→X1
		A1→X2	A1→X2
			A2→X2?
B1→Y1	B1→Y1	B1→Y1	B1→Y1
	B2→Y1	B2→Y1	B2→Y1
		B1→Y2	B1→Y2
			B2→Y2?

Furthermore, experimental data show that the hippocampal region also receives dopaminergic projections, and that the basal ganglia receives cholinergic input. Experimental data show that dopamine is also key for learning in the hippocampus (Rossato et al., 2009; Lisman & Grace, 2005), but this was shown to be related to long-term memory and novelty detection processes. Our model did not simulate performance in these tasks. Further, it was found that acetylcholine in the basal ganglia is key for controlling motor processes (Graybiel, 1998). In our model, we incorporate a simple module of motor learning. A more complex model of initiating and sequencing motor plans will definitely requires the simulating of both acetylcholine an dopamine in the basal ganglia. This is beyond the scope of our current model.

Our model makes several testable predictions. For example, our model predicts that decreasing acetylcholine (using scopolamine) or dopamine (using 6-OHDA) will equally impair acquisition of the acquired equivalence task. Lesioning the hippocampal region should lead to impaired transfer performance (for similar ideas, see Coutureau et al., 2002). As mentioned above, the model also predicts that the acquired equivalence effect is related to the number of training trials of the acquisition phase: The larger the number of trials, the greater the acquired equivalence effect. Future experimental studies are needed to test these predictions in either rodents or humans.

Overall, our model provides a mechanistic account for behavioral performance in the acquired equivalence task in various patient populations, including Alzheimer’s disease, Parkinson’s disease, hypoxia, ACoA aneurysm, and schizophrenia, and through this suggests how the basal ganglia (and dopamine) interact with the hippocampal region (and acetylcholine) in both learning and transfer generalization.

Appendix

Below, we describe details of the acquired equivalence task used in the simulation. We then describe details of the model simulation.

Acquired equivalence task

The acquired equivalence task has two phases: acquisition and transfer (see Table 2). Phase 1 has three sub-phases, in which subjects learn to associate different stimuli with consequents (responses). In this task, four drawings of faces (man, woman, girl, boy) served as the antecedent stimuli. In the model, we represent stimuli using an input of binary values. In the original task, the consequents were four drawings of colored fish, which we also represent using binary values. Subjects (and the model) learn to associate antecedents with consequents, as explained in Table 1. For more details on task description see Myers et al. (2003). The number of trials in the acquisition phase in both the original and simulated task is as follows: 32 (Shaping phase), 64 (Equivalence Training phase), 96 (New Consequents phase). The transfer phase has 16 trials.

Simulation details

As described above, the model has two modules: basal ganglia and hippocampal modules (Figure 1). Below, we describe simulations details for each module.

Basal ganglia module

We model the basal ganglia using the actor-critic architecture, as previously proposed in various models (Berns & Sejnowski, 1995; Houk, 1995b; Moustafa & Gluck, 2010; Moustafa & Maida, 2007; Suri & Schultz, 1998, 1999). The critic is key for reward and feedback-based learning and the actor is key for action selection (motor) learning.

Learning in the striatal module relies on phasic dopamine signals projected from the midbrain (for similar ideas see Suri & Schultz, 1998, 1999). In this model, phasic dopamine signals are important for motor learning. Learning in the basal ganglia module is based on the TD algorithm, which simulates various characteristics of phasic dopamine firing (Schultz et al., 1997; Sutton & Barto, 1987; Sutton & Barto, 1990). Let TD(t) be the temporal difference error signal at time t (also known as the effective reinforcement); R(t) be the reward presented at time t (reward is 1 when reward is presented after correct feedback and is 0 otherwise); P(t) be the reward prediction at time t; γbe the discount factor (which determines how future reward affect reward predictions; is set to 0.99 in all simulation runs presented here). The TD error is computed as follows:

$$TD(t)=R(t)+\gamma P(t)-P(t-1)$$

Let w_i be the weight connecting unit i to the critic node; n be the number of Input nodes; and x_i be activation of input units (which take binary (0,1) values). Reward prediction P(t) is computed by the critic node as follows:

$$P(t)=\sum _{i=1}^n{w}_i(t)x_i(t)$$

Now, we describe the equations of the actor module. Let w_ji be the weight connecting unit i to unit j; δ _ji (t) be the Gaussian noise associated with the weight w_ji. All weights are perturbed using Gaussian noise, which is included to induce exploration in the system. Let u _ji be the perturbed weight connecting unit i to unit j. Perturbed weight values are computed as follows:

$$u_{ji}(t)=w_{ji}(t)+{\delta }_{ji}(t),$$

Activations of all units in the network are computed using a sigmoidal function:

$$f(x)=\frac{1}{1+e^{-gx}}$$

where g is the gain parameter. Let n be the number of Input units. The input units take binary (0,1) values. The activation of a unit j is computed as:

$$A_j(t)=f(\sum _{i=1}^n{u}_{ji}(t)x_i(t))$$

In the model, a winner-take all network computes the unit with the highest activation in the basal ganglia module. In other words, we assume that winner-take all competition among striatal neurons is assumed to be the mechanism underlying the choice of motor responses..

$$A_j^p=\{\begin{array}{l}1if{A}_j>\beta \&A_j>A_i\mathit{for}\mathit{all}i\ne j\hfill \\ 0\mathit{otherwise}\hfill \end{array}$$

where β is a threshold; A_j is the activation of unit j; $A_j^p$ is the activation of unit j resulting from winner-take all computations (for similar ideas see Barto, 1995; Berns & Sejnowski, 1995; Schultz et al., 1997; Suri & Schultz, 1998, 1999).

Learning in the basal ganglia model is based on the three-factor rule of learning—also known as the dopamine-based Hebbian learning rule (for similar ideas, see Guthrie et al., 2009). According to this rule, the phasic dopamine signal is important for strengthening weights linking active nodes. Dopamine phasic signal is also important for weakening weights linking an active node and another inactive node. Also, different computational models incorporate this learning rule (Braver & Cohen, 2000; Guthrie et al., 2009; Suri & Schultz, 1998, 1999).

Let LR_bg be the learning rate, which we assume to correspond to phasic dopamine levels in the model, as previously proposed by Shohamy et al. (Shohamy, Myers, Kalanithi, & Gluck, 2008). Let x_i represents the activation level of the presynaptic node. The weight update rule is,

$$w_{ji}(t+1)=w_{ji}(t)+{LR}_{bg}TD(t)x_i(t)A_j^p$$

Hippocampal module

The hippocampal module is a 2-layer network in which the Input layer is fully connected to the hippocampal layer. The input pattern specifies the values of stimuli and outcomes (which corresponds to Faces and Fishes in the acquired equivalence task). As in the basal ganglia module, all weights are perturbed using Gaussian noise. Activation levels of all units in the model are computed as follows:

$$A_j(t)=f(\sum _{i=1}^n{u}_{ji}(t)x_i(t))$$

where u_ji is the perturbed weight connecting unit i to unit j, n is the number of units in the input; the input units take binary (0,1) values (for similar simulation details see Barto, 1995; Schultz et al., 1997; Suri & Schultz, 1999); t is time step, f is the logistic sigmoid function.

Weights are updated at every time step. Learning in this module is Hebbian. The Hebbian learning algorithm is a model of associative learning, a process ascribed to the hippocampal region function (Bunsey & Eichenbaum, 1995; Henke, Buck, Weber, & Wieser, 1997). It is also a simple model for synaptic change through Long Term Potentiation (LTP) in the hippocampal region (Bilkey, 1996; Bliss & Lomo, 1973). The weight update rule here is as follows:

$$w_{ji}(t+1)=w_{ji}(t)+{LR}_{\mathit{hipp}}{x}_i(t)y_j(t)$$

where LR_hipp is the learning rate for the hippocampal module, and represents acetylcholine levels in the hippocampal region; x_i is the cortical input unit i; y _j is the activation of the unit j in the hippocampal layer. The hippocampal layer consists of many patches of neurons (10 patches and each patch has 20 nodes), each form a separate representational code of the input (for details see Moustafa et al., 2009). Winner-take-all networks are used to simulate lateral inhibitory connections among neurons in each patch.

A hippocampal representation is projected to the basal ganglia module. There is one-toone connection from hippocampal layer to medial temporal cortex, with non-adaptive, fixed weights.

Simulations of neurological and psychiatric disorders

We simulated Parkinson’s disease in the model by decreasing learning rate parameter value in the basal ganglia module ( LR_bg ). Similarly, we simulated ACoA aneurysm by decreasing learning rate value in the hippocampal module ( LR_hipp ). The model shows that decreasing the learning rate value in the hippocampal module affects motor learning in the basal ganglia. This explains how ACoA aneurysm patients show impaired performance in acquisition (stimulus-response) performance, a process that has been ascribed to basal ganglia function. See Table 3 for a summary of parameters manipulated to simulate various brain disorders presented here.

Table 3.

Simulation of all disorders in the model. Lesion means taking out all structure. “Yes” signifies corresponding area is lesioned in the corresponding group. “↓”signifies learning rate (LR) is lower than learning rate used to simulate healthy controls. “-“signifies parameter values used is not different from controls.

	Hippocampal Learning rate	Basal ganglia Learning rate	Hippocampal Lesion	Basal ganglia Lesion
Hippocampal atrophy/Hypoxic/mild Alzheimer’s	-	-	Yes	-
Parkinson’s patients (medicated)	-	↓	-	-
ACoA aneurysm (Basal forebrain amnesia)	↓	-	-	-
Schizophrenia	-	-	Yes	-

We simulated lesioning of the hippocampal region in the model running the simulations without the hippocampal module. Learning in the model in this case depends only on weight update in the basal ganglia module. We assume that lesioning the hippocampal region simulates hippocampal dysfunction in hippocampal atrophy, hypoxic, mild Alzheimer’s disease, and schizophrenic patients. Lesioning the hippocampal region in the model interferes with compression processes (see Figure 2).